You are here: Foswiki>Main Web>TroubleShooting071205guc (15 Dec 2007, MarcoMambelli)Edit Attach

Troubleshooting of Recursive file transfer

Goal is being able to transfer big amounts of data from a directory structure on the Tier2 to one on the Tier3 or vice-versa. Alternatively trasfer datasets using only one command line for each dataset.

Follow different tests and encountered problems

Getting to know the cluster

Here you can find a node matrix for the Tier2 and the Tier3

Given the Matrix the SRM/GSIFTP servers are uct2-dc1 and uct3-edge1

Other caveats:
  • /pnfs/uct3.uchicago.edu/ is a relic from previous installations. Its use is unreliable, DO NOT USE IT

Attempts

DQ2 client utils

These include dq2_put and dq2_get commands.

Attempted to write to /pnfs directory. In order to use it is necessary to add dcap to LD_PRELOAD. These invoke srm-cp to copy to a local directory and overload /pnfs. This usually ends up crashing /pnfs that is no more available on the machine.

Attempted to copy to $HOME being on uct3-edge1 (machine serving $HOME that should be a local directory). The copy was working at reasonable speed but other users coplained about the load. Load ~12, any access to $HOME was very slow. So slow that users complained about not being able to login.

GridFTP

About globus-url-copy:
  • each time you refer to a directory the URL must end with a slash ( '/' )
  • If the server does not support MLSD ftp command, recursive copy is not supported; it will become equivalent to copy the content of a directory

dCache gridftp served does not support MLSt and MLSD (rfc3659). Since I had to copy multiple directories, I decided to copy them one at the time using a script to loop over the directories and copying each one using something like: 
[marco@tier2-06 ~]$ time globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/
pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v130030
08/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/


real    10m27.850s
user    0m0.150s
sys     0m0.027s
That is ~ 10G/10min, 15.2MB/sec (or 121.3 Mbs)

This is the final transfer result using g-u-c v1.18 after some debugging due to an inconsistent behavior of g-u-c, probably a bug of the latest version 1.22:

I've been confused by using 2 different versions of globus-url-copy that behave differently. Charles and I have been doing some testing. And here is a summary of the results.

with globus-url-copy: 3.18 (1120222988-63) 
globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

is equivalent to 
globus-url-copy gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*   gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/

No directory is created, all files are copied but if there are directories, these cause problems. This at least is the behavior with dCache server, that support only NLST

Witn globus-url-copy: 3.22 (1172269086-63) (any version >= 3.20) I tried the same and I get this error: 
/share/osg-client > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.

The same error is given by gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* as source URL

Checking the -dbg execution: NLST /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008 is sent and received OK, but the parsing is returning the error above

Summarizing the problems that I found:
  1. there is no real recursive copy
  2. with newer globus-url-copy the directory parsing is not working and even the copy of one subdirectory is failing

Some failed/successful attempts

These are different tests to try to isolate the error 
[uct3-edge1] /share/osg-client > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.


[uct3-edge1] /share/osg-client > globus-url-copy -dbg -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/

debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
227 OK (128,135,158,238,246,41)

debug: sending command:
NLST /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008

debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
150 Opening ASCII data connection for file list

debug: reading into data buffer 0xb7eb9008, maximum length 262144
debug: data callback, no error, buffer 0xb7eb9008, length 3000, offset=0, eof=true
debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
226 ASCII transfer complete

debug: operation complete

error: No files matched the source url.


[uct3-edge1] /share/osg-client > globus-url-copy -versions
globus-url-copy: 3.22 (1172269086-63)
globus_ftp_client_restart_plugin: 3.6 (1171321665-63)
globus_ftp_client_debug_plugin: 3.6 (1171321665-63)
globus_ftp_client_perf_plugin: 3.6 (1171321665-63)
globus_ftp_client_throughput_plugin: 3.6 (1171321665-63)
globus_ftp_control: 2.5 (1181535824-63)
globus_ftp_client: 3.6 (1171321665-63)
globus_xio_gsi: 0.36 (1163020628-63)
globus_xio_tcp: 0.36 (1163020628-63)
globus_xio_system_select: 0.36 (1163020628-63)
globus_xio_file: 0.36 (1163020628-63)
globus_xio: 0.36 (1163020628-63)
globus_io: 6.0 (1113865852-1)
globus_gsi_callback_module: 0.23 (1121890023-63)
globus_credential: 1.9 (1159917212-63)
globus_gsi_proxy: 2.4 (1113610195-1)
globus_gsi_openssl_error: 0.13 (1170803088-63)
globus_openssl: 0.6 (1113610167-1)
globus_gsi_gssapi: 4.12 (1179416313-63)
globus_sysconfig: 1.20 (1159917212-63)
globus_callout_module: 0.6 (1113865156-1)
globus_gss_assist: 3.23 (1170694562-63)
globus_i_gass_transfer_http: 2.13 (1174421873-63)
globus_extension_module: 7.24 (1181664948-63)
globus_callback_nonthreaded: 7.24 (1181664948-63)
globus_object: 7.24 (1181664948-63)
globus_error: 7.24 (1181664948-63)
globus_common: 7.24 (1181664948-63)
globus_gass_transfer: 2.13 (1174421873-63)
globus_gass_copy: 3.22 (1172269086-63)
globus_thread_common: 7.24 (1181664948-63)
globus_thread_none: 7.24 (1181664948-63)


[uct2-grid6] /home/marco > globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnf
s/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/
 gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

error: No files matched the source url.

[uct2-grid6] /home/marco > . /opt/OLD/CE-0.4.1/setup.sh 
[uct2-grid6] /home/marco > globus-url-copy -versions
globus-url-copy: 3.20 (1133498504-63)
...

[uct3-edge1] /share/osg-client > globus-url-copy -versions
globus-url-copy: 3.22 (1172269086-63)
globus_ftp_client_restart_plugin: 3.6 (1171321665-63)

Successful attempt: 
Copy files in the subdirectory

[marco@tier2-06 jobscheduler]$ globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5d/

Cancelling copy...
[marco@tier2-06 jobscheduler]$ globus-url-copy -versions
globus-url-copy: 3.18 (1120222988-63)
globus_ftp_client_restart_plugin: 3.2 (1120223030-63)
globus_ftp_client_debug_plugin: 3.2 (1120223030-63)
globus_ftp_client_perf_plugin: 3.2 (1120223030-63)
globus_ftp_client_throughput_plugin: 3.2 (1120223030-63)
globus_ftp_control: 2.3 (1113865369-1)
globus_ftp_client: 3.2 (1120223030-63)
globus_xio_gsi: 0.30 (1120662420-63)
globus_xio_tcp: 0.30 (1120662420-63)
globus_xio_system_select: 0.30 (1120662420-63)
globus_xio_file: 0.30 (1120662420-63)
globus_xio: 0.30 (1120662420-63)
globus_io: 6.0 (1113865852-1)
globus_gsi_callback_module: 0.23 (1121890023-63)
globus_credential: 1.8 (1113610001-1)
globus_gsi_proxy: 2.4 (1113610195-1)
globus_gsi_openssl_error: 0.12 (1113610150-1)
globus_openssl: 0.6 (1113610167-1)
globus_gsi_gssapi: 4.8 (1121285874-63)
globus_sysconfig: 1.18 (1115131187-0)
globus_callout_module: 0.6 (1113865156-1)
globus_gss_assist: 3.22 (1115126796-0)
globus_i_gass_transfer_http: 2.9 (1113865293-1)
globus_extension_module: 7.18 (1121807428-63)
globus_callback_nonthreaded: 7.18 (1121807428-63)
globus_object: 7.18 (1121807428-63)
globus_error: 7.18 (1121807428-63)
globus_common: 7.18 (1121807428-63)
globus_gass_transfer: 2.9 (1113865293-1)
globus_gass_copy: 3.18 (1120222988-63)
globus_thread_common: 7.18 (1121807428-63)
globus_thread_none: 7.18 (1121807428-63)

Debug execution of a successful attempt 
debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
227 OK (128,135,158,238,246,41)

debug: sending command:
NLST /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008

debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
150 Opening ASCII data connection for file list

debug: reading into data buffer 0x2e0008, maximum length 262144
debug: data callback, no error, buffer 0x2e0008, length 3000, offset=0, eof=true
debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008:
226 ASCII transfer complete

debug: operation complete
debug: starting to size gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1
debug: sending command:
TYPE I
debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1:
200 Type set to I

debug: sending command:
SIZE /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1

debug: response from gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1:
213 184413247

debug: operation complete
debug: starting to transfer gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1 to gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1
debug: connecting to gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1
debug: response from gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5g/M5.0028940.Default.L1TT-b00000101.ESD.v13003008.part0016._lumi0000._0001.1:
220 GSI FTP Door ready

Other copy operations

First loop

somehow this copied only most of the first directory (several GB anyway) 
[marco@tier2-06 ~]$ time for i in `cat files28940`; do echo "Copying $i"; globus
-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/
ESD/$i/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco
/M5/$i/ >> copylog.txt; done &
[1] 27332
[marco@tier2-06 ~]$ Copying M5.0028940.Default.L1TT-b00000110.ESD.v13003008

Loop to copy files that were missing 

data/users/marco/M5/$i/ ; done 
Copying M5.0028940.Default.L1TT-b00000111.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b00000111.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b00000111.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100001.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100001.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100001.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100011.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100011.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100011.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100100.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100100.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100100.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100101.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100101.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100110.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100110.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100110.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11100111.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11100111.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100111.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11101001.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11101001.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101001.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11101011.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11101011.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101011.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11101101.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11101101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101101.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11101110.ESD.v13003008 (globus-url-copy -r gsif
tp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.De
fault.L1TT-b11101110.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/u
chicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101110.ESD.v13003008/
)
Copying M5.0028940.Default.L1TT-b11101111.ESD.v13003008 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028940.Default.L1TT-b11101111.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
40.Default.L1TT-b11101111.ESD.v13003008/)
Copying M5.0028940.Default.L1TT-b11110001.ESD.v13003008 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028940.Default.L1TT-b11110001.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
40.Default.L1TT-b11110001.ESD.v13003008/)

real    0m47.568s
user    0m1.075s
sys     0m0.160s
This appeared on the terminal a long time after. It seems somehow if the forst copy did not actually stopped/crashed but somehow sopped its activity for a while and then restarted (after the other copy issued in the mean time) 
[marco@tier2-06 ~]$ Copying M5.0028940.Default.L1TT-b00000111.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b00000111.ESD.v13003008/M5.0028940.Default.L1TT-b00000111.ESD.v13003008.part0007._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100001.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100001.ESD.v13003008/M5.0028940.Default.L1TT-b11100001.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100011.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100011.ESD.v13003008/M5.0028940.Default.L1TT-b11100011.ESD.v13003008.part0003._lumi0000._0001.2: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100100.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100100.ESD.v13003008/M5.0028940.Default.L1TT-b11100100.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100101.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100101.ESD.v13003008/M5.0028940.Default.L1TT-b11100101.ESD.v13003008.part0003._lumi0000._0001.2: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100110.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100110.ESD.v13003008/M5.0028940.Default.L1TT-b11100110.ESD.v13003008.part0001._lumi0000._0001.2: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11100111.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11100111.ESD.v13003008/M5.0028940.Default.L1TT-b11100111.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11101001.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101001.ESD.v13003008/M5.0028940.Default.L1TT-b11101001.ESD.v13003008.part0003._lumi0000._0001.2: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11101011.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101011.ESD.v13003008/M5.0028940.Default.L1TT-b11101011.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11101101.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101101.ESD.v13003008/M5.0028940.Default.L1TT-b11101101.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11101110.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101110.ESD.v13003008/M5.0028940.Default.L1TT-b11101110.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11101111.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11101111.ESD.v13003008/M5.0028940.Default.L1TT-b11101111.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)

Copying M5.0028940.Default.L1TT-b11110001.ESD.v13003008

error: globus_ftp_client_state.c:globus_i_ftp_client_response_callback:3144:
the server responded with an error
553 /pnfs/uchicago.edu/data/users/marco/M5/M5.0028940.Default.L1TT-b11110001.ESD.v13003008/M5.0028940.Default.L1TT-b11110001.ESD.v13003008.part0001._lumi0000._0001.1: Cannot create file: CacheException(rc=10008;msg=Pnfs error : File exists)


real    925m24.265s
user    0m4.350s
sys     0m1.019s

Transfer of files from run 28997

The operations involved:
  • estimating the file size
  • creating the directories (g-u-c with 3rd party transfer cannot create directories)
  • transfer of the files

Preliminary operations

From a machine accessing the source file system: 
[uct2-grid6] /home/marco/m5-transfer > ls /pnfs/uchicago.edu/data/ddm1/M5/ESD/ |  grep 28997  > files28997-all
[uct2-grid6] /home/marco/m5-transfer > for i in `cat files28997-all `; do du -s --si /pnfs/uchicago.edu/data/ddm1/M5/ESD/$i; done 
4.8G    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b00000101.ESD.v13003010
374G    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b00000110.ESD.v13003010
14M     /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b00000111.ESD.v13003010
2.5M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11100001.ESD.v13003010
4.6M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11100011.ESD.v13003010
1.7M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11100101.ESD.v13003010
1.7M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11100111.ESD.v13003010
2.5M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11101001.ESD.v13003010
3.0M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11101011.ESD.v13003010
851k    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11101101.ESD.v13003010
2.1M    /pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b11101111.ESD.v13003010
The transfer is of about 380GB

From a machine accessing the destination file system: 
[uct3-edge3] /pnfs/uct3.uchicago.edu/data/users/marco/M5 > for i in `cat ~/files28997-all`; do
> mkdir $i
> done
[uct3-edge3] /pnfs/uct3.uchicago.edu/data/users/marco/M5 > chown :usatlas *
[uct3-edge3] /pnfs/uct3.uchicago.edu/data/users/marco/M5 > chmod g+w *


From a machine with a working globus-url-copy (v3.18 or >v3.9) 
[marco@tier2-06 jobscheduler]$ date
Thu Dec  6 16:33:01 CST 2007
[marco@tier2-06 ~]$ time for i in `cat files28997-all`; do echo "Copying $i (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/$i/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/$i/)"; globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/$i/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/$i/ ; echo "ExitCode: $?"; done 
Copying M5.0028997.Default.L1TT-b00000101.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b00000101.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.0028997.Default.L1TT-b00000101.ESD.v13003010/)
date
ExitCode: 0
Copying M5.0028997.Default.L1TT-b00000110.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028997.Default.L1TT-b00000110.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.0028997.Default.L1TT-b00000110.ESD.v13003010/)

Copy of second run

From a machine with g-u-c 3.18. Copy started Thu Dec  6 16:33:01 CST 2007. 
[marco@tier2-06 ~]$ time for i in `cat files28997-all`; do echo "Copying $i (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uch
icago.edu/data/ddm1/M5/ESD/$i/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/$i/)"; globus-url-copy -r gsiftp:
//uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/M5/ESD/$i/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco
/M5/$i/ ; echo "ExitCode: $?"; done 
Copying M5.0028997.Default.L1TT-b00000101.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b00000101.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b00000101.ESD.v13003010/)
time
date
ExitCode: 0
Copying M5.0028997.Default.L1TT-b00000110.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b00000110.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b00000110.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b00000111.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b00000111.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b00000111.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11100001.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11100001.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11100001.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11100011.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11100011.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11100011.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11100101.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11100101.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11100101.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11100111.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11100111.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11100111.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11101001.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11101001.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11101001.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11101011.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11101011.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11101011.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11101101.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11101101.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11101101.ESD.v13003010/)
ExitCode: 0
Copying M5.0028997.Default.L1TT-b11101111.ESD.v13003010 (globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu:2811/pnfs/uchicago.edu/data/ddm1/
M5/ESD/M5.0028997.Default.L1TT-b11101111.ESD.v13003010/ gsiftp://uct3-edge1.uchicago.edu:2811/pnfs/uchicago.edu/data/users/marco/M5/M5.00289
97.Default.L1TT-b11101111.ESD.v13003010/)
ExitCode: 0

real    474m35.219s
user    0m4.198s
sys     0m1.073s
[marco@tier2-06 ~]$ time

real    0m0.000s
user    0m0.000s
sys     0m0.000s
[marco@tier2-06 ~]$ date
Fri Dec  7 00:27:00 CST 2007

Verify size data transferred 

[uct3-edge3] /pnfs/uct3.uchicago.edu/data/users/marco/M5 > du -s -b M5.0028997.Default.L1TT-b*
4732316388      M5.0028997.Default.L1TT-b00000101.ESD.v13003010
373897616651    M5.0028997.Default.L1TT-b00000110.ESD.v13003010
13876699        M5.0028997.Default.L1TT-b00000111.ESD.v13003010
2494384 M5.0028997.Default.L1TT-b11100001.ESD.v13003010
4577640 M5.0028997.Default.L1TT-b11100011.ESD.v13003010
1643826 M5.0028997.Default.L1TT-b11100101.ESD.v13003010
1698865 M5.0028997.Default.L1TT-b11100111.ESD.v13003010
2491436 M5.0028997.Default.L1TT-b11101001.ESD.v13003010
2935609 M5.0028997.Default.L1TT-b11101011.ESD.v13003010
851004  M5.0028997.Default.L1TT-b11101101.ESD.v13003010
2093471 M5.0028997.Default.L1TT-b11101111.ESD.v13003010

Evaluation of the speed:
  • about 13MB
  • that is ~ 106Megabit/sec 
4732316388+373897616651+13876699+2494384+4577640+1643826+1698865+2491436+2935609+851004+2093471
378662595973
378662595973/(474*60+35.219)/1000^2
13.29796957744205584511
378662595973/(474*60+35.219)/1000^2*8
106.38375661953644676088

PS Check http://en.wikipedia.org/wiki/Mbit/s and http://en.wikipedia.org/wiki/Megabit for unit definition -- MarcoMambelli - 05 Dec 2007
Edit  | Attach | Print version | History: r5 < r4 < r3 < r2 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r5 - 15 Dec 2007, MarcoMambelli
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback