SkimUtil

Developers

Marco

Inputs

File, where to move it to

Outputs

File movement, catalog registration if requested

Task Description

It has been packaged in a package called DMU (Data Moving Utilities) that is currently availabe out of CVS (Panda CVS or as a tar file). Provides a high level interface, both as Python API and Command line user interface

Allows to move files around: to and from the DDM system and user's local storage. Allows transfers to/from ftp, http, gsiftp, either unmanaged or SRM managed. Allows triangular transfer through a cache when no server is available at either ends. DDM interactions include file registration and catalog lookup to retrieve a file/dataset from an unknown location. In other words it allows a Computing Element (Head nodes and Worker Nodes) to interact with the local Storage Element in OSG:

• Find/register files in LRC
• Transfer reliably from/to the storage

And it supports different backends:

• POSIX like (or almost), e.g. NFS, ?
• dCache
• Xrootd
• GSI authenticated FTP
• SRM (including 2.2)

Size and MD5 comparison is supported in most of them to provide reliable transfer.

DMU is used also by managed production (Panda, Pathena)

These utilities will be available on the CEs, pre-installed as part of the DSS package and available on OSG WNs.

External Dependencies

DQ2, some clients (gridftp, SRM) These do not depend on dq2_user_client tools and DQ2 dependencies is limited to the subscription and DS definition/registration features. To rerieve, move and register locally files these are independent dfom DQ2, interacting with the DQ2 site service using http.

Questions/Comments

• Question:Jerry:Is it assumed that these only on the DataServices Server? Could the user submit a grid job and expect to carry over these utilities with them or have them?

I'm expecting as stansard practice DMU to be installed at tier2s but shipping with the job is possible. They are available as python zip packages and Panda is actually sending DMU together with the pilots.

-- MarcoMambelli - 14 Nov 2006