Using DQ2 End User Tools

This is quite old and so not all information is likely to be correct.

Overview

There are different tools for interacting with the DDM system:

• DQ2 end user tools: dq2-get, dq2-put, ...
• DQ2 client tools: allow dataset definition and subscription (more for advanced users)
• Grid client tools (wlcg-client, an extended OSG client) are needed by DQ2

See further descriptions of DQ2 end user tools and clients (dq2-)and instructions in the CERN Twiki, Using DQ2 . A description of the full DQ2 package and system is available at DDM twiki home. This page is specific to DQ2 on OSG. OSG client tools are described here and the wlcg-client package here.

Documentation about the old version of the DQ2 Endusers Tools is available here.

Environment setup

Here is how to set up the tools and use them. You have to setup the Grid middleware and the DQ2 tools:

   $ source /share/wlcg-client/setup.sh
   $ source /share/dq2-client/setup.sh
   $ voms-proxy-init -voms atlas
 

Then you can invoke the desired DQ2 command that will be in the path:

   dq2-get ...

Notes:

• These packages replace the old ones and allows to interact with both OSG and LCG grid environments (LCG interaction is limited, e.g. there is no direct job submission).
• DQ2Clients replaces also the old DQ2 Enduser Tools (dq2_...) with a newer version (e.g. dq2_get becomes dq2-get). If you are familiar with the older version. please check the documentation since these are not exactly equivalent.

Recursive copies

Whenever possible (whenever you know the file name) we recommend using globus-url-copy to copy files. Use the '-r' option to perform recursive copies or with wildcards:

 
      globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/         gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

      globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/*      gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/

A note about recursive copy:

• The recursive copy is not working correctly if the server is a dCache gridftp door (MLST not supported, see TroubleShooting071205guc). In this case it is equivalent to a directory copy (only one level is copied), equivalent to the wildcard command above.

How to make your files DQ2 visible

See PublishingDatasets.

Copy a file from UC Tier3

If you want to copy a file in UC Tier3' dCache system, you can: globus-url-copy gsiftp://uct3-edge1.uchicago.edu/pnfs/... You need to know the exact filename and path.

BNL hosting of user datasets

Policies about hosting keep changing. This may be obsolete: BNL provides space for all US ATLAS users in its dCache pool. Files copied there must be written into a directory following a convention with a directory /pnfs/usatlas.bnl.gov/user01/name/year/week/ where name is your FirstnameLastname. User-defined datasets should have prefix user.FirstnameLastname. The dataset naming convention for ATLAS is defined here.

MWT2 hosting of user datasets

At UofC there is also an area of user files. You can copy your files to the MWT2_UC storage element using a special tier3 subdirectory in the dCache system:

globus-url-copy your_file gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/tier3/user/user.FirstnameLastname/DatasetName/your_file

dCache and the LD_PRELOAD directive

Paths under /pnfs are exported by dCache's PNFS server. You cannot interact normally with files in dCache. dccp is a command that allows one to copy to and from dCache. If dccp is not in your path you can add it to your path (probably it is located in /opt/d-cache/dcap/bin/ and with bash you can use export PATH=$PATH:/opt/d-cache/dcap/bin/).

It is possible to enable dCache interactions setting the LD_PRELOAD variable to /opt/d-cache/dcap/lib/libpdcap.so (e.g. export LD_PRELOAD=/opt/d-cache/dcap/lib/libpdcap.so), but this will limit performance and may crash the PNFS server of dCache. So it is recommended not to interact with dCache this way.

Instead of exporting the environment variable, it can be specified for a single command like this:

LD_PRELOAD=/opt/d-cache/dcap/lib/libpdcap.so tar cfz /tmp/data.tgz /pnfs/uchicago.edu/userdata/mydata

and this will not affect subsequent commands. This is preferable to the 'export' syntax.

How to install the software

To check installation instructions go to InstallingDQ2AtUC (useful for system administrators).

Gotchas

Transferring files from BNL

dq2_get by default uses srmcp. Due to BNL Firewall configuration multiple stream SRM copy does not work with BNL. So in both srmcp and dq2_get SRM streams have to be limited to 1, e.g.:

dq2_get -rv --srmstreams 1 ... 

(information src: Hiro and Wensheng, BNL team)

-- MarcoMambelli - 30 Jul 2007