This is quite old and so not all information is likely to be correct.
There are different tools for interacting with the DDM system:
- DQ2 end user tools:
dq2-get, dq2-put, ...
- DQ2 client tools: allow dataset definition and subscription (more for advanced users)
- Grid client tools (wlcg-client, an extended OSG client) are needed by DQ2
See further descriptions of DQ2 end user tools and clients (
)and instructions in the CERN Twiki, Using DQ2
A description of the full DQ2 package and system is available at DDM twiki home
. This page
is specific to DQ2 on OSG. OSG client tools are described here
and the wlcg-client package here
Documentation about the old version of the DQ2 Endusers Tools is available here
Here is how to set up the tools and use them. You have to setup the Grid middleware and the DQ2 tools:
$ source /share/wlcg-client/setup.sh
$ source /share/dq2-client/setup.sh
$ voms-proxy-init -voms atlas
Then you can invoke the desired DQ2 command that will be in the path:
- These packages replace the old ones and allows to interact with both OSG and LCG grid environments (LCG interaction is limited, e.g. there is no direct job submission).
- DQ2Clients replaces also the old DQ2 Enduser Tools (
dq2_...) with a newer version (e.g.
dq2-get). If you are familiar with the older version. please check the documentation since these are not exactly equivalent.
Whenever possible (whenever you know the file name) we recommend using
to copy files. Use the '-r' option to perform recursive copies or with wildcards:
globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/ gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/
globus-url-copy -r gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/ddm1/M5/ESD/M5.0028940.Default.L1TT-b00000101.ESD.v13003008/* gsiftp://uct3-edge1.uchicago.edu/pnfs/uchicago.edu/data/users/marco/M5f/
A note about recursive copy:
- The recursive copy is not working correctly if the server is a dCache gridftp door (MLST not supported, see TroubleShooting071205guc). In this case it is equivalent to a directory copy (only one level is copied), equivalent to the wildcard command above.
How to make your files DQ2 visible
Copy a file from UC Tier3
If you want to copy a file in UC Tier3' dCache system, you can:
You need to know the exact filename and path.
BNL hosting of user datasets
Policies about hosting keep changing. This may be obsolete:
BNL provides space for all US ATLAS users in its dCache pool. Files copied there must be written into a directory following a convention with a directory
is your FirstnameLastname. User-defined datasets should have prefix
The dataset naming convention for ATLAS is defined here
MWT2 hosting of user datasets
At UofC there is also an area of user files. You can copy your files to the MWT2_UC storage element using a special
subdirectory in the dCache system:
globus-url-copy your_file gsiftp://uct2-dc1.uchicago.edu/pnfs/uchicago.edu/data/tier3/user/user.FirstnameLastname/DatasetName/your_file
dCache and the
are exported by dCache's PNFS server. You cannot interact normally with files in dCache.
is a command that allows one to copy to and from dCache. If
is not in your path you can add it to your path (probably it is located in
and with bash you can use
It is possible to enable dCache interactions setting the
), but this will limit performance and may crash the PNFS server of dCache
. So it is recommended not to interact with dCache this way.
Instead of exporting the environment variable, it can be specified for a single command like this:
LD_PRELOAD=/opt/d-cache/dcap/lib/libpdcap.so tar cfz /tmp/data.tgz /pnfs/uchicago.edu/userdata/mydata
and this will not affect subsequent commands. This is preferable to the 'export' syntax.
How to install the software
To check installation instructions go to InstallingDQ2AtUC
(useful for system administrators
Transferring files from BNL
by default uses
. Due to BNL Firewall configuration multiple stream SRM copy does not work with BNL. So in both
SRM streams have to be limited to 1, e.g.:
dq2_get -rv --srmstreams 1 ...
(information src: Hiro and Wensheng, BNL team)
- 30 Jul 2007