HowToEnableRucio

The following explains how Rucio was enabled on MWT2

Three steps to Rucio

There are only three steps needed to enable Rucio on a site

1. The site must be running SL6
2. DQ2 Client 2.4.0 must be setup by every Panda job
3. The directory "/rucio" is added to then end of "setpath" and/or "seprodpath" in AGIS

Understanding the callouts in the Pilot

The following are the steps performed when a Panda job starts on a site.

$OSG_GRID and $OSG_APP are both defined on the gatekeeper in /etc/osg/config.d/10-storage.ini

$OSG_GRID	grid_dir
$OSG_APP	app_dir

$ATLAS_LOCAL_AREA is defined on the gatekeeper in /etc/osg/config.d/40-localsettings.ini

At many sites $ATLAS_LOCAL_AREA the fully substituted path of $OSG_APP/atlas_app/local

For example, at MWT2, the entries for each of these is

grid_dir = /share/wn-client
app_dir = /osg/mwt2/app

ATLAS_LOCAL_AREA=/osg/mwt2/app/atlas_app/local

A graphical representation of the "source" tree

   wrapper.sh
     |          - source $OSG_GRID/setup.sh
     |
     +---> wrapper.py
            (plugin atlasprodpilotplugin.py)
                - source $OSG_APP/atlas_app/atlas_rel/cctools/latest/setup.sh
                - source $OSG_APP/atlas_app/atlaswn/setup.sh
                - source $OSG_APP/atlas_app/atlas_rel/local/setup.sh
                   |
                   +---> source $ATLAS_LOCAL_AREA/setup.sh
                              |
                              +---> source $ATLAS_LOCAL_AREA/setup.sh.local
                - run {site.envsetup from agis, if any}
                - pilot.py

Must be running SL6

If you are not running SL6 (or RHEL6, etc), then stop. You cannot proceed. You cannot run Rucio on SL5 systems. DQ2 Client 2.4.0 requires python 2.6 and is not supported on SL5 systems. It is important that you upgrade to SL6 ASAP.

Setup DQ2 Client 2.4.0

Each Panda job must setup DQ2 Client 2.4.0 for Rucio to be available. This version of DQ2 is available in the CVMFS atlas.cern.ch repository.

The setup is done with

source /cvmfs/atlas.cern.ch/repo/sw/ddm/2.4.0/setup.sh

It is important the DQ2 be setup for every Panda job via callouts by the Pilot.

There are three places one can place the above "source".

AtlasWN

At MWT2 we put this setup in the same location as the old AtlasWN product which is no longer needed.

On the SL5 systems, the AtlasWN product was used install a python 2.6 and setup DQ2 Client 2.x This was needed since python 2.4 was the default and system python on SL5 system, but DQ2 2.x required python 2.6

Since the default python on SL6 is python 2.6, this product is no longer needed and "must" be removed. However, the pilot still makes a callout to this product via

source $OSG_APP/atlas_app/atlaswn/setup.sh

On MWT2 we removed all files beneath "atlaswn" and created a "setup.sh" which only contains the setup of DQ2 Client 2.4.0

# cat $OSG_APP/atlas_app/atlaswn/setup.sh
#!/bin/sh

source /cvmfs/atlas.cern.ch/repo/sw/ddm/2.4.0/setup.sh

In this way, DQ2 is setup in the same location in the pilot as in previous releases.

$ATLAS_LOCAL_AREA/setup.sh

The second location can be in the $ATLAS_LOCAL_AREA/setup.sh file. This file was previously managed by Alessandro DS via LJSFi 1.0. However, starting with LJSFI 2.0, the files in $ATLAS_LOCAL_AREA are no longer managed by ADS. It would be possible to place the setup of DQ2 in this location.

The best file to add the setup of DQ2 would be in

$ATLAS_LOCAL_AREA/setup.sh.local

which is "source" by the $ATLAS_LOCAL_AREA/setup.sh

AGIS "envsetup"

A third location where one can setup DQ2 would be by using the AGIS parameter "envsetup". For every Panda Q, you would need to modify "envsetup" to include the line

source /cvmfs/atlas.cern.ch/repo/sw/ddm/2.4.0/setup.sh

$OSG_GRID/setup.sh

It is not recommended that the DQ2 setup be put in a path referenced by $OSG_GRID/setup.sh. Since non-Panda jobs source $OSG_GRID/setup.sh, they would pull in the DQ2 Client. This could cause problems for non-Atlas jobs.

Add /rucio to sepath

To complete the switch to Rucio, each Panda Q must be modified to write its file into a "rucio" subdirectory

The AGIS parameter sepath and/or seprodpath must be change to add "/rucio" to the end

For example, at MWT2, the Panda Q MWT2_SL6 has the following values

sepath:   
seprodpath:   /pnfs/uchicago.edu/atlasproddisk

These values were changed to

sepath:   
seprodpath:   /pnfs/uchicago.edu/atlasproddisk/rucio

For the Panda Q ANALY_MWT2_SL6 the changes were from

sepath:   /pnfs/uchicago.edu/atlasuserdisk
seprodpath:   /pnfs/uchicago.edu/atlasuserdisk

sepath:   /pnfs/uchicago.edu/atlasuserdisk/rucio
seprodpath:   /pnfs/uchicago.edu/atlasuserdisk/rucio

Once these changes were made, the jobs running on these Qs being writing there output in Rucio format into the "/rucion" subdirectory on each site.

Clean-PM

With the change to Rucio in seprodpath, all pandamover files will now also be written into "/rucio". If you are running Clean-PM to remove pandamover files, it will need to be change to account for the new location.

CCC and LFC

At this time is it unclear is this change stop writing into the LFC. This could then require changes to CCC.

-- DaveLesny - 13 Aug 2013