This document collects all relevant information related to the proxies needed by Admins.DQ2 site services, and tips for troubleshooting proxy/permission related transfer errors.

dq2-site services no longer runs at T2 sites, so this information is no longer applicable - kept for reference only

X509 cert needed by Admins.DQ2 site services

Admins.DQ2 site services uses a personal cert, not a service cert. At both MWT2_UC and MWT2_IU, the cert used has Subject "/DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209". To switch this to another user, see "Creating the base proxy" below.

Proxies

There are 2 proxies that are used for Admins.DQ2 site services.

Local proxy

One is a "local" proxy which is created via voms-proxy-init. It is stored as /opt/dq2/certs/dq2_proxy.pem, and the environment variable X509_USER_PROXY is set to this location in the dq2 account.

This proxy is used for transfers into space-token-enabled storage, or for transfers which use FTS delegation.

MyProxy proxy

The other is created using myproxy-init and uploaded to a MyProxy server running on myproxy.usatlas.bnl.gov

This proxy is used for transfers into non-space-token storage which do not use FTS delegation.

Potentially troublesome issues

usatlas1 vs usatlas3 and the need for a VOMS proxy

Files which are written by PandaMover or output of production jobs are owned by usatlas1. To be consistent with this we want Admins.DQ2 transfers to also write files as usatlas1. However the personal cert maps to usatlas3, unless a VOMS proxy with Role=production is used.

Using this VOMS proxy adds some additional complications since the VOMS attributes have a relatively short lifetime.

It might be preferable to just use a simple (non-VOMS) grid proxy instead. However, attempts to add a "manual account mapping" to GUMS to force my DN to map to usatlas1 were not successful (this seems to be a limitation in GUMS). If we use a proxy which maps to usatlas3, we run into problems with directories owned by usatlas1 and no group write permission. If newly created directories had write permission for group usatlas, many transfer failures could be circumvented.

If files are being created by Admins.DQ2 site services with usatlas3 ownership, something has gone wrong with the Admins.DQ2 proxies - some VOMS attributes have expired.

dq2-site-services vs osg-client

When you become the dq2 user, (or equivalently, source the setup script for dq2-site-services), some utilities and libraries from /opt/globus are added to $PATH and $LD_LIBRARY_PATH. There is a version of grid-proxy-init but no voms-proxy-init, and the version of myproxy-init is too old:

version MYPROXYv2 (v0.6.1 30 Mar 2004)

which does not support VOMS. So, you need to use the utilities in /share/osg-client. However, when you source the setup file /share/osg-client/setup.sh, this "wrecks" the setup for dq2-site-services. So, you should always start a fresh shell before setting up osg-client, and exit this shell as soon as you are done with voms-proxy- and myproxy- commands.

Keeping proxies up-to-date

Since the VOMS attributes have a lifetime of 96, 24, or 12 hours (depending on the server), the proxies need to be updated frequently. This has to be done in an automated way, i.e. from cron, and in a way which does not require a passphrase.

It was suggested that the -noregen option of voms-proxy-init be used, but this turns out to be problematic (the Subject in the proxy keeps being appended to until it causes dCache failures).

Instead, what we do is to to create a long-lived "base" proxy (without VOMS attributes) and use this to sign the VOMS proxy. The base proxy is stored in a file called dq2_base_proxy.pem.

Creating the base proxy

• Log on to xxt2-grid1
• Become dq2:

    you@xxt2-grid1$ su
    root@xxt2-grid1# su - dq2

• Check the contents of ~dq2/.globus:

    dq2@xxt2-grid1$ grep subj ~/.globus/usercert.pem
        subject=/DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209

    If this is not you, copy your usercert.pem and userkey.pem into =~dq2/.globus=.  Permissions should be:

    -rw-r--r--  1 dq2  root  1720 May 11 15:00 usercert.pem
    -rw-------  1 dq2  root  1906 May 11 15:00 userkey.pem

• Set X509_USER_PROXY, so output will be written to the right place.

    dq2@xxt2-grid1$ X509_USER_PROXY=/opt/dq2/certs/dq2_base_proxy.pem

• Create the proxy

    dq2@xxt2-grid1$ grid-proxy-init -valid 1000000:0

The lifetime of the proxy will be shortened to the lifetime of the X509 certificate.

Creating the VOMS proxy

Note: this only needs to be done once. (Or you can just wait and let the cron job take care of it.)

    dq2@xxt2-grid1$ bash
    dq2@xxt2-grid1$ . /share/osg-client/setup.sh
    dq2@xxt2-grid1$  voms-proxy-init -cert /opt/dq2/certs/dq2_base_proxy.pem -key /opt/dq2/certs/dq2_base_proxy.pem -voms atlas:/atlas/Role=production -valid 96:0  -out /opt/dq2/certs/dq2_proxy.pem

Creating the MyProxy proxy

Note: this only needs to be done once. (Or you can just wait and let the cron job take care of it.)

    dq2@xxt2-grid1$ bash
    dq2@xxt2-grid1$ . /share/osg-client/setup.sh
    dq2@xxt2-grid1$ myproxy-init -S -d -C /opt/dq2/certs/dq2_base_proxy.pem -y /opt/dq2/certs/dq2_base_proxy.pem --voms atlas:/atlas/Role=production

(Enter FTS password)

Cron jobs on uct2-grid1

dq2@xxt2-grid1$ crontab -l
    30 12   * * *  . /share/osg-client/setup.sh;  voms-proxy-init -cert /opt/dq2/certs/dq2_base_proxy.pem -key /opt/dq2/certs/dq2_base_proxy.pem -voms atlas:/atlas/Role=production -valid 96:0  -out /opt/dq2/certs/dq2_proxy.pem
    20 */8  * * *  . /share/osg-client/setup.sh;  myproxy-init -S -d -s myproxy.usatlas.bnl.gov -C /opt/dq2/certs/dq2_base_proxy.pem -y /opt/dq2/certs/dq2_base_proxy.pem --voms atlas:/atlas/Role=production < /opt/dq2home/.secret
    20 */12 * * *  . /opt/glite/setup.sh ; X509_USER_PROXY=/opt/dq2/certs/dq2_proxy.pem glite-delegation-init -f -s https://fts.usatlas.bnl.gov:8443/glite-data-transfer-fts/services/gridsite-delegation

Note: /opt/dq2home/.secret contains the FTS password and is readable only by dq2.

Cron jobs on iut2-grid1

    dq2@iut2-grid1$ crontab -l
    30 12    * * * . /share/osg-client/setup.sh; voms-proxy-init -cert /opt/dq2/certs/dq2_base_proxy.pem -key /opt/dq2/certs/dq2_base_proxy.pem -voms atlas:/atlas/Role=production -valid 96:0 -out /opt/dq2/certs/dq2_proxy.pem
    20 */12 * * *  . /opt/glite/setup.sh ; X509_USER_PROXY=/opt/dq2/certs/dq2_proxy.pem glite-delegation-init -f -s https://fts.usatlas.bnl.gov:8443/glite-data-transfer-fts/services/gridsite-delegation

Note - the myproxy-init is not needed here since this is being done at UC.

Checking proxies

Checking the base proxy

dq2@xxt2-grid1: grid-proxy-info -f /opt/dq2/certs/dq2_base_proxy.pem 
    subject  : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy
    issuer   : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209
    identity : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209
    type     : full legacy globus proxy
    strength : 512 bits
    path     : /opt/dq2/certs/dq2_base_proxy.pem
    timeleft : 6479:30:28  (270.0 days)

The base proxy needs to be renewed about once a year (when the user's X509 cert is renewed).

Checking the local VOMS proxy

dq2@xxt2-grid1$ bash  ##NOTE - start a new shell!
dq2@xxt2-grid1$ . /share/osg-client/setup.sh
dq2@xxt2-grid1$  voms-proxy-info -all -file /opt/dq2/certs/dq2_proxy.pem 
    WARNING: Unable to verify signature! Server certificate possibly not installed.
    Error: Cannot find certificate of AC issuer for vo atlas
    These warnings are harmless
    subject   : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy/CN=proxy
    issuer    : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy
    identity  : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy
    type      : unknown
    strength  : 512 bits
    path      : /opt/dq2/certs/dq2_proxy.pem
    timeleft  : 72:46:54
    === VO atlas extension information ===
    VO        : atlas
    subject   : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209
    issuer    : /DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov
    attribute : /atlas/Role=production/Capability=NULL
    attribute : /atlas/Role=NULL/Capability=NULL
    attribute : /atlas/Role=NULL/Capability=NULL
    attribute : /atlas/lcg1/Role=NULL/Capability=NULL
    timeleft  : 0:46:54

dq2@xxt2-grid1$ exit

Make sure that the first VO attribute listed is

/atlas/Role=production/Capability=NULL

and that both timeleft values (both the cert and the VOMS extensions) are non-zero.

Checking the MyProxy proxy

You need to change the setting of X509_USER_PROXY before retrieving the proxy from the MyProxy server, otherwise you will clobber dq2_proxy.pem

dq2@xxt2-grid1$ bash  ##NOTE - start a new shell!
dq2@xxt2-grid1$ . /share/osg-client/setup.sh
dq2@xxt2-grid1$ export X509_USER_PROXY=/tmp/test_proxy.pem
dq2@xxt2-grid1$ myproxy-logon -d
  Enter MyProxy pass phrase:Note: this is a well-known FTS password, not the Grid passphrase
  A credential has been received for user /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209 in /tmp/test_proxy.pem.

dq2@xxt2-grid1$  voms-proxy-info -all -file /tmp/test_proxy.pem 
    WARNING: Unable to verify signature! Server certificate possibly not installed.
    Error: Cannot find certificate of AC issuer for vo atlas
    subject   : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy/CN=proxy
    issuer    : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy
    identity  : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209/CN=proxy
    type      : unknown
    strength  : 512 bits
    path      : /opt/dq2/certs/dq2_proxy.pem
    timeleft  : 72:46:54
    === VO atlas extension information ===
    VO        : atlas
    subject   : /DC=org/DC=doegrids/OU=People/CN=Charles G Waldman 131209
    issuer    : /DC=org/DC=doegrids/OU=Services/CN=vo.racf.bnl.gov
    attribute : /atlas/Role=production/Capability=NULL
    attribute : /atlas/Role=NULL/Capability=NULL
    attribute : /atlas/Role=NULL/Capability=NULL
    attribute : /atlas/lcg1/Role=NULL/Capability=NULL
    timeleft  : 0:46:54

dq2@xxt2-grid1$ rm /tmp/test_proxy.pem
dq2@xxt2-grid1$ exit

See comments under "checking the local VOMS proxy" above.

glite-delegation-init

 > From: Simone Campana [mailto:Simone.Campana@cern.ch]  
 >  
 > We are running the delegation patch for every cloud served by CERN. The 
 > delegation patch needs to run under the certificate used to transfer 
 > files inside the cloud, which for all clouds apart US is Kors 
 > certificate. The patch is simply running the command  
 >   /opt/glite/bin/glite-delegation-init -f -s \
 >     https://prod-fts-ws.cern.ch:8443/glite-data-transfer-fts/services/gridsite-delegation 
 > periodically (we do it once per hour). For a scenario like in the US, 
 > where site services are "distributed" at T2s, it is a bit more 
 > problematic, you should ask every T2 to run this delegation patch with 
 > the credential they use for the transfers.   

Note - in the US we are using the BNL FTS service, so the above command should use

https://fts.usatlas.bnl.gov:8443/glite-data-transfer-fts/services/gridsite-delegation 

Conversation with Hiro Ito Fri Aug 8 10:28:54 2008 (aim)

(09:59:13) Hiro AIM: I fixed it
(09:59:48) Hiro AIM: I think I know where to look/destroy
(09:59:56) cgwaldman: yes?
(10:00:14) Hiro AIM: I think FTS keeps it from myproxy with 12 hours.
(10:00:56) Hiro AIM: 12 hours is default
(10:01:12) Hiro AIM: in myproxy-init
(10:01:26) cgwaldman: right.  i tried using -t 1000 but still only got 12 hours
(10:01:30) Hiro AIM: myproxy provides the client with 12 hour proxy to use.
(10:01:55) Hiro AIM: so, FTS just keeps it until it is not valid.
(10:02:01) cgwaldman: i see
(10:02:13) Hiro AIM: anyway, that is my guess.
(10:02:20) cgwaldman: so, FTS got hold of a proxy that did not have VOMS
(10:02:26) Hiro AIM: yes.
(10:02:28) cgwaldman: and it took 12 hours for this to "clear" ?
(10:02:32) Hiro AIM: and keeps it for 12 hours.
(10:02:37) Hiro AIM: right.
(10:02:48) cgwaldman: ok.   how did FTS get this bad proxy?  do we know?
(10:03:09) Hiro AIM: FTS gets it from myproxy
(10:03:13) cgwaldman: well, how come FTS had this proxy with expired VOMS?  I guess because myproxy expired
(10:03:48) Hiro AIM: right. or when you/I are testing myproxy with/without voms.  
(10:04:06) cgwaldman: ok.
(10:05:18) Hiro AIM: when you delegate thing like proxy, it needs to tell the duration.
(10:05:29) Hiro AIM: we should use it shorter.
(10:05:42) cgwaldman: shorter than 12 hours ???
(10:05:56) Hiro AIM: well.  We should start using FTS delegation.
(10:06:02) Hiro AIM: which works.
(10:06:20) Hiro AIM: in FTS, we have 2 proxy stored for you.
(10:06:30) Hiro AIM: one from myproxy
(10:06:43) Hiro AIM: one from FTS delegation for space token area.
(10:06:49) cgwaldman: aha
(10:07:14) Hiro AIM: for the space token one, it gets directly from your local proxy of the client.
(10:07:30) cgwaldman: i see
(10:07:34) Hiro AIM: for non-space token area, it uses the one from myproxy.
(10:08:21) Hiro AIM: since myproxy tells client to keep for 12 hours, FTS just keep for that time.
(10:08:31) cgwaldman: ok
(10:08:31) Hiro AIM: that is my theory.
(10:08:33) cgwaldman: ok
(10:08:41) cgwaldman: what do we need to change?  anything?
(10:08:59) Hiro AIM: no.  just make sure no bad proxy is stored in myproxy :)
(10:09:02) cgwaldman: ok
(10:09:11) cgwaldman: and do the myproxy-init in cron job
(10:09:27) Hiro AIM: otherwise, I can clean it since now I know what I look.
(10:09:33) cgwaldman: ok
(10:09:41) cgwaldman: you can delete the cached proxy in fts ?
(10:09:42) Hiro AIM: for FTS delegation, you can destory
(10:10:02) Hiro AIM: maybe.
(10:10:05) cgwaldman: ok
(10:10:16) Hiro AIM: you probably need to do the following (not tested).
(10:10:25) Hiro AIM: 1 destroy myproxy proxy
(10:10:31) Hiro AIM: myproxy-destroy
(10:10:46) Hiro AIM: 2.  try the transfer to make sure FTS talks to myproxy.
(10:10:52) Hiro AIM: which will fail.
(10:11:05) Hiro AIM: 3.  do myproxy-init
(10:11:10) Hiro AIM: 4.  then transfer.
(10:11:16) Hiro AIM: well.  that is my guess.
(10:11:26) cgwaldman: ok
(10:11:29) cgwaldman: one last question
(10:11:56) cgwaldman: i've got this cron job doing the myproxy-init every 8 hours.  does that seem like a good idea?
(10:12:09) Hiro AIM: sounds ok.
(10:12:35) Hiro AIM: I am sure you can do every hour if you like.
(10:13:35) Hiro AIM: well.  I am not sure how long the actual duration of role in your proxy stored in myproxy server.
(10:13:54) Hiro AIM: I guess you should do less than 12 hours just be in safe side.
(10:14:00) cgwaldman: ok, that's what i'll do
(10:14:21) cgwaldman: thanks for all the help

--CharlesWaldman 13 Aug 2008