Here some information on Globus troubleshooting

Check also ReplicaCheckCE

x509 certificates

There are 3 versions of certificates: old, GT3 (RFC draft) and RFC 3820

You can see more about differences and which versions are available and default in different Globus versions in http://dev.globus.org/wiki/Security/ProxyCertTypes

OSG 1.2 uses GT 4.0 that even if it supports RFC proxies, it defaults to GT3 ones. GT3 proxies are little tested and in some cases the support may even be almost dropped.

In commands like voms-proxy-init the version of the proxy may affect the communication with the VOMS server depending on its version. ATLAS has 3 servers (2 at CERN and one at BNL) so the error may seem intermittent depending on the server contacted.

voms-proxy-init -quiet -voms atlas:/atlas -vomses /osg/mwt2/app/atlas_app/local/vomses -key $X509_USER_PROXY -cert $X509_USER_PROXY -out $XrdSecGSIUSERPROXY -limited -noregen

has been reported to fail if the -noregen is not included

-noregen should only tell to voms-proxy-init to use the created proxy instead of using the certificate and key but its behavior is not clear when -cert and -key are in the command line. Doing some tests:

• old proxy (ver2) and RFC proxy seem to work always fine with and without -noregen
• GT3 proxy (default in OSG 1.2) works only with -noregen, otherwise fails at CERN (Error: Could not establish authenticated connection with the server. globus_gss_assist token :-1: read failure: Operation not permitted) and hangs at BNL

It is not clear why -noregen affects the behavior. Note that the error goes away in OSG 3 installations, using GT5 that by default uses RFC proxies

tmp_x509up_u20003_23271 is an example problematic proxy failing at the VOMS servers.

CA Certificates

The Globus tools will trust certificates issued by a CA if (and only if) it can find information about the CA in the trusted certificates directory.

The trusted certificates directory is located as described below and exists either on a per-machine or on a per-installation basis.

X509_CERT_DIR is the environment variable used to specify the path to the trusted certificates directory. This directory contains information about which CAs are trusted (including the CA certificates themselves) and, in some cases, configuration information used by grid-cert-request to formulate certificate requests. The location of the trusted certificates directory is looked for in the following order:

1. value of the X509_CERT_DIR environment variable
2. $HOME/.globus/certificates
3. /etc/grid-security/certificates exists
4. $GLOBUS_LOCATION/share/certificates

Doc source: http://www.globus.org/toolkit/docs/4.2/4.2.1/security/gsic/pi/#id2554930

From the source file of globus_gsi_sysconfig_get_cert_dir (in globus_gsi_system_config.c): Get the Trusted Certificate Directory containing the trusted Certificate Authority certificates. This directory is determined in the order shown below. Failure in one method results in attempting the next.

On UNIX:

• X509_CERT_DIR environment variable - if this is set, the trusted certificates will be searched for in that directory. This variable allows the end user to specify the location of trusted certificates.
• $HOME/.globus/certificates - If this directory exists, and the previous methods of determining the trusted certs directory failed, this directory will be used.
• /etc/grid-security/certificates - This location is intended to be independent of the globus installation ($GLOBUS_LOCATION), and is generally only writeable by the host system administrator.
• $GLOBUS_LOCATION/share/certificates

On Windows

• X509_CERT_DIR environment variable - if this is set, the trusted certificates will be searched for in that directory. This variable allows the end user to specify the location of trusted certificates.
• x509_cert_dir registry key - If this registry key is set on windows, the directory it points to should contain the trusted certificates. The path to the registry key is software\\Globus\\GSI
• \\<user home directory>\\.globus\\certificates - If this directory exists, and the previous methods of determining the trusted certs directory failed, this directory will be used.
• Host Trusted Cert Dir - This location is intended to be independent of the globus installation ($GLOBUS_LOCATION), and is generally only writeable by the host system administrator.
• Globus Install Trusted Cert Dir - this is $GLOBUS_LOCATION\\share\\certificates.

Globus 4.2 has grid-cert-diagnostics (not in VDT 2.0): http://www.globus.org/toolkit/docs/4.2/4.2.0/security/gsic/user/grid-cert-diagnostics.html

Some other documents:

• MyProxy rustroots: http://grid.ncsa.illinois.edu/myproxy/trustroots/
• http://www.globus.org/toolkit/docs/3.2/gsi/admin/configuration.html
• Generic introduction: http://www.globus.org/toolkit/docs/4.0/security/key-index.html
• http://globus-proxy-utils.sourcearchive.com/documentation/2.5-4/grid__cert__diagnostics_8c-source.html

Updating manually CA certificates and CRLs

vdt-ca-manage fetchCRL works only if it is in the cron as well (else it is refusing to run)

vdt-control --enable fetch-crl
vdt-control --on fetch-crl
vdt-ca-manage fetchCRL

Alternatively you can go to the directory and run manually: cd into it and run $VDT_LOCATION/fetch-crl/sbin/fetch-crl

To update the CA certs and check the status:

vdt-update-certs
vdt-ca-certs-status

Host certificates

Note that:

• host certificate must be owned by root
• permission must be 444 on the certificate, 400 on the key (nothing different)

Failure to complain will give sometime misleading errors on the clients trying to connect

Time synchronization

Synchronization is essential for security

ntpd -q -g
or
ntpd -q -g -x

ntpq -p
less /etc/ntp.conf 
man ntpdate
man ntpd 
ntpd -q
date
/etc/init.d/ntpd start
chkconfig 
chkconfig --list ntpd
chkconfig --add ntpd
chkconfig --list ntpd
chkconfig --level 3,5 ntpd on
chkconfig --level 3 5 ntpd on
chkconfig --level 35 ntpd on
chkconfig --list ntpd
cat /etc/resolv.conf 
less /etc/ntp.conf 
date

-- MarcoMambelli - 01 Apr 2010

• tmp_x509up_u20003_23271: problematic proxy

• tmp120511-voms_cert_tests.txt: tmp120511-voms_cert_tests.txt