WRK0PRV [atlasadmin@ascwrk0prv ~]$ source setup-xrootd-vdt.sh ...Type localSetupDQ2Client to use DQ2 Client ...Type localSetupGanga to use Ganga ...Type localSetupGcc to use alternate gcc ...Type localSetupGLite to use GLite ...Type localSetupPacman to use Pacman ...Type localSetupPandaClient to use Panda Client ...Type localSetupROOT to setup (standalone) ROOT ...Type localSetupWlcgClientLite to use wlcg-client-lite ...Type saveSnapshot [--help] to save your settings ...Type showVersions to show versions of installed software ...Type createRequirements [--help] to create requirements/setup files ...Type setupDBRelease to use an alternate DBRelease ...Type diagnostics for diagnostic tools Setting up xrootd version beta-1 [atlasadmin@ascwrk0prv ~]$ ps auxwww | grep xr 501 10082 0.0 0.0 110940 4100 ? Sl 00:34 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//xrootd -l /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/var/logs/xrdlog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 10110 0.0 0.0 46016 3884 ? Sl 00:34 0:05 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//cmsd -l /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/var/logs/cmslog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13714 0.0 0.0 61180 736 pts/1 S+ 17:59 0:00 grep xr [atlasadmin@ascwrk0prv ~]$ vdt-control --non-root --list Service | Type | Desired State ------------------------+--------+-------------- xrootd | init | enable [atlasadmin@ascwrk0prv ~]$ vdt-control --non-root --off xrootd vdt-control running as a non-root user - 'inetd' services will not be started. disabling init service xrootd as non-root user... Killed [atlasadmin@ascwrk0prv ~]$ ps auxwww | grep xr 501 13767 0.0 0.0 61180 736 pts/1 S+ 18:00 0:00 grep xr [atlasadmin@ascwrk0prv ~]$ vdt-control --non-root --on xrootd vdt-control running as a non-root user - 'inetd' services will not be started. enabling init service xrootd as non-root user... ok [atlasadmin@ascwrk0prv ~]$ ps auxwww | grep xr 501 13801 0.0 0.0 41204 3924 pts/1 Sl 18:00 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//xrootd -l /var/log/xrootd/xrdlog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13843 0.0 0.0 108468 3804 pts/1 Sl 18:00 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//cmsd -l /var/log/xrootd/cmslog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13859 0.0 0.0 61180 736 pts/1 S+ 18:00 0:00 grep xr WRK1PRV [atlasadmin@ascwrk1prv ~]$ source setup-xrootd-vdt.sh ...Type localSetupDQ2Client to use DQ2 Client ...Type localSetupGanga to use Ganga ...Type localSetupGcc to use alternate gcc ...Type localSetupGLite to use GLite ...Type localSetupPacman to use Pacman ...Type localSetupPandaClient to use Panda Client ...Type localSetupROOT to setup (standalone) ROOT ...Type localSetupWlcgClientLite to use wlcg-client-lite ...Type saveSnapshot [--help] to save your settings ...Type showVersions to show versions of installed software ...Type createRequirements [--help] to create requirements/setup files ...Type setupDBRelease to use an alternate DBRelease ...Type diagnostics for diagnostic tools Setting up xrootd version beta-1 [atlasadmin@ascwrk1prv ~]$ ps auxwww | grep xr 501 13492 0.0 0.0 61160 736 pts/2 S+ 18:01 0:00 grep xr [atlasadmin@ascwrk1prv ~]$ vdt-control --non-root --on xrootd vdt-control running as a non-root user - 'inetd' services will not be started. enabling init service xrootd as non-root user... ok [atlasadmin@ascwrk1prv ~]$ ps auxwww | grep xr 501 13526 0.0 0.0 46460 4060 pts/2 Sl 18:02 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//xrootd -l /var/log/xrootd/xrdlog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13569 0.0 0.0 42932 3796 pts/2 Sl 18:02 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//cmsd -l /var/log/xrootd/cmslog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13585 0.0 0.0 61160 736 pts/2 S+ 18:02 0:00 grep xr WRK2PRV [atlasadmin@ascwrk2prv ~]$ source setup-xrootd-vdt.sh ...Type localSetupDQ2Client to use DQ2 Client ...Type localSetupGanga to use Ganga ...Type localSetupGcc to use alternate gcc ...Type localSetupGLite to use GLite ...Type localSetupPacman to use Pacman ...Type localSetupPandaClient to use Panda Client ...Type localSetupROOT to setup (standalone) ROOT ...Type localSetupWlcgClientLite to use wlcg-client-lite ...Type saveSnapshot [--help] to save your settings ...Type showVersions to show versions of installed software ...Type createRequirements [--help] to create requirements/setup files ...Type setupDBRelease to use an alternate DBRelease ...Type diagnostics for diagnostic tools Setting up xrootd version beta-1 [atlasadmin@ascwrk2prv ~]$ ps auxwww | grep xr 501 13308 0.0 0.0 61176 740 pts/3 S+ 18:03 0:00 grep xr [atlasadmin@ascwrk2prv ~]$ vdt-control --non-root --on xrootd vdt-control running as a non-root user - 'inetd' services will not be started. enabling init service xrootd as non-root user... ok [atlasadmin@ascwrk2prv ~]$ ps auxwww | grep xr 501 13342 0.0 0.0 46460 4052 pts/3 Sl 18:03 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//xrootd -l /var/log/xrootd/xrdlog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13384 0.0 0.0 42932 3800 pts/3 Sl 18:03 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//cmsd -l /var/log/xrootd/cmslog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 13401 0.0 0.0 61176 736 pts/3 S+ 18:03 0:00 grep xr INT1Y -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/test-test0604-5 Last server error 3010 ('Unable to create /xrootd/test-test0604-5; Permission denied') Error accessing path/file for xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/test-test0604-5 -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/marco/test-test0604-51 [xrootd] Total 0.00 MB |====================| 100.00 % [0.0 MB/s] -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk0prv.hep.anl.gov:1093//xrootd/marco/test-test0604-52 [xrootd] Total 0.00 MB |====================| 100.00 % [0.0 MB/s] -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk0prv.hep.anl.gov:1093//xrootd/test-test0604-53 [xrootd] Total 0.00 MB |====================| 100.00 % [0.0 MB/s] -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk1prv.hep.anl.gov:1093//xrootd/test-test0604-54 Last server error 3010 ('Unable to create /xrootd/test-test0604-54; Permission denied') Error accessing path/file for xroot://ascwrk1prv.hep.anl.gov:1093//xrootd/test-test0604-54 -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk1prv.hep.anl.gov:1093//xrootd/marco/test-test0604-55 [xrootd] Total 0.00 MB |====================| 100.00 % [0.0 MB/s] Email: Hi Doug, I think there are NFS problems. The config file seen on some hosts is different: on ascwrk0prv [atlasadmin@ascwrk0prv ~]$ less /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg #COMMON INFORMATION #We assume that all xrootd instances using this file are installed in the same directory set xrootdlocation = /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd set xrdr = ascheadprv.hep.anl.gov all.export /xrootd all.adminpath ${xrootdlocation}/var/admin all.manager $(xrdr):1213 cms.allow host *.hep.anl.gov xrootd.fslib ${xrootdlocation}/lib/libXrdOfs.so if $(xrdr) xrd.port 1094 all.role manager else xrd.port 1093 all.role server xrootd.chksum max 3 adler32 ${xrootdlocation}/bin/xrdadler32 set osscachepath = /data/xrootd oss.cache public $(osscachepath)/* xa oss.usage log ${xrootdlocation}/var/admin fi /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg Then I did this test: on ascwrk2prv: cp /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg.from-ascwr2prv on ascwrk0prv: [atlasadmin@ascwrk0prv ~]$ cp /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg.from-ascwr0prv [atlasadmin@ascwrk0prv ~]$ diff /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg.from-ascwr0prv /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg.from-ascwr2prv 31c31,40 < --- > # ENABLE_SECURITY_BEGIN > xrootd.seclib /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/lib/libXrdSec.so > # this specify that we use the 'unix' authentication module, additional one can be specified. > sec.protocol /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/lib unix > # this is the authorization file > acc.authdb /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/auth_file > ofs.authorize > sec.protbind *.hep.anl.gov unix > > # ENABLE_SECURITY_END Cheers, Marco Previous email: Hi Doug, not completely true that they do not work. I did a quick test on ascwrk2prv trying to be non invasive. I did the following as atlasadmin: $ source setup-xrootd-vdt.sh ... $ vdt-control --non-root --on vdt-control running as a non-root user - 'inetd' services will not be started. enabling init service xrootd as non-root user... ok And as marco: -bash-3.2$ ps auxwww | grep xr 501 12688 0.0 0.0 46460 4044 pts/3 Sl 16:33 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//xrootd -l /var/log/xrootd/xrdlog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg 501 12731 0.0 0.0 111552 3872 pts/3 Sl 16:33 0:00 /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/bin//cmsd -l /var/log/xrootd/cmslog -c /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/beta-1/xrootd/etc/xrootd.cfg marco 12762 0.0 0.0 61176 736 pts/2 S+ 16:36 0:00 grep xr -bash-3.2$ source /export/share/atlas/ATLASLocalRootBase/x86_64/xrootd/current/setup.sh -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/marco/test-test0604-4 [xrootd] Total 0.00 MB |====================| 100.00 % [0.0 MB/s] -bash-3.2$ xrdcp /users/marco/testfile xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/nonmarco/test-test0604-4 Last server error 3010 ('Unable to create /xrootd/nonmarco/test-test0604-4; Permission denied') Error accessing path/file for xroot://ascwrk2prv.hep.anl.gov:1093//xrootd/nonmarco/test-test0604-4 Then I turned it off As you can see the data server is using the correct log file that you modified and it is also behaving correctly. Checking the log file, there is the correct security initialization. How do you start/stop the data servers? Cheers, Marco