You are here: Foswiki>Main Web>TestSkim080909Tutorial (13 Sep 2008, MarcoMambelli)Edit Attach

Introduction

Test to prepare the page for the tutorial at CERN

Exercise 1.1

I repeated the exercise 1.1 from the July tutorial. So far submission is succesful.

I submitted 2 version:
  • AODtoDPD_mm.py job option starting from AODtoDPD.py from release 14.2.10
  • AODtoDPD_mm2.py job option starting from AODtoDPD.py from PAT $GroupArea of the tutorial as in https://twiki.cern.ch/twiki/bin/view/Atlas/PatTutorial140210 (/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODtoDPD.py)
In both cases I applied the modification to use the TAG file from ELSSI as input. In both cases I used release 14.2.10 without special patches/tags (except Pathena). 

[lxplus223] /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10/pathena_run > diff ./AODtoDPD_mm2.py /afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODtoDPD.py 
61,64c61
< #svcMgr.EventSelector.InputCollections =[ "/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODs/valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496/AOD.022496._00001.pool.root.1"]
< svcMgr.EventSelector.InputCollections =[ "fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root" ]
< svcMgr.EventSelector.CollectionType = "ExplicitROOT"
< svcMgr.PoolSvc.ReadCatalog += ["xmlcatalog_file:/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/TAG/PoolFileCatalogRun52280Egamma.xml"]
---
> svcMgr.EventSelector.InputCollections =[ "/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODs/valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496/AOD.022496._00001.pool.root.1"]

Setup

ATLAS release setup as in https://twiki.cern.ch/twiki/bin/view/Atlas/RegularComputingTutorial#DAY_1_THE_BASICS 
mkdir cmthome_tut080820
cd cmthome_tut080820
cat <<EOF > requirements
set   CMTSITE  CERN
set   SITEROOT /afs/cern.ch
macro ATLAS_DIST_AREA ${SITEROOT}/atlas/software/dist
macro ATLAS_TEST_AREA ${HOME}/scratch0/
use AtlasLogin AtlasLogin-* $(ATLAS_DIST_AREA) 
EOF
mkdir scratch0_0808
ln -s scratch0_0808 scratch0
mkdir tut0808
cat <<EOF > tut0808/setup1_cmt.sh
#the cmt config command must run from within the directory containing the requirement file
source /afs/cern.ch/sw/contrib/CMT/v1r20p20080222/mgr/setup.sh
cd $HOME/cmthome_tut0808/
cmt config 
EOF
cat <<EOF > tut0808/setup2_release.sh
cd $HOME
source cmthome_tut080820/setup.sh -tag=14.2.10,setup,32 
EOF
. tut0808/setup1_cmt.sh
. tut0808/setup2_release.sh
echo $TestArea 
mkdir $TestArea

Pathena setup as in https://twiki.cern.ch/twiki/bin/view/Atlas/DAonPanda#1_Setup 
cd $TestArea
cmt co PhysicsAnalysis/DistributedAnalysis/PandaTools
cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
cmt config
. ./setup.sh 
make

I needed a Grid setup but it is not necessary (better to skip it if possible).
  • Enter your password after voms-proxy-init
  • you need your certificate/s in your .globus directory 
source /afs/cern.ch/project/gd/LCG-share/current/etc/profile.d/grid_env.sh
voms-proxy-init

WARNING:
  • Pathena requires python 2.5. Both the system and the gLite python are 2.3, you may require to modify your PATH: 
export PATH=/afs/cern.ch/sw/lcg/external/Python/2.5/slc4_ia32_gcc34/bin/:$PATH
  • The run directory for Pathena jobs needs to be a subdirectory of the base Pathena installation ($TestArea or /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10 here), else Pathena will fail with an error like ERROR : you need to run pathena in a directory under /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10

Run directory

Run directory setup (input files):
  • fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root is a file produced using ELSSI
  • AODtoDPD_mm.py needs to be edited to add TAG selection lines 
mkdir $TestArea/pathena_run
cd $TestArea/pathena_run
cp /afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/TAG/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root ./
echo "fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root" > myinputs.txt
get_files -jo DPDUtils/AODtoDPD.py
cp AODtoDPD.py AODtoDPD_mm.py
vi AODtoDPD_mm.py

Second jobOption file:
  • also AODtoDPD_mm2.py needs to be edited to add TAG selection lines 
cp /afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODtoDPD.py AODtoDPD_mm2.py
vi AODtoDPD_mm2.py

Grid submission

Pathena submission: 
pathena --shipInput --inputFileList=myinputs.txt --outDS user08.MarcoMambelli.test.tag.skim.tut080820.1 AODtoDPD_mm.py 
pathena --shipInput --inputFileList=myinputs.txt --outDS user08.MarcoMambelli.test.tag.skim.tut080820.2 -site=ANALY_MWT2 AODtoDPD_mm.py 
pathena --shipInput --inputFileList=myinputs.txt --outDS user08.MarcoMambelli.test.tag.skim.tut080820.2 --site=ANALY_MWT2 AODtoDPD_mm.py
...
pathena --shipInput --inputFileList=myinputs.txt --outDS user08.MarcoMambelli.test.tag.skim.tut080820b.5 --site=ANALY_AGLT2 AODtoDPD_mm2.py
...

Results

The first version (with AODtoDPD.py from the release is failing (Athena crash) everywhere.

The second version is completing succesfully in some CEs (e.g. BNL) and failing in other because the input files (AODs) are not everywhere.

Ganga example

The Ganga example from https://twiki.cern.ch/twiki/bin/view/Atlas/RegularComputingTutorial#Running_physics_analysis_on_the is not working correctly.

First I set up the environment as described in https://twiki.cern.ch/twiki/bin/view/Atlas/GangaShortTutorial5 
. tut0808/setup1_cmt.sh
cd
. tut0808/setup2_release.sh
cd scratch0/AtlasOffline-14.2.10/
mkdir ganga_run
cd ganga_run
ln -s ../pathena_run/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root  ./
get_files AnalysisSkeleton_topOptions.py

Setup and run/initialize 
source /afs/cern.ch/sw/ganga/install/etc/setup-atlas.sh
ganga

Preparing IPython script cat <gangaTAG.py j=Job() j.splitter = TNTJobSplitter() j.splitter.query = '' j.splitter.src_collection_name = '$HOME/scratch0/AtlasOffline-14.2.10/ganga_run/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user' # Note that you have to omit the .root in the line above, for reasons that aren't entirely clear! j.splitter.src_collection_type = 'RootCollection' j.splitter.match_ce = True j.splitter.runQuery() j.application=Athena() j.application.exclude_from_user_area=["*.o","*.root*","*.exe"] j.application.prepare(athena_compile=False) j.application.option_file=['$HOME/scratch0/AtlasOffline-14.2.10/ganga_run/AnalysisSkeleton_topOptions.py' ] j.outputdata=DQ2OutputDataset() j.outputdata.outputdata=['AnalysisSkeleton.aan.root' ] j.merger=AthenaOutputMerger() j.backend=LCG() j.backend.iocache='srm-atlas.cern.ch' j.submit() EOF

Run 
ganga gangaTAG.py

I think TNT is tailored for the FDR that required explicit StreamAOD_ref pointer, so it is failing because here the AOD is in the default stream ref. Below is the error message in case you are interested. The changes should be easy but I think they have to be done in TNT (I don't know how to change the reference name) 
/afs/cern.ch/sw/ganga/external/ipython/0.6.13/noarch/lib/python/IPython/genutils.py:1404: Warning: 'with' will become a reserved keyword in Python 2.6
GangaAtlas                         : INFO     Tasks read from file
Ganga.GPIDev.Lib.JobRegistry       : INFO     Found 0 jobs in "templates", completed in 0 seconds
GangaAtlas.Lib.ATLASDataset        : INFO     **** Welcome to TNT! ****
GangaAtlas.Lib.ATLASDataset        : INFO     Using /afs/cern.ch/user/m/mambelli/gangadir/workspace/mambelli/LocalAMGA/1/input/ to store intermediary files
GangaAtlas.Lib.ATLASDataset        : INFO     TNT.log-12590
GangaAtlas.Lib.ATLASDataset        : INFO     authentication.xml found in /afs/cern.ch/user/m/mambelli/scratch0_0808/AtlasOffline-14.2.10/ganga_run
GangaAtlas.Lib.ATLASDataset        : INFO     Executing query '' on tag database...
GangaAtlas.Lib.ATLASDataset        : INFO     XMLFileCatalog     Info Connecting to the catalog

GangaAtlas.Lib.ATLASDataset        : INFO     PoolXMLFileCatalog     Info Xerces-c initialization Number 0

GangaAtlas.Lib.ATLASDataset        : INFO     XMLFileCatalog     Info Connecting to the catalog

GangaAtlas.Lib.ATLASDataset        : INFO     XMLFileCatalog     Info Connecting to the catalog

GangaAtlas.Lib.ATLASDataset        : INFO     XMLFileCatalog     Info Connecting to the catalog

GangaAtlas.Lib.ATLASDataset        : INFO     RootCollection     Info Opening Collection File /afs/cern.ch/user/m/mambelli/gangadir/workspace/mambelli/LocalAMGA/1/input/myEvents.root in mode: READ

GangaAtlas.Lib.ATLASDataset        : INFO     Error in : file /afs/cern.ch/user/m/mambelli/gangadir/workspace/mambelli/LocalAMGA/1/input/myEvents.root does not exist

GangaAtlas.Lib.ATLASDataset        : INFO       CollAppend  Warning Destination Collection `/afs/cern.ch/user/m/mambelli/gangadir/workspace/mambelli/LocalAMGA/1/input/myEvents' does not exist; A new one will be created; 

GangaAtlas.Lib.ATLASDataset        : INFO     RootCollection     Info Opening Collection File /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10/ganga_run/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root in mode: READ

GangaAtlas.Lib.ATLASDataset        : INFO     RootCollection     Info File /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10/ganga_run/fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root opened

GangaAtlas.Lib.ATLASDataset        : INFO     RootCollection     Info Root collection opened, size = 69

GangaAtlas.Lib.ATLASDataset        : INFO     pool::Exception: Column with name `StreamAOD_ref' does NOT exist. ( POOL : "CollectionDescription::column" from "CollectionBase" )

Pathena example with TAG files

Instead of having a TAG file produced by ELSSI there the inputs are TAG files (one or more, e.g. a dataset) where the TAG selection is done in a SQL query

The Pathena job is very similar. The submission string has the TAG dataset as input: 
[lxplus232] /afs/cern.ch/user/m/mambelli/scratch0/AtlasOffline-14.2.10/pathena_run > pathena --inDS fdr08_run2.0052280.physics_Egamma.merge.TAG.o3_f8_m10 --outDS user08.MarcoMambelli.test.tag.skim.tut080820c.3 --site=ANALY_BNL_ATLAS_1 AODtoDPD_mm3.py

Or, if you want to split the job (one for each file in the input dataset), add the option --nFilesPerJob=1 
pathena --inDS fdr08_run2.0052280.physics_Egamma.merge.TAG.o3_f8_m10 --outDS user08.MarcoMambelli.test.tag.skim.tut080820h.2 --site=ANALY_BNL_ATLAS_1 --nFilesPerJob=1 AODtoDPD_mm3.py

The jobOption is very similar to the previous one, it adds
  • a query
  • a bogus input file 
diff ./AODtoDPD_mm3.py /afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODtoDPD.py 
61,68c61
< #svcMgr.EventSelector.InputCollections =[ "/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODs/valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496/AOD.022496._00001.pool.root.1"]
< #svcMgr.EventSelector.InputCollections =[ "TAG_user.root" ]
< svcMgr.EventSelector.InputCollections =[ "fdr08_run2.0052280.physics_Egamma.merge.TAG.o3_f8_m10._0001.1" ]
< svcMgr.EventSelector.CollectionType = "ExplicitROOT"
< #svcMgr.PoolSvc.ReadCatalog += ["xmlcatalog_file:/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/TAG/PoolFileCatalogRun52280Egamma.xml"]
< #svcMgr.EventSelector.Query = "abs(JetEta1)<2.5 && abs(JetEta2)<2.5 && abs(JetEta3)<2.5 && abs(JetEta4)<2.5"
< #svcMgr.EventSelector.Query = "NLooseElectron==1"
< svcMgr.EventSelector.Query = "NLooseElectron==1 && abs(LooseElectronPt1)>25000.0 && NJet>3 && abs(JetPt4)>25000.0 && abs(MissingET)>25000.0"
---
> svcMgr.EventSelector.InputCollections =[ "/afs/cern.ch/atlas/groups/PAT/Tutorial/AtlasOffline-14.2.10/AODs/valid1.005200.T1_McAtNlo_Jimmy.recon.AOD.e322_s435_r432_tid022496/AOD.022496._00001.pool.root.1"]

Instead of having a TAG file produced by ELSSI there the inputs are TAG files (one or more, e.g. a dataset) where the TAG selection is done in a SQL query

Some successful jobs can be checked in Panda:

Other remarks:
  • Successful jobs ran at: ANALY_BNL_ATLAS_1, ANALY_MWT2
  • Failed jobs ran at: ANALY_AGLT2

  • A single job took about
  • Splitting in 6 (1 job per input file), build job took 3min, main jobs took from 35min to 1 hour.
  • With a less selective query (NLooseElectron==1) main jobs took from 2h45 to 3h40.

Failed attempts

The jobOption file has the query and a different input collection (ignore the name because it will be replaced by Pathena with files from the dataset). 
< svcMgr.EventSelector.InputCollections =[ "TAG_user.root" ]
---
> svcMgr.EventSelector.InputCollections =[ "fdr08_run2_physics_Egamma_o3_f8_m10_1216280899_user.root" ]
65,67d64
< #svcMgr.EventSelector.Query = "abs(JetEta1)<2.5 && abs(JetEta2)<2.5 && abs(JetEta3)<2.5 && abs(JetEta4)<2.5"
< #svcMgr.EventSelector.Query = "NLooseElectron==1 AND abs(LooseElectronPt1)>25000.0 AND NJet>3 AND abs(JetPt4)>25000.0 AND abs(MissingET)>25000.0"
< svcMgr.EventSelector.Query = "NLooseElectron==1"

I had several failed attempts
  • mainly I was using a wrong syntax for the query: AND instead of &&
  • some unknown failures at ANALY_MWT2 (no log file available, unknown reason)
  • some failure for missing file at ANALY_SWT2
  • some 'failed connection to DB'?

Setting svcMgr.EventSelector.InputCollections =[ "TAG_user.root" ] was giving an error because of the missing file, but it was not fatal. Isn't Pathena replacing the input collection? Leaving the input collection empty (commenting the line) was causing a Patena error in job submission. At the end I decided to put one of the files that are part of the input dataset. Should I have been putting all of them?

The error when I was having the wrong query 
20 Aug 2008 18:04:21| !!FAILED!!3000!! Job failed: Non-zero failed job return code: 40
- Errors from athena_stdout.txt -
 -+- 82  25 T class AllowIgnoreConfigError(JobProperty):
 -+- 82  25 C class AllowIgnoreConfigError(JobProperty):
 -+- 82  63 T     AllowIgnoreConfigError,
 -+-  7 129   #MessageSvc.OutputLevel = ERROR
EventSelector.s...  ERROR Could not construct TTreeFormula object, probably because of incorrect ROOT predicate syntax in expression `NLooseElectron==1 AND abs(LooseElectronPt1)>25000.0 AND NJet>3 AND abs(JetPt4)>25000.0 AND abs(MissingET)>25000.0' ( POOL : "RootCollectionQuery::execute" from "RootCollection" )
ServiceManager      ERROR Unable to initialize service "EventSelector"
ProxyProviderSvc    ERROR ServiceLocatorHelper::createService: can not create service EventSelector of type EventSelectorAthenaPool
ProxyProviderSvc    ERROR  getting Address Provider EventSelectorAthenaPool/EventSelector
JobOptionsSvc       ERROR Unable to set the property 'ProviderNames' of 'ProxyProviderSvc'. Check option and algorithm names, type and bounds.
ServiceManager      ERROR Unable to initialize service "ProxyProviderSvc"
DetectorStore       ERROR ServiceLocatorHelper::createService: can not create service ProxyProviderSvc
DetectorStore       ERROR Could not get pointer to ProxyProvider Service
ServiceManager      ERROR Unable to initialize service "DetectorStore"
GeoModelSvc         ERROR ServiceLocatorHelper::createService: can not create service DetectorStore
ServiceManager      ERROR Unable to initialize Service: GeoModelSvc
Error in : file TAG_user.root does not exist
Error in :  Bad numerical expression : "1ANDabs(LooseElectronPt1)"
-rwxr-xr-x  1 usatlas1 usatlas   12808 Aug  7 12:31 PilotErrors.py
-rw-r--r--  1 usatlas1 usatlas   13944 Aug 20 14:02 PilotErrors.pyc
- athena errors from athena_stdout.txt -
execute script: Running athena failed : 8448

- Walltime -
jobRetrival=0, StageIn=8, Execution=84, StageOut=0, CleanUp=7
UHURA (25b)

Later attempts

These seem to work, at least they have been running for a while so far...

TODO

  • test other CEs
  • test multiple dataset input
  • test trigger query (prerun?)
  • compare execution times:
    • this way
    • filter in athena (no tag)
    • filter the TAG files with POOL utilities, produce a file with only the events that pass the query, then run Pathena

-- MarcoMambelli - 09 Sep 2008
Edit  | Attach | Print version | History: r1 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r1 - 13 Sep 2008, MarcoMambelli
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback