You are here: Foswiki>Main Web>WikiUsers>MarcoMambelli>MarcoRepWeekly>MarcoWR0808>TestSkim080221 (18 Mar 2008, MarcoMambelli)Edit Attach

TestSkim080221

Intro

Directory /local/workdir/mm-s1/ and subdirectories

README file with info

DB links:

Some TNT links:

Some Ganga links:

Query 





 Manual job splitting 

Found a bug in splitting exe, working with Caitriana.
Workaround is to use CollCopy (or CollAppend for 13.X) before invoking CollSplitByGUID.exe



[marco@tier2-06 splitting]$ CollCopy.exe -src streamtest_inclJet_v3_1203442639_user RootCollection \
> -dst myTest RootCollection -queryopt 'SELECT RunNumber, EventNumber'
CollCopy: Finished copying input collection(s) `streamtest_inclJet_v3_1203442639_user:RootCollection' to output collection(s) `myTest:RootCollection' 
[marco@tier2-06 splitting]$ ls

[marco@tier2-06 splitting]$ /local/workdir/skimming-athena/splitting/CollSplitByGUID.exe -src /local/workdir/skimming-athena/splitting/myTest RootCollection -minevents 100 





 Manual DQ2 info exploration 


from dq2.common.Config import Config
from dq2.clientapi.DQ2 import DQ2
dq=DQ2('http://atlddmcat.cern.ch:80/','https://atlddmcat.cern.ch:443/','http://atlddmcat.cern.ch:80/','https://atlddmcat.cern.ch:443/','http://atlddmcat.cern.ch:80/','https://atlddmcat.cern.ch:443/','http://atlddmcat.cern.ch:80/','https://atlddmcat.cern.ch:443/')
dq.contentClient.queryDatasetsWithFileByGUID('04A50ABB-D82F-DC11-89AE-00E08127C853')
['6848bb7d-5922-4850-b6e7-9312eee3697d']
>>> dataset = dq.repositoryClient.resolveVUID('6848bb7d-5922-4850-b6e7-9312eee3697d') 
 print dataset
 {'dsn': 'streamtest.004979.inclJet.merge.AOD.v12000699', 'version': 1}







 Manual Skim on subcollection 1 (i file) 

It ran successfully (lot of output, some minutes to complete).
The output AOD is smaller than the input one.
Count events to verify (or better check content) 

[marco@tier2-06 skim_exe]$ cp ../splitting/sub_collection_1.root ./
[marco@tier2-06 skim_exe]$ ln -s sub_collection_1.root tag.pool.root
[marco@tier2-06 skim_exe]$ less ../README 
[marco@tier2-06 skim_exe]$ CollListFileGUID.exe -src sub_collection_1 RootCollection
04A50ABB-D82F-DC11-89AE-00E08127C853
[marco@tier2-06 skim_exe]$ curl http://uct2-grid1.uchicago.edu:8000/dq2/lrc/PoolFileCatalog/?guids=04A50ABB-D82F-DC11-89AE-00E08127C853

...      
    
...

[marco@tier2-06 skim_exe]$ pool_insertFileToCatalog streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root
 streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root.1
-rw-rw-r--  1 marco marco 596882983 Feb 21 15:20 streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root.1
-rw-r--r--  1 marco marco     10433 Feb 21 13:41 sub_collection_1.root
lrwxrwxrwx  1 marco marco        21 Feb 21 13:42 tag.pool.root -> sub_collection_1.root
-rw-r--r--  1 marco marco 109822871 Feb 21 15:41 test080221outputAOD





 Pathena on subcollection 1 (i file) 


[marco@tier2-06 skim_pathena]$ ls -l
total 28
-rw-rw-r--  1 marco marco   383 Feb 21 15:47 PoolFileCatalog-inputfile.xml
-rw-rw-r--  1 marco marco   383 Feb 21 15:49 PoolFileCatalog.xml
-rw-rw-r--  1 marco marco   588 Feb 21 15:47 myJobOption.py
lrwxrwxrwx  1 marco marco    86 Feb 21 15:47 streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root -> ../skim_exe/streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root.1
-rw-r--r--  1 marco marco 10433 Feb 21 15:47 sub_collection_1.root
lrwxrwxrwx  1 marco marco    21 Feb 21 15:48 tag.pool.root -> sub_collection_1.root









[marco@tier2-06 skim_pathena]$ . /local/inst/pandajs/setup.sh 
[marco@tier2-06 skim_pathena]$ source /share/app/atlas_app/atlas_rel/13.0.30/cmtsite/setup.sh -tag=13.0.30.15,AtlasPoint1
[marco@tier2-06 skim_pathena]$ ls    
PoolFileCatalog-inputfile.xml  streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root
PoolFileCatalog.xml            sub_collection_1.root
myJobOption.py                 tag.pool.root
[marco@tier2-06 skim_pathena]$ mkdir pathena
[marco@tier2-06 skim_pathena]$ cd pathena/
[marco@tier2-06 pathena]$ export CMTPATH=`pwd`:${CMTPATH}
[marco@tier2-06 pathena]$ export CVSROOT=:ext:mambelli@atlas-sw.cern.ch:/atlascvs;
[marco@tier2-06 pathena]$ export CVS_RSH=ssh; 
[marco@tier2-06 pathena]$ export PATHENA_GRID_SETUP_SH=/local/inst/pandajs/setup.sh 
[marco@tier2-06 pathena]$ cmt co PhysicsAnalysis/DistributedAnalysis/PandaTools;
# ================= working on package PandaTools version PandaTools-00-00-07 (At head)  path PhysicsAnalysis/DistributedAnalysis in /local/workdir/mm-s1/skim_pathena/pathena/PhysicsAnalysis/DistributedAnalysis/PandaTools
  # get top files 
cvs update: Updating .
Creating setup scripts.
Creating cleanup scripts.
[marco@tier2-06 pathena]$ cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
[marco@tier2-06 cmt]$ source setup.sh
make [marco@tier2-06 cmt]$ make 
...
------> (constituents.make) install_python_modules done
 all ok.
[marco@tier2-06 cmt]$ cd /local/workdir/mm-s1/skim_pathena/pathena/
[marco@tier2-06 pathena]$ ls
InstallArea  PhysicsAnalysis
[marco@tier2-06 pathena]$ mkdir run
[marco@tier2-06 pathena]$ cd run/
[marco@tier2-06 run]$ cp ../../myJobOption.py ../../sub_collection_1.root ./
[marco@tier2-06 run]$ ln -s sub_collection_1.root  tag.pool.root
[marco@tier2-06 run]$ ls
myJobOption.py  sub_collection_1.root  tag.pool.root
[marco@tier2-06 run]$ pathena myJobOption.py --inDS streamtest.004979.inclJet.merge.AOD.v12000699 --outDS user.MarcoMambelli.test.skimtest080221.v1







 Pathena on subcollection 1 (i file), ATLAS Rel12 


[marco@tier2-06 skim_pathena]$ cd ..
[marco@tier2-06 mm-s1]$ cd skim_pathena12/
[marco@tier2-06 skim_pathena12]$ ln -s ../skim_exe/streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root.1 streamtest.004979.inclJet.merge.AOD.v12000699_tid000000._00002.pool.root
[marco@tier2-06 skim_pathena12]$ mkdir rum
[marco@tier2-06 skim_pathena12]$ mkdir run
[marco@tier2-06 skim_pathena12]$ rmdir rum
[marco@tier2-06 skim_pathena12]$ . /local/inst/pandajs/setup.sh 
[marco@tier2-06 skim_pathena12]$ source /share/app/atlas_app/atlas_rel/12.0.6/cmtsite/setup.sh -tag=12.0.6,AtlasOffline 
AtlasLogin: Warning - SLC3 development support not available
[marco@tier2-06 skim_pathena12]$ export CMTPATH=`pwd`:${CMTPATH}
[marco@tier2-06 skim_pathena12]$ export CVSROOT=:ext:mambelli@atlas-sw.cern.ch:/atlascvs;
[marco@tier2-06 skim_pathena12]$ export CVS_RSH=ssh; 
[marco@tier2-06 skim_pathena12]$ export PATHENA_GRID_SETUP_SH=/local/inst/pandajs/setup.sh 
[marco@tier2-06 skim_pathena12]$ cmt co PhysicsAnalysis/DistributedAnalysis/PandaTools;

# ================= working on package PandaTools version PandaTools-00-00-08 (At head)  path PhysicsAnalysis/DistributedAnalysis in /local/workdir/mm-s1/skim_pathena12/PhysicsAnalysis/DistributedAnalysis/PandaTools
  # get top files 
cvs update: Updating .
Creating setup scripts.
Creating cleanup scripts.

[marco@tier2-06 skim_pathena12]$ cd PhysicsAnalysis/DistributedAnalysis/PandaTools/cmt
[marco@tier2-06 cmt]$ source setup.sh
,ake[marco@tier2-06 cmt]$ make
------> (Makefile.header) Rebuilding constituents.make
------> (constituents.make) Rebuilding setup.make i686-slc3-gcc323-opt.make
..
------> (constituents.make) install_python_modules done
 all ok.


[marco@tier2-06 run]$ pathena myJobOption.py --inDS streamtest.004979.inclJet.merge.AOD.v12000699 --outDS user.MarcoMambelli.test.skimtest080221.v1
extracting run configuration
ConfigExtractor > Input=COLL
ConfigExtractor > Output=AOD
archive sources
archive InstallArea
post sources/jobO
query files in dataset:streamtest.004979.inclJet.merge.AOD.v12000699
submit
===================
 JobID  : 15
 Status : 0
  > build
    PandaID=7723271
  > run
    PandaID=7723272
[marco@tier2-06 run]$ 


The first submission failed due to BNL problem (seems): out file to SE failed

[marco@tier2-06 run]$ pathena_util
>>> status(15)
----------------------
buildJob   : failed
----------------------
runAthena  :
          ----
----------------------
>>> retry(15)
No retry: status of buildJob 7723271 is failed (!= finished)


Retry seems not to work, resubmitted the whole job

[marco@tier2-06 run]$ pathena myJobOption.py --inDS streamtest.004979.inclJet.merge.AOD.v12000699 --outDS user.MarcoMambelli.test.skimtest080221.v1
extracting run configuration
ConfigExtractor > Input=COLL
ConfigExtractor > Output=AOD
archive sources
archive InstallArea
post sources/jobO
query files in dataset:streamtest.004979.inclJet.merge.AOD.v12000699
query files in dataset:user.MarcoMambelli.test.skimtest080221.v1
submit
===================
 JobID  : 16
 Status : 0
  > build
    PandaID=7723323
  > run
    PandaID=7723324


Failed again. There is an error with BNL storage:  
Then BNL is closed due to snowstorm today


Must ship the input:

pathena --shipInput --inputFileList input_list --outDS user.MarcoMambelli.test080227 myJobOption.py





-- MarcoMambelli - 21 Feb 2008   




Edit  | Attach | Print version | History: r7 < r6 < r5 < r4 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r7 - 18 Mar 2008, MarcoMambelli








 This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors. 
 Ideas, requests, problems regarding Foswiki? Send feedback