DSS

Definitions

Kyle brought up the terms skimming, slimming, and thinning during a PAT talk at software week. I've been trying to understand the implications on our tool set. Here's my take.

Skimming, n., the extraction of events of interest from the data store. - this is more or less what the TAG's are made for, and what we've prototyped with our extraction utilities/browser.

Slimming, n., limiting the itemlist during a Skim to a specified list of container#key combinations useful for the physics. - this could be done if we had an actual list of what container#key combinations were in AOD,ESD,... It's not as easy as it should be. The list cannot be compiled from the output because the job options may have container#*. To get it from the job options requires actual execution of python, since it is built from multiple fragments, and varies from release to release. It would be nice if it were saved in the metadata system for each release.

Thinning, n., limiting the objects in each container in the itemlist based on physics principles. - I'm not sure we can help here. Slimming just selects containers, but thinning actually alters their state and does require some code to build the list, which may be hard to generalize. I think we should give good pointers on how to do it, but leave everything else to the physicist.

Summary: We should provide the best skimming help we can, and help with slimming if we can.

New Oracle DB format and Connecting to CERN Oracle

Oracle connection: lxplus:~cranshaw/public/authentication.xml

Script to dump attributes from a Collection: http://atlas-sw.cern.ch/cgi-bin/viewcvs-atlas.cgi/offline/Database/AthenaPOOL/POOLCollectionTools/share/DumpItemList.py?revision=1.2 &view=markup

This is the script I use to dump the item list. If you have set up an Atlas release then do

python DumpItemList.py

Caveats: - You need to have a RecOpts.py in your directory. - The output is more verbose that I would like, but I haven't found the flags to reduce it. - I don't know that this will work with all releases.

Job execution

https://twiki.cern.ch/twiki/bin/view/Atlas/DAonPanda

Patena: Hi Jack,

I've just updated pathena on CERN CVS. If you have InputCollections = [ "streamtest_exclEle1373032551.root" ] in your jobO and the root file exists on the current directoy,

pathena --shipInput ...

will export the root file to WNs and Athena will run on the file. You can use --nEventsPerJob=??? when you want to split the job.

� � Tadashi

From: Jack Cranshaw <cranshaw@anl.gov> Subject: TAG toys Date: Thu, 14 Jun 2007 10:33:49 -0500

Hi Tadashi,

http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/php/TAGS/DavidM/list_columns.php

This is a prototype page with limited functionality for doing a query on a relational TAG database for the streamtest. It allows a certain amount of browsing. It also allows the user to generate a root TAG file based on the selection. For example, the one here

http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/php/TAGS/DavidM/output/streamtest_exclEle1373032551.root.

What would be the easiest way to use this as input to a pathena job since I want to use DA, but it's not in a dataset?

Thanks,

Jack

Other from Jack

Streaming test: https://twiki.cern.ch/twiki/bin/view/Atlas/StreamsTestDistributedAnalysisExamples#Using_Relational_Tags_with_PanDA

Subject: loaded a file Command I was using ~/maxidisk/workarea/data/inclEle/tag % CollListAttrib.exe -src streamtest1 RelationalCollection -srcconnect oracle://int8r/atlas_tags_test

You can find the files that I've been transferring for the streams test in

/afs/cern.ch/user/c/cranshaw/maxidisk/workarea/data/incl*/tag /afs/cern.ch/user/c/cranshaw/maxidisk/workarea/data/excl*/tag

I've been running a script in

/afs/cern.ch/user/c/cranshaw/maxidisk/workarea/data/test.sh

to do the loading by hand.

I forgot to send the connection information for the CERN db.

http://twiki.mwt2.org/bin/view/DataServices/TagDBMysql

Docs

https://twiki.cern.ch/twiki/bin/view/Atlas/StreamsTestRelationalTagDatabase https://twiki.cern.ch/twiki/bin/view/Atlas/StreamsTestDistributedAnalysisExamples https://twiki.cern.ch/twiki/bin/view/Atlas/StreamingTestForUsers https://twiki.cern.ch/twiki/bin/view/Atlas/StreamingTest2006 https://twiki.cern.ch/twiki/bin/view/Main/SampleTagQueries1 http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/php/TAGS/listtables.php https://twiki.cern.ch/twiki/bin/view/Atlas/TagForEventSelection https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasEventTagTools

Please feel free to add, but it might be nice to have this for the discussion.

Cheers,

Jack

On Jun 14, 2007, at 7:55 AM, Helen McGlone wrote:

'>
'>
'>
'>
'> Hi all,
'>
'> https://twiki.cern.ch/twiki/bin/view/Atlas/StreamsTestRelationalTagDatabase
'>
-- MarcoMambelli - 09 Aug 2007