Links

Monitoring

• production dashboard: http://panda.cern.ch:25880/server/pandamon/query?dash=prod (alt port http://panda.cern.ch:25980/server/pandamon/query?dash=prod)
• panda job error summary (past 24 hours): http://panda.cern.ch:25880/server/pandamon/query?days=1&overview=errorlist
• panda job error summary 2: http://panda.cern.ch:25880/server/pandamon/query?hours=6&overview=errorlist
• production job blocks (5 day view): http://panda.cern.ch:25880/server/pandamon/query?overview=prodlist&type=production&hours=120
• dashboard: http://dashb-atlas-prodsys.cern.ch/dashboard/request.py/shifters-display
• Task list: http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=listtask
• PROD server, main panda monitor page: http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query
• DEV server: http://pandamondev.usatlas.bnl.gov:25880/
• CERN server: http://panda.cern.ch:25880/
• pandamon.usatlas.bnl.gov:25880 became panda.cern.ch:25880
• Redirector: http://panda.atlascomp.org/

Monitoring2

• nagios main page (BNL): https://www.usatlas.bnl.gov/nagios/
  • Tier2 services: https://www.usatlas.bnl.gov/nagios/tier2.html
  • BNL services: https://www.usatlas.bnl.gov/nagios/sla_array.html
• NOT-WORKING obsolete info Hiro's dq2 status pages: http://www.usatlas.bnl.gov/dq2/monitor/dq2Pings
• fts status: https://www.usatlas.bnl.gov/fts/
• Panglia: http://gridinfo.triumf.ca/panglia/
• Physics DB service at CERN: http://phydb.web.cern.ch/phydb/
• GOC downtime: https://goc.gridops.org/downtime
• CERN Site Services (VOBOXES): https://sls.cern.ch/sls/service.php?id=ATLAS_DDM_VOBOXES (need NICE login)
• DQ2 Central Catalogues: http://phydb.web.cern.ch/phydb/
• Gstat: http://goc.grid.sinica.edu.tw/gstat//Region.html

Logs and tickets

• ADCoS shift log: https://prod-grid-logger.cern.ch/elog/ATLAS+Computer+Operations+Logbook/ (old logbook: http://atlas003.uta.edu:8080/ADCoS/ )
• savannah validation bug reporting: https://savannah.cern.ch/projects/validation/
• RT ticketing system (BNL): https://rt-racf.bnl.gov/rt/
• ggus reporting system: https://gus.fzk.de/pages/home.php
• ADC eLog: https://prod-grid-logger.cern.ch/elog/ATLAS+Computer+Operations+Logbook/ (same as new ADCoS)
• My VDT tickets: http://crt.cs.wisc.edu/Search/Simple.html?user=guest&pass=guest&q=marco%40hep.uchicago.edu

Documents

• panda wiki (click on "system operations," then "shift guide" for more shift details): https://twiki.cern.ch/twiki/bin/view/Atlas/PandaShiftGuide
• atlas-wide shift wiki: https://twiki.cern.ch/twiki/bin/view/Atlas/ADCoS
• Panda docs: https://twiki.cern.ch/twiki/bin/view/Atlas/PanDA
• Shift calendar/booking
  • http://grid.uchicago.edu/cgi/plans.cgi?cal_id=1
  • OTP: https://pptevm.cern.ch/mao/client/cern.ppt.mao.app.gwt.MaoClient/MaoClient.html#home
  • EU calendar: http://atlas01.pic.es/Shift_Calendar/cal.php
• Shift report: http://lhcweb01.pic.es/atlas/ (was http://atlas01.pic.es/Shift_Summaries/dailycheck_form.htm )
• Pilot blog: http://pandapilotblog.blogspot.com/
• Authorship: http://atlas-authdb.web.cern.ch/atlas-authdb/
• Access policies: https://twiki.cern.ch/twiki/bin/view/Atlas/AccessProtectedInformation

ATLAS at BNL

• hypernews request: https://www.racf.bnl.gov/experiments/usatlas/analysis/hypernews
• Tier 3 Hypernews: https://www.usatlas.bnl.gov/HyperNews/racf/get/Tier3Support.html
• Membership at CERN:
  • https://atglance.web.cern.ch/atglance/membership/memberDetails.php?id=802
  • http://atglance.web.cern.ch/atglance/cgi-bin/glanceAplic
  • Registration Form: http://atlassec.web.cern.ch/atlassec/Registration.htm

Production caches and important files:

In the 1st one are the release Distribution Kits and AtlasPoint1, in the 2nd one are the TRFs. The 3rd one is a cache.

• http://atlas-computing.web.cern.ch/atlas-computing/links/kitsDirectory/projects/cache/
• http://atlas-computing.web.cern.ch/atlas-computing/links/kitsDirectory/Production/cache/
• http://classis01.roma1.infn.it/pacman/Production/cache
• Schedule: http://atlas-computing.web.cern.ch/atlas-computing/projects/releases/status/
• TiersOfATLAS: http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/TiersOfATLASCache.py
• Alt ToA: http://atlas.web.cern.ch/Atlas/GROUPS/DATABASE/project/ddm/releases/testbed/TiersOfATLASCache.py
• OSG Pacman cache: http://software.grid.iu.edu/pacman/
• MWT2 Pacman cache: http://www.mwt2.org/caches/

Analysis

Old servers were http://panda.cern.ch:25980/server/pandamon/ and http://pandamon.usatlas.bnl.gov:25880/.

• https://twiki.cern.ch/twiki/bin/view/AtlasProtected/PathenaAnalysisQueues (was https://twiki.cern.ch/twiki/bin/view/Atlas/PathenaAnalysisQueues)
• User list: http://panda.cern.ch:25980/server/pandamon/query?ui=users&days=7&sort=latest
• Marco Mambelli:
  • http://panda.cern.ch:25980/server/pandamon/query?ui=user&name=Marco%20Mambelli (was http://pandamon.usatlas.bnl.gov:25880/?ui=user&name=Marco%20Mambelli)
• http://atlas-sw.cern.ch/cgi-bin/viewcvs-atlas.cgi/offline/Tools/Scripts/share/setupWorkArea.py
• http://panda.cern.ch:25980/server/pandamon/query?dash=analysis
• Elog
  • http://atlas003.uta.edu:8080/ADCoS/?mode=summary
  • https://prod-grid-logger.cern.ch/elog/ATLAS+Computer+Operations+Logbook/
• File Replication (subscription): http://panda.cern.ch:25980/server/pandamon/query?mode=reqsubs0 (was: http://panda.cern.ch:25980/server/pandamon/query?mode=reqsubs0 )
• User registration to Panda server and DDM replication (marco): http://panda.cern.ch:25980/server/pandamon/query?mode=login&action=info&nickname=marco&email=marco%40hep.uchicago.edu&submit=GetInfo

Clients

• Reference pages
  • http://www.usatlas.bnl.gov/twiki/bin/view/Admins/WlcgClient
  • http://www.usatlas.bnl.gov/twiki/bin/view/Admins/AtlasWorkerNode
  • http://www.usatlas.bnl.gov/twiki/bin/view/Admins/DQ2ClientsInOSG
  • https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/ClientInstallationGuide
• Others: https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2Clients
• Some answers: MarcoUserInfo
• DQ2Clients bugzilla: https://savannah.cern.ch/bugs/?group=atlas-ddm
• https://twiki.cern.ch/twiki/bin/view/Atlas/DQ2ClientsTroubleshooting
• ATLAS Space Tokens(If you act on behalf of a (physics, combined performance or detector) group, you can ask for data replication to a GROUPDISK. Your own data will be on a SCRATCHDISK as it is produced, and you have to move it to a LOCALUSERDISK where you have write access (i.e. you are a local user).)
  • http://lcg.in2p3.fr/wiki/index.php/Atlas:StorageRequirements
  • https://twiki.cern.ch/twiki/bin/view/Atlas/SpaceTokens
  • http://www.nikhef.nl/~bosk/presentations/2009/StorageIssues.pdf

Meetings

• US-ATLAS TB meeting: http://www.usatlas.bnl.gov/twiki/bin/view/Admins/IntegrationProgram
• http://www.usatlas.bnl.gov/twiki/bin/view/Admins/MinutesJul9
• ACD Shift meeting agenda: http://indico.cern.ch/categoryDisplay.py?categId=1706
• AG calendar: https://agschedule.ncsa.uiuc.edu/default.asp
• Campfire IM: http://integrationcloud.campfirenow.com/

Monitoring notes and gotchas

Some notes about time and timestamps in different monitoring sites:

• Panda monitor is using UTC (date -u: Chicago -6, -5 if DST)
• ARDA dashboard (DDM) is using CERN time (Chicago -7)

ATLAS membership

See the documents.

Tasks

Check site

To check if a site is available

• Downtime announcements: https://goc.gridops.org/downtime
• Panda monitor
• Twiki page with analysis sites

Report Panda monitor problems

Sometimes you can see a number of issues with Panda-monitoring, for example: errors when you press some buttons, time-outs, slow response, etc. In order to help the experts to debug and fix the issue, if you see such a problem in your browser next time could you please report the following information:

1. Any error in Panda monitor received through the browser.

• please cut and paste the error-text in your browser
• URL-string you use in the browser
• your ip-address
• type of the browser you use

2. Time-outs in Panda-monitor/browser

• cut and paste time-out error, including the time if available
• whether the time-out is reproducible
• URL-string you use in the browser
• your ip-address
• type of the browser you use

3. Slow response in Panda-monitor (no error)

• How long does it take to execute the function in the browser
• URL-string you use in the browser
• button you use in Panda-mon (or the query)
• your IP-address and the type of the browser

From Yury, email, 230608

Pilot queue control

This command controls the queue length (tpmes=setnqueue&nqueue=). Here sets nqueue=10 for queue UC_ATLAS_MWT2-condor

$ curl 'http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?tpmes=setnqueue&nqueue=10&queue=UC_ATLAS_MWT2-condor'                             

Each queue can be set manual/auto (defines how queue parameters are updated) and online/offline (defines if ATLAS jobs are assigned to the queue), so possible commands are: tpmes= setmanual setauto setoffline setonline

curl -sS  'http://gridui07.usatlas.bnl.gov:25880/server/pandamon/query?tpmes=setmanual&queue=Taiwan-LCG2-lcg00125-atlas-lcgpbs'
curl -sS  'http://gridui07.usatlas.bnl.gov:25880/server/pandamon/query?tpmes=setoffline&queue=Taiwan-LCG2-lcg00125-atlas-lcgpbs'
curl -sS 'http://gridui07.usatlas.bnl.gov:25880/server/pandamon/query?tpmes=setonline&queue=Taiwan-LCG2-lcg00125-atlas-lcgpbs'
curl -sS  'http://gridui07.usatlas.bnl.gov:25880/server/pandamon/query?tpmes=setauto&queue=Taiwan-LCG2-lcg00125-atlas-lcgpbs'

Find task bugs

If it is not site-related issue. E.g. all jobs of this task failed with that error up to 7 attempts each at different sites: MWT2_UC,BNL_ATLAS_1,SLACXRD,OU_OCHEP_SWT2,AGLT2,etc. Probably this task needs to be ABORTED.

Before submitting a new bug about the task in question you can always check whether the bug related to this task is already in Savannah. You can use Panda montor, "Quck Search" in the left yellow bar. Just type the task ID 23381 in "Task request", you will get http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?qTID=23381&mode=taskquery&qsubmit=QuerySubmit "Bug report" shows the corresponding # in Savannah (previous one was overwritten by the current one).

There is another way to check whether the bug was already submitted - it's a direct search in Savannah: https://savannah.cern.ch/bugs/?func=search&group=validation

IM use

There is an 'atlasshift' chat room with no password in Google Talk (using partychat)

To join it:

1. If you have no Google account go to http://www.google.com/talk/ download a client or open the web one and register
2. open your favorite Google Talk client
3. add partychat#@gmail.com (# is any digit 0-9) as your buddy: this is not a real person; it is a service gateway. Steps 2-3 are documented also at http://techwalla.googlepages.com/
4. open a message window to talk with partychat#@gmail.com (double click, IM, ...)
5. send the message '/join atlasshift' . If you are using Pidgin or another client that supports IRC like commands (you will see 'Unknown command error' instead of

your message) then type '/say /join atlasshift' (this protects your message for partychat). Partychat will reply to you saying that you joined the chat room

1. '/list' to see who is there
2. type something to talk with all the other users in the chat room
3. '/exit' to go out (exit the chat room)

'atlasshift' is a temporary name. If you don't like it I can change it, as long as we agree on something.

Production servers

Here is a document on login on RACF (BNL) machines: https://www.racf.bnl.gov/docs/howto/interactive/login

I neded to contact Dantong and provide my public key. atlasgw is forwarding ssh-agent (key used to login there).

gridui03,7,9 are the ones that run autopilots. gridui05,6,7 also need healthy proxies, the Panda server and monitor run there (the monitor needs proxy for DQ2 and LFC access and logfile retrieval). gridui01 is now just an alias, the former gridui01 is now a development machine. The new servers gridui05+ require that each person requiring access to sm be specifically authorized. John or Dantong can do this (and John has said shifters can and should do this) - Torre

Release updates

Releases installes from 14.1 can be checked in BDII

To know which tags (options) are possible in ATLAS kit setup you can check AtlasLogin and AtlasSetup files: https://twiki.cern.ch/twiki/bin/view/Atlas/AtlasLogin

Sometime these files need to be updated

Pacman commands

pacman -update-check can be used to check for updates in packages. It is not intrusive (writes to log files only). For more info on carefully update: http://physics.bu.edu/pacman/htmls/Updating.html

To update selectively you can mention the package names:

                                                                                                                                                                                                      
% pacman -update AtlasSettings AtlasLogin                                                                                                                                                                                            

Subversion (SVN)

Help:

• https://twiki.cern.ch/twiki/bin/view/Atlas/SoftwareDevelopmentWorkBookSVN
• http://svn.web.cern.ch/svn/howto.php#accessing-clients

Browsing:

• https://svnweb.cern.ch/trac/atlasoff/browser
• https://svnweb.cern.ch/cern/wsvn/atlasoff

export SVNROOT=svn+ssh://mambelli@svn.cern.ch/reps/atlasoff
cmt co [-r ] [/]
svn co $SVNROOT/[/]/trunk 

Dist Analysis

For support: hn-atlas-dist-analysis-help@cern.ch

MWT2 info

• System info: http://www.mwt2.org/sys/

Best practices

Check for installed releases

https://atlas-install.roma1.infn.it/atlas_install/

Ticket submission

GGUS ticket in the weekend are not useful (they are not routed until Monday). Better to route the ticket directly and/or add in CC the site contact: Site will receive the request and GGUS will still track the problem

Notes

Sure we can add, but could you be more specific? I did not follow your                                                                                                    
discussion so I need more explanations which services you want to send                                                                                                    
notifications to this address.                                                                                                                                            
                                                                                                                                                                          
And by the way: there are already pages which show various open tickets                                                                                                   
on www, without the need to logon to rt. go to rt page                                                                                                                    
                                                                                                                                                                          
https://rt-racf.bnl.gov/rt/                                                                                                                                               
                                                                                                                                                                          
and select one of the links on the bottom half of the page.                                                                                                               
                                                                                                                                                                          
Or you can go directly to                                                                                                                                                 
                                                                                                                                                                          
http://rt-racf.bnl.gov/rss/tier2.rss                                                                                                                                      
                                                                                                                                                                          
for tier2 related tickets,                                                                                                                                                
                                                                                                                                                                          
to                                                                                                                                                                        
http://rt-racf.bnl.gov/rss/nagios.rss                                                                                                                                     
                                                                                                                                                                          
for nagios tickets, and to                                                                                                                                                
http://rt-racf.bnl.gov/rss/itb.rss                                                                                                                                        
                                                                                                                                                                          
for osg tickets.                                                                                                                                                          
                                                                                                                                                                          
You can see the text of the original ticket and the first response. This                                                                                                  
should be enough for fast browsing. To see more you will have to logon                                                                                                    
to rt.                                                                                                                                                                    
                                                                                                                                                                          
Please let me know if this is what you want.                                                                                                                              

To do

• check cacti
• Differential page

• Error page with results like:

http://pandamon.usatlas.bnl.gov:25880/server/pandamon/query?mode=archive&site=MWT2_IU&hours=12&sort=endTime http://gridui05.usatlas.bnl.gov:25880/server/pandamon/query?hours=6&overview=errorlist -- MarcoMambelli - 06 Jun 2008