PeeringLHCONE

LHCONE sites and info

• https://twiki.cern.ch/twiki/bin/view/LHCONE/SiteList
• http://lhcone.web.cern.ch/
• https://twiki.cern.ch/twiki/bin/view/LHCONE/LhcOneHowToConnect

LHCONE and PerfSONAR dashboards

• https://perfsonar.racf.bnl.gov:8443/exda/?page=25&cloudName=LHCONE
• http://uct2-net1.uchicago.edu/toolkit/ - latency
• http://uct2-net2.uchicago.edu/toolkit/ - bandwidth

ESnet monitors

• https://stats.es.net/graphite/
• OSCARS circuit for vlan 2622 BNL to Chicago es.net-2707 1G reservation BNL ATLAS to UOC.
• This circuit reservation is 1Gbps and the below graphite graph indicates that the load is actually peaking at 3Gbps. Beyond 1Gbps packet loss may occur during periods of congestion.
• The esnet interfaces supporting this circuit:

  bnl-mr3:xe-8/1/0
  bnl-mr3:xe-8/2/0.0
  aofa-sdn1:xe-9/1/0.0
  aofa-sdn1:xe-9/0/0.0
  newy-sdn1:xe-7/2/0.0
  newy-sdn1:xe-1/1/0.0
  wash-sdn2:xe-2/2/0.0
  wash-sdn2:xe-1/1/0.0
  clev-sdn1:xe-1/2/0.0
  clev-sdn1:xe-7/1/0.0
  chic-sdn2:xe-7/0/0.0
  chic-sdn2:xe-7/1/0
  

• The chic-sdn2:xe-7/1/0 interface is the circuit handoff with CICnet at the OMNIPOP in Chicago.
• screenshot_03.jpg:
  

ATLAS monitors

• http://adc-monitoring.cern.ch/
• WLCG transfers involving MWT2 endpoints
• Popularity - we should see rates increasing here:
  • CERN-PROD
  • FZK-LCG2

MWT2 - UChicago

uct2-6509 to Campus

• 6509 trunks
• Te1/1: http://www.mwt2.org/cacti/graph.php?action=view&local_graph_id=2908&rra_id=all
• Te1/2: http://www.mwt2.org/cacti/graph.php?action=view&local_graph_id=4318&rra_id=all
• Bond: http://www.mwt2.org/cacti/graph.php?action=view&local_graph_id=4319&rra_id=all

Tracepaths pre-peering (before 9/7/12)

iut2-s4.iu.edu

rwg@uct2-mgt ~]$ tracepath iut2-s4.iu.edu
 1:  uct2-mgt.uchicago.edu (128.135.158.232)                0.147ms pmtu 1500
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       0.445ms 
 2:  10.4.247.237 (10.4.247.237)                            0.462ms 
 3:  10.4.247.225 (10.4.247.225)                          asymm  2   1.053ms 
 4:  149.165.227.21 (149.165.227.21)                        5.691ms 
 5:  iut2-s4.iu.edu (149.165.225.218)                       8.114ms reached
     Resume: pmtu 1500 hops 5 back 5 

head01.aglt2.org

[root@uct2-s4 ~]# tracepath head01.aglt2.org
 1:  uct2-s4.uchicago.edu (128.135.158.184)                 0.064ms pmtu 9000
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       1.477ms 
 2:  10.4.247.229 (10.4.247.229)                            1.692ms 
 3:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125) asymm  4   5.652ms 
 4:  xe-0-1-3.0.rtr.newy32aoa.net.internet2.edu (64.57.28.115) asymm  5  28.560ms 
 5:  198.32.11.70 (198.32.11.70)                           39.761ms 
 6:  192.65.196.241 (192.65.196.241)                      asymm  4  34.945ms 
 7:  192.65.196.241 (192.65.196.241)                      asymm  4  30.910ms 
 8:  192.65.196.246 (192.65.196.246)                      asymm  5  35.396ms 
 9:  head01.aglt2.org (192.41.230.44)                     asymm  6  33.068ms !H
     Resume: pmtu 9000 

nettest10g.usatlas.bnl.gov

 [rwg@uct2-s1 ~]$ traceroute nettest10g.usatlas.bnl.gov
traceroute to nettest10g.usatlas.bnl.gov (192.12.15.25), 30 hops max, 40 byte packets
 1  b98-b102-200-v624.uchicago.edu (128.135.158.131)  0.478 ms  0.531 ms  0.665 ms
 2  10.4.247.229 (10.4.247.229)  0.425 ms 10.4.247.237 (10.4.247.237)  0.489 ms 10.4.247.229 (10.4.247.229)  0.474 ms
 3  10.4.247.224 (10.4.247.224)  0.536 ms omnipop-atlas-bnl-10g-router.uchicago.edu (128.135.247.129)  25.007 ms  25.071 ms
 4  nettest10g.usatlas.bnl.gov (192.12.15.25)  24.453 ms  24.455 ms  24.515 ms

perfsonar2-de-kit.gridka.de

rwg@uct3-s1:$ tracepath perfsonar2-de-kit.gridka.de
 1:  uct3-s1.uchicago.edu (128.135.158.245)                 0.068ms pmtu 9000
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       2.524ms 
 2:  10.4.247.237 (10.4.247.237)                            1.217ms 
 3:  10.4.247.224 (10.4.247.224)                          asymm  2   1.365ms 
 4:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125)   1.236ms 
 5:  ge-2-3-0.0.rtr.wash.net.internet2.edu (64.57.28.201) 351.129ms 
 6:  dfn-gw.rt1.fra.de.geant.net (62.40.124.34)           111.312ms 
 7:  xr-fzk1-te2-3.x-win.dfn.de (188.1.145.50)            112.385ms 
 8:  xr-fzk1-te2-3.x-win.dfn.de (188.1.145.50)            asymm  7 114.519ms pmtu 1500
 9:  tr-v700-rig1.gridka.de (192.108.46.129)              117.360ms 
10:  tr-v700-rig1.gridka.de (192.108.46.129)              asymm  9 117.272ms 
11:  tn-msfc-vrf-gtn3-1.gridka.de (192.108.46.137)        asymm 10 119.105ms 
12:  tn1480-vrf-gtn51-ospf47.gridka.de (192.108.46.214)   asymm 11 115.566ms 
13:  perfsonar2-de-kit.gridka.de (192.108.47.12)          asymm 11 117.415ms reached
     Resume: pmtu 1500 hops 13 back 11 

ps-bandwidth.lhcopn-mon.triumf.ca

[root@uct2-s4 ~]# tracepath ps-bandwidth.lhcopn-mon.triumf.ca
 1:  uct2-s4.uchicago.edu (128.135.158.184)                 0.056ms pmtu 9000
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       1.020ms 
 2:  10.4.247.237 (10.4.247.237)                            2.131ms 
 3:  10.4.247.224 (10.4.247.224)                          asymm  2   1.963ms 
 4:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125)  12.271ms 
 5:  ae-0.10.rtr.kans.net.internet2.edu (64.57.28.36)      46.579ms 
 6:  ge-6-0-0.0.rtr.salt.net.internet2.edu (64.57.28.24)   65.612ms 
 7:  207.231.240.21 (207.231.240.21)                       53.153ms 
 8:  c4-bcnet.canet4.net (205.189.32.193)                 asymm  9  52.657ms 
 9:  no reply
10:  no reply
11:  no reply

ps-latency.scinet.utoronto.ca

[rwg@uct2-s1 ~]$ traceroute ps-latency.scinet.utoronto.ca
traceroute to ps-latency.scinet.utoronto.ca (142.150.188.69), 30 hops max, 40 byte packets
 1  b98-b102-200-v624.uchicago.edu (128.135.158.131)  1.540 ms  1.841 ms  2.211 ms
 2  10.4.247.237 (10.4.247.237)  2.038 ms 10.4.247.229 (10.4.247.229)  5.503 ms 10.4.247.237 (10.4.247.237)  1.624 ms
 3  10.4.247.224 (10.4.247.224)  1.993 ms omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125)  39.181 ms  39.172 ms
 4  toro1rtr1.canarie.ca (205.189.32.98)  12.328 ms  12.317 ms  12.304 ms
 5  toro1rtr1.canarie.ca (205.189.32.98)  12.294 ms 205.189.32.42 (205.189.32.42)  12.806 ms  12.751 ms
 6  be201.p01-toro.orion.on.ca (66.97.16.21)  12.734 ms  11.547 ms 205.189.32.42 (205.189.32.42)  11.242 ms
 7  be201.p01-toro.orion.on.ca (66.97.16.21)  11.602 ms GTANET-ORION-RNE.DIST1-TORO.IP.orion.on.ca (66.97.23.58)  11.750 ms be201.p01-toro.orion.on.ca (66.97.16.21)  11.577 ms
 8  GTANET-ORION-RNE.DIST1-TORO.IP.orion.on.ca (66.97.23.58)  11.725 ms utoronto-ut-hub-if-re.gtanet.ca (205.211.94.234)  12.316 ms  11.688 ms
 9  scinet-keele-if.gw.utoronto.ca (128.100.200.198)  12.351 ms utoronto-ut-hub-if-re.gtanet.ca (205.211.94.234)  11.884 ms scinet-keele-if.gw.utoronto.ca (128.100.200.198)  12.328 ms
10  ps-latency.scinet.utoronto.ca (142.150.188.69)  11.821 ms scinet-keele-if.gw.utoronto.ca (128.100.200.198)  12.186 ms  12.470 ms
[rwg@uct2-s1 ~]$ 

lcg-lrz-perfs2.grid.lrz.de

rwg@uct3-s1:$ tracepath lcg-lrz-perfs2.grid.lrz.de
 1:  uct3-s1.uchicago.edu (128.135.158.245)                 0.064ms pmtu 9000
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       0.524ms 
 2:  10.4.247.237 (10.4.247.237)                            1.256ms 
 3:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125) asymm  4   1.183ms 
 4:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125)   1.327ms 
 5:  abilene-wash.rt1.fra.de.geant.net (62.40.125.17)     asymm  6 110.625ms 
 6:  dfn-gw.rt1.fra.de.geant.net (62.40.124.34)           112.035ms 
 7:  dfn-gw.rt1.fra.de.geant.net (62.40.124.34)           asymm  6 110.301ms 
 8:  xr-gar1-te2-2.x-win.dfn.de (188.1.145.54)            asymm  7 121.842ms 
 9:  kr-lrz-muenchen2.x-win.dfn.de (188.1.37.90)          asymm  8 121.579ms 
10:  vl-3016.vss1-2wr.lrz.de (129.187.0.254)              asymm  9 119.874ms 
11:  no reply
12:  no reply

srm-eosatlas.cern.ch

[root@uct2-s4 ~]# tracepath srm-eosatlas.cern.ch
 1:  uct2-s4.uchicago.edu (128.135.158.184)                 0.057ms pmtu 9000
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       1.380ms 
 2:  10.4.247.237 (10.4.247.237)                            1.472ms 
 3:  omnipop-mcleod-10g-router.uchicago.edu (128.135.247.125) asymm  4   1.498ms 
 4:  xe-0-1-3.0.rtr.newy32aoa.net.internet2.edu (64.57.28.115) asymm  5  28.625ms 
 5:  xe-0-1-3.0.rtr.newy32aoa.net.internet2.edu (64.57.28.115)  28.763ms 
 6:  peer-cern-via-manlan.newy32aoa.net.internet2.edu (198.32.11.86) asymm  5 181.100ms 
 7:  no reply
30:  no reply
31:  no reply
     Too many hops: pmtu 9000
     Resume: pmtu 9000 

Restoring dedicated BNL-UC circuit (9/20/2012)

[rwg@uct2-mgt ~]$ tracepath nettest10g.usatlas.bnl.gov
 1:  uct2-mgt.uchicago.edu (128.135.158.232)                0.133ms pmtu 1500
 1:  b98-b102-200-v624.uchicago.edu (128.135.158.131)       0.444ms 
 2:  10.4.247.237 (10.4.247.237)                            0.423ms 
 3:  omnipop-atlas-bnl-10g-router.uchicago.edu (128.135.247.129)  24.895ms 
 4:  nettest10g.usatlas.bnl.gov (192.12.15.25)             60.177ms reached
     Resume: pmtu 1500 hops 4 back 4 

References

• http://lhcone.web.cern.ch/
• In MWT2 Jira system, http://jira.usatlasfacility.org/browse/MW-386
• https://twiki.cern.ch/twiki/bin/view/LHCONE/LhcOneHowToConnect
• Talk on policy based routing by Mike O'Conner here
• Just for reference, BNL-UC links from before:
  • From April 18, 2012, https://stats.es.net/graphite/ :
    
  • From October 14, 2011
     

Connectivity diagram (June 2012)

• image001.jpg: