NodesAllocation

Existing machines

Name Primary Job Notes
itb2.uchicago.edu
 
NFS server for home directories
itb3.uchicago.edu
 
MediaWiki
itb1.uchicago.edu
 
 
itb4.uchicago.edu
 
ITB Robot host

New machine node matrix

Node Matrix:
Name VM Job RAM/Storage needed Public IP Provisioned Notes
itb-kvm1
*
storage node
NA
N
DONE
Host OS for kvm on storage node, has 2 MD1200 attached, VMs for pool only need internal ips
edit
 
itbv-dpool1
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-dpool2
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-dpool3
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool1
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool2
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool3
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool1
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool2
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool3
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-xrd
xrootd redirector
1G/10G
N
DONE
 
edit
 
itbv-hdfs1
hdfs primary name server
2G/10G
N
DONE
 
edit
 
itbv-hdfs2
hdfs secondary name server
2G/10G
N
DONE
 
edit
itb-kvm2
*
Host for KVM
 
N
DONE
R610, initially used for io testing
edit
 
itbv-ce-pbs
CE for ITB cluster
2G/50G
Y
DONE
points to PBS scheduler
edit
 
itbv-ce-condor
CE for ITB cluster
2G/50G
Y
DONE
points to Condor scheduler
edit
 
vtbv-ce
CE for VTB
2G/10G
Y
DONE
 
edit
 
itbv-mon
Cluster monitoring
1G/10G
Y
DONE
RSV / nagios central monitoring
edit
 
itbv-web
Webserver for ITB/VTB resources
<1G/10G
Y
DONE
Home for ITB robot website / VTB cache / SVN webservices
edit
itb-kvm3
*
Host for KVM
NA
N
DONE
R610
edit
 
itbv-dc1
dcache headnode1
2G/10G
Y
DONE
 
edit
 
itbv-dc2
dcache headnode2
2G/10G
Y
DONE
hosting pnfs for dcache
edit
 
itbv-pbs
PBS scheduler
1G/10G
N
DONE
 
edit
 
itbv-condor
Condor scheduler
1G/10G
N
DONE
 
edit
 
itbv-bmx1
Bestman2 gateway for xrootd
1G/10G
Y
DONE
Link to xrootd install
edit
 
itbv-bmh1
Bestman2 gateway for hdfs
1G/10G
Y
DONE
Link to hdfs install
edit
 
ui-vofe
VO frontend (for Glidein-WMS job submission)
 
 
 
(1)
edit
 
ui-eng
Engagement submit host (with OSG MM submission)
 
 
 
(1)
edit
 
ui-cr
Campus router
 
 
 
(1)
edit
itb-esx3
*
Test system
NA
 
 
Initially used for KVM vs. VM vs. physical performance studies and KVM testing (physical vs. file vs direct i/o)
edit
itb-cloud
*
Machine for cloud services
 
Y
DONE
Nimbus/VM shipping, setup as a PBS worker node as well
edit
itb-c001
*
Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c002
*
Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c003
*
Virtual Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c004
*
Virtual Worker node
 
N
DONE
PBS/condor overlay -- priority to Condor, setup for htpc for condor
edit
itb-kvm4
*
Host for VT3, test system at the beginning
 
N
DONE
 
edit
 
gc2-ce
Virtual Tier 3 Gatekeeper (CE)
 
 
 
 
edit
 
gc2-se
Virtual Tier 3 Storage element
 
 
 
 
edit
 
gc2-hn
Virtual Tier 3 Condor headnode (negotiator, )
 
 
 
 
edit
 
gc2-gums
Virtual Tier 3 GUMS server
 
 
 
 
edit
 
gc2-ui
Virtual Tier 3 UI
 
 
 
 
edit

Notes:
  • (1) Public IP is needed if we want to allow outside users direct access

How to create a KVM host with cobbler:

http://twiki.mwt2.org/bin/view/Admins/NewCobblerNode

New machines 1 (R610 with 2 MD1200)

NameSorted ascending Primary Job Cabling Notes
itb-kvm1.uchicago.edu
KVM host
int/ext
hosts storage systems: dCache, Hadoop, xrootd

The available disk on the MD1200 boxes is allocated in 3 equal parts (dCache, Hadoop, xrootd). The R610 will use one drive to run the host OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) will be used for the virtual machines. There are 9 VMs, 3 each for dCache, Hadoop, xrootd

Storage (virtual) nodes include:
  • 3 VMs for dcache pools
  • 3 VMs for xrootd pools
  • 3 VMs for hdfs pools

New machines 2 (8 R610)

Name Primary Job Cabling Notes
itb-esx3.uchicago.edu
ESXi host
int/ext
vmware ESXi to compare VM performance (and replace esx1 and 2 eventually)
itb-kvm2.uchicago.edu
KVM host
int/ext
hosts headnode services
itb-kvm3.uchicago.edu
KVM host
int
hosts worker nodes for the ITB clusters
itb-kvm4.uchicago.edu
KVM host
int
hosts worker nodes for the ITB clusters
itb-c001.uchicago.edu
worker node
int
worker node for the ITB clusters
itb-c002.uchicago.edu
worker node
int
worker node for the ITB clusters
itb-dyn.uchicago.edu
cloud
int/ext
test dynamic provisioning - on top of KVM? nimbus?
unnamed7
R610
int/ext
unused

These R610 servers will be installed with:
  • VMware ESXi on itb-esx3 (installation CD image available from vmware.com)
  • SL 5.5 and KVM in itb-kvmX: one drive to run the host OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) will be used for the virtual machines
  • SL 5.5 like Tier 2/3 worker nodes on itb-cXXX: probably one drive to run the OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) for the scratch space
  • for itb-dyn check what is needed for dynamic provisioning (create/destroy VM dynamically); check Nimbus requirements (http://www.nimbusproject.org/ )

Headnode services for ITB and the virtual tier 3 include:
  • PBS scheduler
  • Condor headnode (collector/negotiator)
  • GUMS
  • ITB CE Headnode
  • VTB CE Headnode
  • Tier 3 CE
  • Tier 3 SE (server varies)
  • Tier 3 GUMS
  • Tier 3 UI
  • VO frontend (for Glidein-WMS job submission)
  • Engagement submit host (with OSG MM submission)
  • Campus router

We have currently 12 services in the list above. We may consider allocating one of the 2 unallocated R610 to host headnode services as well (itb-kvm6.uchicago.edu, with int/ext cabling).

We discussed having one management/service node (possibly one of the previous ITB nodes) to use as:
  • login node
  • NFSv4 server for home directories
  • Ganglia monitor

Services "borrowed" from the Tier 2/Tier 3 infrastructure:
  • Cobbler
  • OSG_APP with ATLAS releases

IP Reallocation

IPs from the grid school machines have been reallocated as follows:
Old hostname New hostname Changed? IP Subnet Gateway Notes
gs-gratia.uchicago.edu
itbv-web.uchicago.edu
DONE
128.135.158.177
255.255.255.128
128.135.158.129
 
edit
gsu1.uchicago.edu
itbv-ce-pbs.uchicago.edu
DONE
128.135.158.176
255.255.255.128
128.135.158.129
 
edit
gs-bdii.uchicago.edu
vtbv-ce.uchicago.edu
DONE
128.135.158.175
255.255.255.128
128.135.158.129
Name change requested
edit
gs-ress.uchicago.edu
itbv-ce-condor.uchicago.edu
DONE
128.135.158.140
255.255.255.128
128.135.158.129
Name change requested
edit
gs-mm.uchicago.edu
itbv-bmx.uchicago.edu
DONE
128.135.158.139
255.255.255.128
128.135.158.129
Name change requested
edit

Testing

Interesting tests that we discussed.

IO testing

  • IO comparison:
    • local IO (on host OS) vs
    • direct IO from VM (aka raw disks, access to the HW partition) vs
    • virtual disks on VM

Worker nodes

  • Compare real WN with virtual WN

Dynamic Provisioning

Dynamic allocation tests (http://www.nimbusproject.org/)

Virtual machine infrastructure

Compare different virtualization software frameworks, i.e. ESXi vs KVM.


-- MarcoMambelli - 20 Jan 2011
Topic revision: r25 - 15 Jun 2011, SuchandraThapa
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback