You are here: Foswiki>ITB Web>NodesAllocation (15 Jun 2011, SuchandraThapa)Edit Attach

NodesAllocation

Existing machines

Name Primary Job Notes
itb2.uchicago.edu
 
NFS server for home directories
itb3.uchicago.edu
 
MediaWiki
itb1.uchicago.edu
 
 
itb4.uchicago.edu
 
ITB Robot host

New machine node matrix

Node Matrix:
Name VM Job RAM/Storage needed Public IP Provisioned Notes
itb-kvm1
*
storage node
NA
N
DONE
Host OS for kvm on storage node, has 2 MD1200 attached, VMs for pool only need internal ips
edit
 
itbv-dpool1
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-dpool2
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-dpool3
dcache storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool1
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool2
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-xpool3
xrootd storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool1
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool2
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-hpool3
hadoop storage pool
1G/10G
N
DONE
 
edit
 
itbv-xrd
xrootd redirector
1G/10G
N
DONE
 
edit
 
itbv-hdfs1
hdfs primary name server
2G/10G
N
DONE
 
edit
 
itbv-hdfs2
hdfs secondary name server
2G/10G
N
DONE
 
edit
itb-kvm2
*
Host for KVM
 
N
DONE
R610, initially used for io testing
edit
 
itbv-ce-pbs
CE for ITB cluster
2G/50G
Y
DONE
points to PBS scheduler
edit
 
itbv-ce-condor
CE for ITB cluster
2G/50G
Y
DONE
points to Condor scheduler
edit
 
vtbv-ce
CE for VTB
2G/10G
Y
DONE
 
edit
 
itbv-mon
Cluster monitoring
1G/10G
Y
DONE
RSV / nagios central monitoring
edit
 
itbv-web
Webserver for ITB/VTB resources
<1G/10G
Y
DONE
Home for ITB robot website / VTB cache / SVN webservices
edit
itb-kvm3
*
Host for KVM
NA
N
DONE
R610
edit
 
itbv-dc1
dcache headnode1
2G/10G
Y
DONE
 
edit
 
itbv-dc2
dcache headnode2
2G/10G
Y
DONE
hosting pnfs for dcache
edit
 
itbv-pbs
PBS scheduler
1G/10G
N
DONE
 
edit
 
itbv-condor
Condor scheduler
1G/10G
N
DONE
 
edit
 
itbv-bmx1
Bestman2 gateway for xrootd
1G/10G
Y
DONE
Link to xrootd install
edit
 
itbv-bmh1
Bestman2 gateway for hdfs
1G/10G
Y
DONE
Link to hdfs install
edit
 
ui-vofe
VO frontend (for Glidein-WMS job submission)
 
 
 
(1)
edit
 
ui-eng
Engagement submit host (with OSG MM submission)
 
 
 
(1)
edit
 
ui-cr
Campus router
 
 
 
(1)
edit
itb-esx3
*
Test system
NA
 
 
Initially used for KVM vs. VM vs. physical performance studies and KVM testing (physical vs. file vs direct i/o)
edit
itb-cloud
*
Machine for cloud services
 
Y
DONE
Nimbus/VM shipping, setup as a PBS worker node as well
edit
itb-c001
*
Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c002
*
Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c003
*
Virtual Worker node
 
N
DONE
PBS/condor overlay
edit
itb-c004
*
Virtual Worker node
 
N
DONE
PBS/condor overlay -- priority to Condor, setup for htpc for condor
edit
itb-kvm4
*
Host for VT3, test system at the beginning
 
N
DONE
 
edit
 
gc2-ce
Virtual Tier 3 Gatekeeper (CE)
 
 
 
 
edit
 
gc2-se
Virtual Tier 3 Storage element
 
 
 
 
edit
 
gc2-hn
Virtual Tier 3 Condor headnode (negotiator, )
 
 
 
 
edit
 
gc2-gums
Virtual Tier 3 GUMS server
 
 
 
 
edit
 
gc2-ui
Virtual Tier 3 UI
 
 
 
 
edit

Notes:
  • (1) Public IP is needed if we want to allow outside users direct access

How to create a KVM host with cobbler:

http://twiki.mwt2.org/bin/view/Admins/NewCobblerNode

New machines 1 (R610 with 2 MD1200)

Name Primary Job Cabling Notes
itb-kvm1.uchicago.edu
KVM host
int/ext
hosts storage systems: dCache, Hadoop, xrootd

The available disk on the MD1200 boxes is allocated in 3 equal parts (dCache, Hadoop, xrootd). The R610 will use one drive to run the host OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) will be used for the virtual machines. There are 9 VMs, 3 each for dCache, Hadoop, xrootd

Storage (virtual) nodes include:
  • 3 VMs for dcache pools
  • 3 VMs for xrootd pools
  • 3 VMs for hdfs pools

New machines 2 (8 R610)

Name Primary Job Cabling Notes
itb-esx3.uchicago.edu
ESXi host
int/ext
vmware ESXi to compare VM performance (and replace esx1 and 2 eventually)
itb-kvm2.uchicago.edu
KVM host
int/ext
hosts headnode services
itb-kvm3.uchicago.edu
KVM host
int
hosts worker nodes for the ITB clusters
itb-kvm4.uchicago.edu
KVM host
int
hosts worker nodes for the ITB clusters
itb-c001.uchicago.edu
worker node
int
worker node for the ITB clusters
itb-c002.uchicago.edu
worker node
int
worker node for the ITB clusters
itb-dyn.uchicago.edu
cloud
int/ext
test dynamic provisioning - on top of KVM? nimbus?
unnamed7
R610
int/ext
unused

These R610 servers will be installed with:
  • VMware ESXi on itb-esx3 (installation CD image available from vmware.com)
  • SL 5.5 and KVM in itb-kvmX: one drive to run the host OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) will be used for the virtual machines
  • SL 5.5 like Tier 2/3 worker nodes on itb-cXXX: probably one drive to run the OS (SL 5.5) and the other 5 drives (RAID configuration - LVS) for the scratch space
  • for itb-dyn check what is needed for dynamic provisioning (create/destroy VM dynamically); check Nimbus requirements (http://www.nimbusproject.org/ )

Headnode services for ITB and the virtual tier 3 include:
  • PBS scheduler
  • Condor headnode (collector/negotiator)
  • GUMS
  • ITB CE Headnode
  • VTB CE Headnode
  • Tier 3 CE
  • Tier 3 SE (server varies)
  • Tier 3 GUMS
  • Tier 3 UI
  • VO frontend (for Glidein-WMS job submission)
  • Engagement submit host (with OSG MM submission)
  • Campus router

We have currently 12 services in the list above. We may consider allocating one of the 2 unallocated R610 to host headnode services as well (itb-kvm6.uchicago.edu, with int/ext cabling).

We discussed having one management/service node (possibly one of the previous ITB nodes) to use as:
  • login node
  • NFSv4 server for home directories
  • Ganglia monitor

Services "borrowed" from the Tier 2/Tier 3 infrastructure:
  • Cobbler
  • OSG_APP with ATLAS releases

IP Reallocation

IPs from the grid school machines have been reallocated as follows:
Old hostname New hostname Changed? IP Subnet Gateway Notes
gs-gratia.uchicago.edu
itbv-web.uchicago.edu
DONE
128.135.158.177
255.255.255.128
128.135.158.129
 
edit
gsu1.uchicago.edu
itbv-ce-pbs.uchicago.edu
DONE
128.135.158.176
255.255.255.128
128.135.158.129
 
edit
gs-bdii.uchicago.edu
vtbv-ce.uchicago.edu
DONE
128.135.158.175
255.255.255.128
128.135.158.129
Name change requested
edit
gs-ress.uchicago.edu
itbv-ce-condor.uchicago.edu
DONE
128.135.158.140
255.255.255.128
128.135.158.129
Name change requested
edit
gs-mm.uchicago.edu
itbv-bmx.uchicago.edu
DONE
128.135.158.139
255.255.255.128
128.135.158.129
Name change requested
edit

Testing

Interesting tests that we discussed.

IO testing

  • IO comparison:
    • local IO (on host OS) vs
    • direct IO from VM (aka raw disks, access to the HW partition) vs
    • virtual disks on VM

Worker nodes

  • Compare real WN with virtual WN

Dynamic Provisioning

Dynamic allocation tests (http://www.nimbusproject.org/)

Virtual machine infrastructure

Compare different virtualization software frameworks, i.e. ESXi vs KVM.


-- MarcoMambelli - 20 Jan 2011
Edit  | Attach | Print version | History: r26 < r25 < r24 < r23 | Backlinks | View wiki text | Edit wiki text | More topic actions
Topic revision: r25 - 15 Jun 2011, SuchandraThapa
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback