KernelBuild

Background

• Thanks to Charles Waldman, MWT2 was able to determine the correct tunings for our own custom kernel and later a packaged one merged with UL3 to create UL4.
• This work has been integrated into the Ultra Light Kernel: UltraLightKernel

Configuration and installation of MWT2 kernel from source

Download and prepare source

• download 2.6 kernel source from ftp.us.kernel.org. As of this writing the most recent stable 2.6 kerkel is linux-2.6.33.4
• extract tarfile to /usr/src/linux-2.6.33.4, symlink this to /usr/src/linux
• copy SL config file from /usr/src/kernels/2.6.18-164.11.1.el5-x86_64/.config to /usr/src/linux/.config
• cd /usr/src/linux; make oldconfig to import previous configuration
• make menuconfig, judiciously choosing options you want to enable/disable

Configuration choices

• Set Processor family to Opteron/Athlon64/Hammer/K8 (for AMD hosts) and Core 2/newer Xeon for the Dells
• Set Preeemption model and some soft-lockup detection features:

  CONFIG_PREEMPT_NONE:                                        
                                   
     Select this option if you are building a kernel for a server or 
     scientific/computation system, or if you want to maximize the
     raw processing power of the kernel, irrespective of scheduling
     latencies.                             
  
  CONFIG_DETECT_SOFTLOCKUP:
     Say Y here to enable the kernel to detect "soft lockups",
     which are bugs that cause the kernel to loop in kernel 
     mode for more than 60 seconds, without giving other tasks a
     chance to run.
  
     When a soft-lockup is detected, the kernel will print the
     current stack trace (which you should report), but the
     system will stay locked up. This feature has negligible
     overhead.
  
     (Note that "hard lockups" are separate type of bugs that 
      can be detected via the NMI-watchdog, on platforms that
      support it.)

• Note this is an obsolete flag as per this note from olga.kornievskaia, "Btw. CONFIG_DETECT_SOFTLOCKUP is obsolete since 2.6.35, it's now CONFIG_LOCKUP_DETECTOR."

  CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC: 
     Say Y here to enable the kernel to panic on "soft lockups",
     which are bugs that cause the kernel to loop in kernel 
     mode for more than 60 seconds, without giving other tasks a
     chance to run. 
  
     The panic can be used in combination with panic_timeout, 
     to cause the system to reboot automatically after a
     lockup has been detected. This feature is useful for
     high-availability systems that have uptime guarantees and
     where a lockup must be resolved ASAP.
  
  CONFIG_BOOTPARAM_HUNG_TASK_PANIC: 
     Say Y here to enable the kernel to panic on "hung tasks",
     which are bugs that cause the kernel to leave a task stuck
     in uninterruptible "D" state.
  
     The panic can be used in combination with panic_timeout,
     to cause the system to reboot automatically after a
     hung task has been detected. This feature is useful for
     high-availability systems that have uptime guarantees and  
     where a hung tasks must be resolved ASAP.
  
  

• Make sure drivers for disk controllers are built in to the kernel (Y) rather than module (M). This makes booting easier (no initrd required).
  • CONFIG_SCSI_MPT2SAS=y (used in Dell nodes)
  • CONFIG_SATA_NV=y (used in KOI nodes)
  • CONFIG_SATA_SIL=y (used in ACT nodes)
• Set the default I/O scheduler to deadline (cfq is still available and can be selected at run-time)
  • CONFIG_IOSCHED_DEADLINE=y
  • CONFIG_DEFAULT_DEADLINE=y
  • CONFIG_DEFAULT_IOSCHED="deadline"
  • CONFIG_IOSCHED_CFQ=y
  • CONFIG_DEFAULT_CFQ is not set
• Make configuration of running kernel visible via /proc/config.gz
  • CONFIG_IKCONFIG=y
  • CONFIG_IKCONFIG_PROC=y
• Some unused driver modules have been disabled simply to speed up the kernel build.
• Config files here:
  • MWT2-AMD.config: Kernel config file for AMD
  • MWT2-xeon.config: Kernel config file for Intel (xeon)

Building the kernel

• cd /usr/src/linux
• make -j8 bzImage; make -j8 modules;  make modules_install
• cd /; tar cfz /home/aaron/Kernel/lib-modules-2.6.33.4-mwt2-2-$ARCH.tar.gz /lib/modules/2.6.33.4-mwt2-2
• cd /; tar cfz /home/aaron/Kernel/lib-firmware-2.6.33.4-mwt2-2-$ARCH.tar.gz /lib/firmware

Installing and booting

• Extract modules and firmware from tar files, copy the kernel to /boot/vmlinuz-2.6.33.4, edit /boot/grub/grub.conf
• On the first attempt, the boot failed with init: cannot open an intial console
• Since we are booting without initrd, we have to make sure that /dev/zero, /dev/null, and /dev/console are present before udev starts. This only needs to be done once. For resolving udev issues, this document was indespensible: http://docs.fedoraproject.org/udev/

  kill -9 `pidof udevd`
  mkdir -p /tmp/dev
  mount --move /dev /tmp/dev
  MAKEDEV zero null console
  mount --move /tmp/dev /dev
  

Post-install

• The name of the real-time clock device seems to have changed from rtc to rtc0, making ntpd fail. Fix (one-time):

  /etc/init.d/ntpd stop
  cd /dev
  ln -s rtc0 rtc
  ntpdate tick.usno.navy.mil
  hwclock --systohc
  /etc/init.d/ntpd start
  

Scripts

• /home/cgw/Kernel/install.sh detects CPU type and all the above steps (including grub.conf). postinstall.sh fixes ntp. All supporting files are in /home/cgw/Kernel

Config wishlist

• Configuration settings to apply next time we build a kernel
  • CONFIG_USB_SUSPEND=y
  • CONFIG_TASK_IO_ACCOUNTING=y