leyland

Occasionally we compile a new kernel and would like to test booting our machine with it. The choice would be easy if the machine is local and within your physical reach. But what if the machine is located somewhere remote, on the other side of the planet? Then we risk booting into a box that would panic!
Ok, you say pass the panic=5 kernel parameter when booting it so that it would auto reboot itself within 5 seconds if it does panic. What if it hangs and not panic?
Assuming you do have some kind of control to hard reset the machine (say via IPMI), there still may be a way to do this.
The grub bootloader can be configure to test boot a particular kernel only once and choose back the original default kernel in all subsequent boots. That way if the new kernel panic or even hang, as long as you can hard reset the box, all you need to do is to reboot the box again and viola it will go back to the original test and true old kernel.
To do this, first run the grub shell, then...
grub> savedefault --default=2 --once
assuming that your new kernel is the 3rd kernel in the /boot/grub/menu.lst config file.
Actually, grub also support another option call fallback, however, after trying a few box, I find that some box may fail that option, depending on the grub version.