Tag Archives: sysadm

Rescuing hosed server

This post describes the steps that I took to restore an old server which had died some months back.

I had a linux server running debian pretty reliably. It had failed because of a hardware problems (broken power supply). I replaced the power supply and proceeded to make it current with security fixes. But as it happens when you update after a long time, (or it could also be due to my own itch to remove old kernels), I hosed the system. By that I mean that the system was running but somehow all the kernels had been removed (including the current one that I was running!). Anyway, in summary, the system would not boot because there was no kernel to do so.

Step 1: Okay, I thought I will just reinstall it with Ubuntu (which BTW is my new favorite distribution). But the problem was that the machine was using LVM and Ubuntu installer did not recognize the LVM volumes (even though it is documented to do so).

Step 2: So next stop was latest and greatest Knoppix which recognizes LVM. When I booted knoppix, I was able to see all the old partitions and data (70 gigs of legal *cough* songs, audio books and movies). Success!

Step 3: Next step was to chroot into the old root partition and use apt-get to get the kernel package installed back again. I confirmed that there was no kernel-image-* package installed! apt-get update would fail, because procfs was not mounted, so from out of the chroot, mounted the procfs. After that I was able to update the packages, but the install scripts failed because of errors. The specific error was /dev/null: Permission Denied. I was stumped. The device looked okay and the permissions also were okay (the script runs as root afterall). Upon some search I found out that it is because the default mounts in knoppix do not mount devices (nodev option). Once I fixed that, all things fell right in place.