Ceph

Researched this a bit this afternoon. I have used LKCD in the past and
find now it's fairly old. Here someone has gathered information on LCKD
and the more recent kdump mechanism, and it appears to contain fairly
modern information on the subject:
http://www.dedoimedo.com/computers/crash-book.html

So far I'm just scanned through this (and some other) material, but we'll
need to be sure certain configuration options are set for built kernels.
We'll also need to ensure a few packages are installed. Dumping to a
repository over the network is supported.

I'm going to want to set up a system locally and fool around with it
a bit to verify I've got things done right.

I have been able to generate a core dump on an Ubuntu system.
I have transferred the result using scp to another host, along
with a copy of the debug kernel corresponding to the core, and
am able to use the "crash" program on the result to examine the
core.

Getting the core to dump was not straightforward, and it
involves a number of tweaks and modifications to configuration
files on the system. In a few cases I may have done more than
was required, but have no way of knowing which (if any) of these
things can be skipped without going through the process again
and attempting to refine it.. I can re-install my target
system and re-do the steps I took in order to narrow things
down, but I think that time would be better spent trying to
automate the process and transfer it over to the target
test hardware.

Transferring this capability to the sepia/plana hardware
is going to take a great deal of close knowledge of how
that stuff operates, and I don't believe I am the right
person to implement that portion of this task. It would
probably be better to consult with (I presume) Tommi.

I have documented what I did and will be adding that to
this issue in a separate commit.

So I guess I suggest we need to discuss how to proceed
with this task at this point.

Oh, I forgot to mention I also wrote a little program that extracts
identifying information from a dump file that "kdump" generates.
There could be something out there that does this, but I couldn't
readily find it.

I will check it in at some point. But here's what running it looks like:

src/ceph-kdump-info ~/kernel_cores/testor/201203132346/dump.201203132346
DUMP_FILE="/home/elder/kernel_cores/testor/201203132346/dump.201203132346"
DUMP_TYPE="KDUMP V4"
DUMP_SYSNAME="Linux"
DUMP_NODENAME="testor"
DUMP_RELEASE="3.0.0-16-generic"
DUMP_VERSION="#29-Ubuntu SMP Tue Feb 14 12:48:51 UTC 2012"
DUMP_MACHINE="x86_64"
DUMP_DOMAINNAME="(none)"
DUMP_TIMESTAMP="1331700359.000000"
DUMP_TIME="Tue Mar 13 23:45:59 2012"

Wed Mar 14 11:14:50 CDT 2012

OK, I got kernel core dumps and crash working in Ubuntu 11.10.

A lot of what I used in figuring this out came from here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/710733

It isn't completely seamless, and at this point I'm not fully sure
what I did that was possibly unnecessary.

There were a few possible problems that I addressed, and some of
them were done preemptively and others (like separate /boot) might
not even be an issue for sepia machines.

- First you need to have the kdump-tools and kexec-tools packages
installed. I also added crash, since that allows the dumps to be
used for local debugging if desired.
sudo apt-get install kdump-tools
sudo apt-get install kexec-tools
sudo apt-get install crash
And apport is an application failure report, and it also
ties into the kdump stuff a bit, but I really didn't invstigate:
sudo apt-get install apport
Finally, this one may capture some of the above, I'm not sure:
sudo apt-get install linux-crashdump

- My machine had a separate /boot partition, and this was reportedly
the source of a possible problem. To remedy this, I unmounted
/boot and copied its contents over to the root filesystem under
/boot. These files will be hidden when /boot is mounted, but if
it is not mounted (which is maybe the case when the kexec kernel
is running) they're there if needed. # Only if /boot is separate from /:
pushd /
DV=$(df /boot | awk '/^Filesystem/ { next;} {print $1}')
mkdir /a$$
umount /boot
mount ${DV} /a$$
cp a /a$$/. /boot
umount /a$$
mount ${DV} /boot
rmdir /a$$
unset DV
popd /
-> It looks like the plana machines don't have a separate /boot
partition, so this is probably moot.

- By default, the boot command line which includes information about
the amount of memory to set aside for the kexec kernel (which is
what actually dumps memory, not the crashed kernel) is not by
default setting aside enough memory for a normal Ubuntu kernel.

So in /etc/grub.d/10_linux, in the definition of GRUB_CMDLINE_EXTRA,
  the "64M" specified should be increased to 128M.  I also doubled
  the 128M:

old: GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-2G:64M,2G-:128M"
new: GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-2G:128M,2G-:256M"

(The 384M-2G:128M,2G-:256M syntax means "if there is between 384MB
  and 2GB of system memory, set aside 128 MB of memory for the crash
  kernel, and if there is 2GB or above of system memory, set aside
  256 MB for the crash kernel.)

- One of the programs required is /usr/bin/makedumpfile. However
that is a dynamically-linked version of the program. A statically
linked version, /bin/makedumpfile-static, is available via the
makedumpfile-static package. Note that /etc/init.d/kdump (as well
as /etc/grub.d/10_linux) currently specifies /usr/bin/makedumpfile
explicitly. Rather than edit that, I replaced it in the filesystem.
mv /usr/bin/makedumpfile /usr/bin/makedumpfile-dynamic
cp /bin/makedumpfile-static /usr/bin/makedumpfile

- Apparently, once you've changed the makedumpfile program you may
need to rebuild your initramdisks.
update-initramfs -k all -u

- /etc/default/kdump-tools, I changed the value assigned to
"USE_KDUMP" to be 1 (it had been 0). I'm not sure why this was
necessary (and it may not be). This allows the kdump-tools
service script to operate, meaning it will save a core and reboot
if run within the kexec kernel, or will load the kexec kernel for
a "normal" boot.

- /etc/default/apport, I changed the value assigned to "enabled" to
1 (it was 0). But as I indicated above, I didn't really do
anything with apport.

- Finally, you need to get the debug symbols for the version of
the Linux kernel that is running:
sudo apt-get install linux-image-$(uname r)-dbgsym
This will place files under /usr/lib/debug/{boot,lib}/ associated
with the kernel. In particular, this is needed:
/usr/lib/debug/boot/vmlinux$(uname -r)
e.g.
/usr/lib/debug/boot/vmlinux-3.0.0-16-generic
Apparently debug/symbol-laden versions of the kernel modules
are alsso available under the lib directory, e.g.:
/usr/lib/debug/lib/modules/3.0.0-16-generic/kernel/fs/ceph/ceph.ko

- To manually generate a crash, use the "magic sysrq" /proc
interface:
echo c > /proc/sysrq-trigger
Whether this works is defined by another entry in /proc. If it's
disabled, this will enable it again:
echo 1 > /proc/sys/kernel/sysrq

- The crash dump for me took a longer time than I expected. However,
eventually it completed, and the system rebooted. At that point
the crash dump is found in a compressed kdump file, here:
/var/crash/<datestring>/dump.<datestring>
e.g.
/var/crash/201203132346/dump.201203132346
Again, in order to be useful, make sure the kernel debug file
(vmlinux-$(uname r)) is also available. I copied it into the
crash directory.
/usr/lib/debug/boot/vmlinux$(uname -r)
Of course, the kernel running at the time the crash is copied
could be different from the kernel. For that I wrote a program
"kdump_info" that extracts information from the kdump file.

1. cd /var/crash/201203132346/
2. eval $(kdump_info dump.201203132346)
3. echo $DUMP_RELEASE
   3.0.0-16-generic
   cp /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
4. ls
   dump.201203132346 vmlinux-3.0.0-16-generic #

- With the kdump file and the matching vmlinux image, the "crash"
program can be used to debug the result.

crash /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
    crash /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
  This could/should be
  done at boot time--copying the result into a host-specific
  directory on the server, and then deleting it when done.  It could
  be there is something available (like apport?) that does this, but
  I didn't come across it.  Anyway, do something like:
        cd /var/crash/
        ssh ${SERVER} "mkdir -p $(hostname)" 
        for i in 2012*; do
        scp -r $i ${SERVER}:${SPATH}/$(hostname) && rm -rf $i
    vmlinux-3.0.0-16-generic
    dump.201203132346
    done

- These files can be copied (e.g., using scp -r") to a server that
is meant to hold these crash images. I was able to run the same
crash program on a different system this way, although the two
machines involved both had the same architecture so there could be
some gotchas there.

http://linux.die.net/man/8/netdump

this mechanism looks simpler?

I seem to remember seeing a reference to 'mkcrashrd', a mkinitrd type script that generates the initrd image the crash kernel boot into. Presumably that generates some simple script to copy the core based on what /etc/kdump.conf (or whatever it is) says. Probably that is the thing to modify to get different behavior?

The ideal situation would probably be to extend that to look for options in kdump.conf that specify a remote host/location and, say, and ssh key to use to scp it there. I suspect the hard part is figuring out how to configure the network in the crash kernel, though... that may be why it only does local storage, though.