Project

General

Profile

Feature #2127

Save kernel core dumps on all of our test machines

Added by Anonymous almost 11 years ago. Updated about 3 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
qa
Target version:
-
% Done:

50%

Source:
Tags:
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

The claim is that there is a netdump module that will UDP-squirt kernel coredumps to a waiting server, which is probably exactly what we want to do.
(1) investigate this possibility and prototype to validate that we can make it work.
(2) update our standard plana systems to include this capability
(3) bring up a dump server (probably on one of our lab VM servers)
(4) initially store the coredumps on that server
(5) create a new feature to provide more scalable storage for coredumps, logs etc (e.g. a Ceph cloud)


Subtasks

Subtask #2238: vm for coredump archiveRejected

Subtask #2239: install + configure package everywhereNew

History

#1 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_position set to 8

#2 Updated by Alex Elder almost 11 years ago

Researched this a bit this afternoon. I have used LKCD in the past and
find now it's fairly old. Here someone has gathered information on LCKD
and the more recent kdump mechanism, and it appears to contain fairly
modern information on the subject:
http://www.dedoimedo.com/computers/crash-book.html

So far I'm just scanned through this (and some other) material, but we'll
need to be sure certain configuration options are set for built kernels.
We'll also need to ensure a few packages are installed. Dumping to a
repository over the network is supported.

I'm going to want to set up a system locally and fool around with it
a bit to verify I've got things done right.

#3 Updated by Sage Weil almost 11 years ago

  • Target version set to v0.45
  • translation missing: en.field_position deleted (20)
  • translation missing: en.field_position set to 6

#4 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_position deleted (6)
  • translation missing: en.field_position set to 8

#5 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_story_points set to 8
  • translation missing: en.field_position deleted (7)
  • translation missing: en.field_position set to 7

#6 Updated by Alex Elder almost 11 years ago

I have been able to generate a core dump on an Ubuntu system.
I have transferred the result using scp to another host, along
with a copy of the debug kernel corresponding to the core, and
am able to use the "crash" program on the result to examine the
core.

Getting the core to dump was not straightforward, and it
involves a number of tweaks and modifications to configuration
files on the system. In a few cases I may have done more than
was required, but have no way of knowing which (if any) of these
things can be skipped without going through the process again
and attempting to refine it.. I can re-install my target
system and re-do the steps I took in order to narrow things
down, but I think that time would be better spent trying to
automate the process and transfer it over to the target
test hardware.

Transferring this capability to the sepia/plana hardware
is going to take a great deal of close knowledge of how
that stuff operates, and I don't believe I am the right
person to implement that portion of this task. It would
probably be better to consult with (I presume) Tommi.

I have documented what I did and will be adding that to
this issue in a separate commit.

So I guess I suggest we need to discuss how to proceed
with this task at this point.

#7 Updated by Alex Elder almost 11 years ago

Oh, I forgot to mention I also wrote a little program that extracts
identifying information from a dump file that "kdump" generates.
There could be something out there that does this, but I couldn't
readily find it.

I will check it in at some point. But here's what running it looks like:

src/ceph-kdump-info ~/kernel_cores/testor/201203132346/dump.201203132346
DUMP_FILE="/home/elder/kernel_cores/testor/201203132346/dump.201203132346"
DUMP_TYPE="KDUMP V4"
DUMP_SYSNAME="Linux"
DUMP_NODENAME="testor"
DUMP_RELEASE="3.0.0-16-generic"
DUMP_VERSION="#29-Ubuntu SMP Tue Feb 14 12:48:51 UTC 2012"
DUMP_MACHINE="x86_64"
DUMP_DOMAINNAME="(none)"
DUMP_TIMESTAMP="1331700359.000000"
DUMP_TIME="Tue Mar 13 23:45:59 2012"

#8 Updated by Alex Elder almost 11 years ago

Wed Mar 14 11:14:50 CDT 2012

OK, I got kernel core dumps and crash working in Ubuntu 11.10.

A lot of what I used in figuring this out came from here:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/710733

It isn't completely seamless, and at this point I'm not fully sure
what I did that was possibly unnecessary.

There were a few possible problems that I addressed, and some of
them were done preemptively and others (like separate /boot) might
not even be an issue for sepia machines.

- First you need to have the kdump-tools and kexec-tools packages
installed. I also added crash, since that allows the dumps to be
used for local debugging if desired.
sudo apt-get install kdump-tools
sudo apt-get install kexec-tools
sudo apt-get install crash
And apport is an application failure report, and it also
ties into the kdump stuff a bit, but I really didn't invstigate:
sudo apt-get install apport
Finally, this one may capture some of the above, I'm not sure:
sudo apt-get install linux-crashdump

- My machine had a separate /boot partition, and this was reportedly
the source of a possible problem. To remedy this, I unmounted
/boot and copied its contents over to the root filesystem under
/boot. These files will be hidden when /boot is mounted, but if
it is not mounted (which is maybe the case when the kexec kernel
is running) they're there if needed. # Only if /boot is separate from /:
pushd /
DV=$(df /boot | awk '/^Filesystem/ { next;} {print $1}')
mkdir /a$$
umount /boot
mount ${DV} /a$$
cp -a /a$$/. /boot
umount /a$$
mount ${DV} /boot
rmdir /a$$
unset DV
popd /
--> It looks like the plana machines don't have a separate /boot
partition, so this is probably moot.

- By default, the boot command line which includes information about
the amount of memory to set aside for the kexec kernel (which is
what actually dumps memory, not the crashed kernel) is not by
default setting aside enough memory for a normal Ubuntu kernel.

So in /etc/grub.d/10_linux, in the definition of GRUB_CMDLINE_EXTRA,
the "64M" specified should be increased to 128M. I also doubled
the 128M:

old: GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-2G:64M,2G-:128M"
new: GRUB_CMDLINE_EXTRA="$GRUB_CMDLINE_EXTRA crashkernel=384M-2G:128M,2G-:256M"

(The 384M-2G:128M,2G-:256M syntax means "if there is between 384MB
and 2GB of system memory, set aside 128 MB of memory for the crash
kernel, and if there is 2GB or above of system memory, set aside
256 MB for the crash kernel.)

- One of the programs required is /usr/bin/makedumpfile. However
that is a dynamically-linked version of the program. A statically
linked version, /bin/makedumpfile-static, is available via the
makedumpfile-static package. Note that /etc/init.d/kdump (as well
as /etc/grub.d/10_linux) currently specifies /usr/bin/makedumpfile
explicitly. Rather than edit that, I replaced it in the filesystem.
mv /usr/bin/makedumpfile /usr/bin/makedumpfile-dynamic
cp /bin/makedumpfile-static /usr/bin/makedumpfile

- Apparently, once you've changed the makedumpfile program you may
need to rebuild your initramdisks.
update-initramfs -k all -u

- /etc/default/kdump-tools, I changed the value assigned to
"USE_KDUMP" to be 1 (it had been 0). I'm not sure why this was
necessary (and it may not be). This allows the kdump-tools
service script to operate, meaning it will save a core and reboot
if run within the kexec kernel, or will load the kexec kernel for
a "normal" boot.

- /etc/default/apport, I changed the value assigned to "enabled" to
1 (it was 0). But as I indicated above, I didn't really do
anything with apport.

- Finally, you need to get the debug symbols for the version of
the Linux kernel that is running:
sudo apt-get install linux-image-$(uname -r)-dbgsym
This will place files under /usr/lib/debug/{boot,lib}/ associated
with the kernel. In particular, this is needed:
/usr/lib/debug/boot/vmlinux-$(uname -r)
e.g.
/usr/lib/debug/boot/vmlinux-3.0.0-16-generic
Apparently debug/symbol-laden versions of the kernel modules
are alsso available under the lib directory, e.g.:
/usr/lib/debug/lib/modules/3.0.0-16-generic/kernel/fs/ceph/ceph.ko

- To manually generate a crash, use the "magic sysrq" /proc
interface:
echo c > /proc/sysrq-trigger
Whether this works is defined by another entry in /proc. If it's
disabled, this will enable it again:
echo 1 > /proc/sys/kernel/sysrq

- The crash dump for me took a longer time than I expected. However,
eventually it completed, and the system rebooted. At that point
the crash dump is found in a compressed kdump file, here:
/var/crash/<datestring>/dump.<datestring>
e.g.
/var/crash/201203132346/dump.201203132346
Again, in order to be useful, make sure the kernel debug file
(vmlinux-$(uname -r)) is also available. I copied it into the
crash directory.
/usr/lib/debug/boot/vmlinux-$(uname -r)
Of course, the kernel running at the time the crash is copied
could be different from the kernel. For that I wrote a program
"kdump_info" that extracts information from the kdump file.

  1. cd /var/crash/201203132346/
  2. eval $(kdump_info dump.201203132346)
  3. echo $DUMP_RELEASE
    3.0.0-16-generic
    cp /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
  4. ls
    dump.201203132346 vmlinux-3.0.0-16-generic #

- With the kdump file and the matching vmlinux image, the "crash"
program can be used to debug the result.

crash /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
crash /usr/lib/debug/boot/vmlinux-${DUMP_RELEASE} .
This could/should be
done at boot time--copying the result into a host-specific
directory on the server, and then deleting it when done. It could
be there is something available (like apport?) that does this, but
I didn't come across it. Anyway, do something like:
cd /var/crash/
ssh ${SERVER} "mkdir -p $(hostname)"
for i in 2012*; do
scp -r $i ${SERVER}:${SPATH}/$(hostname) && rm -rf $i
vmlinux-3.0.0-16-generic
dump.201203132346
done

- These files can be copied (e.g., using scp -r") to a server that
is meant to hold these crash images. I was able to run the same
crash program on a different system this way, although the two
machines involved both had the same architecture so there could be
some gotchas there.

#9 Updated by Sage Weil almost 11 years ago

http://linux.die.net/man/8/netdump

this mechanism looks simpler?

#10 Updated by Alex Elder almost 11 years ago

  • Status changed from New to In Progress

I finally have crash dumps getting packaged and sent over to a
remote machine reliably. The problem is that it doesn't work
quite as we'd hoped. In the event of a panic or other crash,
the kernel core dump file gets generated inside a separate
kernel that's launched from the crashing kernel. I believe
the crash dump is placed in the filesystem in (by default)
/var/crash by this crash kernel.

We had hoped that this was the time that we would be able to
copy the core dump to the remote machine, but in fact that
is done in a separate step, during the next reboot of the
"outer" kernel (the one that had crashed).

So as I said, I have this all sorted out and have scripts
that bundle up what we'd want and push it to a remote host.

I have not yet found the exact place in the kernel (or even
user space) code that dumps the core though, and therefore
don't know what the chance are I'll be able to move these
copying steps in, closer to the event of the actual crash.

It would be nice to starting using this method in the mean
time, if it's a workable solution.

Tomorrow I will try to reproduce what I've done to date,
which has been on a local machine in my home, on one of
the plana nodes.

#11 Updated by Sage Weil almost 11 years ago

I seem to remember seeing a reference to 'mkcrashrd', a mkinitrd type script that generates the initrd image the crash kernel boot into. Presumably that generates some simple script to copy the core based on what /etc/kdump.conf (or whatever it is) says. Probably that is the thing to modify to get different behavior?

The ideal situation would probably be to extend that to look for options in kdump.conf that specify a remote host/location and, say, and ssh key to use to scp it there. I suspect the hard part is figuring out how to configure the network in the crash kernel, though... that may be why it only does local storage, though.

#12 Updated by Sage Weil almost 11 years ago

  • Target version changed from v0.45 to v0.46

#13 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_position deleted (22)
  • translation missing: en.field_position set to 1
  • translation missing: en.field_position changed from 1 to 1178

#14 Updated by Sage Weil almost 11 years ago

  • Target version deleted (v0.46)
  • translation missing: en.field_position deleted (1181)
  • translation missing: en.field_position set to 1

#15 Updated by Sage Weil almost 11 years ago

  • Target version set to v0.47
  • translation missing: en.field_position deleted (1)
  • translation missing: en.field_position set to 1181

#16 Updated by Sage Weil almost 11 years ago

  • Target version deleted (v0.47)

#17 Updated by Sage Weil almost 11 years ago

  • translation missing: en.field_position deleted (1211)
  • translation missing: en.field_position set to 11

#18 Updated by Alex Elder over 10 years ago

  • Assignee changed from Alex Elder to Anonymous

My work on this was pretty much complete a few months ago.

It included a shell script that leverages Ubuntu kdump
mechanism for collecting and summarizing crash information
and copying it to a designated remote host. It also includes
an init script that runs that script when the system boots
in order to capture a crashed system.

At that point it was decided that the deployment of this (or
something like it) ought to wait until Tommi's work on
using virtual machines in teuthology was further along so
it could mesh well with that.

So I believe this bug really falls on Tv now, and as such
I am reassigning it to him.

#19 Updated by Sage Weil over 7 years ago

  • Status changed from In Progress to 12

#20 Updated by Patrick Donnelly about 3 years ago

  • Status changed from 12 to New

Also available in: Atom PDF