Bug #6139
closedkernel panic in vms during disk benchmarking
0%
Description
Hi
I am having regular issues with virtual machines running heavy disk io benchmarks.
My ceph setup:
Ceph version is 0.67.2 running on Ubuntu 12.04 with 3.5.0-39-generic kernel. 5 mons, 2 osd servers with 8 osds each. Each osd server has 1 ssd disk to provide journalling for 4 osds. The other 4 osds have journals on the disks. VM host servers are also Ubuntu 12.04 with the same kernel. Version of Qemu is 1.5.0+dfsg-3ubuntu2 backported from Ubuntu 13.04 and librbd1 is 0.67.2-1precise. Libvirt is backported from Ubuntu 13.04 version 1.0.2-0ubuntu11.13.04.1~precise1~ppa1.
I am using CloudStack with RBD primary storage. By default it uses virtio drivers and cache=none when starting virtual machines.
I am using phoronix-test-suite for benchmarking vm disk throughput by running "phoronix-test-suite benchmark pts/disk" command with the default settings. I've managed to complete the test only once out of 5 runs I've tried. Four times I had kernel panic in the console. I can't find the kernel panic message in the logs - i only have two screenshot (attached).
During the benchmark test I am also seeing these messages (not sure if they are related to the kernel panic):
Aug 27 08:05:39 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.695992] ata2: lost interrupt (Status 0x58)
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.700047] ata2: drained 65536 bytes to clear DRQ
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.967579] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.985443] sr 1:0:0:0: CDB:
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.985453] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.985459] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31542.985459] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31543.004340] ata2.00: status: { DRDY }
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31543.014190] ata2: soft resetting link
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31543.168974] ata2.00: configured for MWDMA2
Aug 27 08:05:40 Ubuntu-ceph-ubuntu-host-1 kernel: [31543.169208] ata2: EH complete
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31668.762971] ata2: lost interrupt (Status 0x58)
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31668.766914] ata2: drained 65536 bytes to clear DRQ
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.033942] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.056926] sr 1:0:0:0: CDB:
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.056926] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.056926] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.056926] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.080817] ata2.00: status: { DRDY }
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.092873] ata2: soft resetting link
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.248961] ata2.00: configured for MWDMA2
Aug 27 08:07:46 Ubuntu-ceph-ubuntu-host-1 kernel: [31669.249195] ata2: EH complete
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.587971] ata2: lost interrupt (Status 0x58)
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.591971] ata2: drained 65536 bytes to clear DRQ
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.870048] ata2.00: limiting speed to MWDMA1:PIO4
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.870054] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.896136] sr 1:0:0:0: CDB:
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.896136] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.897054] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.897054] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.924657] ata2.00: status: { DRDY }
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31804.940617] ata2: soft resetting link
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31805.097018] ata2.00: configured for MWDMA1
Aug 27 08:10:02 Ubuntu-ceph-ubuntu-host-1 kernel: [31805.097257] ata2: EH complete
Could you please let me know how to obtain more information for debugging this issue.
P.S. I've not seen kernel panics when running vms on 0.61.7 release using the same hardware. I've seen hang tasks messages, but the benchmarks would always complete without crashing the vm.
Files
Updated by Andrei Mikhailovsky over 10 years ago
I've updated kernel to the latest linux-image-lts-raring and I am also having vm crashes. I can get the crash by running phoronix-test-suite using pts/tiobenchmark by running: phoronix-test-suite benchmark pts/tiobench. I've managed to capture syslog output this time:
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7421.859970] ata2: lost interrupt (Status 0x58)
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7421.863891] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619878] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619896] sr 1:0:0:0: CDB:
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619898] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619909] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619909] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.619916] ata2.00: status: { DRDY }
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.649907] ata2: soft resetting link
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.805013] ata2.00: configured for MWDMA2
Aug 29 23:50:23 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7422.808179] ata2: EH complete
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.013121] ata2: lost interrupt (Status 0x58)
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.224494] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346619] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346629] sr 1:0:0:0: CDB:
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346630] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346639] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346639] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346646] ata2.00: status: { DRDY }
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.346731] ata2: soft resetting link
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.610246] ata2.00: configured for MWDMA2
Aug 29 23:51:10 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7469.610450] ata2: EH complete
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7512.005250] ata2: lost interrupt (Status 0x58)
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.151221] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477496] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477506] sr 1:0:0:0: CDB:
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477507] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477517] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477517] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477523] ata2.00: status: { DRDY }
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.477607] ata2: soft resetting link
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.632999] ata2.00: configured for MWDMA2
Aug 29 23:52:01 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7520.633228] ata2: EH complete
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7571.518135] ata2: lost interrupt (Status 0x58)
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7571.564600] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035381] ata2.00: limiting speed to MWDMA1:PIO4
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035384] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035395] sr 1:0:0:0: CDB:
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035400] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035409] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035409] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035416] ata2.00: status: { DRDY }
Aug 29 23:52:52 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.035521] ata2: soft resetting link
Aug 29 23:52:53 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.286599] ata2.00: configured for MWDMA1
Aug 29 23:52:53 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7572.286825] ata2: EH complete
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7693.810160] ata2: lost interrupt (Status 0x58)
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7693.854474] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353249] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353258] sr 1:0:0:0: CDB:
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353259] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353268] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353268] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353274] ata2.00: status: { DRDY }
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.353353] ata2: soft resetting link
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.529103] ata2.00: configured for MWDMA1
Aug 29 23:54:55 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7694.529354] ata2: EH complete
Aug 29 23:55:01 Ubuntu-ceph-ubuntu-host-1 CRON12602: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792189] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=15002 jiffies, g=55725, c=55724, q=9140)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792210] sending NMI to all CPUs:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] NMI backtrace for cpu 4
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] CPU 4
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] Pid: 0, comm: swapper/4 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] RIP: 0010:[<ffffffff81045136>] [<ffffffff81045136>] native_safe_halt+0x6/0x10
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] RSP: 0018:ffff880119b1fec8 EFLAGS: 00000246
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] RAX: 0000000000000000 RBX: ffff880119b1ffd8 RCX: ffff88011fd00000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] RDX: ffffffff81c3d680 RSI: 0000000000000000 RDI: 0000000000000092
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] RBP: ffff880119b1fec8 R08: 0000000000000000 R09: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] FS: 0000000000000000(0000) GS:ffff88011fd00000(0000) knlGS:0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] CR2: 00007f2fb73a2000 CR3: 0000000115832000 CR4: 00000000000006e0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] Process swapper/4 (pid: 0, threadinfo ffff880119b1e000, task ffff880119b15d00)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] Stack:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] ffff880119b1fef8 ffffffff8101cc33 ffff880119b1ffd8 ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] 0000000000000000 0000000000000000 ffff880119b1ff28 ffffffff8101dad9
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] ffff880119b1ff18 722d0495907806b2 0000000000000000 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] Call Trace:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] [<ffffffff8101cc33>] default_idle+0x53/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] [<ffffffff816d10fe>] start_secondary+0xc3/0xc5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.105167] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] NMI backtrace for cpu 0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] CPU 0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] Pid: 0, comm: swapper/0 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] RIP: 0010:[<ffffffff8103ef5f>] [<ffffffff8103ef5f>] flat_send_IPI_all+0xaf/0xd0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] RSP: 0018:ffff88011fc03d28 EFLAGS: 00010006
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] RAX: 0000000000000000 RBX: 0000000000000046 RCX: ffffffff81e304c0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] RDX: 00000000000020a8 RSI: 0000000000000082 RDI: 0000000000000300
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] RBP: ffff88011fc03d48 R08: 000000000000000a R09: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] R10: 0000000000000203 R11: 0000000000000202 R12: 0000000000000c00
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] R13: 000000003f000000 R14: ffffffff81ce6fc0 R15: 0000000000000082
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] FS: 0000000000000000(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] CR2: 00007f8d7b3f8000 CR3: 0000000115832000 CR4: 00000000000006f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15440)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] Stack:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] 00000000000020a3 0000000000002710 0000000000000001 00000000000023b4
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] ffff88011fc03d68 ffffffff8103aa61 000000000000d9ac ffffffff81c3f400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] ffff88011fc03da8 ffffffff810f5b6b ffff88011fc03dc8 ffffffff81c3f400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] Call Trace:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] <IRQ>
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff8103aa61>] arch_trigger_all_cpu_backtrace+0x61/0xa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810f5b6b>] print_other_cpu_stall+0x21b/0x280
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810f67f0>] __rcu_pending+0x1a0/0x210
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810f68bd>] rcu_pending+0x5d/0x80
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810f86a3>] rcu_check_callbacks+0xa3/0x160
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff8106bfd8>] update_process_times+0x48/0x90
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810b6b63>] tick_sched_handle.isra.8+0x33/0x70
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810b6c8c>] tick_sched_timer+0x4c/0x80
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81083978>] __run_hrtimer+0x78/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810b6c40>] ? tick_nohz_handler+0xa0/0xa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff810affec>] ? ktime_get_update_offsets+0x4c/0xd0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81084207>] hrtimer_interrupt+0xf7/0x240
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff816fe6d9>] smp_apic_timer_interrupt+0x69/0x99
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff816fd61d>] apic_timer_interrupt+0x6d/0x80
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] <EOI>
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81045136>] ? native_safe_halt+0x6/0x10
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff8101cc33>] default_idle+0x53/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff816c6db2>] rest_init+0x72/0x80
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81d05c4f>] start_kernel+0x3d1/0x3de
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81d057ff>] ? pass_bootoption.constprop.2+0xd3/0xd3
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81d05397>] x86_64_start_reservations+0x131/0x135
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81d05120>] ? early_idt_handlers+0x120/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] [<ffffffff81d05468>] x86_64_start_kernel+0xcd/0xdc
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.796173] Code: 8b 05 e6 7c ca 00 41 c1 e5 18 44 8b 60 3c ff 90 50 01 00 00 44 89 2c 25 10 73 5f ff 41 81 cc 00 04 00 00 44 89 24 25 00 73 5f ff <48> 89 df 57 9d 0f 1f 44 00 00 48 83 c4 08 5b 41 5c 41 5d 5d c3
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] NMI backtrace for cpu 3
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] CPU 3
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] Pid: 0, comm: swapper/3 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] RIP: 0010:[<ffffffff81045136>] [<ffffffff81045136>] native_safe_halt+0x6/0x10
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] RSP: 0018:ffff880119b1dec8 EFLAGS: 00000246
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] RAX: 0000000000000000 RBX: ffff880119b1dfd8 RCX: ffff88011fcc0000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] RDX: ffffffff81c3d680 RSI: 0000000000000000 RDI: 0000000000000092
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] RBP: ffff880119b1dec8 R08: 0000000000000000 R09: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] FS: 0000000000000000(0000) GS:ffff88011fcc0000(0000) knlGS:0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] CR2: 00007f2fb73ad000 CR3: 0000000115832000 CR4: 00000000000006e0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] Process swapper/3 (pid: 0, threadinfo ffff880119b1c000, task ffff880119b145c0)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] Stack:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] ffff880119b1def8 ffffffff8101cc33 ffff880119b1dfd8 ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] 0000000000000000 0000000000000000 ffff880119b1df28 ffffffff8101dad9
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] ffff880119b1df18 f13726c70ba4a28a 0000000000000000 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] Call Trace:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] [<ffffffff8101cc33>] default_idle+0x53/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] [<ffffffff816d10fe>] start_secondary+0xc3/0xc5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7844.085255] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] NMI backtrace for cpu 5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] CPU 5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] Pid: 0, comm: swapper/5 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] RIP: 0010:[<ffffffff81045136>] [<ffffffff81045136>] native_safe_halt+0x6/0x10
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] RSP: 0018:ffff880119b29ec8 EFLAGS: 00000246
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] RAX: 0000000000000000 RBX: ffff880119b29fd8 RCX: 00000001001cc8f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] RDX: ffff880119becf68 RSI: 0000000000000082 RDI: 0000000000000092
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] RBP: ffff880119b29ec8 R08: 0000000000000000 R09: 0000000000000001
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] R13: 0000000000000005 R14: 0000000000000000 R15: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] FS: 0000000000000000(0000) GS:ffff88011fd40000(0000) knlGS:0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] CR2: 00007f00e624a000 CR3: 0000000115832000 CR4: 00000000000006e0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] Process swapper/5 (pid: 0, threadinfo ffff880119b28000, task ffff880119b20000)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] Stack:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] ffff880119b29ef8 ffffffff8101cc33 ffff880119b29fd8 ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] 0000000000000000 0000000000000000 ffff880119b29f28 ffffffff8101dad9
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] ffff880119b29f18 7dcc54d0989c7462 0000000000000000 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] Call Trace:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] [<ffffffff8101cc33>] default_idle+0x53/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] [<ffffffff816d10fe>] start_secondary+0xc3/0xc5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.792121] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] NMI backtrace for cpu 1
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] CPU 1
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] Pid: 0, comm: swapper/1 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] RIP: 0010:[<ffffffff81045136>] [<ffffffff81045136>] native_safe_halt+0x6/0x10
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] RSP: 0018:ffff880119b19ec8 EFLAGS: 00000246
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] RAX: 0000000000000000 RBX: ffff880119b19fd8 RCX: 00000001001cc8db
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] RDX: ffff880119b49108 RSI: 0000000000000082 RDI: 0000000000000092
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] RBP: ffff880119b19ec8 R08: 0000000000000000 R09: 0000000000000001
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] FS: 0000000000000000(0000) GS:ffff88011fc40000(0000) knlGS:0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] CR2: 00007fffc6993f68 CR3: 0000000115832000 CR4: 00000000000006e0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] Process swapper/1 (pid: 0, threadinfo ffff880119b18000, task ffff880119b11740)
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] Stack:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] ffff880119b19ef8 ffffffff8101cc33 ffff880119b19fd8 ffffffff81ce6fa0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] 0000000000000000 0000000000000000 ffff880119b19f28 ffffffff8101dad9
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] ffff880119b19f18 d94b2b5b66b125a6 0000000000000000 0000000000000000
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] Call Trace:
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] [<ffffffff8101cc33>] default_idle+0x53/0x1f0
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] [<ffffffff8101dad9>] cpu_idle+0xd9/0x120
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] [<ffffffff816d10fe>] start_secondary+0xc3/0xc5
Aug 29 23:57:26 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.807776] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 f4 5d c3 66 0f 1f 84
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] NMI backtrace for cpu 2
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] CPU 2
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] Pid: 11307, comm: kworker/2:2 Tainted: GF 3.8.0-29-generic #42~precise1-Ubuntu Bochs Bochs
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] RIP: 0010:[<ffffffff81365b6a>] [<ffffffff81365b6a>] ioread8+0x1a/0x40
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] RSP: 0018:ffff880036e41838 EFLAGS: 00000006
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] RAX: ffffffff81ca7950 RBX: ffff88011603c000 RCX: 00000000d2524ab9
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] RDX: 0000000000000177 RSI: 0000000000000000 RDI: 0000000000010177
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] RBP: ffff880036e41838 R08: 0000000000000001 R09: ffff88011603c188
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] R10: 0000000000004008 R11: 0000000000000002 R12: 0000000000000000
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] R13: 00000000000003e8 R14: 0000000000000004 R15: 0000000000000004
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] FS: 0000000000000000(0000) GS:ffff88011fc80000(0000) knlGS:0000000000000000
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] CR2: 00007ff0f2e0f000 CR3: 0000000115832000 CR4: 00000000000006e0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] Process kworker/2:2 (pid: 11307, threadinfo ffff880036e40000, task ffff8801158c2e80)
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] Stack:
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] ffff880036e41848 ffffffff814ddcd2 ffff880036e41878 ffffffff814e0f5e
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] ffff880116768620 ffff88011603c150 ffff88011603c000 ffff88011603dc60
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] ffff880036e418b8 ffffffff814e1032 0000000000000020 ffff880036e41c8e
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] Call Trace:
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814ddcd2>] ata_sff_check_status+0x12/0x20
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814e0f5e>] ata_dev_select.constprop.21+0x3e/0xd0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814e1032>] ata_sff_qc_issue+0x42/0x1e0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814e120a>] ata_bmdma_qc_issue+0x3a/0x1a0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814cd56f>] ata_qc_issue+0x18f/0x2d0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814d28f0>] ? ata_gen_ata_sense+0x190/0x190
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814d2287>] ata_scsi_translate+0xa7/0x180
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814d5eeb>] ata_scsi_queuecmd+0xab/0x2b0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814a8106>] scsi_dispatch_cmd+0x136/0x2e0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814af722>] scsi_request_fn+0x382/0x450
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8117aa1a>] ? alloc_pages_current+0xba/0x170
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8132b547>] __blk_run_queue+0x37/0x50
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81336351>] blk_execute_rq_nowait+0x81/0x100
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8133645d>] blk_execute_rq+0x8d/0x140
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff811d2561>] ? bio_phys_segments+0x21/0x30
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81332370>] ? blk_rq_bio_prep+0x30/0xb0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81335ead>] ? blk_rq_append_bio+0x5d/0x70
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814ae7ae>] scsi_execute+0xee/0x180
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814af98d>] scsi_execute_req+0xbd/0x130
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814c1f98>] sr_get_events+0x78/0xf0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81097d93>] ? update_curr+0x143/0x1f0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff816f3d1e>] ? _raw_spin_lock_irqsave+0x2e/0x40
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814c205c>] sr_check_events+0x4c/0x1e0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81471a39>] ? __pm_runtime_resume+0x69/0x90
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8150048c>] cdrom_check_events+0x1c/0x40
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff814c24b9>] sr_block_check_events+0x39/0x50
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8133ad6b>] disk_check_events+0x5b/0x160
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8133ae86>] disk_events_workfn+0x16/0x20
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81078ce1>] process_one_work+0x141/0x490
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81079ca8>] worker_thread+0x168/0x400
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff81079b40>] ? manage_workers+0x120/0x120
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8107f1b0>] kthread+0xc0/0xd0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8107f0f0>] ? flush_kthread_worker+0xb0/0xb0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff816fc82c>] ret_from_fork+0x7c/0xb0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] [<ffffffff8107f0f0>] ? flush_kthread_worker+0xb0/0xb0
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7845.850153] Code: 00 00 5d c3 0f 1f 44 00 00 66 8b 07 0f b7 c0 5d c3 55 48 81 ff ff ff 03 00 48 89 e5 77 2b 48 81 ff 00 00 01 00 76 0a 0f b7 d7 ec <0f> b6 c0 5d c3 90 48 c7 c6 6c 00 a6 81 e8 a4 fd ff ff b8 ff 00
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7861.808670] ata2: lost interrupt (Status 0x58)
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7861.812614] ata2: drained 65536 bytes to clear DRQ
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334165] ata2.00: limiting speed to PIO4
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334169] ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334180] sr 1:0:0:0: CDB:
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334185] Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334217] ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334217] res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334224] ata2.00: status: { DRDY }
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.334306] ata2: soft resetting link
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.668421] ata2.00: configured for PIO4
Aug 29 23:57:43 Ubuntu-ceph-ubuntu-host-1 kernel: [ 7862.669285] ata2: EH complete
at which point the vm is not responding and needs a reboot.
Updated by Sage Weil over 10 years ago
- Status changed from New to Need More Info
- Assignee deleted (
Josh Durgin)
please try virtio. this looks like a combination of a bad ata driver and rados being slow...
Updated by Andrei Mikhailovsky over 10 years ago
Sage, I am using virtio to start my vms. by default cloudstack starts vms with virtio, however, the cache settings are cache=none. I can't change that unfortunately.
Updated by Sage Weil over 10 years ago
- Status changed from Need More Info to Closed
Andrei Mikhailovsky wrote:
Sage, I am using virtio to start my vms. by default cloudstack starts vms with virtio, however, the cache settings are cache=none. I can't change that unfortunately.
hmm, i'm not sure then what to say about the kernel crash. if a slow device makes 3.8 crash that is problematic. i don't see anything here pointing to a bug, though. you might try a newer kernel and/or make sure you are using a recent qemu that includes the aio_flush fixes?
Updated by Nico Schottelius over 9 years ago
- File ceph-debian-crash-20141227.png ceph-debian-crash-20141227.png added
- File ceph-redmine-20141227.png ceph-redmine-20141227.png added
Hello,
It seems I experience exactly the same problems on ceph 0.80.7 with various kernels within the VMs.
Attached are the latest screenshots.
The VMs are running Debian, jessie or wheezy with kernel 3.16.0-4-amd64 and 3.2.0-4-amd64.
I am using opennebula instead of openstack and the qemu command line is seen below.
Even though I see that this might be a Linux, QEMU or opennebula bug, it happens only in relation with ceph/rbd backend and it looks very similar to what was reported before.
libvirt+ 5640 8.7 9.0 4087000 1868348 ? Sl Dec21 739:58 qemu-system-x86_64 -enable-kvm -name one-180 -S -machine pc-i440fx-trusty,accel=kvm,usb=off -m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid ... -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-180.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device lsi,id=scsi0,bus=pci.0,addr=0x4 -drive file=rbd:one/one-49:id=libvirt:key=...:auth_supported=cephx\;none:mon_host=kaffee.private.ungleich.ch\;wein.private.ungleich.ch\;tee.private.ungleich.ch,if=none,id=drive-scsi0-0-0,format=raw,cache=none -device scsi-hd,bus=scsi0.0,scsi-id=0,drive=drive-scsi0-0-0,id=scsi0-0-0,bootindex=1 -drive file=/var/lib/one//datastores/0/180/disk.1,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=25,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=02:00:4d:6d:96:97,bus=pci.0,addr=0x3 -vnc 0.0.0.0:180 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -incoming fd:23 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5