Ceph

Here is a stack trace of an OSD crash, after the OSD crashed it caused all of the hosts using the RBD kernel client to reboot.

Nov 28 00:42:02 d44-1e-a1-3a-a2-50 kernel: [19830344.673997] ceph-osd²³¹⁴⁸ general protection ip:7f3a2e915df7 sp:7f3a01137700 error:0 in libtcmalloc.so.0.1.0[7f3a2e8f7000+3f000]
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940146] INFO: task ceph-osd:23017 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940370] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940696] ceph-osd D 0000000000000001 0 23017 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940709] ffff880101d71cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940730] ffff880101d71fd8 ffff880101d71fd8 ffff880101d71fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940745] ffff880809518000 ffff8800030e96f0 ffff8800030e96f0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940760] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940779] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940793] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940803] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940814] [<ffffffff8109d8bf>] ? __unqueue_futex+0x3f/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940826] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940836] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940846] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940858] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940867] [<ffffffff8106653b>] ? do_fork+0x15b/0x2e0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940877] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940885] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940894] [<ffffffff8107d1c2>] ? set_current_blocked+0x52/0x70
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940903] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940913] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.940925] INFO: task ceph-osd:23018 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941459] ceph-osd D 0000000000000007 0 23018 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941469] ffff881844849cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941485] ffff881844849fd8 ffff881844849fd8 ffff881844849fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941500] ffff88200c0f44d0 ffff881ce7208000 ffff881ce7208000 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941514] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941523] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941532] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941541] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941550] [<ffffffff8109d8bf>] ? __unqueue_futex+0x3f/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941560] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941569] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941578] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941588] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941597] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941606] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941615] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941625] [<ffffffff81177fa7>] ? sys_write+0x67/0x90
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941633] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941642] INFO: task ceph-osd:23019 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.941849] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942175] ceph-osd D 0000000000000015 0 23019 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942184] ffff881843513cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942198] ffff881843513fd8 ffff881843513fd8 ffff881843513fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942213] ffff88007d0aade0 ffff881ce72096f0 ffff881ce72096f0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942227] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942235] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942244] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942253] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942263] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942273] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942282] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942292] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942301] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942309] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942318] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942327] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942335] INFO: task ceph-osd:23020 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942541] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942867] ceph-osd D 000000000000000c 0 23020 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942876] ffff8818410d3cc8 0000000000000082 ffff8818410d3ca8 ffffffff8104c308
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942890] ffff8818410d3fd8 ffff8818410d3fd8 ffff8818410d3fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942905] ffff880e15865bc0 ffff881ce720ade0 0000000000000282 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942919] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942929] [<ffffffff8104c308>] ? __wake_up_common+0x58/0x90
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942938] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942946] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942956] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942966] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942975] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942984] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.942994] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943003] [<ffffffff810a0a16>] ? do_futex+0x146/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943012] [<ffffffff8106bc2b>] ? do_exit+0x26b/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943020] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943030] [<ffffffff810562ca>] ? finish_task_switch+0x4a/0xf0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943040] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943049] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943057] INFO: task ceph-osd:23035 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943590] ceph-osd D 0000000000000007 0 23035 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943598] ffff881843d65cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943613] ffff881843d65fd8 ffff881843d65fd8 ffff881843d65fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943627] ffff882007acdbc0 ffff8820075e5bc0 ffff8820075e5bc0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943641] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943649] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943658] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943667] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943677] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943687] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943696] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943706] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943715] [<ffffffff8106653b>] ? do_fork+0x15b/0x2e0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943723] [<ffffffff810a0978>] ? do_futex+0xa8/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943732] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943740] [<ffffffff8107d1c2>] ? set_current_blocked+0x52/0x70
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943750] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943759] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943767] INFO: task ceph-osd:23036 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.943973] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944328] ceph-osd D ffffffff81806240 0 23036 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944337] ffff8818435f1cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944352] ffff8818435f1fd8 ffff8818435f1fd8 ffff8818435f1fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944382] ffff88080c5f8000 ffff88200c0716f0 ffff88200c0716f0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944414] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944429] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944445] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944461] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944476] [<ffffffff8109d8bf>] ? __unqueue_futex+0x3f/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944492] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944513] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944528] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944545] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944561] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944576] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944592] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944607] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944619] INFO: task ceph-osd:23037 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.944831] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945154] ceph-osd D ffffffff81806240 0 23037 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945164] ffff8818426f9cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945179] ffff8818426f9fd8 ffff8818426f9fd8 ffff8818426f9fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945193] ffff88080c56c4d0 ffff882007acade0 ffff882007acade0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945207] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945216] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945225] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945235] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945244] [<ffffffff8109d8bf>] ? __unqueue_futex+0x3f/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945254] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945263] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945272] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945282] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945292] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945300] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945310] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945319] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945327] INFO: task ceph-osd:23038 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945536] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945859] ceph-osd D ffffffff81806240 0 23038 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945868] ffff881843151cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945882] ffff881843151fd8 ffff881843151fd8 ffff881843151fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945896] ffff88080c53dbc0 ffff882007acdbc0 ffff882007acdbc0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945910] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945918] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945928] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945937] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945947] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945957] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945966] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945976] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945984] [<ffffffff8106653b>] ? do_fork+0x15b/0x2e0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.945994] [<ffffffff8107a56b>] ? recalc_sigpending+0x1b/0x50
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946005] [<ffffffff8107ac67>] ? __set_task_blocked+0x37/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946014] [<ffffffff8107d1c2>] ? set_current_blocked+0x52/0x70
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946025] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946041] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946053] INFO: task ceph-osd:23039 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946264] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946588] ceph-osd D 0000000000000010 0 23039 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946597] ffff881844819cc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946612] ffff881844819fd8 ffff881844819fd8 ffff881844819fd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946626] ffff88000c36ade0 ffff882007ac96f0 ffff882007ac96f0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946641] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946649] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946658] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946668] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946676] [<ffffffff8109d8bf>] ? __unqueue_futex+0x3f/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946686] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946696] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946704] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946715] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946723] [<ffffffff810a098c>] ? do_futex+0xbc/0x1d0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946732] [<ffffffff810a0baa>] ? sys_futex+0x10a/0x1a0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946741] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946750] [<ffffffff816647d0>] int_signal+0x12/0x17
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946758] INFO: task ceph-osd:23040 blocked for more than 120 seconds.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.946965] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947290] ceph-osd D ffffffff81806240 0 23040 1 0x00000000
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947299] ffff881845befcc8 0000000000000082 0000000000000000 ffffffffffffffe0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947313] ffff881845beffd8 ffff881845beffd8 ffff881845beffd8 0000000000013780
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947327] ffff88080c4ddbc0 ffff882007acc4d0 ffff882007acc4d0 ffff8808096e7380
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947341] Call Trace:
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947349] [<ffffffff81659fdf>] schedule+0x3f/0x60
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947358] [<ffffffff8106b915>] exit_mm+0x85/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947367] [<ffffffff8106bb2e>] do_exit+0x16e/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947377] [<ffffffff8107a2ca>] ? __dequeue_signal+0x6a/0xb0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947387] [<ffffffff8106bf84>] do_group_exit+0x44/0xa0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947396] [<ffffffff8107ce0c>] get_signal_to_deliver+0x21c/0x420
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947406] [<ffffffff81013865>] do_signal+0x45/0x130
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947414] [<ffffffff8106653b>] ? do_fork+0x15b/0x2e0
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947424] [<ffffffff8107a56b>] ? recalc_sigpending+0x1b/0x50
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947434] [<ffffffff8107ac67>] ? __set_task_blocked+0x37/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947443] [<ffffffff8107d1c2>] ? set_current_blocked+0x52/0x70
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947452] [<ffffffff81013b15>] do_notify_resume+0x65/0x80
Nov 28 00:44:22 d44-1e-a1-3a-a2-50 kernel: [19830483.947461] [<ffffffff816647d0>] int_signal+0x12/0x17