Ceph » Linux kernel client

ceph3, rsync workload.
unstable circa 5/25

[258282.844588] ------------[ cut here ]------------
[258282.852409] WARNING: at lib/list_debug.c:48 list_del+0x38/0x91()
[258282.858572] Hardware name: H8SSL
[258282.861921] list_del corruption. prev->next should be ffff8800095969c0, but was 00000090ef814855
[258282.870848] Modules linked in: aes_x86_64 aes_generic ceph fan ac battery psmouse ide_pci_generic ehci_hcd ohci_hcd thermal button processor
[258282.883985] Pid: 4081, comm: umount Not tainted 2.6.34 #29
[258282.889599] Call Trace:
[258282.892188]  [<ffffffff81228c68>] ? list_del+0x38/0x91
[258282.897462]  [<ffffffff810358a8>] warn_slowpath_common+0x77/0x8f
[258282.903614]  [<ffffffff81035935>] warn_slowpath_fmt+0x64/0x66
[258282.909490]  [<ffffffff81057f1c>] ? mark_held_locks+0x49/0x64
[258282.915382]  [<ffffffff8105804a>] ? trace_hardirqs_on_caller+0x113/0x13e
[258282.922215]  [<ffffffff81058082>] ? trace_hardirqs_on+0xd/0xf
[258282.928104]  [<ffffffff81073001>] ? call_rcu_sched+0x10/0x12
[258282.933892]  [<ffffffff8107300c>] ? call_rcu+0x9/0xb
[258282.939004]  [<ffffffff81228c68>] list_del+0x38/0x91
[258282.944095]  [<ffffffff810bffec>] shrink_dcache_for_umount_subtree+0x1b6/0x270
[258282.951481]  [<ffffffff810c0a86>] shrink_dcache_for_umount+0x37/0x47
[258282.957962]  [<ffffffff810b1b3b>] generic_shutdown_super+0x20/0xcd
[258282.964287]  [<ffffffff810b1c33>] kill_anon_super+0x11/0x50
[258282.970011]  [<ffffffffa0071bfc>] ceph_kill_sb+0x41/0x7a [ceph]
[258282.976071]  [<ffffffff810b2379>] deactivate_super+0x68/0x7d
[258282.981860]  [<ffffffff810c599d>] mntput_no_expire+0x78/0xb3
[258282.987666]  [<ffffffff810c5eed>] sys_umount+0x2c5/0x2f2
[258282.993125]  [<ffffffff81002a2b>] system_call_fastpath+0x16/0x1b
[258282.999328] ---[ end trace b45c5ffc42aedb15 ]---
[258283.004076] general protection fault: 0000 [#1] PREEMPT SMP 
[258283.008065] last sysfs file: /sys/kernel/uevent_seqnum
[258283.008065] CPU 1 
[258283.008065] Modules linked in: aes_x86_64 aes_generic ceph fan ac battery psmouse ide_pci_generic ehci_hcd ohci_hcd thermal button processor
[258283.008065] 
[258283.008065] Pid: 4081, comm: umount Tainted: G        W  2.6.34 #29 H8SSL/H8SSL
[258283.008065] RIP: 0010:[<ffffffff81228c6b>]  [<ffffffff81228c6b>] list_del+0x3b/0x91
[258283.008065] RSP: 0018:ffff8800243d1dc8  EFLAGS: 00010296
[258283.008065] RAX: 6b6b6b6b6b6b6b6b RBX: ffff8800095969c0 RCX: ffff8800243d1ca8
[258283.008065] RDX: ffff880002800000 RSI: ffff8800243d19e8 RDI: 0000000000000009
[258283.008065] RBP: ffff8800243d1dd8 R08: ffff8800243d1b58 R09: 0000000000000000
[258283.008065] R10: ffffffff8104eaf8 R11: ffffffff8104e537 R12: ffff8800da5bd070
[258283.008065] R13: ffff8800095968b8 R14: 0000000000031e0a R15: 0000000000000000
[258283.008065] FS:  00007f4128d00720(0000) GS:ffff880002800000(0000) knlGS:0000000000000000
[258283.008065] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[258283.008065] CR2: 00000000006b97f8 CR3: 00000000f6fc9000 CR4: 00000000000006e0
[258283.008065] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[258283.008065] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[258283.008065] Process umount (pid: 4081, threadinfo ffff8800243d0000, task ffff880023cb8c00)
[258283.008065] Stack:
[258283.008065]  ffff88000958c818 ffff880009596930 ffff8800243d1e18 ffffffff810bffec
[258283.008065] <0> ffff8800f7f61350 ffff8800f7f61308 ffff8800f7df2508 ffff8800f7f61000
[258283.008065] <0> ffffffffa00a5f80 ffff8800f7f2b838 ffff8800243d1e38 ffffffff810c0a86
[258283.008065] Call Trace:
[258283.008065]  [<ffffffff810bffec>] shrink_dcache_for_umount_subtree+0x1b6/0x270
[258283.008065]  [<ffffffff810c0a86>] shrink_dcache_for_umount+0x37/0x47
[258283.008065]  [<ffffffff810b1b3b>] generic_shutdown_super+0x20/0xcd
[258283.008065]  [<ffffffff810b1c33>] kill_anon_super+0x11/0x50
[258283.008065]  [<ffffffffa0071bfc>] ceph_kill_sb+0x41/0x7a [ceph]
[258283.008065]  [<ffffffff810b2379>] deactivate_super+0x68/0x7d
[258283.008065]  [<ffffffff810c599d>] mntput_no_expire+0x78/0xb3
[258283.008065]  [<ffffffff810c5eed>] sys_umount+0x2c5/0x2f2
[258283.008065]  [<ffffffff81002a2b>] system_call_fastpath+0x16/0x1b
[258283.008065] Code: 48 8b 00 48 39 f8 74 20 49 89 c0 48 89 f9 48 c7 c2 3d 1c 59 81 be 30 00 00 00 48 c7 c7 77 1c 59 81 31 c0 e8 69 cc e0 ff 48 8b 03 <48> 8b 40 08 48 39 d8 74 20 49 89 c0 48 89 d9 48 c7 c2 88 1c 59 
[258283.008065] RIP  [<ffffffff81228c6b>] list_del+0x3b/0x91
[258283.008065]  RSP <ffff8800243d1dc8>
[258283.251107] ---[ end trace b45c5ffc42aedb16 ]---