Ceph » rbd

Jul 16 19:29:18 172.20.0.13 ceph-osd: 2013-07-16 19:29:18.072133 7f0237757700  0 -- 172.20.0.13:6800/1576 >> 172.20.0.13:0/301618590 pipe(0x7f0253fc4c80 sd=28 :6800 s=0 pgs=0 cs=0 l=0).accept peer addr is really 172.20.0.13:0/301618590 (socket is 172.20.0.13:34293/0)
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719077] rbd: obj_request read result -108 xferred 0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719077] 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719518] end_request: I/O error, dev rbd1, sector 0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719739] 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719739] Assertion failure in rbd_img_obj_callback() at line 1736:
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719739] 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719739]   rbd_assert(more ^ (which == img_request->obj_request_count));
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.719739] 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.720819] ------------[ cut here ]------------
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.721040] kernel BUG at drivers/block/rbd.c:1736!
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.721262] invalid opcode: 0000 [#1] SMP 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.721487] Modules linked in: dm_snapshot ebt_redirect vhost_net macvtap macvlan ebt_arp ebt_ip tun xt_CHECKSUM xt_mark xt_connmark xt_nat iptable_mangle ip6table_filter ip6_tables ebtable_nat ebtables nbd dm_mod xt_physdev sch_fq_codel cls_u32 sch_htb bridge stp llc 8021q openvswitch(O) xt_LOG xt_conntrack xt_CT iptable_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_filter ip_tables sd_mod crc_t10dif coretemp hwmon kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel cryptd iTCO_wdt iTCO_vendor_support microcode serio_raw e1000e i2c_i801 ahci lpc_ich i2c_core libahci ptp mfd_core pps_core vmsfs(O) vmsfs_impl(O) vmsfs_impl_2_6_2333(O) netconsole configfs
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.723601] CPU 3 
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.723608] Pid: 37, comm: kworker/3:1 Tainted: G           O 3.9.10 #1 Supermicro X9SCL/X9SCM/X9SCL/X9SCM
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.724247] RIP: 0010:[<ffffffff813f0730>]  [<ffffffff813f0730>] rbd_img_obj_callback+0x281/0x2db
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.724685] RSP: 0018:ffff88042a1c7b98  EFLAGS: 00010096
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.724909] RAX: 000000000000007b RBX: ffff8803e8b74b40 RCX: 0000000000000007
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.725134] RDX: 0000000000000006 RSI: 0000000000000046 RDI: ffff88043fd8ced0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.725360] RBP: ffff88042a1c7bd8 R08: 0000000000000000 R09: 000000000000ffa0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.725586] R10: 000000000000ffa0 R11: 0000000000001e00 R12: ffff8803e8b74b68
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.725810] R13: 0000000000000001 R14: 00000000ffffff94 R15: 0000000000000000
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.726031] FS:  0000000000000000(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.726460] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.726684] CR2: 00007f33c141fa30 CR3: 0000000401976000 CR4: 00000000000407e0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.726910] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.727135] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.727362] Process kworker/3:1 (pid: 37, threadinfo ffff88042a1c6000, task ffff88042a1beac0)
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.727796] Stack:
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.728010]  ffff8803f480d848 ffff8803e8b74b88 ffff88042a1c7bd8 ffff880404effc00
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.733773]  ffff88042a636400 0000000000000010 0000000000000000 0000000000000001
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.734213]  ffff88042a1c7c08 ffffffff813ef35a 00000000ffffff94 ffff88042a636400
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.734648] Call Trace:
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.734866]  [<ffffffff813ef35a>] rbd_osd_req_callback+0x296/0x2a9
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.735091]  [<ffffffff81548af4>] handle_reply+0x449/0x4e4
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.735311]  [<ffffffff81549ee6>] dispatch+0x43/0x79
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.735529]  [<ffffffff815400f4>] process_message+0x13e/0x156
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.735755]  [<ffffffff815434de>] ? read_partial_message+0x3ac/0x476
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.735978]  [<ffffffff81473a98>] ? kernel_recvmsg+0x3d/0x49
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.736197]  [<ffffffff815402f5>] ? ceph_tcp_recvmsg+0x4a/0x57
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.736417]  [<ffffffff815439a0>] try_read+0x3f8/0x501
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.736634]  [<ffffffff81543c06>] con_work+0x15d/0x224
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.736855]  [<ffffffff8104913e>] process_one_work+0x1a5/0x28e
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.737076]  [<ffffffff81049f1c>] worker_thread+0x14c/0x1e5
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.737298]  [<ffffffff81049dd0>] ? manage_workers+0xea/0xea
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.737520]  [<ffffffff8104dcb2>] kthread+0x8d/0x95
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.737742]  [<ffffffff8104dc25>] ? kthread_freezable_should_stop+0x3e/0x3e
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.737967]  [<ffffffff8157ef5c>] ret_from_fork+0x7c/0xb0
Jul 16 19:29:18 172.20.0.13 kernel: [ 1868.738187]  [<ffffffff8104dc25>] ? kthread_freezable_should_stop+0x3e/0x3e