Ceph » CephFS

Mar 16 09:43:10 teuthology kernel: [3307568.154466] libceph: osd52 down
Mar 16 09:43:18 teuthology kernel: [3307576.262229] libceph: osd52 up
Mar 16 09:43:50 teuthology kernel: [3307608.808578] ceph: mds0 reconnect denied
Mar 16 09:43:50 teuthology kernel: [3307608.825439] ceph:  dropping dirty+flushing Fw state for ffff880003cc4bb0 1099676077574
Mar 16 09:43:50 teuthology kernel: [3307608.833684] ceph:  dropping dirty data for ffff880003cc4bb0 1099676077574
Mar 16 09:43:50 teuthology kernel: [3307608.847329] ceph:  dropping dirty+flushing Fw state for ffff8800394ccbb0 1099676117445
Mar 16 09:43:50 teuthology kernel: [3307608.855594] ceph:  dropping dirty+flushing Fw state for ffff8800394ce620 1099676117446
...
Mar 16 09:43:59 teuthology kernel: [3307617.220234] ceph:  dropping dirty+flushing Fw state for ffff880297b62870 1099676094931
Mar 16 09:43:59 teuthology kernel: [3307617.228569] ceph:  dropping dirty+flushing Fw state for ffff8801ce6add50 1099676087465
Mar 16 09:43:59 teuthology kernel: [3307617.236811] ceph:  dropping dirty+flushing Fw state for ffff8800341bd480 1099676070642
Mar 16 09:43:59 teuthology kernel: [3307617.245116] ceph:  dropping dirty+flushing Fw state for ffff880020c842e0 1099676080523
Mar 16 09:43:59 teuthology kernel: [3307617.253313] ceph:  dropping dirty+flushing Fw state for ffff880020d70e00 1099676086396
Mar 16 09:43:59 teuthology kernel: [3307617.261511] ceph:  dropping dirty+flushing Fw state for ffff88025d5ca870 1099676114877
Mar 16 09:43:59 teuthology kernel: [3307617.269902] ceph:  dropping dirty+flushing Fw state for ffff88013e2a8530 1099676114901
Mar 16 09:43:59 teuthology kernel: [3307617.278239] ceph:  dropping dirty+flushing Fw state for ffff88016c8f2870 1099676113436
Mar 16 09:43:59 teuthology kernel: [3307617.286501] ceph:  dropping dirty+flushing Fw state for ffff8801eb25e620 1099676075091
Mar 16 09:43:59 teuthology kernel: [3307617.294703] ceph:  dropping dirty+flushing Fw state for ffff8802726a4bb0 1099676112540
Mar 16 09:43:59 teuthology kernel: [3307617.302889] ceph:  dropping dirty+flushing Fw state for ffff8801d4a39fa0 1099676084375
Mar 16 09:43:59 teuthology kernel: [3307617.311198] ceph:  dropping dirty+flushing Fw state for ffff8801d275a870 1099675847146
Mar 16 09:43:59 teuthology kernel: [3307617.319540] ceph:  dropping dirty+flushing Fw state for ffff880280465480 1099676095542
Mar 16 09:43:59 teuthology kernel: [3307617.327793] ceph:  dropping dirty+flushing Fw state for ffff88000537dd50 1099676111500
Mar 16 09:43:59 teuthology kernel: [3307617.336054] ceph:  dropping dirty+flushing Fw state for ffff880263423a10 1099676112633
Mar 16 09:43:59 teuthology kernel: [3307617.344335] ceph:  dropping dirty+flushing Fw state for ffff8800179c96d0 1099676084597
Mar 16 09:43:59 teuthology kernel: [3307617.352672] ceph:  dropping dirty+flushing Fw state for ffff8801efa65480 1099676045528
Mar 16 09:43:59 teuthology kernel: [3307617.360881] ceph:  dropping dirty+flushing Fw state for ffff88017ea1a870 1099676111315
Mar 16 09:43:59 teuthology kernel: [3307617.369135] ceph:  dropping dirty+flushing Fw state for ffff88001ba54bb0 1099676049451
Mar 16 09:43:59 teuthology kernel: [3307617.377467] [sched_delayed] sched: RT throttling activated
Mar 16 09:44:00 teuthology kernel: [3307617.728291] ------------[ cut here ]------------
Mar 16 09:44:00 teuthology kernel: [3307617.733107] kernel BUG at /srv/autobuild-ceph/gitbuilder.git/build/fs/ceph/addr.c:1024!
Mar 16 09:44:00 teuthology kernel: [3307617.741397] invalid opcode: 0000 [#1] SMP 
Mar 16 09:44:00 teuthology kernel: [3307617.745751] Modules linked in: ipmi_devintf(E) ipmi_si(E) ipmi_msghandler(E) ip6table_filter(E) ip6_tables(E) ebtable_nat(E) ebtables(E) ipt_MASQUERADE(E) iptable_nat(E) nf_nat_ipv4(E) nf_nat(E) nf_conntrack_ipv4(E) nf_defrag_ipv4(E) xt_state(E) nf_conntrack(E) ipt_REJECT(E) xt_CHECKSUM(E) iptable_mangle(E) xt_tcpudp(E) iptable_filter(E) ip_tables(E) x_tables(E) bridge(E) stp(E) llc(E) ceph(E) libceph(E) fscache(E) gpio_ich(E) psmouse(E) serio_raw(E) i7core_edac(E) edac_core(E) joydev(E) tpm_infineon(E) tpm_tis(E) lpc_ich(E) xfs(E) lp(E) parport(E) hid_generic(E) usbhid(E) hid(E) btrfs(E) e1000e(E) ahci(E) ptp(E) raid6_pq(E) libahci(E) pps_core(E) arcmsr(E) xor(E) libcrc32c(E)
Mar 16 09:44:00 teuthology kernel: [3307617.807403] CPU: 3 PID: 14523 Comm: python Tainted: G            E 3.16.3-ceph-00306-g76c7fd1 #1
Mar 16 09:44:00 teuthology kernel: [3307617.816499] Hardware name: Supermicro X8SIL/X8SIL, BIOS 1.1 05/27/2010
Mar 16 09:44:00 teuthology kernel: [3307617.823237] task: ffff8803353da1f0 ti: ffff88036c174000 task.ti: ffff88036c174000
Mar 16 09:44:00 teuthology kernel: [3307617.830955] RIP: 0010:[<ffffffffa0397e0a>]  [<ffffffffa0397e0a>] ceph_update_writeable_page+0x40a/0x480 [ceph]
Mar 16 09:44:00 teuthology kernel: [3307617.841190] RSP: 0018:ffff88036c177b18  EFLAGS: 00010246
Mar 16 09:44:00 teuthology kernel: [3307617.846745] RAX: 02ffff0000000005 RBX: ffffea000d764840 RCX: ffffea000d764840
Mar 16 09:44:00 teuthology kernel: [3307617.854194] RDX: 0000000000000b23 RSI: 0000000000220b23 RDI: ffff8801269e0540
Mar 16 09:44:00 teuthology kernel: [3307617.861581] RBP: ffff88036c177ba8 R08: 0000000000000001 R09: 0000000000000000
Mar 16 09:44:00 teuthology kernel: [3307617.868928] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801c6488e00
Mar 16 09:44:00 teuthology kernel: [3307617.876273] R13: ffff8801269e0540 R14: ffff880425859950 R15: 000000000000006c
Mar 16 09:44:00 teuthology kernel: [3307617.883654] FS:  00007fafc644f700(0000) GS:ffff88043fcc0000(0000) knlGS:0000000000000000
Mar 16 09:44:00 teuthology kernel: [3307617.891997] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Mar 16 09:44:00 teuthology kernel: [3307617.897930] CR2: 00007f1256380000 CR3: 0000000169569000 CR4: 00000000000007e0
Mar 16 09:44:00 teuthology kernel: [3307617.905307] Stack:
Mar 16 09:44:00 teuthology kernel: [3307617.907511]  ffff8801c6488fe0 0000006c00000000 0000000000220b23 00000ab700000b23
Mar 16 09:44:00 teuthology kernel: [3307617.915199]  0000000000220000 ffff8801269e0540 0000000000001000 0000000000000220
Mar 16 09:44:00 teuthology kernel: [3307617.922891]  ffff88036c177b78 0000000000220ab7 ffff88036c177b88 ffffffff81166536
Mar 16 09:44:00 teuthology kernel: [3307617.930601] Call Trace:
Mar 16 09:44:00 teuthology kernel: [3307617.933372]  [<ffffffff81166536>] ? wait_for_stable_page+0x16/0x50
Mar 16 09:44:00 teuthology kernel: [3307617.939747]  [<ffffffffa0397eec>] ceph_write_begin+0x6c/0xc0 [ceph]
Mar 16 09:44:00 teuthology kernel: [3307617.946321]  [<ffffffff8115a9e6>] generic_perform_write+0xc6/0x1e0
Mar 16 09:44:00 teuthology kernel: [3307617.952726]  [<ffffffffa03936b0>] ceph_write_iter+0x890/0xa30 [ceph]
Mar 16 09:44:00 teuthology kernel: [3307617.959268]  [<ffffffff817295fb>] ? _raw_spin_unlock+0x2b/0x40
Mar 16 09:44:00 teuthology kernel: [3307617.965348]  [<ffffffff81187454>] ? do_wp_page+0x374/0x740
Mar 16 09:44:00 teuthology kernel: [3307617.971021]  [<ffffffff81189efc>] ? handle_mm_fault+0x86c/0x1060
Mar 16 09:44:00 teuthology kernel: [3307617.977335]  [<ffffffff81046a19>] ? __do_page_fault+0x159/0x580
Mar 16 09:44:00 teuthology kernel: [3307617.983512]  [<ffffffff811cdc13>] ? vfs_write+0x1d3/0x1f0
Mar 16 09:44:00 teuthology kernel: [3307617.989111]  [<ffffffff811cdc13>] ? vfs_write+0x1d3/0x1f0
Mar 16 09:44:00 teuthology kernel: [3307617.994711]  [<ffffffff811ccdcb>] new_sync_write+0x7b/0xb0
Mar 16 09:44:00 teuthology kernel: [3307618.000386]  [<ffffffff811cdb07>] vfs_write+0xc7/0x1f0
Mar 16 09:44:00 teuthology kernel: [3307618.005842]  [<ffffffff811ce012>] SyS_write+0x52/0xb0
Mar 16 09:44:00 teuthology kernel: [3307618.011081]  [<ffffffff81729cd6>] system_call_fastpath+0x1a/0x1f
Mar 16 09:44:00 teuthology kernel: [3307618.018451] Code: fe ff ff 48 89 da 48 c7 c6 90 5b 3b a0 48 c7 c7 08 38 3c a0 31 c0 e8 46 c7 02 e1 31 c0 e9 ca fc ff ff 0f 1f 80 00 00 00 00 0f 0b <0f> 0b 4c 89 e9 48 89 da 48 c7 c6 58 5b 3b a0 48 c7 c7 e0 37 3c 
Mar 16 09:44:00 teuthology kernel: [3307618.038808] RIP  [<ffffffffa0397e0a>] ceph_update_writeable_page+0x40a/0x480 [ceph]
Mar 16 09:44:00 teuthology kernel: [3307618.046715]  RSP <ffff88036c177b18>
Mar 16 09:44:00 teuthology kernel: [3307618.050701] ---[ end trace 8bcfe0f2cc71a190 ]---
Mar 16 09:44:01 teuthology kernel: [3307618.959925] ------------[ cut here ]------------