Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2017-05-16T07:10:05Z
Ceph
Redmine
devops - Bug #19938 (Resolved): Duplicate logrotate entries if ceph-base and ceph-common are inst...
https://tracker.ceph.com/issues/19938
2017-05-16T07:10:05Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
devops - Bug #19937 (Duplicate): ceph-common needs dependency to psmisc package (Ubuntu 16.04)
https://tracker.ceph.com/issues/19937
2017-05-16T07:07:34Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>ceph-common includes a logrotate script that uses killall to inform ceph daemons to reopen their logfiles.</p>
<p>killall is part of the psmisc package, which is optional and might not be present on a system. As a result logrotate is not able to rotate log correctly.</p>
Ceph - Bug #19823 (Won't Fix): ceph-disk prepare/activate does not work properly using setuser/se...
https://tracker.ceph.com/issues/19823
2017-05-02T11:38:14Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>If 'ceph-disk prepare ...' is invoked on a new host without any existing OSDs, ceph-disk uses</p>
<p>ceph-osd --check-allows-journal -i 0 --cluster ceph --setuser ceph --setgroup ceph</p>
<p>or a similar command to check whether journals are allowed for the OSD backend. This fails with</p>
<p>unable to stat setuser_match_path /var/lib/ceph/$type/$cluster-$id: (2) No such file or directory</p>
<p>if the corresponding directory does not exists. Faking the directory (manual mkdir + chown) results in ceph-disk prepare to succeed.</p>
<p>The attempt to activate the new OSD using ceph-disk activate /dev/sdb in this example also fails due to the missing directory:<br />got monmap epoch 23<br />mount_activate: Failed to activate<br />ceph-disk: Error: ['ceph-osd', '--cluster', 'ceph', ...., '--setuser', 'ceph', '--setgroup', 'ceph'] failed : unable to stat setuser_match_path /var/lib/ceph/$type/$cluster-$id: (2) No such file or directory</p>
<p>Expected: preparation / activation of new OSD<br />Got: error due to missing OSD directory</p>
Linux kernel client - Bug #18671 (Resolved): kernel 4.8.15: BUG: soft lockup
https://tracker.ceph.com/issues/18671
2017-01-25T15:13:47Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>Running kernel 4.8.15 from Ubuntu mainline PPA, a machine is stuck in a kernel bug:</p>
<p>[Wed Jan 25 15:32:46 2017] NMI watchdog: BUG: soft lockup - CPU#88 stuck for 22s! [jellyfish:157790]<br />[Wed Jan 25 15:32:46 2017] Modules linked in: ceph libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache sunrpc veth xt_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_CHECKSUM openvswitch iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_ipv6 nf_nat_ipv6 iptable_nat nf_conntrack_ipv4 xt_tcpudp nf_defrag_ipv4 nf_nat_ipv4 bridge iptable_filter ip_tables nf_defrag_ipv6 x_tables nf_nat nf_conntrack libcrc32c 8021q garp mrp stp llc bonding ipmi_ssif intel_powerclamp binfmt_misc coretemp ipmi_si joydev input_leds hpilo crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i7core_edac aesni_intel gpio_ich aes_x86_64 lrw glue_helper ablk_helper cryptd lpc_ich intel_cstate kvm_intel serio_raw ipmi_msghandler acpi_power_meter edac_core shpchp mac_hid kvm irqbypass autofs4 amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt hid_generic fb_sys_fops usbhid hpsa psmouse drm hid pata_acpi scsi_transport_sas netxen_nic wmi fjes<br />[Wed Jan 25 15:32:46 2017] CPU: 88 PID: 157790 Comm: jellyfish Tainted: G L 4.8.15-040815-generic #201612151231<br />[Wed Jan 25 15:32:46 2017] Hardware name: HP ProLiant DL980 G7, BIOS P66 08/16/2015<br />[Wed Jan 25 15:32:46 2017] task: ffff8c4c95e11a00 task.stack: ffff8eb05c1c8000<br />[Wed Jan 25 15:32:46 2017] RIP: 0010:[<ffffffffa76ceb44>] [<ffffffffa76ceb44>] native_queued_spin_lock_slowpath+0x114/0x1a0<br />[Wed Jan 25 15:32:46 2017] RSP: 0018:ffff8eb05c1cbb10 EFLAGS: 00000246<br />[Wed Jan 25 15:32:46 2017] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff8c4f3fc99d40<br />[Wed Jan 25 15:32:46 2017] RDX: 0000000000000011 RSI: 0000000000480000 RDI: ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:46 2017] RBP: ffff8eb05c1cbb10 R08: 0000000001640000 R09: 0000000000000000<br />[Wed Jan 25 15:32:46 2017] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:46 2017] R13: 00000000ffffffff R14: ffff8c1a4c0b5f18 R15: ffff8ecea5f4bc00<br />[Wed Jan 25 15:32:46 2017] FS: 00007f6e68743700(0000) GS:ffff8c4f3fc80000(0000) knlGS:0000000000000000<br />[Wed Jan 25 15:32:46 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />[Wed Jan 25 15:32:46 2017] CR2: 00007f35feffd9d0 CR3: 000000bee402c000 CR4: 00000000000006e0<br />[Wed Jan 25 15:32:46 2017] Stack:<br />[Wed Jan 25 15:32:46 2017] ffff8eb05c1cbb20 ffffffffa7e833b0 ffff8eb05c1cbc50 ffffffffc0a1ce09<br />[Wed Jan 25 15:32:46 2017] ffff8ecea5f4bca8 ffff8c1a4c0b6260 ffff8c1a4c0b5f18 ffff8eb05c1cbbd8<br />[Wed Jan 25 15:32:46 2017] ffff8c1a4c0b5f28 0000000000000000 0000000000000000 0000000000000000<br />[Wed Jan 25 15:32:46 2017] Call Trace:<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e833b0>] _raw_spin_lock+0x20/0x30<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1ce09>] ceph_check_caps+0x89/0xaa0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a196d5>] ? __cap_is_valid+0x25/0xc0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a196d5>] ? __cap_is_valid+0x25/0xc0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1bf64>] ? __ceph_caps_mds_wanted+0x54/0x80 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1afcb>] ? __ceph_caps_issued+0x7b/0xe0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a11bdb>] ceph_renew_caps+0xbb/0x1c0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1f32f>] ceph_get_caps+0x29f/0x3b0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76c6cf0>] ? wake_atomic_t_function+0x60/0x60<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a16b50>] ceph_filemap_fault+0xb0/0x460 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa77d6514>] __do_fault+0x84/0x170<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76f58cc>] ? hrtimer_try_to_cancel+0x2c/0x120<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa77dad8a>] handle_mm_fault+0xdba/0x13c0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e827d6>] ? do_nanosleep+0x96/0xf0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76f657b>] ? hrtimer_nanosleep+0xdb/0x210<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa766b37b>] __do_page_fault+0x1db/0x4d0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa766b692>] do_page_fault+0x22/0x30<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e84898>] page_fault+0x28/0x30<br />[Wed Jan 25 15:32:46 2017] Code: 41 c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 9d 01 00 48 03 04 d5 20 83 55 a8 48 89 08 8b 41 08 85 c0 75 09 f3 90 <8b> 41 08 85 c0 74 f7 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 <br />[Wed Jan 25 15:32:46 2017] NMI watchdog: BUG: soft lockup - CPU#89 stuck for 22s! [jellyfish:157787]<br />[Wed Jan 25 15:32:46 2017] Modules linked in: ceph libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache sunrpc veth xt_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_CHECKSUM openvswitch iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_ipv6 nf_nat_ipv6 iptable_nat nf_conntrack_ipv4 xt_tcpudp nf_defrag_ipv4 nf_nat_ipv4 bridge iptable_filter ip_tables nf_defrag_ipv6 x_tables nf_nat nf_conntrack libcrc32c 8021q garp mrp stp llc bonding ipmi_ssif intel_powerclamp binfmt_misc coretemp ipmi_si joydev input_leds hpilo crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i7core_edac aesni_intel gpio_ich aes_x86_64 lrw glue_helper ablk_helper cryptd lpc_ich intel_cstate kvm_intel serio_raw ipmi_msghandler acpi_power_meter edac_core shpchp mac_hid kvm irqbypass autofs4 amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt hid_generic fb_sys_fops usbhid hpsa psmouse drm hid pata_acpi scsi_transport_sas netxen_nic wmi fjes<br />[Wed Jan 25 15:32:46 2017] CPU: 89 PID: 157787 Comm: jellyfish Tainted: G L 4.8.15-040815-generic #201612151231<br />[Wed Jan 25 15:32:46 2017] Hardware name: HP ProLiant DL980 G7, BIOS P66 08/16/2015<br />[Wed Jan 25 15:32:46 2017] task: ffff8c4c95e14e00 task.stack: ffff8ece98b64000<br />[Wed Jan 25 15:32:46 2017] RIP: 0010:[<ffffffffa76ceb44>] [<ffffffffa76ceb44>] native_queued_spin_lock_slowpath+0x114/0x1a0<br />[Wed Jan 25 15:32:46 2017] RSP: 0018:ffff8ece98b67bd8 EFLAGS: 00000246<br />[Wed Jan 25 15:32:46 2017] RAX: 0000000000000000 RBX: ffff8c1a4c0b5f28 RCX: ffff8c4f3fcd9d40<br />[Wed Jan 25 15:32:46 2017] RDX: 0000000000000057 RSI: 0000000001600000 RDI: ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:46 2017] RBP: ffff8ece98b67bd8 R08: 0000000001680000 R09: 0000000000000000<br />[Wed Jan 25 15:32:46 2017] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c1a4c0b5f18<br />[Wed Jan 25 15:32:46 2017] R13: ffff8c1a4c0b6260 R14: ffff8ecea5f4bca8 R15: 0000000000000800<br />[Wed Jan 25 15:32:46 2017] FS: 00007f6e69f46700(0000) GS:ffff8c4f3fcc0000(0000) knlGS:0000000000000000<br />[Wed Jan 25 15:32:46 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />[Wed Jan 25 15:32:46 2017] CR2: 00007f36a6794ab8 CR3: 000000bee402c000 CR4: 00000000000006e0<br />[Wed Jan 25 15:32:46 2017] Stack:<br />[Wed Jan 25 15:32:46 2017] ffff8ece98b67be8 ffffffffa7e833b0 ffff8ece98b67c88 ffffffffc0a1c023<br />[Wed Jan 25 15:32:46 2017] ffff8ece98b67ce4 ffff8ece98b67ce0 ffff8ecea5f4bc00 0000040098b67c50<br />[Wed Jan 25 15:32:46 2017] ffffffffffffffff 00000000cd2db6ed ffff8c1a4c0b6260 ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:46 2017] Call Trace:<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e833b0>] _raw_spin_lock+0x20/0x30<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1c023>] try_get_cap_refs+0x93/0x5c0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a1f1a5>] ceph_get_caps+0x115/0x3b0 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76c6cf0>] ? wake_atomic_t_function+0x60/0x60<br />[Wed Jan 25 15:32:46 2017] [<ffffffffc0a16b50>] ceph_filemap_fault+0xb0/0x460 [ceph]<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa77d6514>] __do_fault+0x84/0x170<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76f58cc>] ? hrtimer_try_to_cancel+0x2c/0x120<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa77dad8a>] handle_mm_fault+0xdba/0x13c0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e827d6>] ? do_nanosleep+0x96/0xf0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa76f657b>] ? hrtimer_nanosleep+0xdb/0x210<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa766b37b>] __do_page_fault+0x1db/0x4d0<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa766b692>] do_page_fault+0x22/0x30<br />[Wed Jan 25 15:32:46 2017] [<ffffffffa7e84898>] page_fault+0x28/0x30<br />[Wed Jan 25 15:32:46 2017] Code: 41 c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 9d 01 00 48 03 04 d5 20 83 55 a8 48 89 08 8b 41 08 85 c0 75 09 f3 90 <8b> 41 08 85 c0 74 f7 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 <br />[Wed Jan 25 15:32:50 2017] NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [jellyfish:157830]<br />[Wed Jan 25 15:32:50 2017] Modules linked in: ceph libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache sunrpc veth xt_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_CHECKSUM openvswitch iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_ipv6 nf_nat_ipv6 iptable_nat nf_conntrack_ipv4 xt_tcpudp nf_defrag_ipv4 nf_nat_ipv4 bridge iptable_filter ip_tables nf_defrag_ipv6 x_tables nf_nat nf_conntrack libcrc32c 8021q garp mrp stp llc bonding ipmi_ssif intel_powerclamp binfmt_misc coretemp ipmi_si joydev input_leds hpilo crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i7core_edac aesni_intel gpio_ich aes_x86_64 lrw glue_helper ablk_helper cryptd lpc_ich intel_cstate kvm_intel serio_raw ipmi_msghandler acpi_power_meter edac_core shpchp mac_hid kvm irqbypass autofs4 amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt hid_generic fb_sys_fops usbhid hpsa psmouse drm hid pata_acpi scsi_transport_sas netxen_nic wmi fjes<br />[Wed Jan 25 15:32:50 2017] CPU: 2 PID: 157830 Comm: jellyfish Tainted: G L 4.8.15-040815-generic #201612151231<br />[Wed Jan 25 15:32:50 2017] Hardware name: HP ProLiant DL980 G7, BIOS P66 08/16/2015<br />[Wed Jan 25 15:32:50 2017] task: ffff8ec6c9f40d00 task.stack: ffff8eb05c388000<br />[Wed Jan 25 15:32:50 2017] RIP: 0010:[<ffffffffa76ceb44>] [<ffffffffa76ceb44>] native_queued_spin_lock_slowpath+0x114/0x1a0<br />[Wed Jan 25 15:32:50 2017] RSP: 0018:ffff8eb05c38bbd8 EFLAGS: 00000246<br />[Wed Jan 25 15:32:50 2017] RAX: 0000000000000000 RBX: ffff8c1a4c0b5f28 RCX: ffff8c4f3f899d40<br />[Wed Jan 25 15:32:50 2017] RDX: 0000000000000058 RSI: 0000000001640000 RDI: ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:50 2017] RBP: ffff8eb05c38bbd8 R08: 00000000000c0000 R09: 0000000000000000<br />[Wed Jan 25 15:32:50 2017] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c1a4c0b5f18<br />[Wed Jan 25 15:32:50 2017] R13: ffff8c1a4c0b6260 R14: ffff8ecea5f4bca8 R15: 0000000000000800<br />[Wed Jan 25 15:32:50 2017] FS: 00007f6e5471b700(0000) GS:ffff8c4f3f880000(0000) knlGS:0000000000000000<br />[Wed Jan 25 15:32:50 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />[Wed Jan 25 15:32:50 2017] CR2: 00007f35f67e7ea8 CR3: 000000bee402c000 CR4: 00000000000006e0<br />[Wed Jan 25 15:32:50 2017] Stack:<br />[Wed Jan 25 15:32:50 2017] ffff8eb05c38bbe8 ffffffffa7e833b0 ffff8eb05c38bc88 ffffffffc0a1c023<br />[Wed Jan 25 15:32:50 2017] ffff8eb05c38bce4 ffff8eb05c38bce0 ffff8ecea5f4bc00 000004005c38bc50<br />[Wed Jan 25 15:32:50 2017] ffffffffffffffff 0000000003927d82 ffff8c1a4c0b6260 ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:50 2017] Call Trace:<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa7e833b0>] _raw_spin_lock+0x20/0x30<br />[Wed Jan 25 15:32:50 2017] [<ffffffffc0a1c023>] try_get_cap_refs+0x93/0x5c0 [ceph]<br />[Wed Jan 25 15:32:50 2017] [<ffffffffc0a1f1a5>] ceph_get_caps+0x115/0x3b0 [ceph]<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa76c6cf0>] ? wake_atomic_t_function+0x60/0x60<br />[Wed Jan 25 15:32:50 2017] [<ffffffffc0a16b50>] ceph_filemap_fault+0xb0/0x460 [ceph]<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa77d6514>] __do_fault+0x84/0x170<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa76f58cc>] ? hrtimer_try_to_cancel+0x2c/0x120<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa77dad8a>] handle_mm_fault+0xdba/0x13c0<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa7e827d6>] ? do_nanosleep+0x96/0xf0<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa76f657b>] ? hrtimer_nanosleep+0xdb/0x210<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa766b37b>] __do_page_fault+0x1db/0x4d0<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa766b692>] do_page_fault+0x22/0x30<br />[Wed Jan 25 15:32:50 2017] [<ffffffffa7e84898>] page_fault+0x28/0x30<br />[Wed Jan 25 15:32:50 2017] Code: 41 c1 ea 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 9d 01 00 48 03 04 d5 20 83 55 a8 48 89 08 8b 41 08 85 c0 75 09 f3 90 <8b> 41 08 85 c0 74 f7 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 <br />[Wed Jan 25 15:32:54 2017] NMI watchdog: BUG: soft lockup - CPU#91 stuck for 22s! [jellyfish:157783]<br />[Wed Jan 25 15:32:54 2017] Modules linked in: ceph libceph rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache sunrpc veth xt_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_CHECKSUM openvswitch iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_ipv6 nf_nat_ipv6 iptable_nat nf_conntrack_ipv4 xt_tcpudp nf_defrag_ipv4 nf_nat_ipv4 bridge iptable_filter ip_tables nf_defrag_ipv6 x_tables nf_nat nf_conntrack libcrc32c 8021q garp mrp stp llc bonding ipmi_ssif intel_powerclamp binfmt_misc coretemp ipmi_si joydev input_leds hpilo crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i7core_edac aesni_intel gpio_ich aes_x86_64 lrw glue_helper ablk_helper cryptd lpc_ich intel_cstate kvm_intel serio_raw ipmi_msghandler acpi_power_meter edac_core shpchp mac_hid kvm irqbypass autofs4 amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt hid_generic fb_sys_fops usbhid hpsa psmouse drm hid pata_acpi scsi_transport_sas netxen_nic wmi fjes<br />[Wed Jan 25 15:32:54 2017] CPU: 91 PID: 157783 Comm: jellyfish Tainted: G L 4.8.15-040815-generic #201612151231<br />[Wed Jan 25 15:32:54 2017] Hardware name: HP ProLiant DL980 G7, BIOS P66 08/16/2015<br />[Wed Jan 25 15:32:54 2017] task: ffff8e3d245f2700 task.stack: ffff8eb47316c000<br />[Wed Jan 25 15:32:54 2017] RIP: 0010:[<ffffffffa76ceb47>] [<ffffffffa76ceb47>] native_queued_spin_lock_slowpath+0x117/0x1a0<br />[Wed Jan 25 15:32:54 2017] RSP: 0018:ffff8eb47316fbd8 EFLAGS: 00000246<br />[Wed Jan 25 15:32:54 2017] RAX: 0000000000000000 RBX: ffff8c1a4c0b5f28 RCX: ffff8ccebfad9d40<br />[Wed Jan 25 15:32:54 2017] RDX: 0000000000000063 RSI: 0000000001900000 RDI: ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:54 2017] RBP: ffff8eb47316fbd8 R08: 0000000001700000 R09: 0000000000000000<br />[Wed Jan 25 15:32:54 2017] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8c1a4c0b5f18<br />[Wed Jan 25 15:32:54 2017] R13: ffff8c1a4c0b6260 R14: ffff8ecea5f4bca8 R15: 0000000000000800<br />[Wed Jan 25 15:32:54 2017] FS: 00007f6e6bf4a700(0000) GS:ffff8ccebfac0000(0000) knlGS:0000000000000000<br />[Wed Jan 25 15:32:54 2017] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />[Wed Jan 25 15:32:54 2017] CR2: 0000000000a80118 CR3: 000000bee402c000 CR4: 00000000000006e0<br />[Wed Jan 25 15:32:54 2017] Stack:<br />[Wed Jan 25 15:32:54 2017] ffff8eb47316fbe8 ffffffffa7e833b0 ffff8eb47316fc88 ffffffffc0a1c023<br />[Wed Jan 25 15:32:54 2017] ffff8eb47316fce4 ffff8eb47316fce0 ffff8ecea5f4bc00 000004007316fc50<br />[Wed Jan 25 15:32:54 2017] ffffffffffffffff 0000000034dd7540 ffff8c1a4c0b6260 ffff8c1a4c0b5f28<br />[Wed Jan 25 15:32:54 2017] Call Trace:<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa7e833b0>] _raw_spin_lock+0x20/0x30<br />[Wed Jan 25 15:32:54 2017] [<ffffffffc0a1c023>] try_get_cap_refs+0x93/0x5c0 [ceph]<br />[Wed Jan 25 15:32:54 2017] [<ffffffffc0a1f1a5>] ceph_get_caps+0x115/0x3b0 [ceph]<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa76c6cf0>] ? wake_atomic_t_function+0x60/0x60<br />[Wed Jan 25 15:32:54 2017] [<ffffffffc0a16b50>] ceph_filemap_fault+0xb0/0x460 [ceph]<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa77d6514>] __do_fault+0x84/0x170<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa76f58cc>] ? hrtimer_try_to_cancel+0x2c/0x120<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa77dad8a>] handle_mm_fault+0xdba/0x13c0<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa7e827d6>] ? do_nanosleep+0x96/0xf0<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa76f657b>] ? hrtimer_nanosleep+0xdb/0x210<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa766b37b>] __do_page_fault+0x1db/0x4d0<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa766b692>] do_page_fault+0x22/0x30<br />[Wed Jan 25 15:32:54 2017] [<ffffffffa7e84898>] page_fault+0x28/0x30<br />[Wed Jan 25 15:32:54 2017] Code: 12 83 e0 03 83 ea 01 48 c1 e0 04 48 63 d2 48 05 40 9d 01 00 48 03 04 d5 20 83 55 a8 48 89 08 8b 41 08 85 c0 75 09 f3 90 8b 41 08 <85> c0 74 f7 4c 8b 09 4d 85 c9 74 08 41 0f 18 09 eb 02 f3 90 8b</p>
<p>The machine hosts a LXC container for executing HPC jobs. The physical host is not accessible, but ssh to the LXC container succeeds after several minutes. Access to the /sys/kernel/debug filesystem is not possible from within the container. The machine currently has a high load, either due to several kernel threads spinning on their locks or user space applications:</p>
<ol>
<li>cat /proc/loadavg<br />430.48 431.23 430.97 263/2850 106806</li>
</ol>
<p>Listing the user process is not possible. The machine is stuck in this situation for about half a day; the MDS does not list an active session for it anymore (using ceph daemon mds.XXX session ls). The kernel log further indicates that the machine had some trouble with mon and mds connections yesterday. Unfortunately most of the kernel log file entries are truncated.</p>
<p>We will have to reboot the machine (or find a better way to recover it), so we will be unable to provide more information in this case.</p>
CephFS - Bug #17212 (Resolved): Unable to remove symlink / fill_inode badness on ffff88025f049f88
https://tracker.ceph.com/issues/17212
2016-09-05T13:09:19Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>We have a symlink in our filesystem that we cannot remove.</p>
<p>Ceph MDS version: 10.2.2<br />Kernel version: 4.4.0-34-generic #53~14.04.1-Ubuntu, 4.4.0-24-generic #43~14.04.1-Ubuntu (running ubuntu trusty with xenial backport kernels)<br />ceph-fuse version: 10.2.2-1trusty</p>
<p>cephfs (kernel):</p>
<ol>
<li>ls -al save/<br />ls: cannot access save/stdout.json: Invalid argument<br />total 0<br />drwxrwxr-x 1 XYZ students 0 Sep 5 14:33 .<br />drwxr-xr-x 1 root root 0 Sep 5 14:34 ..<br />l????????? ? ? ? ? ? stdout.json</li>
</ol>
<p>kernel log:</p>
<p>[Mon Sep 5 14:46:39 2016] ceph: fill_inode badness on ffff88025f049f88<br />[Mon Sep 5 14:46:39 2016] ------------[ cut here ]------------<br />[Mon Sep 5 14:46:39 2016] WARNING: CPU: 3 PID: 32673 at /build/linux-lts-xenial-D8Bcmp/linux-lts-xenial-4.4.0/fs/ceph/inode.c:811 fill_inode.isra.16+0xb2a/0xc00 [ceph]()<br />[Mon Sep 5 14:46:39 2016] Modules linked in: ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs ceph libceph cts 8021q garp mrp stp llc openvswitch nf_defrag_ipv6 nf_conntrack libcrc32c ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper serio_raw cryptd sb_edac lpc_ich hpilo edac_core wmi 8250_fintek shpchp ipmi_si ipmi_msghandler ioatdma acpi_power_meter mac_hid parport_pc ppdev rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lp lockd grace sunrpc fscache parport btrfs xor raid6_pq ixgbe dca vxlan psmouse ip6_udp_tunnel tg3 udp_tunnel ptp hpsa pps_core scsi_transport_sas mdio fjes<br />[Mon Sep 5 14:46:39 2016] CPU: 3 PID: 32673 Comm: kworker/3:2 Tainted: G W 4.4.0-24-generic #43~14.04.1-Ubuntu<br />[Mon Sep 5 14:46:39 2016] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015<br />[Mon Sep 5 14:46:39 2016] Workqueue: ceph-msgr ceph_con_workfn [libceph]<br />[Mon Sep 5 14:46:39 2016] 0000000000000000 ffff881fb8943a50 ffffffff813ce10c 0000000000000000<br />[Mon Sep 5 14:46:39 2016] ffffffffc05ff3b0 ffff881fb8943a88 ffffffff8107d736 00000000ffffffea<br />[Mon Sep 5 14:46:39 2016] 0000000000000000 0000000000000000 ffff88025f04a528 ffff881febe4d931<br />[Mon Sep 5 14:46:39 2016] Call Trace:<br />[Mon Sep 5 14:46:39 2016] [<ffffffff813ce10c>] dump_stack+0x63/0x87<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8107d736>] warn_slowpath_common+0x86/0xc0<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8107d82a>] warn_slowpath_null+0x1a/0x20<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc05d799a>] fill_inode.isra.16+0xb2a/0xc00 [ceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffff81216275>] ? inode_init_always+0x105/0x1b0<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc05d7b87>] ceph_fill_trace+0x117/0x970 [ceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc05f5d19>] handle_reply+0x429/0xca0 [ceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc05f816e>] dispatch+0xae/0xaf0 [ceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc059cad3>] try_read+0x443/0x1120 [libceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffff810963a0>] ? try_to_grab_pending+0xb0/0x150<br />[Mon Sep 5 14:46:39 2016] [<ffffffff810b2f75>] ? put_prev_entity+0x35/0x670<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8102c666>] ? __switch_to+0x1d6/0x570<br />[Mon Sep 5 14:46:39 2016] [<ffffffffc059d862>] ceph_con_workfn+0xb2/0x5d0 [libceph]<br />[Mon Sep 5 14:46:39 2016] [<ffffffff81095820>] process_one_work+0x150/0x3f0<br />[Mon Sep 5 14:46:39 2016] [<ffffffff81095f9a>] worker_thread+0x11a/0x470<br />[Mon Sep 5 14:46:39 2016] [<ffffffff817eb829>] ? __schedule+0x359/0x980<br />[Mon Sep 5 14:46:39 2016] [<ffffffff81095e80>] ? rescuer_thread+0x310/0x310<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8109b6d9>] kthread+0xc9/0xe0<br />[Mon Sep 5 14:46:39 2016] [<ffffffff810804fb>] ? do_exit+0x65b/0xae0<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8109b610>] ? kthread_park+0x60/0x60<br />[Mon Sep 5 14:46:39 2016] [<ffffffff817efa4f>] ret_from_fork+0x3f/0x70<br />[Mon Sep 5 14:46:39 2016] [<ffffffff8109b610>] ? kthread_park+0x60/0x60<br />[Mon Sep 5 14:46:39 2016] ---[ end trace 03e9b82c2c95e5ab ]---</p>
<p>ceph-fuse:</p>
<ol>
<li>ls -al save/<br />total 1<br />drwxrwxr-x 1 XYZ students 0 Sep 5 14:33 .<br />drwxr-xr-x 1 root root 0 Sep 5 14:34 ..<br />lrwxrwxrwx 1 XYZ students 0 Sep 5 13:20 stdout.json -> /dev/stdout</li>
</ol>
<p>(the file is already moved to a save location)</p>
<p>We had a number of messages in the MDS logs over the weekend:</p>
<p>2016-09-03 07:41:17.419313 7fc7fffff700 0 log_channel(cluster) log [WRN] : slow request 122911.110094 seconds old, received at 2016-09-01 21:32:46.309124: client_request(client.3810003:4637318 lo<br />okup #100072025ce/BlastParser.groovy 2016-09-01 21:32:46.307463) currently failed to rdlock, waiting</p>
<p>and</p>
<p>2016-09-03 14:19:25.750649 7fc803468700 0 log_channel(cluster) log [WRN] : client.4031546 does not advance its oldest_client_tid (3341229), 100002 completed requests recorded in session</p>
<p>Finally today there was a problem with the session map, resulting in the first MDS restart:</p>
<p>2016-09-05 13:18:12.151833 7fc7fdffb700 -1 mds/SessionMap.cc: In function 'virtual void C_IO_SM_Save::finish(int)' thread 7fc7fdffb700 time 2016-09-05 13:18:11.755592<br />mds/SessionMap.cc: 319: FAILED assert(r == 0)</p>
<pre><code>ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7fc80b032915]<br /> 2: (()+0x45efc1) [0x7fc80ae96fc1]<br /> 3: (MDSIOContextBase::complete(int)+0xa4) [0x7fc80ae98334]<br /> 4: (Finisher::finisher_thread_entry()+0x27d) [0x7fc80af61e9d]<br /> 5: (()+0x7dc5) [0x7fc809bf4dc5]<br /> 6: (clone()+0x6d) [0x7fc8086bfced]</code></pre>
<p>I don't know when the symlink was created, but the next crash semms to be related to it:</p>
<pre><code><del>5> 2016-09-05 13:47:29.823859 7f3ad7aa6700 5 -</del> op tracker -- seq: 449313, time: 2016-09-05 13:47:29.823859, event: killing request, op: client_request(client.4010513:1442135 create #10006da65c1/stdout.json 2016-09-05 13:22:09.431415)<br /> <del>4> 2016-09-05 13:47:29.823912 7f3ad07fe700 5 mds.0.log _submit_thread 4425743152308~230 : ESession client.4010513 134.176.31.195:0/3840219 close cmapv 154523073 (630 inos, v12770540)<br /> -3> 2016-09-05 13:47:29.823941 7f3ad7aa6700 5 -</del> op tracker -- seq: 449313, time: 2016-09-05 13:47:29.823941, event: cleaned up request, op: client_request(client.4010513:1442135 create #10006da65c1/stdout.json 2016-09-05 13:22:09.431415)<br /> <del>2> 2016-09-05 13:47:29.823976 7f3ad07fe700 1 -</del> 192.168.6.129:6801/43283 --> 192.168.6.134:6816/31271 -- osd_op(mds.0.197986:376340 8.3ee7dd50 300.001019cb [write 1651892~250 [fadvise_dontneed]] snapc 0=[] ondisk+write+known_if_redirected+full_force e421445) v7 -- ?+0 0x7f3b217bb440 con 0x7f3ae9da6100<br /> <del>1> 2016-09-05 13:47:29.826297 7f3ac33ff700 1 -</del> 192.168.6.129:6801/43283 <== osd.88 192.168.6.134:6816/31271 2584 ==== osd_op_reply(376340 300.001019cb [write 1651892~250 [fadvise_dontneed]] v421445'361223 uv361223 ondisk = 0) v7 ==== 132+0+0 (3052469259 0 0) 0x7f3b2654c580 con 0x7f3ae9da6100<br /> 0> 2016-09-05 13:47:29.856106 7f3ad23fb700 -1 mds/Locker.cc: In function 'bool Locker::check_inode_max_size(CInode*, bool, bool, uint64_t, bool, uint64_t, utime_t)' thread 7f3ad23fb700 time 2016-09-05 13:47:29.853913<br />mds/Locker.cc: 2190: FAILED assert(in->is_file())</code></pre>
<pre><code>ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f3adf6a0915]<br /> 2: (Locker::check_inode_max_size(CInode*, bool, bool, unsigned long, bool, unsigned long, utime_t)+0x15fd) [0x7f3adf43bafd]<br /> 3: (Locker::remove_client_cap(CInode*, client_t)+0xb1) [0x7f3adf43c051]<br /> 4: (Server::_session_logged(Session*, unsigned long, bool, unsigned long, interval_set&lt;inodeno_t&gt;&, unsigned long)+0x515) [0x7f3adf2f5cc5]<br /> 5: (C_MDS_session_finish::finish(int)+0x35) [0x7f3adf34d855]<br /> 6: (MDSInternalContextBase::complete(int)+0x1eb) [0x7f3adf50600b]<br /> 7: (C_MDL_Flushed::finish(int)+0x13) [0x7f3adf51add3]<br /> 8: (MDSIOContextBase::complete(int)+0xa4) [0x7f3adf506334]<br /> 9: (Finisher::finisher_thread_entry()+0x216) [0x7f3adf5cfe36]<br /> 10: (()+0x7dc5) [0x7f3ade262dc5]<br /> 11: (clone()+0x6d) [0x7f3adcd2dced]</code></pre>
<p>Further attempts to delete the file results in further MDS crashes. I know how to remove the symlink (removing the omap entry), but<br />I'll keep it around in case you want more debug output to track down the root of the problem.</p>
<p>Ceph MDS log available on request.</p>
CephFS - Bug #16013 (Resolved): Failing file operations on kernel based cephfs mount point leaves...
https://tracker.ceph.com/issues/16013
2016-05-25T07:28:13Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>After some number of operations on files (which could not be traced for reproduction), we end up with a broken directory:</p>
<ol>
<li>ls -al<br />ls: cannot access DensityMap: Invalid argument<br />total 0<br />drwxr-sr-x 1 XXXX XXXX 0 May 24 17:50 .<br />drwxr-sr-x 1 XXXX XXXX 260874151 May 24 17:55 ..<br />l????????? ? ? ? ? ? DensityMap</li>
</ol>
<p>Accessing the directory results in a kernel error message:</p>
<ol>
<li>uname -a<br />Linux waas 4.4.0-22-generic #39~14.04.1-Ubuntu SMP Thu May 5 19:19:06 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux</li>
</ol>
<p>[Wed May 25 09:08:23 2016] ------------[ cut here ]------------<br />[Wed May 25 09:08:23 2016] WARNING: CPU: 8 PID: 9700 at /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/ceph/inode.c:811 fill_inode.isra.16+0xb2a/0xc00 [ceph]()<br />[Wed May 25 09:08:23 2016] Modules linked in: tcp_diag(E) inet_diag(E) ceph(E) libceph(E) usblp(E) parport_pc(E) ppdev(E) ufs(E) qnx4(E) hfsplus(E) hfs(E) minix(E) ntfs(E) msdos(E) jfs(E) xfs(E) iptable_filter ip_tables x_tables cts 8021q garp mrp stp llc openvswitch nf_defrag_ipv6 nf_conntrack libcrc32c rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache intel_rapl x86_pkg_temp_thermal ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw input_leds hpilo gf128mul glue_helper joydev ablk_helper shpchp ioatdma cryptd lpc_ich sb_edac serio_raw edac_core acpi_power_meter 8250_fintek wmi mac_hid ipmi_si ipmi_msghandler lp parport btrfs xor raid6_pq hid_generic usbhid hid ixgbe dca psmouse vxlan tg3 ip6_udp_tunnel hpsa udp_tunnel ptp pps_core scsi_transport_sas mdio fjes [last unloaded: libceph]<br />[Wed May 25 09:08:23 2016] CPU: 8 PID: 9700 Comm: kworker/8:0 Tainted: G W E 4.4.0-22-generic #39~14.04.1-Ubuntu<br />[Wed May 25 09:08:23 2016] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015<br />[Wed May 25 09:08:23 2016] Workqueue: ceph-msgr ceph_con_workfn [libceph]<br />[Wed May 25 09:08:23 2016] 0000000000000000 ffff881fde657a28 ffffffff813cde6c 0000000000000000<br />[Wed May 25 09:08:23 2016] ffffffffc08003b0 ffff881fde657a60 ffffffff8107d856 00000000ffffffea<br />[Wed May 25 09:08:23 2016] 0000000000000000 0000000000000000 ffff880422556308 ffff881f1e5e053a<br />[Wed May 25 09:08:23 2016] Call Trace:<br />[Wed May 25 09:08:23 2016] [<ffffffff813cde6c>] dump_stack+0x63/0x87<br />[Wed May 25 09:08:23 2016] [<ffffffff8107d856>] warn_slowpath_common+0x86/0xc0<br />[Wed May 25 09:08:23 2016] [<ffffffff8107d94a>] warn_slowpath_null+0x1a/0x20<br />[Wed May 25 09:08:23 2016] [<ffffffffc07d899a>] fill_inode.isra.16+0xb2a/0xc00 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07d65e0>] ? ceph_mount+0x810/0x810 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07d9604>] ceph_readdir_prepopulate+0x224/0x8c0 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07f7342>] handle_reply+0xa32/0xca0 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07f918e>] dispatch+0xae/0xaf0 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc0618ac3>] try_read+0x443/0x1120 [libceph]<br />[Wed May 25 09:08:23 2016] [<ffffffff81036a09>] ? sched_clock+0x9/0x10<br />[Wed May 25 09:08:23 2016] [<ffffffff810b31c5>] ? put_prev_entity+0x35/0x670<br />[Wed May 25 09:08:23 2016] [<ffffffff8102c696>] ? __switch_to+0x1d6/0x570<br />[Wed May 25 09:08:23 2016] [<ffffffffc0619852>] ceph_con_workfn+0xb2/0x5d0 [libceph]<br />[Wed May 25 09:08:23 2016] [<ffffffff810959cd>] process_one_work+0x14d/0x3f0<br />[Wed May 25 09:08:23 2016] [<ffffffff8109614a>] worker_thread+0x11a/0x470<br />[Wed May 25 09:08:23 2016] [<ffffffff817ebe19>] ? __schedule+0x359/0x970<br />[Wed May 25 09:08:23 2016] [<ffffffff81096030>] ? rescuer_thread+0x310/0x310<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b882>] kthread+0xd2/0xf0<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b7b0>] ? kthread_park+0x50/0x50<br />[Wed May 25 09:08:23 2016] [<ffffffff817f004f>] ret_from_fork+0x3f/0x70<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b7b0>] ? kthread_park+0x50/0x50<br />[Wed May 25 09:08:23 2016] ---[ end trace 99ae552d517bb8d0 ]---<br />[Wed May 25 09:08:23 2016] ceph: fill_inode badness on ffff880422556308<br />[Wed May 25 09:08:23 2016] ------------[ cut here ]------------<br />[Wed May 25 09:08:23 2016] WARNING: CPU: 8 PID: 9700 at /build/linux-lts-xenial-7RlTta/linux-lts-xenial-4.4.0/fs/ceph/inode.c:811 fill_inode.isra.16+0xb2a/0xc00 [ceph]()<br />[Wed May 25 09:08:23 2016] Modules linked in: tcp_diag(E) inet_diag(E) ceph(E) libceph(E) usblp(E) parport_pc(E) ppdev(E) ufs(E) qnx4(E) hfsplus(E) hfs(E) minix(E) ntfs(E) msdos(E) jfs(E) xfs(E) iptable_filter ip_tables x_tables cts 8021q garp mrp stp llc openvswitch nf_defrag_ipv6 nf_conntrack libcrc32c rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl nfs lockd grace sunrpc fscache intel_rapl x86_pkg_temp_thermal ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul aesni_intel aes_x86_64 lrw input_leds hpilo gf128mul glue_helper joydev ablk_helper shpchp ioatdma cryptd lpc_ich sb_edac serio_raw edac_core acpi_power_meter 8250_fintek wmi mac_hid ipmi_si ipmi_msghandler lp parport btrfs xor raid6_pq hid_generic usbhid hid ixgbe dca psmouse vxlan tg3 ip6_udp_tunnel hpsa udp_tunnel ptp pps_core scsi_transport_sas mdio fjes [last unloaded: libceph]<br />[Wed May 25 09:08:23 2016] CPU: 8 PID: 9700 Comm: kworker/8:0 Tainted: G W E 4.4.0-22-generic #39~14.04.1-Ubuntu<br />[Wed May 25 09:08:23 2016] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015<br />[Wed May 25 09:08:23 2016] Workqueue: ceph-msgr ceph_con_workfn [libceph]<br />[Wed May 25 09:08:23 2016] 0000000000000000 ffff881fde657a50 ffffffff813cde6c 0000000000000000<br />[Wed May 25 09:08:23 2016] ffffffffc08003b0 ffff881fde657a88 ffffffff8107d856 00000000ffffffea<br />[Wed May 25 09:08:23 2016] 0000000000000000 0000000000000000 ffff880422556e48 ffff881f1e5e0130<br />[Wed May 25 09:08:23 2016] Call Trace:<br />[Wed May 25 09:08:23 2016] [<ffffffff813cde6c>] dump_stack+0x63/0x87<br />[Wed May 25 09:08:23 2016] [<ffffffff8107d856>] warn_slowpath_common+0x86/0xc0<br />[Wed May 25 09:08:23 2016] [<ffffffff8107d94a>] warn_slowpath_null+0x1a/0x20<br />[Wed May 25 09:08:23 2016] [<ffffffffc07d899a>] fill_inode.isra.16+0xb2a/0xc00 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffff812167c5>] ? inode_init_always+0x105/0x1b0<br />[Wed May 25 09:08:23 2016] [<ffffffffc07d8b87>] ceph_fill_trace+0x117/0x970 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07f6d39>] handle_reply+0x429/0xca0 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc07f918e>] dispatch+0xae/0xaf0 [ceph]<br />[Wed May 25 09:08:23 2016] [<ffffffffc0618ac3>] try_read+0x443/0x1120 [libceph]<br />[Wed May 25 09:08:23 2016] [<ffffffff810b31c5>] ? put_prev_entity+0x35/0x670<br />[Wed May 25 09:08:23 2016] [<ffffffff8102c696>] ? __switch_to+0x1d6/0x570<br />[Wed May 25 09:08:23 2016] [<ffffffffc0619852>] ceph_con_workfn+0xb2/0x5d0 [libceph]<br />[Wed May 25 09:08:23 2016] [<ffffffff810959cd>] process_one_work+0x14d/0x3f0<br />[Wed May 25 09:08:23 2016] [<ffffffff8109614a>] worker_thread+0x11a/0x470<br />[Wed May 25 09:08:23 2016] [<ffffffff817ebe19>] ? __schedule+0x359/0x970<br />[Wed May 25 09:08:23 2016] [<ffffffff81096030>] ? rescuer_thread+0x310/0x310<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b882>] kthread+0xd2/0xf0<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b7b0>] ? kthread_park+0x50/0x50<br />[Wed May 25 09:08:23 2016] [<ffffffff817f004f>] ret_from_fork+0x3f/0x70<br />[Wed May 25 09:08:23 2016] [<ffffffff8109b7b0>] ? kthread_park+0x50/0x50<br />[Wed May 25 09:08:23 2016] ---[ end trace 99ae552d517bb8d1 ]---<br />[Wed May 25 09:08:23 2016] ceph: fill_inode badness ffff880422556e48 100041dee77.fffffffffffffffe</p>
<p>Attempts to remove the affected file or moving the directory itself of the the way also fail with the similar error messages.</p>
<p>How do I resolve this problem?</p>
Linux kernel client - Bug #15845 (Resolved): Kernel crash after unmounting CephFS mountpoint
https://tracker.ceph.com/issues/15845
2016-05-11T12:00:09Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>We had a server crash several hours after unmounting a CephFS mountpoint.</p>
<p>OS: Ubuntu 14.04 with Xenial LTS backport kernel 4.4.0-18<br />Ceph: ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432)</p>
<p>1. umount /srv/ceph</p>
<p>Error message in terminal: Segmentation fault<br />Kernel log:</p>
<p>May 11 10:46:52 waas kernel: [2328493.378160] VFS: Busy inodes after unmount of ceph. Self-destruct in 5 seconds. Have a nice day...<br />May 11 10:46:52 waas kernel: [2328493.378313] FS-Cache: Cookie 'CEPH.fsid' still has children<br />May 11 10:46:54 waas kernel: [2328493.415074] ------------[ cut here ]------------<br />May 11 10:46:54 waas kernel: [2328493.451388] kernel BUG at /build/linux-lts-xenial-WD7MTT/linux-lts-xenial-4.4.0/fs/fscache/cookie.c:524!<br />May 11 10:46:54 waas kernel: [2328493.525716] invalid opcode: 0000 [<a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a>] SMP <br />May 11 10:46:54 waas kernel: [2328493.562257] Modules linked in: rbd ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs ceph libceph cts 8021q garp mrp stp llc openvswitch nf_defrag_ipv6 nf_conntrack libcrc32c ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretem<br />p kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul joydev aesni_intel rpcsec_gss_krb5 nfsd hpilo auth_rpcgss nfs_acl input_leds serio_raw nfs shpchp sb_edac lockd grace 8250_fintek edac_core mac_hid ioatdma sunrpc acpi_power_meter wmi ipmi_si aes_x86_64 lpc_ich lp l<br />rw gf128mul glue_helper ipmi_msghandler fscache ablk_helper cryptd parport btrfs xor hid_generic usbhid hid raid6_pq ixgbe dca psmouse vxlan tg3 ip6_udp_tunnel udp_tunnel ptp hpsa pps_core scsi_transport_sas mdio fjes<br />May 11 10:46:54 waas kernel: [2328493.944655] CPU: 10 PID: 15196 Comm: umount Not tainted 4.4.0-18-generic #34~14.04.1-Ubuntu<br />May 11 10:46:54 waas kernel: [2328494.022091] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015<br />May 11 10:46:54 waas kernel: [2328494.060144] task: ffff881fe9ee2940 ti: ffff880413ce0000 task.ti: ffff880413ce0000<br />May 11 10:46:54 waas kernel: [2328494.136513] RIP: 0010:[<ffffffffc0114d60>] [<ffffffffc0114d60>] __fscache_disable_cookie+0x1a0/0x1e0 [fscache]<br />May 11 10:46:54 waas kernel: [2328494.216464] RSP: 0018:ffff880413ce3dd0 EFLAGS: 00010296<br />May 11 10:46:54 waas kernel: [2328494.255946] RAX: 000000000000002f RBX: ffff881fab61c000 RCX: 00000000ffffffff<br />May 11 10:46:54 waas kernel: [2328494.334802] RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246<br />May 11 10:46:54 waas kernel: [2328494.413225] RBP: ffff880413ce3df8 R08: 0000000000000000 R09: ffff881fed375220<br />May 11 10:46:54 waas kernel: [2328494.491477] R10: 0000000000005000 R11: 00000000000015af R12: ffff881fab61c000<br />May 11 10:46:54 waas kernel: [2328494.572308] R13: 000000000000001d R14: ffff881fe9ee2940 R15: 00000000016e41b0<br />May 11 10:46:54 waas kernel: [2328494.651741] FS: 00007f1e81e0f880(0000) GS:ffff881fff480000(0000) knlGS:0000000000000000<br />May 11 10:46:54 waas kernel: [2328494.735018] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />May 11 10:46:54 waas kernel: [2328494.777006] CR2: 00007f1e81439cf0 CR3: 0000001fd37e8000 CR4: 00000000001406e0<br />May 11 10:46:54 waas kernel: [2328494.859892] Stack:<br />May 11 10:46:54 waas kernel: [2328494.899193] ffff881fab61c000 ffff881f1a3eb000 000000000000001d ffff881fe9ee2940<br />May 11 10:46:54 waas kernel: [2328494.979408] 00000000016e41b0 ffff880413ce3e10 ffffffffc0115559 ffff881f1a3eb000<br />May 11 10:46:54 waas kernel: [2328495.058058] ffff880413ce3e28 ffffffffc06509f1 ffff881f1a3eb000 ffff880413ce3e40<br />May 11 10:46:54 waas kernel: [2328495.138089] Call Trace:<br />May 11 10:46:54 waas kernel: [2328495.176902] [<ffffffffc0115559>] __fscache_relinquish_cookie+0x39/0x140 [fscache]<br />May 11 10:46:54 waas kernel: [2328495.256600] [<ffffffffc06509f1>] ceph_fscache_unregister_fs+0x31/0x40 [ceph]<br />May 11 10:46:54 waas kernel: [2328495.335208] [<ffffffffc062a06e>] destroy_fs_client+0x1e/0xb0 [ceph]<br />May 11 10:46:54 waas kernel: [2328495.376211] [<ffffffffc062a147>] ceph_kill_sb+0x47/0x70 [ceph]<br />May 11 10:46:54 waas kernel: [2328495.415226] [<ffffffff811ff853>] deactivate_locked_super+0x43/0x70<br />May 11 10:46:54 waas kernel: [2328495.455126] [<ffffffff811ff8c6>] deactivate_super+0x46/0x60<br />May 11 10:46:54 waas kernel: [2328495.492709] [<ffffffff8121b93f>] cleanup_mnt+0x3f/0x80<br />May 11 10:46:54 waas kernel: [2328495.532423] [<ffffffff8121b9d2>] __cleanup_mnt+0x12/0x20<br />May 11 10:46:54 waas kernel: [2328495.569888] [<ffffffff81099ec7>] task_work_run+0x77/0x90<br />May 11 10:46:54 waas kernel: [2328495.606028] [<ffffffff81078806>] exit_to_usermode_loop+0x73/0xa2<br />May 11 10:46:54 waas kernel: [2328495.641044] [<ffffffff81003a5e>] syscall_return_slowpath+0x4e/0x60<br />May 11 10:46:54 waas kernel: [2328495.676108] [<ffffffff817eea58>] int_ret_from_sys_call+0x25/0x8f<br />May 11 10:46:54 waas kernel: [2328495.708901] Code: 0f ba 74 24 48 03 72 07 31 d2 e9 66 ff ff ff ba 01 00 00 00 e9 5c ff ff ff 48 8b 77 20 31 c0 48 c7 c7 18 b1 11 c0 e8 9f b8 06 c1 <0f> 0b 48 c7 c7 88 c2 11 c0 31 c0 e8 8f b8 06 c1 48 c7 c7 96 c2 <br />May 11 10:46:54 waas kernel: [2328495.812838] RIP [<ffffffffc0114d60>] __fscache_disable_cookie+0x1a0/0x1e0 [fscache]<br />May 11 10:46:54 waas kernel: [2328495.878970] RSP <ffff880413ce3dd0><br />May 11 10:46:54 waas kernel: [2328495.957741] ---[ end trace d46a6ec086d7f531 ]---</p>
<p>The mount point was used to backup files from CephFS on tape, and held several millions caps according to the MDS session status.</p>
<p>2. Re-mount /srv/ceph<br />Kernel log:<br />May 11 10:50:33 waas kernel: [2328714.423366] libceph: client2735548 fsid 49098879-85ac-4c5d-aac0-e1a2658a680b<br />May 11 10:50:33 waas kernel: [2328714.494295] libceph: mon1 192.168.6.133:6789 session established</p>
<p>After remounting the mount point was idle, with just a dozend capabilities held.</p>
<p>3. Two hours later: complete server freeze, soft lockup messages etc. Had to restart the machine.<br />Kernel log:<br />May 11 12:54:08 waas kernel: [2336127.416016] BUG: unable to handle kernel NULL pointer dereference at (null)<br />May 11 12:54:08 waas kernel: [2336127.475968] IP: [<ffffffffc064dafd>] ceph_mdsc_handle_map+0x6d/0x650 [ceph]<br />May 11 12:54:08 waas kernel: [2336127.506619] PGD 0 <br />May 11 12:54:08 waas kernel: [2336127.537923] Oops: 0000 [<a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:2178 (Closed)" href="https://tracker.ceph.com/issues/2">#2</a>] SMP <br />May 11 12:54:08 waas kernel: [2336127.568024] Modules linked in: rbd ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs ceph libceph cts 8021q garp mrp stp llc openvswitch nf_defrag_ipv6 nf_conntrack libcrc32c ipmi_ssif intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul joydev aesni_intel rpcsec_gss_krb5 nfsd hpilo auth_rpcgss nfs_acl input_leds serio_raw nfs shpchp sb_edac lockd grace 8250_fintek edac_core mac_hid ioatdma sunrpc acpi_power_meter wmi ipmi_si aes_x86_64 lpc_ich lp lrw gf128mul glue_helper ipmi_msghandler fscache ablk_helper cryptd parport btrfs xor hid_generic usbhid hid raid6_pq ixgbe dca psmouse vxlan tg3 ip6_udp_tunnel udp_tunnel ptp hpsa pps_core scsi_transport_sas mdio fjes<br />May 11 12:54:08 waas kernel: [2336127.898223] CPU: 3 PID: 9696 Comm: kworker/3:0 Tainted: G D 4.4.0-18-generic #34~14.04.1-Ubuntu<br />May 11 12:54:08 waas kernel: [2336127.971855] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015<br />May 11 12:54:08 waas kernel: [2336128.009882] Workqueue: ceph-msgr ceph_con_workfn [libceph]<br />May 11 12:54:08 waas kernel: [2336128.047523] task: ffff880103d644c0 ti: ffff880243a28000 task.ti: ffff880243a28000<br />May 11 12:54:08 waas kernel: [2336128.120833] RIP: 0010:[<ffffffffc064dafd>] [<ffffffffc064dafd>] ceph_mdsc_handle_map+0x6d/0x650 [ceph]<br />May 11 12:54:08 waas kernel: [2336128.197396] RSP: 0018:ffff880243a2bc18 EFLAGS: 00010282<br />May 11 12:54:08 waas kernel: [2336128.235299] RAX: 5d4cac8579880949 RBX: ffff881a55591e94 RCX: ffff881a55591c10<br />May 11 12:54:08 waas kernel: [2336128.311435] RDX: 0b688a65a2e1c0aa RSI: ffff880243a2bc70 RDI: 0000000000000000<br />May 11 12:54:08 waas kernel: [2336128.388958] RBP: ffff880243a2bcc0 R08: 000000000001a000 R09: ffffffff816d626a<br />May 11 12:54:08 waas kernel: [2336128.467017] R10: ffff88077dbde900 R11: ffffea00049ab200 R12: 0000000000000015<br />May 11 12:54:08 waas kernel: [2336128.546565] R13: ffff881eedcfe440 R14: 0000000000000000 R15: 0000000000000001<br />May 11 12:54:08 waas kernel: [2336128.625745] FS: 0000000000000000(0000) GS:ffff881fff2c0000(0000) knlGS:0000000000000000<br />May 11 12:54:08 waas kernel: [2336128.708279] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033<br />May 11 12:54:08 waas kernel: [2336128.749349] CR2: 0000000000000000 CR3: 0000000001c0c000 CR4: 00000000001406e0<br />May 11 12:54:08 waas kernel: [2336128.828880] Stack:<br />May 11 12:54:08 waas kernel: [2336128.868506] 0000000000000000 0000000000000000 0000000000004040 0000000000000000<br />May 11 12:54:08 waas kernel: [2336128.945246] ffff880243a2bc70 ffffffffc05ddff0 ffff880126ac8800 ffff881eedcfe2d8<br />May 11 12:54:08 waas kernel: [2336129.022320] 0000000000000001 ffff881eedcfe2b8 ffff881a55591c10 5d4cac8579880949<br />May 11 12:54:08 waas kernel: [2336129.100669] Call Trace:<br />May 11 12:54:08 waas kernel: [2336129.138602] [<ffffffffc05ddff0>] ? read_partial.isra.25+0x50/0x70 [libceph]<br />May 11 12:54:08 waas kernel: [2336129.215472] [<ffffffffc05e0209>] ? read_partial_message+0x419/0x890 [libceph]<br />May 11 12:54:08 waas kernel: [2336129.293238] [<ffffffffc062a73d>] extra_mon_dispatch+0x1d/0x30 [ceph]<br />May 11 12:54:08 waas kernel: [2336129.332335] [<ffffffffc05e45ca>] dispatch+0x30a/0x720 [libceph]<br />May 11 12:54:08 waas kernel: [2336129.370846] [<ffffffffc05e0ac3>] try_read+0x443/0x1120 [libceph]<br />May 11 12:54:08 waas kernel: [2336129.410031] [<ffffffff81096510>] ? try_to_grab_pending+0xb0/0x150<br />May 11 12:54:08 waas kernel: [2336129.448015] [<ffffffff810b3175>] ? put_prev_entity+0x35/0x670<br />May 11 12:54:08 waas kernel: [2336129.485085] [<ffffffff8102c696>] ? __switch_to+0x1d6/0x550<br />May 11 12:54:08 waas kernel: [2336129.521170] [<ffffffffc05e1852>] ceph_con_workfn+0xb2/0x5d0 [libceph]<br />May 11 12:54:08 waas kernel: [2336129.556734] [<ffffffff8109598d>] process_one_work+0x14d/0x3f0<br />May 11 12:54:08 waas kernel: [2336129.591955] [<ffffffff8109610a>] worker_thread+0x11a/0x470<br />May 11 12:54:08 waas kernel: [2336129.626262] [<ffffffff817eaa49>] ? __schedule+0x359/0x970<br />May 11 12:54:08 waas kernel: [2336129.659578] [<ffffffff81095ff0>] ? rescuer_thread+0x310/0x310<br />May 11 12:54:08 waas kernel: [2336129.692627] [<ffffffff8109b842>] kthread+0xd2/0xf0<br />May 11 12:54:08 waas kernel: [2336129.725096] [<ffffffff8109b770>] ? kthread_park+0x50/0x50<br />May 11 12:54:08 waas kernel: [2336129.756811] [<ffffffff817eec8f>] ret_from_fork+0x3f/0x70<br />May 11 12:54:08 waas kernel: [2336129.787102] [<ffffffff8109b770>] ? kthread_park+0x50/0x50<br />May 11 12:54:08 waas kernel: [2336129.817415] Code: d8 48 29 c8 48 83 f8 17 0f 86 b0 00 00 00 48 8b 01 48 8b 51 08 49 89 fe 48 8d 74 24 58 48 83 c1 10 48 89 4c 24 50 48 89 44 24 58 <48> 8b 07 48 89 54 24 60 48 8b 78 10 e8 f2 d6 f8 ff 85 c0 0f 88 <br />May 11 12:54:08 waas kernel: [2336129.910566] RIP [<ffffffffc064dafd>] ceph_mdsc_handle_map+0x6d/0x650 [ceph]<br />May 11 12:54:08 waas kernel: [2336129.973138] RSP <ffff880243a2bc18><br />May 11 12:54:08 waas kernel: [2336130.002901] CR2: 0000000000000000<br />May 11 12:54:08 waas kernel: [2336130.077904] ---[ end trace d46a6ec086d7f532 ]---</p>
<p>The only messages in the mon log at the time is a dump of the mds map:</p>
<p>2016-05-11 12:54:05.659616 7f97792f1700 0 mon.ceph-storage-03@0(leader).mds e187276 print_map<br />epoch 187276<br />flags 0<br />created 2014-10-22 19:19:45.462330<br />modified 2016-05-11 12:54:05.648986<br />tableserver 0<br />root 0<br />session_timeout 60<br />session_autoclose 300<br />max_file_size 1099511627776<br />last_failure 186368<br />last_failure_osd_epoch 352275<br />compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table}<br />max_mds 1<br />in 0<br />up {0=2718471}<br />failed <br />stopped <br />data_pools 7,12<br />metadata_pool 8<br />inline_data disabled<br />2718471: 192.168.6.129:6801/12700 'ceph-storage-01' mds.0.111 up:active seq 253459</p>
Linux kernel client - Bug #15780 (Resolved): Applications using kernel based cephfs and mmap fail...
https://tracker.ceph.com/issues/15780
2016-05-09T13:38:58Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>OS/Kernel: Ubuntu Trusty with Xenial LTS backport kernel 4.4.0-21-generic<br />Ceph: 0.94.6</p>
<p>The PTRACE_ATTACH syscall used by strace and gdb may interrupt the cephfs mmap handler, resulting in a SIGBUS for the application.</p>
<p>See e.g. <a class="external" href="https://github.com/torvalds/linux/blob/master/fs/ceph/addr.c#L1342">https://github.com/torvalds/linux/blob/master/fs/ceph/addr.c#L1342</a></p>
<p>Steps to reproduce:</p>
<p>1. have a rather slow ceph cluster (higher chance to hit the mmap handler)<br />2. flush all caches (echo 3 > /proc/sys/vm/drop_caches)<br />3. run an application making heavy used of mmap'ed memory<br />4. try to attach a debugger or strace with the right timing</p>
<p>Expected result:<br />- debugger prompt or strace output</p>
<p>Got result:<br />application:<br />Bus error (core dumped)</p>
<p>strace:<br />~# strace <del>p 60724<br />Process 60724 attached<br />--</del> SIGBUS {si_signo=SIGBUS, si_code=BUS_ADRERR, si_addr=0x7fbda9063c1c} ---<br />+<ins>+ killed by SIGBUS (core dumped) +</ins>+</p>
<hr />
<p>The mmap handler in the code referred to above returns SIGBUS on several occasions. It is unclear whether the SIGBUS originates from line 1342, 1366 or 1382. Is it possible to activate debug output implemented with dout(..) on a running kernel, which may allow to rule out the first occurrence?</p>
CephFS - Bug #13926 (Closed): lockup in multithreaded application
https://tracker.ceph.com/issues/13926
2015-12-01T09:56:46Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>A multithreaded applications end up in a blocked state when multiple threads try to access the same file.</p>
<ol>
<li>apt-cache policy ceph-fuse<br />ceph-fuse:<br /> Installed: 0.94.5-1trusty</li>
</ol>
<p>ceph.conf:</p>
<p>[global]<br />osd_pool_default_pgp_num = 512<br />osd_pool_default_min_size = 2<br />auth_service_required = cephx<br />mon_initial_members = <one monitor><br />fsid = <fs id><br />cluster_network = <network><br />auth_supported = cephx<br />auth_cluster_required = cephx<br />mon_host = <monitor hosts><br />auth_client_required = cephx<br />osd_pool_default_size = 3<br />osd_pool_default_pg_num = 512<br />public_network = <network><br />#fuse_use_invalidate_cb = True<br />debug_client=20/20</p>
<p>The locked up process is consuming 100% CPU in system call at that time (40 CPU cores):</p>
<p>top - 10:38:26 up 7 min, 1 user, load average: 0.99, 0.64, 0.30<br />Tasks: 40 total, 2 running, 38 sleeping, 0 stopped, 0 zombie<br />%Cpu(s): 0.1 us, 2.5 sy, 0.0 ni, 97.4 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st<br />KiB Mem: 26411547+total, 2673076 used, 26144240+free, 43872 buffers<br />KiB Swap: 26855424+total, 0 used, 26855424+free. 687924 cached Mem</p>
<pre><code>PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br /> 4039 blinke 20 0 16.198g 38060 2152 R 100.1 0.0 2:41.25 hammer</code></pre>
<p>Trying to terminate the process (e.g. CTRL-C) kills the worker thread, but the main thread keeps running. Accessing the list of file handles associated with the process (/proc/$PID/fd) also blocks.</p>
<p>Debug output is available with ceph-post-file id a8eb75d5-cc13-430a-bed8-428c8a33d6d8</p>
CephFS - Bug #13640 (Resolved): CephFS and page cache handling
https://tracker.ceph.com/issues/13640
2015-10-29T10:03:16Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>CephFS (both ceph-fuse and kernel client) flush the content of files from page cache upon closing the file.</p>
<p>This behaviour is undesirable in case the same file has to be read by a subsequent process, which might be the case in certain<br />HPC workloads.</p>
<p>A patch was supplied that disabled the cache invalidation:</p>
<p><a class="external" href="https://github.com/ceph/ceph/pull/6380">https://github.com/ceph/ceph/pull/6380</a><br /><a class="external" href="https://github.com/ceph/ceph-client/commit/dfbb503e4e12580fc3d2952269104f293b0ec7e8">https://github.com/ceph/ceph-client/commit/dfbb503e4e12580fc3d2952269104f293b0ec7e8</a></p>
<p>fuse needs to be instructed to delegate cache invalidation to ceph-fuse using "fuse use invalidate cb" configuration setting<br />(default fuse behaviour is dropping content from cache upon open() call).</p>
<p>Some questions remain whether this patch might lead to dead locks:</p>
<p>"...as long as nobody blocks waiting for invalidate while holding a lock <br />(client_lock?) that could prevent other fuse ops like write (pretty sure <br />that was the deadlock we saw before). I worry this could still happen <br />with a writer (or reader?) getting stuck in a check_caps() type situation <br />while the invalidate cb is waiting on a page lock held by the calling <br />kernel syscall..."</p>
<p>Although simple tests on a single host with single interactions works well with the patch,<br />the correct behaviour in the mentioned scenario should be check prior to use this patch in<br />production environments.</p>
CephFS - Bug #12617 (Resolved): Missing handling of supplementary groups in ceph-fuse -> renderin...
https://tracker.ceph.com/issues/12617
2015-08-05T08:06:04Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>During investigating a user's bug report I've stumbled accross</p>
<p><a class="external" href="https://github.com/ceph/ceph/blob/master/src/client/fuse_ll.cc#L981">https://github.com/ceph/ceph/blob/master/src/client/fuse_ll.cc#L981</a></p>
<p>ceph-fuse does not handle supplementary user groups AT ALL.</p>
<p>This is a SHOW STOPPER for everyone trying to use ceph-fuse in a multi user shared environment.</p>
<p>Since the callback is only used in Client::check_permissions (<a class="external" href="https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L4546">https://github.com/ceph/ceph/blob/master/src/client/Client.cc#L4546</a>), the fix would be trivial:</p>
<p>Release the memory allocated for the group list in the callback.</p>
CephFS - Backport #12350 (Resolved): Provided logrotate setup does not handle ceph-fuse correctly
https://tracker.ceph.com/issues/12350
2015-07-16T08:59:51Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p><a class="external" href="https://github.com/ceph/ceph/pull/7110">https://github.com/ceph/ceph/pull/7110</a></p>
CephFS - Bug #12297 (Resolved): ceph-fuse 0.94.2-1trusty segfaults / aborts
https://tracker.ceph.com/issues/12297
2015-07-13T10:33:43Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>Running CephFS for data volumes on a compute cluster.</p>
<p>ceph-fuse aborts and leaves the mount point unusable (transport endpoint not connected).</p>
<p>Except of client log incl. 20 latest ops:<br /><pre>
-20> 2015-07-13 12:08:13.905733 7f60be7fc700 20 client.412413 trim_caps counting as trimmed: 100015d4d13.head(ref=2 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.875809 caps=pAsLsXsFscr(0=pAsLsXsFscr) 0x7f60ac8c6320)
-19> 2015-07-13 12:08:13.905744 7f60be7fc700 10 client.412413 put_inode on 100015d4d13.head(ref=2 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.875809 caps=pAsLsXsFscr(0=pAsLsXsFscr) 0x7f60ac8c6320)
-18> 2015-07-13 12:08:13.905755 7f60be7fc700 20 client.412413 trying to trim dentries for 100015d4d14.head(ref=2 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.893082 caps=pAsLsXsFscr(0=pAsLsXsFscr) parents=0x7f5fb43408e0 0x7f60acc44830)
-17> 2015-07-13 12:08:13.905767 7f60be7fc700 15 client.412413 trim_dentry unlinking dn raw_reads.136.raw_reads.296.N2.las in dir 100012fa7e7
-16> 2015-07-13 12:08:13.905770 7f60be7fc700 15 client.412413 unlink dir 0x7f60ac59d5a0 'raw_reads.136.raw_reads.296.N2.las' dn 0x7f5fb43408e0 inode 0x7f60acc44830
-15> 2015-07-13 12:08:13.905773 7f60be7fc700 20 client.412413 unlink inode 0x7f60acc44830 parents now
-14> 2015-07-13 12:08:13.905774 7f60be7fc700 10 client.412413 put_inode on 100015d4d14.head(ref=3 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.893082 caps=pAsLsXsFscr(0=pAsLsXsFscr) 0x7f60acc44830)
-13> 2015-07-13 12:08:13.905785 7f60be7fc700 15 client.412413 unlink removing 'raw_reads.136.raw_reads.296.N2.las' dn 0x7f5fb43408e0
-12> 2015-07-13 12:08:13.905789 7f60be7fc700 20 client.412413 trim_caps counting as trimmed: 100015d4d14.head(ref=2 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.893082 caps=pAsLsXsFscr(0=pAsLsXsFscr) 0x7f60acc44830)
-11> 2015-07-13 12:08:13.905801 7f60be7fc700 10 client.412413 put_inode on 100015d4d14.head(ref=2 ll_ref=1 cap_refs={} open={} mode=120777 size=74/0 mtime=2015-07-13 12:08:05.893082 caps=pAsLsXsFscr(0=pAsLsXsFscr) 0x7f60acc44830)
-10> 2015-07-13 12:08:13.905840 7f60a3fff700 10 client.412413 _async_dentry_invalidate 'ch' ino 0 in dir 100009de0f2.head
-9> 2015-07-13 12:08:13.905849 7f606bfff700 15 client.412413 de raw_reads.124.raw_reads.296.C2.las off 6170202972637346 = 0
-8> 2015-07-13 12:08:13.906701 7f60a06f7700 2 -- 192.168.2.22:0/20333 >> 192.168.6.5:6810/4247 pipe(0x7f607c030cf0 sd=2 :37085 s=2 pgs=28199 cs=1 l=1 c=0x7f607c034f90).reader couldn't read tag, (11) Resource temporarily unavailable
-7> 2015-07-13 12:08:13.906738 7f60a06f7700 2 -- 192.168.2.22:0/20333 >> 192.168.6.5:6810/4247 pipe(0x7f607c030cf0 sd=2 :37085 s=2 pgs=28199 cs=1 l=1 c=0x7f607c034f90).fault (11) Resource temporarily unavailable
-6> 2015-07-13 12:08:13.906808 7f60be7fc700 1 client.412413.objecter ms_handle_reset on osd.61
-5> 2015-07-13 12:08:13.906818 7f60be7fc700 1 -- 192.168.2.22:0/20333 mark_down 0x7f607c034f90 -- pipe dne
-4> 2015-07-13 12:08:13.907127 7f60be7fc700 10 monclient: renew_subs
-3> 2015-07-13 12:08:13.907135 7f60be7fc700 10 monclient: _send_mon_message to mon.mon-i1 at 192.168.6.50:6789/0
-2> 2015-07-13 12:08:13.907141 7f60be7fc700 1 -- 192.168.2.22:0/20333 --> 192.168.6.50:6789/0 -- mon_subscribe({mdsmap=4312+,monmap=14+,osdmap=175910}) v2 -- ?+0 0x7f60ac68f8b0 con 0x3b50160
-1> 2015-07-13 12:08:13.907279 7f606bfff700 -1 *** Caught signal (Segmentation fault) **
in thread 7f606bfff700
ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
1: ceph-fuse() [0x6235ca]
2: (()+0x10340) [0x7f60c9dd8340]
3: (std::string::assign(std::string const&)+0x1c) [0x7f60c8ff748c]
4: (Client::_readdir_cache_cb(dir_result_t*, int (*)(void*, dirent*, stat*, int, long), void*)+0x39a) [0x55263a]
5: (Client::readdir_r_cb(dir_result_t*, int (*)(void*, dirent*, stat*, int, long), void*)+0xfc5) [0x596045]
6: ceph-fuse() [0x546d2d]
7: (()+0x13e76) [0x7f60ca20fe76]
8: (()+0x1522b) [0x7f60ca21122b]
9: (()+0x11e49) [0x7f60ca20de49]
10: (()+0x8182) [0x7f60c9dd0182]
11: (clone()+0x6d) [0x7f60c875547d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2015-07-13 12:08:13.908290 7f60be7fc700 1 -- 192.168.2.22:0/20333 <== mon.3 192.168.6.50:6789/0 266 ==== osd_map(175910..175910 src has 172194..175910) v3 ==== 222+0+0 (8575416 0 0) 0x7f60b00008c0 con 0x3b50160
</pre></p>
<p>The OSD mentioned in OP 8 is up and running, but might take some time to respond due to current backfilling operations. The compute jobs are setup to write output to the same file, so several cephfs clients are trying to open a single file for writing.</p>
CephFS - Bug #12189 (Won't Fix): Editing / Creating files fails for NFS-over-CephFS on EC pool wi...
https://tracker.ceph.com/issues/12189
2015-06-30T14:33:20Z
Burkhard Linke
Burkhard.Linke@computational.bio.uni-giessen.de
<p>Ubuntu 14.04, Kernel 3.13.0-55-generic<br />Standard kernel-based NFS server<br />Ceph Hammer release <br />~# ceph version<br />ceph version 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)</p>
<p>Mount point for cephfs via fstab:<br />client_mountpoint=/volumes /srv/ceph fuse.ceph defaults,_netdev 0 0</p>
<p>~# mount | grep ceph<br />ceph-fuse on /srv/ceph type fuse.ceph-fuse (rw,nosuid,nodev,allow_other,default_permissions)</p>
Mount point is exported to kerberized desktop machines in another network:<br />~# cat /etc/exports | grep ceph
<ol>
<li>export ceph to desktop machines only<br />/srv/ceph XX.XX.XX.XX/<abbr title="fsid=2,sec=krb5p,rw,no_subtree_check">YY</abbr></li>
</ol>
<p>On client machine export CephFS is mounted as /ceph:<br />~$ mount | grep ceph<br />nfs-homes:/srv/ceph on /ceph type nfs (rw,noatime,fsc,nfsvers=4,sec=krb5p,intr,ac,sloppy,addr=XX.XX.XX.XX,clientaddr=XX.XX.XX.XX)</p>
<p>CephFS uses three data pools:<br />~# ceph fs ls<br />name: cephfs, metadata pool: cephfs_test_metadata, data pools: [cephfs_test_data cephfs_two_rep_data ec_ssd_cache ]</p>
Two pools are replicated pools, one pool is cache tier of an erasure coding pool. The default pool is the EC pool (resp. its cache tier):<br />~# getfattr -n ceph.dir.layout /srv/ceph/ <br />getfattr: Removing leading '/' from absolute path names
<ol>
<li>file: srv/ceph/<br />ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=ec_ssd_cache"</li>
</ol>
<p><sub># getfattr -n ceph.dir.layout /srv/ceph/adm/temp/test/<br />/srv/ceph/adm/temp/test/: ceph.dir.layout: No such attribute<br /></sub># getfattr -n ceph.dir.layout /srv/ceph/adm/temp/ <br />/srv/ceph/adm/temp/: ceph.dir.layout: No such attribute<br />~# getfattr -n ceph.dir.layout /srv/ceph/adm/<br />/srv/ceph/adm/: ceph.dir.layout: No such attribute</p>
<p>Creating files and/or editing them on the server works:</p>
<p>:/srv/ceph/adm/temp/test$ dd if=/dev/zero of=test bs=1 count=10<br />10+0 records in<br />10+0 records out<br />10 bytes (10 B) copied, 0.0046936 s, 2.1 kB/s</p>
<p>Creating a file in the same way on a NFS client also works:</p>
<p>:/ceph/adm/temp/test$ dd if=/dev/zero of=test2 bs=1 count=10 <br />10+0 records in<br />10+0 records out<br />10 bytes (10 B) copied, 0.0332668 s, 0.3 kB/s</p>
<p><em>Editing</em> the file on the server is also ok:</p>
<p>:/srv/ceph/adm/temp/test$ vi test<br />blinke@waas:/srv/ceph/adm/temp/test$ ls <del>al<br />total 2<br />drwxr-xr-x 1 blinke cb 20 Jun 30 16:25 .<br />drwxrwxrwt 1 blinke support 20733315 Jun 30 16:11 ..<br />-rw-r--r-</del> 1 blinke cb 10 Jun 30 16:23 test<br /><del>rw-r--r-</del> 1 blinke cb 10 Jun 30 16:24 test2</p>
<p><em>Editing</em> the file on the NFS client FAILS:</p>
<p>:/ceph/adm/temp/test$ vi test2</p>
<p>E325: ATTENTION<br />Found a swap file by the name ".test2.swp" <br /> owned by: blinke dated: Mon Dec 11 03:21:47 1972<br /> [cannot be opened]<br />While opening file "test2" <br /> dated: Tue Jun 30 16:24:29 2015<br /> NEWER than swap file!</p>
<p>(1) Another program may be editing the same file. If this is the case,<br /> be careful not to end up with two different instances of the same<br /> file when making changes. Quit, or continue with caution.<br />(2) An edit session for this file crashed.<br /> If this is the case, use ":recover" or "vim -r test2" <br /> to recover the changes (see ":help recovery").<br /> If you did this already, delete the swap file ".test2.swp" <br /> to avoid this message.</p>
<p>Swap file ".test2.swp" already exists!<br />[O]pen Read-Only, (E)dit anyway, (R)ecover, (D)elete it, (Q)uit, (A)bort: -> q</p>
<p>:/ceph/adm/temp/test$ vi test2<br />blinke@fb08-bcf-pc01:/ceph/adm/temp/test$ ls <del>al<br />total 2<br />drwxr-xr-x 1 blinke cb 20 Jun 30 16:26 .<br />drwxrwxrwt 1 blinke support 20733315 Jun 30 16:11 ..<br />-rw-r--r-</del> 1 blinke cb 10 Jun 30 16:23 test<br /><del>rw-r--r-</del> 1 blinke cb 10 Jun 30 16:24 test2<br />---------- 1 blinke cb 0 Dec 11 1972 .test2.swo<br />---------- 1 blinke cb 0 Dec 11 1972 .test2.swp</p>
<p>Editing a files stored on a replicated pool is OK</p>
<p>Copying a file to the directory on the server is OK:<br />:/srv/ceph/adm/temp/test$ cp ~/wf1.out .<br />:/srv/ceph/adm/temp/test$ rm wf1.out</p>
<p>Copying a file to the directory on the client FAILS:<br />:/ceph/adm/temp/test$ cp ~/wf1.out .<br />cp: cannot create regular file './wf1.out': Permission denied<br />:/ceph/adm/temp/test$ ls <del>al<br />total 2<br />drwxr-xr-x 1 blinke cb 20 Jun 30 16:30 .<br />drwxrwxrwt 1 blinke support 20728068 Jun 30 16:11 ..<br />-rw-r--r-</del> 1 blinke cb 10 Jun 30 16:23 test<br /><del>rw-r--r-</del> 1 blinke cb 10 Jun 30 16:24 test2<br />---------- 1 blinke cb 0 Dec 11 1972 .test2.swo<br />---------- 1 blinke cb 0 Dec 11 1972 .test2.swp<br />---------- 1 blinke cb 0 Dec 11 1972 wf1.out</p>
<p>It is created as empty file....</p>