Ceph » Linux kernel client

While running xfstests vs. a ceph vstart cluster, I hitting softlockups in test generic/095. Looks like a we're cycling over and over again on a splice write? Unfortunately, the VM was completely frozen so I couldn't get more than the stack traces from the soft lockups. The kernel is the current ceph-client testing branch (commit b818a11fe + some d_revalidate changes, which I don't think would be related here):

[ 7579.947190] run fstests generic/094 at 2016-12-02 13:08:11
[ 7580.237678] run fstests generic/095 at 2016-12-02 13:08:11
[ 7581.157345] libceph: mon0 192.168.1.3:6789 session established
[ 7581.159992] libceph: client4249 fsid ad1604e3-173c-4298-89d6-a5ed61dba811
[ 7582.529699] libceph: mon0 192.168.1.3:6789 session established
[ 7582.532220] libceph: client4250 fsid ad1604e3-173c-4298-89d6-a5ed61dba811
[ 7608.369471] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7608.370481] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7608.380623] CPU: 0 PID: 16914 Comm: fio Tainted: G           OE   4.9.0-rc6+ #24
[ 7608.381550] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7608.382606] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7608.383388] RIP: 0010:[<ffffffffac3f7ba6>]  [<ffffffffac3f7ba6>] iov_iter_advance+0x36/0x380
[ 7608.384445] RSP: 0018:ffff9b3e88a1fa50  EFLAGS: 00000202
[ 7608.385161] RAX: 0000000000001000 RBX: 0000000000000000 RCX: ffff9b3e88a1fbb0
[ 7608.386062] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7608.386961] RBP: ffff9b3e88a1faa0 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7608.387866] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000000
[ 7608.388795] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7608.389707] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7608.390714] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7608.391482] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7608.392399] Stack:
[ 7608.392792]  ffff9b3e88a1fa60 0000000000000018 0000000000000002 0000000000000000
[ 7608.393821]  00000000137a28c7 0000000000000000 0000000000000002 ffff8b23f5687de0
[ 7608.394834]  0000000000000000 0000000000000001 ffff9b3e88a1fc10 ffffffffc09590ea
[ 7608.395868] Call Trace:
[ 7608.396341]  [<ffffffffc09590ea>] ceph_direct_read_write+0x36a/0xc60 [ceph]
[ 7608.397251]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7608.398104]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7608.398944]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7608.399707]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7608.400554]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7608.401291]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7608.402121]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7608.402979] Code: 41 55 41 54 53 48 83 ec 28 8b 17 f6 c2 08 0f 85 21 01 00 00 48 8b 47 10 48 85 c0 74 78 48 39 c6 4c 8b 67 08 48 0f 47 f0 f6 c2 04 <49> 89 f0 75 74 83 e2 02 48 8b 77 18 0f 85 8d 02 00 00 4c 8b 4e 
[ 7636.370045] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7636.371116] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7636.381824] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7636.382855] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7636.384024] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7636.384883] RIP: 0010:[<ffffffffac3f7c1d>]  [<ffffffffac3f7c1d>] iov_iter_advance+0xad/0x380
[ 7636.386016] RSP: 0018:ffff9b3e88a1faa0  EFLAGS: 00000286
[ 7636.386803] RAX: 0000000000001000 RBX: 0000000000000000 RCX: ffff9b3e88a1fbb0
[ 7636.387782] RDX: 0000000000000005 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7636.388766] RBP: ffff9b3e88a1faa0 R08: 0000000000000000 R09: 0000000000000000
[ 7636.389750] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7636.390728] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7636.391693] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7636.392750] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7636.393576] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7636.394541] Stack:
[ 7636.394973]  ffff9b3e88a1fc10 ffffffffc09590ea ffff8b2400000000 ffff8b2400000002
[ 7636.396018]  ffff8b2400002201 ffff8b2400000064 ffff8b24704bdaa0 ffffffff00000001
[ 7636.397069]  ffffffffffffffff ffff9b3e00000000 ffff8b246f8d6400 ffff9b3e88a1fd30
[ 7636.398114] Call Trace:
[ 7636.398611]  [<ffffffffc09590ea>] ceph_direct_read_write+0x36a/0xc60 [ceph]
[ 7636.399556]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7636.400456]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7636.401335]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7636.402130]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7636.403007]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7636.403778]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7636.404640]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7636.405523] Code: 00 48 89 fa 48 89 79 18 48 29 f2 48 c1 fa 04 48 29 51 20 4c 29 c0 4c 89 61 08 48 89 41 10 48 83 c4 28 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 4c 8b 57 18 41 89 f5 45 31 c9 45 31 f6 49 c7 c3 80 5c c7 
[ 7643.185184] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 7643.186125]     (detected by 0, t=60002 jiffies, g=647181, c=647180, q=0)
[ 7643.187046] All QSes seen, last rcu_sched kthread activity 60002 (4302310160-4302250158), jiffies_till_next_fqs=3, root ->qsmask 0x0
[ 7643.188750] fio             R  running task        0 16914  16897 0x00000008
[ 7643.189755]  ffff8b247fc03e40 ffffffffac0cf783 ffff8b247fc1a400 fffffffface4cb00
[ 7643.190816]  ffff8b247fc03ea8 ffffffffac10c6b0 0000000000000000 ffff8b247960d880
[ 7643.191878]  ffff8b247fc1a400 ffffffff00000000 0000000000000000 ffff8b247960d880
[ 7643.192954] Call Trace:
[ 7643.193454]  <IRQ> [ 7643.193674]  [<ffffffffac0cf783>] sched_show_task+0xd3/0x140
[ 7643.194507]  [<ffffffffac10c6b0>] rcu_check_callbacks+0x8c0/0x8d0
[ 7643.195368]  [<ffffffffac121d80>] ? tick_sched_do_timer+0x50/0x50
[ 7643.196241]  [<ffffffffac111b8f>] update_process_times+0x2f/0x60
[ 7643.197091]  [<ffffffffac1216d5>] tick_sched_handle.isra.17+0x25/0x60
[ 7643.197983]  [<ffffffffac121dbd>] tick_sched_timer+0x3d/0x70
[ 7643.198793]  [<ffffffffac11260e>] __hrtimer_run_queues+0xee/0x260
[ 7643.199631]  [<ffffffffac112d9a>] hrtimer_interrupt+0x9a/0x180
[ 7643.200442]  [<ffffffffac04b978>] local_apic_timer_interrupt+0x38/0x60
[ 7643.201324]  [<ffffffffac80c72d>] smp_apic_timer_interrupt+0x3d/0x50
[ 7643.202182]  [<ffffffffac80b8ec>] apic_timer_interrupt+0x8c/0xa0
[ 7643.203011]  <EOI> [ 7643.203238]  [<ffffffffac3f7b79>] ? iov_iter_advance+0x9/0x380
[ 7643.204064]  [<ffffffffc09590ea>] ceph_direct_read_write+0x36a/0xc60 [ceph]
[ 7643.204979]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7643.205840]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7643.206700]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7643.207476]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7643.208337]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7643.209083]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7643.209903]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7643.210765] rcu_sched kthread starved for 60002 jiffies! g647181 c647180 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0
[ 7643.211990] rcu_sched       R  running task        0     7      2 0x00000000
[ 7643.212957]  ffff8b246ef2da40 0000000000000000 ffff8b247abf8000 ffff8b247fc19600
[ 7643.213981]  ffff8b23f5a3bb00 ffff9b3e8065fd78 ffffffffac804985 ffff8b247a400000
[ 7643.214993]  00ff8b247a400080 ffff8b247fc19600 ffff9b3e8065fdc0 ffff8b247abf8000
[ 7643.216012] Call Trace:
[ 7643.216467]  [<ffffffffac804985>] ? __schedule+0x225/0x6d0
[ 7643.217228]  [<ffffffffac804e66>] schedule+0x36/0x80
[ 7643.217932]  [<ffffffffac80823b>] schedule_timeout+0x1db/0x3f0
[ 7643.218726]  [<ffffffffac110040>] ? del_timer_sync+0x50/0x50
[ 7643.219511]  [<ffffffffac10a139>] rcu_gp_kthread+0x4d9/0x960
[ 7643.220285]  [<ffffffffac109c60>] ? _synchronize_rcu_expedited.constprop.65+0x320/0x320
[ 7643.221288]  [<ffffffffac0c1959>] kthread+0xd9/0xf0
[ 7643.221980]  [<ffffffffac0c1880>] ? kthread_park+0x60/0x60
[ 7643.222745]  [<ffffffffac809dd5>] ret_from_fork+0x25/0x30
[ 7668.370705] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7668.371756] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7668.382576] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7668.383554] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7668.384736] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7668.385609] RIP: 0010:[<ffffffffc09590e2>]  [<ffffffffc09590e2>] ceph_direct_read_write+0x362/0xc60 [ceph]
[ 7668.386939] RSP: 0018:ffff9b3e88a1fab0  EFLAGS: 00000246
[ 7668.387775] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[ 7668.388793] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7668.389816] RBP: ffff9b3e88a1fc10 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7668.390842] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7668.391875] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7668.392869] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7668.393976] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7668.394841] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7668.395861] Stack:
[ 7668.396331]  ffff8b2400000000 ffff8b2400000002 ffff8b2400002201 ffff8b2400000064
[ 7668.397375]  ffff8b24704bdaa0 ffffffff00000001 ffffffffffffffff ffff9b3e00000000
[ 7668.398491]  ffff8b246f8d6400 ffff9b3e88a1fd30 000000005841b88e 0000000032314610
[ 7668.399616] Call Trace:
[ 7668.400155]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7668.401094]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7668.402053]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7668.402916]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7668.403859]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7668.404687]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7668.405608]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7668.406534] Code: 85 c9 0f 8e 0d 02 00 00 45 31 ff 4c 8b ac 24 e8 00 00 00 4c 89 b4 24 a0 00 00 00 eb 36 4c 63 f0 48 8d bc 24 00 01 00 00 4c 89 f6 <4c> 29 f3 e8 86 ea a9 eb 48 8b 94 24 f8 00 00 00 49 8d 84 16 ff 
[ 7696.371279] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7696.372376] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7696.383272] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7696.384342] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7696.385520] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7696.386431] RIP: 0010:[<ffffffffac3fa562>]  [<ffffffffac3fa562>] iov_iter_get_pages+0x102/0x340
[ 7696.387666] RSP: 0018:ffff9b3e88a1fa60  EFLAGS: 00000246
[ 7696.388536] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[ 7696.389601] RDX: 0000000000000000 RSI: ffff8b23f5687de8 RDI: ffff9b3e88a1fbb0
[ 7696.390615] RBP: ffff9b3e88a1faa0 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7696.391585] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7696.392542] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7696.393528] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7696.394603] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7696.395461] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7696.396429] Stack:
[ 7696.396875]  ffff9b3e88a1fbb0 0000000000000000 00000000137a28c7 0000000000000000
[ 7696.397940]  0000000000000002 ffff8b23f5687de0 0000000000000000 0000000000000001
[ 7696.399053]  ffff9b3e88a1fc10 ffffffffc0959130 ffff8b2400000000 ffff8b2400000002
[ 7696.400184] Call Trace:
[ 7696.400699]  [<ffffffffc0959130>] ceph_direct_read_write+0x3b0/0xc60 [ceph]
[ 7696.401694]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7696.402610]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7696.403529]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7696.404356]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7696.405237]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7696.406002]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7696.406881]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7696.407789] Code: 0f 45 da 48 89 d8 49 2b 06 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 43 02 00 00 48 83 c4 18 5b 41 5c 41 5d 41 5e 41 5f 5d c3 <31> c0 eb da 85 d2 74 f8 49 8b 4c 24 18 8b 41 08 29 f0 39 d0 48 
[ 7724.371852] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7724.372945] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7724.383794] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7724.384850] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7724.386062] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7724.386943] RIP: 0010:[<ffffffffc09590fa>]  [<ffffffffc09590fa>] ceph_direct_read_write+0x37a/0xc60 [ceph]
[ 7724.388212] RSP: 0018:ffff9b3e88a1fab0  EFLAGS: 00000286
[ 7724.389019] RAX: 0000000000000fff RBX: 0000000000000000 RCX: ffff9b3e88a1fbb0
[ 7724.390003] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7724.390979] RBP: ffff9b3e88a1fc10 R08: 0000000000000000 R09: 0000000000000000
[ 7724.391951] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7724.392918] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7724.393892] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7724.394971] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7724.395800] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7724.396768] Stack:
[ 7724.397202]  ffff8b2400000000 ffff8b2400000002 ffff8b2400002201 ffff8b2400000064
[ 7724.398270]  ffff8b24704bdaa0 ffffffff00000001 ffffffffffffffff ffff9b3e00000000
[ 7724.399348]  ffff8b246f8d6400 ffff9b3e88a1fd30 000000005841b88e 0000000032314610
[ 7724.400402] Call Trace:
[ 7724.400908]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7724.401810]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7724.402715]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7724.403548]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7724.404428]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7724.405215]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7724.406087]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7724.406981] Code: 00 00 00 eb 36 4c 63 f0 48 8d bc 24 00 01 00 00 4c 89 f6 4c 29 f3 e8 86 ea a9 eb 48 8b 94 24 f8 00 00 00 49 8d 84 16 ff 0f 00 00 <48> c1 e8 0c 41 01 c7 45 39 fc 0f 8e ba 01 00 00 49 63 c7 44 89 
[ 7752.372427] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7752.373512] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7752.384304] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7752.385330] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7752.386526] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7752.387417] RIP: 0010:[<ffffffffc095911d>]  [<ffffffffc095911d>] ceph_direct_read_write+0x39d/0xc60 [ceph]
[ 7752.388725] RSP: 0018:ffff9b3e88a1fab0  EFLAGS: 00000202
[ 7752.389542] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000002
[ 7752.390530] RDX: 0000000000000000 RSI: ffff8b23f5687de8 RDI: ffff9b3e88a1fbb0
[ 7752.391518] RBP: ffff9b3e88a1fc10 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7752.392489] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7752.393458] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7752.394426] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7752.395499] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7752.396323] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7752.397286] Stack:
[ 7752.397721]  ffff8b2400000000 ffff8b2400000002 ffff8b2400002201 ffff8b2400000064
[ 7752.398770]  ffff8b24704bdaa0 ffffffff00000001 ffffffffffffffff ffff9b3e00000000
[ 7752.399827]  ffff8b246f8d6400 ffff9b3e88a1fd30 000000005841b88e 0000000032314610
[ 7752.400895] Call Trace:
[ 7752.401396]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7752.402281]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7752.403167]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7752.403991]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7752.404917]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7752.405688]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7752.406568]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7752.407453] Code: 49 8d 84 16 ff 0f 00 00 48 c1 e8 0c 41 01 c7 45 39 fc 0f 8e ba 01 00 00 49 63 c7 44 89 e1 4c 8d 84 24 f8 00 00 00 49 8d 74 c5 00 <48> 8d bc 24 00 01 00 00 44 29 f9 48 89 da e8 30 13 aa eb 85 c0 
[ 7780.373001] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:16914]
[ 7780.374091] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7780.384722] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7780.385748] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7780.386958] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7780.387848] RIP: 0010:[<ffffffffac3fa46c>]  [<ffffffffac3fa46c>] iov_iter_get_pages+0xc/0x340
[ 7780.389035] RSP: 0018:ffff9b3e88a1fa80  EFLAGS: 00000202
[ 7780.389914] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
[ 7780.390948] RDX: 0000000000000000 RSI: ffff8b23f5687de8 RDI: ffff9b3e88a1fbb0
[ 7780.391969] RBP: ffff9b3e88a1faa0 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7780.392950] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7780.393939] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7780.394930] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7780.396026] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7780.396864] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7780.397840] Stack:
[ 7780.398277]  0000000000000002 ffff8b23f5687de0 0000000000000000 0000000000000001
[ 7780.399340]  ffff9b3e88a1fc10 ffffffffc0959130 ffff8b2400000000 ffff8b2400000002
[ 7780.400420]  ffff8b2400002201 ffff8b2400000064 ffff8b24704bdaa0 ffffffff00000001
[ 7780.401467] Call Trace:
[ 7780.401971]  [<ffffffffc0959130>] ceph_direct_read_write+0x3b0/0xc60 [ceph]
[ 7780.402967]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7780.403874]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7780.404767]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7780.405589]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7780.406481]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7780.407262]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7780.408122]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7780.409017] Code: 85 c0 0f 85 f6 fe ff ff 4d 85 ed 0f 84 ed fe ff ff eb c1 66 90 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 <53> 48 89 d3 48 83 ec 18 65 48 8b 04 25 28 00 00 00 48 89 45 d0 
[ 7808.373575] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [fio:16914]
[ 7808.374678] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7808.385453] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7808.386521] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7808.387735] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7808.388625] RIP: 0010:[<ffffffffac3f7b71>]  [<ffffffffac3f7b71>] iov_iter_advance+0x1/0x380
[ 7808.389765] RSP: 0018:ffff9b3e88a1faa0  EFLAGS: 00000246
[ 7808.390560] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[ 7808.391542] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7808.392536] RBP: ffff9b3e88a1fc10 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7808.393510] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7808.394474] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7808.395433] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7808.396488] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7808.397309] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7808.398272] Stack:
[ 7808.398710]  ffff9b3e88a1fc10 ffffffffc09590ea ffff8b2400000000 ffff8b2400000002
[ 7808.399760]  ffff8b2400002201 ffff8b2400000064 ffff8b24704bdaa0 ffffffff00000001
[ 7808.400821]  ffffffffffffffff ffff9b3e00000000 ffff8b246f8d6400 ffff9b3e88a1fd30
[ 7808.401879] Call Trace:
[ 7808.402381]  [<ffffffffc09590ea>] ? ceph_direct_read_write+0x36a/0xc60 [ceph]
[ 7808.403362]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7808.404291]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7808.405216]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7808.406045]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7808.406938]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7808.407722]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7808.408586]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7808.409465] Code: ff 74 eb 48 8b 16 48 85 c0 4d 89 c1 49 0f 44 d0 48 39 f9 4c 0f 45 c9 48 09 d0 4c 09 c8 48 29 f9 0f 84 54 fe ff ff eb c6 66 90 55 <48> 89 f9 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 28 8b 17 
[ 7823.193878] INFO: rcu_sched detected stalls on CPUs/tasks:
[ 7823.194828]     (detected by 0, t=240007 jiffies, g=647181, c=647180, q=0)
[ 7823.195790] All QSes seen, last rcu_sched kthread activity 240007 (4302490165-4302250158), jiffies_till_next_fqs=3, root ->qsmask 0x0
[ 7823.197484] fio             R  running task        0 16914  16897 0x00000008
[ 7823.198486]  ffff8b247fc03e40 ffffffffac0cf783 ffff8b247fc1a400 fffffffface4cb00
[ 7823.199546]  ffff8b247fc03ea8 ffffffffac10c6b0 0000000000000000 ffff8b247960d880
[ 7823.200620]  ffff8b247fc1a400 ffffffff00000000 0000000000000000 ffff8b247960d880
[ 7823.201698] Call Trace:
[ 7823.202192]  <IRQ> [ 7823.202411]  [<ffffffffac0cf783>] sched_show_task+0xd3/0x140
[ 7823.203239]  [<ffffffffac10c6b0>] rcu_check_callbacks+0x8c0/0x8d0
[ 7823.204097]  [<ffffffffac121d80>] ? tick_sched_do_timer+0x50/0x50
[ 7823.204972]  [<ffffffffac111b8f>] update_process_times+0x2f/0x60
[ 7823.205848]  [<ffffffffac1216d5>] tick_sched_handle.isra.17+0x25/0x60
[ 7823.206777]  [<ffffffffac121dbd>] tick_sched_timer+0x3d/0x70
[ 7823.207635]  [<ffffffffac11260e>] __hrtimer_run_queues+0xee/0x260
[ 7823.208496]  [<ffffffffac112d9a>] hrtimer_interrupt+0x9a/0x180
[ 7823.209314]  [<ffffffffac04b978>] local_apic_timer_interrupt+0x38/0x60
[ 7823.210194]  [<ffffffffac80c72d>] smp_apic_timer_interrupt+0x3d/0x50
[ 7823.211052]  [<ffffffffac80b8ec>] apic_timer_interrupt+0x8c/0xa0
[ 7823.211881]  <EOI> [ 7823.212115]  [<ffffffffc09590d4>] ? ceph_direct_read_write+0x354/0xc60 [ceph]
[ 7823.213102]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7823.213962]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7823.214824]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7823.215602]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7823.216467]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7823.217215]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7823.218059]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7823.218919] rcu_sched kthread starved for 240007 jiffies! g647181 c647180 f0x2 RCU_GP_WAIT_FQS(3) ->state=0x0
[ 7823.220147] rcu_sched       R  running task        0     7      2 0x00000000
[ 7823.221117]  ffff8b246ef2da40 0000000000000000 ffff8b247abf8000 ffff8b247fc19600
[ 7823.222136]  ffff8b23f5a3bb00 ffff9b3e8065fd78 ffffffffac804985 ffff8b247a400000
[ 7823.223211]  00ff8b247a400080 ffff8b247fc19600 ffff9b3e8065fdc0 ffff8b247abf8000
[ 7823.224274] Call Trace:
[ 7823.224758]  [<ffffffffac804985>] ? __schedule+0x225/0x6d0
[ 7823.225515]  [<ffffffffac804e66>] schedule+0x36/0x80
[ 7823.226220]  [<ffffffffac80823b>] schedule_timeout+0x1db/0x3f0
[ 7823.227015]  [<ffffffffac110040>] ? del_timer_sync+0x50/0x50
[ 7823.227784]  [<ffffffffac10a139>] rcu_gp_kthread+0x4d9/0x960
[ 7823.228572]  [<ffffffffac109c60>] ? _synchronize_rcu_expedited.constprop.65+0x320/0x320
[ 7823.229595]  [<ffffffffac0c1959>] kthread+0xd9/0xf0
[ 7823.230300]  [<ffffffffac0c1880>] ? kthread_park+0x60/0x60
[ 7823.231082]  [<ffffffffac809dd5>] ret_from_fork+0x25/0x30
[ 7848.374400] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [fio:16914]
[ 7848.375451] Modules linked in: ceph(OE) libceph loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_security ip6table_raw ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 iptable_security iptable_raw iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables nfsd crct10dif_pclmul crc32_pclmul ghash_clmulni_intel acpi_cpufreq auth_rpcgss tpm_tis tpm_tis_core nfs_acl lockd tpm ppdev joydev floppy i2c_piix4 virtio_balloon pcspkr parport_pc parport qemu_fw_cfg grace sunrpc xfs libcrc32c qxl drm_kms_helper ttm drm virtio_net virtio_console virtio_blk crc32c_intel ata_generic virtio_pci pata_acpi serio_raw virtio_ring virtio [last unloaded: libceph]
[ 7848.385913] CPU: 0 PID: 16914 Comm: fio Tainted: G           OEL  4.9.0-rc6+ #24
[ 7848.386910] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.9.1-1.fc24 04/01/2014
[ 7848.388009] task: ffff8b247960d880 task.stack: ffff9b3e88a1c000
[ 7848.388817] RIP: 0010:[<ffffffffac3fa55e>]  [<ffffffffac3fa55e>] iov_iter_get_pages+0xfe/0x340
[ 7848.389919] RSP: 0018:ffff9b3e88a1fa98  EFLAGS: 00000286
[ 7848.390666] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000001
[ 7848.391598] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9b3e88a1fbb0
[ 7848.392547] RBP: ffff9b3e88a1faa0 R08: ffff9b3e88a1fba8 R09: 0000000000000000
[ 7848.393470] R10: ffff8b2477744d10 R11: ffffffffacc75c80 R12: 0000000000000002
[ 7848.394401] R13: ffff8b23f5687de0 R14: 0000000000000000 R15: 0000000000000001
[ 7848.395337] FS:  00007fca3ee3ba80(0000) GS:ffff8b247fc00000(0000) knlGS:0000000000000000
[ 7848.396358] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7848.397156] CR2: 000000000046efd0 CR3: 0000000137845000 CR4: 00000000000406f0
[ 7848.398113] Stack:
[ 7848.398537]  0000000000000001 ffff9b3e88a1fc10 ffffffffc0959130 ffff8b2400000000
[ 7848.399548]  ffff8b2400000002 ffff8b2400002201 ffff8b2400000064 ffff8b24704bdaa0
[ 7848.400582]  ffffffff00000001 ffffffffffffffff ffff9b3e00000000 ffff8b246f8d6400
[ 7848.401636] Call Trace:
[ 7848.402116]  [<ffffffffc0959130>] ceph_direct_read_write+0x3b0/0xc60 [ceph]
[ 7848.403067]  [<ffffffffc0959c94>] ceph_write_iter+0x2b4/0xbe0 [ceph]
[ 7848.403954]  [<ffffffffac06961b>] ? get_user_pages_fast+0x10b/0x1a0
[ 7848.404808]  [<ffffffffac2505f2>] vfs_iter_write+0x92/0x110
[ 7848.405622]  [<ffffffffac2842c9>] iter_file_splice_write+0x279/0x3f0
[ 7848.406601]  [<ffffffffac284bb6>] SyS_splice+0x326/0x7f0
[ 7848.407385]  [<ffffffffac05cf1e>] ? kvm_clock_get_cycles+0x1e/0x20
[ 7848.408307]  [<ffffffffac809b77>] entry_SYSCALL_64_fastpath+0x1a/0xa9
[ 7848.409219] Code: 44 39 f8 48 0f 45 da 48 89 d8 49 2b 06 48 8b 75 d0 65 48 33 34 25 28 00 00 00 0f 85 43 02 00 00 48 83 c4 18 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 31 c0 eb da 85 d2 74 f8 49 8b 4c 24 18 8b 41 08 29