Actions
Bug #58126
openkclient: WARNING: fs/crypto/keyring.c:242 fscrypt_destroy_keyring+0x7e/0xd0
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
The xfstests-dev generic/595 failed, in another test the generic/580 failed:
2022-11-30T03:47:31.465 INFO:teuthology.orchestra.run.smithi201.stdout:generic/592 [not run] kernel does not support encryption policy: '-c 1 -n 4 -v 2 -f 8' 2022-11-30T03:47:33.856 INFO:teuthology.orchestra.run.smithi201.stdout:generic/593 2s 2022-11-30T03:47:40.285 INFO:teuthology.orchestra.run.smithi201.stdout:generic/595 _check_dmesg: something found in dmesg (see /tmp/tmp.P7Ufk1sVOjxfstests-dev/results//generic/595.dmesg)
And the kernel dmesg logs:
2022-11-30T03:47:35.348815+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729482 2022-11-30T03:47:35.767327+00:00 smithi201 kernel: 595 (166596): drop_caches: 2 2022-11-30T03:47:35.827301+00:00 smithi201 kernel: fscrypt: ceph: 1 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:35.860288+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729481 2022-11-30T03:47:35.909075+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:35.940948+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:35.991842+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:36.028291+00:00 smithi201 kernel: fscrypt: ceph: 1 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729481 2022-11-30T03:47:36.057318+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:36.089317+00:00 smithi201 kernel: fscrypt: ceph: 1 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729987 2022-11-30T03:47:36.121286+00:00 smithi201 kernel: fscrypt: ceph: 2 inode(s) still busy after removing key with identifier 69b2f6edeee720cce0577937eb8a6751, including ino 1099511729481 2022-11-30T03:47:39.891918+00:00 smithi201 kernel: ------------[ cut here ]------------ 2022-11-30T03:47:39.891992+00:00 smithi201 kernel: WARNING: CPU: 4 PID: 168846 at fs/crypto/keyring.c:242 fscrypt_destroy_keyring+0x7e/0xd0 2022-11-30T03:47:39.948750+00:00 smithi201 kernel: Modules linked in: essiv authenc ceph libceph dns_resolver fscache netfs veth nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_compat nf_tables nfnetlink bridge stp llc xfs libcrc32c intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal coretemp kvm_intel kvm ipmi_ssif iTCO_wdt irqbypass iTCO_vendor_support acpi_ipmi crct10dif_pclmul sunrpc crc32_pclmul ipmi_si i2c_i801 ghash_clmulni_intel lpc_ich sha512_ssse3 joydev ipmi_devintf wmi i2c_smbus mfd_core ipmi_msghandler mei_me acpi_pad mei acpi_power_meter ioatdma ixgbe igb nvme mdio crc32c_intel ptp nvme_core i2c_algo_bit pps_core dca fuse 2022-11-30T03:47:39.958648+00:00 smithi201 kernel: CPU: 4 PID: 168846 Comm: umount Tainted: G S 6.1.0-rc5-ceph-g72ead199864c #1 2022-11-30T03:47:39.966177+00:00 smithi201 kernel: Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015 2022-11-30T03:47:39.971512+00:00 smithi201 kernel: RIP: 0010:fscrypt_destroy_keyring+0x7e/0xd0 2022-11-30T03:47:39.990345+00:00 smithi201 kernel: Code: e8 c7 f6 ff ff 48 89 df e8 5f fe ff ff 4d 85 ff 74 35 4c 89 fb 48 83 eb 08 74 2c 8b 83 b0 00 00 00 4c 8b 7b 08 83 f8 01 74 ba <0f> 0b 8b 83 b4 00 00 00 83 f8 01 74 b8 0f 0b 8b 83 d0 00 00 00 85 2022-11-30T03:47:39.995775+00:00 smithi201 kernel: RSP: 0018:ffffc9000b277e28 EFLAGS: 00010202 2022-11-30T03:47:39.995846+00:00 smithi201 kernel: RAX: 0000000000000002 RBX: ffff88810d52ac00 RCX: ffff88810b56aa00 2022-11-30T03:47:40.010286+00:00 smithi201 kernel: RDX: 0000000080000000 RSI: ffffffff822f3a09 RDI: ffff888108f59000 2022-11-30T03:47:40.010356+00:00 smithi201 kernel: RBP: ffff8881d394fb88 R08: 0000000000000028 R09: 0000000000000000 2022-11-30T03:47:40.024757+00:00 smithi201 kernel: R10: 0000000000000001 R11: 11ff4fe6834fcd91 R12: ffff8881d394fc40 2022-11-30T03:47:40.032000+00:00 smithi201 kernel: R13: ffff888108f59000 R14: ffff8881d394f800 R15: 0000000000000000 2022-11-30T03:47:40.040201+00:00 smithi201 kernel: FS: 00007fd83f6f1080(0000) GS:ffff88885fd00000(0000) knlGS:0000000000000000 2022-11-30T03:47:40.046084+00:00 smithi201 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 2022-11-30T03:47:40.053315+00:00 smithi201 kernel: CR2: 00007f918d417000 CR3: 000000017f89a005 CR4: 00000000003706e0 2022-11-30T03:47:40.060538+00:00 smithi201 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2022-11-30T03:47:40.067802+00:00 smithi201 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 2022-11-30T03:47:40.067880+00:00 smithi201 kernel: Call Trace: 2022-11-30T03:47:40.072515+00:00 smithi201 kernel: <TASK> 2022-11-30T03:47:40.077085+00:00 smithi201 kernel: generic_shutdown_super+0x47/0x120 2022-11-30T03:47:40.080949+00:00 smithi201 kernel: kill_anon_super+0x14/0x30 2022-11-30T03:47:40.085172+00:00 smithi201 kernel: ceph_kill_sb+0x36/0x90 [ceph] 2022-11-30T03:47:40.085249+00:00 smithi201 kernel: deactivate_locked_super+0x29/0x60 2022-11-30T03:47:40.093312+00:00 smithi201 kernel: cleanup_mnt+0xb8/0x140 2022-11-30T03:47:40.097009+00:00 smithi201 kernel: task_work_run+0x67/0xb0 2022-11-30T03:47:40.097089+00:00 smithi201 kernel: exit_to_user_mode_prepare+0x23d/0x240 2022-11-30T03:47:40.114628+00:00 smithi201 kernel: syscall_exit_to_user_mode+0x25/0x60 2022-11-30T03:47:40.114701+00:00 smithi201 kernel: do_syscall_64+0x40/0x80 2022-11-30T03:47:40.123441+00:00 smithi201 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd 2022-11-30T03:47:40.127152+00:00 smithi201 kernel: RIP: 0033:0x7fd83dc39e9b 2022-11-30T03:47:40.127221+00:00 smithi201 kernel: Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d e4 5f 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d bd 5f 38 00 f7 d8 64 89 01 48 2022-11-30T03:47:40.153759+00:00 smithi201 kernel: RSP: 002b:00007ffc71fdaf58 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 2022-11-30T03:47:40.161021+00:00 smithi201 kernel: RAX: 0000000000000000 RBX: 000055e46a189310 RCX: 00007fd83dc39e9b 2022-11-30T03:47:40.168267+00:00 smithi201 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055e46a197ce0 2022-11-30T03:47:40.168351+00:00 smithi201 kernel: RBP: 0000000000000000 R08: 000055e46a189010 R09: 0000000000000000 2022-11-30T03:47:40.182749+00:00 smithi201 kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 000055e46a197ce0 2022-11-30T03:47:40.189982+00:00 smithi201 kernel: R13: 00007fd83f258184 R14: 000055e46a1894f0 R15: 00000000ffffffff 2022-11-30T03:47:40.192317+00:00 smithi201 kernel: </TASK> 2022-11-30T03:47:40.192387+00:00 smithi201 kernel: irq event stamp: 9805 2022-11-30T03:47:40.204332+00:00 smithi201 kernel: hardirqs last enabled at (9817): [<ffffffff81166d4e>] __up_console_sem+0x5e/0x70 2022-11-30T03:47:40.213007+00:00 smithi201 kernel: hardirqs last disabled at (9828): [<ffffffff81166d33>] __up_console_sem+0x43/0x70 2022-11-30T03:47:40.221452+00:00 smithi201 kernel: softirqs last enabled at (9706): [<ffffffff81e0031b>] __do_softirq+0x31b/0x47e 2022-11-30T03:47:40.229870+00:00 smithi201 kernel: softirqs last disabled at (9699): [<ffffffff810ea8ad>] irq_exit_rcu+0xdd/0x110 2022-11-30T03:47:40.234619+00:00 smithi201 kernel: ---[ end trace 0000000000000000 ]--- 2022-11-30T03:47:40.741296+00:00 smithi201 kernel: libceph: mon0 (1)172.21.15.32:6789 session established 2022-11-30T03:47:40.741454+00:00 smithi201 kernel: libceph: client5415 fsid c3ace4c5-8016-41d3-8d88-66882dbcad2f 2022-11-30T03:47:41.086665+00:00 smithi201 kernel: libceph: mon0 (1)172.21.15.32:6789 session established 2022-11-30T03:47:41.086776+00:00 smithi201 kernel: libceph: client5418 fsid c3ace4c5-8016-41d3-8d88-66882dbcad2f 2022-11-30T03:47:41.210323+00:00 smithi201 kernel: libceph: mon0 (1)172.21.15.32:6789 session established 2022-11-30T03:47:41.210432+00:00 smithi201 kernel: libceph: client5421 fsid c3ace4c5-8016-41d3-8d88-66882dbcad2f 2022-11-30T03:47:41.252146+00:00 smithi201 kernel: xfs_io (pid 169182) is setting deprecated v1 encryption policy; recommend upgrading to v2. 2022-11-30T03:47:41.349318+00:00 smithi201 kernel: libceph: mon0 (1)172.21.15.32:6789 session established 2022-11-30T03:47:41.357316+00:00 smithi201 kernel: libceph: client5424 fsid c3ace4c5-8016-41d3-8d88-66882dbcad2f 2022-11-30T03:47:41.464286+00:00 smithi201 kernel: libceph: mon0 (1)172.21.15.32:6789 session established 2022-11-30T03:47:41.464411+00:00 smithi201 kernel: libceph: client5427 fsid c3ace4c5-8016-41d3-8d88-66882dbcad2f 2022-11-30T03:47:41.701510+00:00 smithi201 kernel: fscrypt_msg: 155 callbacks suppressed
Updated by Xiubo Li over 1 year ago
By reading the code it should be a bug in fs/crypto/ code. When closing the file it will be delayed in kernel space by adding it into the delayed_fput_list delay queue.
And if that queue is delayed for some reasons and when unmounting the mountpoint it will skip evicting the corresponding inode in evict_inodes(). So the fscrypt_put_encryption_info(), which will decrease the mk->mk_active_refs reference count, will be missed. And at last in generic_shutdown_super() will hit this warning.
Updated by Xiubo Li over 1 year ago
Xiubo Li wrote:
By reading the code it should be a bug in fs/crypto/ code. When closing the file it will be delayed in kernel space by adding it into the delayed_fput_list delay queue.
And if that queue is delayed for some reasons and when unmounting the mountpoint it will skip evicting the corresponding inode in evict_inodes(). So the fscrypt_put_encryption_info(), which will decrease the mk->mk_active_refs reference count, will be missed. And at last in generic_shutdown_super() will hit this warning.
Still reading the code to see whether could I fix this in ceph layer.
Updated by Xiubo Li over 1 year ago
- Related to Feature #46690: Add fscrypt support to the kernel CephFS client added
Updated by Xiubo Li over 1 year ago
- Status changed from In Progress to Fix Under Review
Updated by Xiubo Li over 1 year ago
Another call trace with some debug logs:
<6>[67807.661294] libceph: client4254 fsid 9439b448-7468-43ce-b766-9c46f02e67d5 <4>[67807.663476] ceph: test_dummy_encryption mode enabled <4>[67809.989345] fscrypt_destroy_keyring: mk_active_refs = 2 <4>[67809.989405] ------------[ cut here ]------------ <4>[67809.989442] WARNING: CPU: 17 PID: 106889 at fs/crypto/keyring.c:245 fscrypt_destroy_keyring+0x19d/0x1b0 <4>[67809.989459] Modules linked in: ceph libceph dns_resolver netfs tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_compat nf_nat_tftp nft_objref nf_conntrack_tftp bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 rfkill ip_set nf_tables nfnetlink sunrpc squashfs loop intel_rapl_msr intel_rapl_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 i2c_piix4 joydev pcspkr xfs libcrc32c sd_mod t10_pi sr_mod crc64_rocksoft cdrom crc64 bochs drm_vram_helper sg drm_kms_helper ata_generic syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper virtio_net ttm net_failover ata_piix crc32c_intel virtio_scsi serio_raw virtio_console failover drm libata dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse <4>[67809.989769] CPU: 17 PID: 106889 Comm: umount Tainted: G W 6.1.0+ #189 <4>[67809.989778] Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014 <4>[67809.989784] RIP: 0010:fscrypt_destroy_keyring+0x19d/0x1b0 <4>[67809.989793] Code: 54 57 a3 e8 8a f5 a6 00 be 04 00 00 00 4c 89 ff e8 68 9d ee ff 4c 89 ff e8 60 90 ee ff 41 8b 45 40 83 f8 01 0f 84 44 ff ff ff <0f> 0b e9 3d ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 <4>[67809.989801] RSP: 0018:ffffc9000a397d78 EFLAGS: 00010202 <4>[67809.989809] RAX: 0000000000000002 RBX: 0000000000000000 RCX: dffffc0000000000 <4>[67809.989815] RDX: 0000000000000003 RSI: 0000000000000004 RDI: ffff8881246c2840 <4>[67809.989820] RBP: ffff888117da2138 R08: ffffffffa2442490 R09: ffff8881246c2843 <4>[67809.989827] R10: ffffed10248d8508 R11: 0000000000000001 R12: ffff888117da2408 <4>[67809.989832] R13: ffff8881246c2800 R14: ffff888117da2000 R15: ffff8881246c2840 <4>[67809.989839] FS: 0000149fd85e56c0(0000) GS:ffff8887cd280000(0000) knlGS:0000000000000000 <4>[67809.989848] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[67809.989854] CR2: 00005593b8fd2048 CR3: 000000010de7a003 CR4: 00000000007706e0 <4>[67809.989859] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <4>[67809.989886] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 <4>[67809.989891] PKRU: 55555554 <4>[67809.989895] Call Trace: <4>[67809.989899] <TASK> <4>[67809.989907] generic_shutdown_super+0x72/0x1d0 <4>[67809.989919] kill_anon_super+0x1e/0x40 <4>[67809.989929] ceph_kill_sb+0x7f/0xd0 [ceph] <4>[67809.990066] deactivate_locked_super+0x52/0xd0 <4>[67809.990087] cleanup_mnt+0x187/0x1f0 <4>[67809.990103] task_work_run+0xf6/0x170 <4>[67809.990114] ? task_work_cancel+0x20/0x20 <4>[67809.990125] ? __x64_sys_umount+0xe3/0x100 <4>[67809.990138] exit_to_user_mode_loop+0x121/0x130 <4>[67809.990151] exit_to_user_mode_prepare+0xbd/0x100 <4>[67809.990162] syscall_exit_to_user_mode+0x12/0x30 <4>[67809.990172] do_syscall_64+0x48/0x90 <4>[67809.990183] entry_SYSCALL_64_after_hwframe+0x63/0xcd <4>[67809.990194] RIP: 0033:0x149fd6a39dfb <4>[67809.990202] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 50 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 50 38 00 f7 d8 64 89 01 48 <4>[67809.990209] RSP: 002b:00007ffe5d751c38 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 <4>[67809.990218] RAX: 0000000000000000 RBX: 00005593b8fc2310 RCX: 0000149fd6a39dfb <4>[67809.990224] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 00005593b8fc7d20 <4>[67809.990229] RBP: 0000000000000000 R08: 00005593b8fd0f80 R09: 00005593b8fc2010 <4>[67809.992308] R10: 0000149fd6dbfbc0 R11: 0000000000000246 R12: 00005593b8fc7d20 <4>[67809.992315] R13: 0000149fd8058184 R14: 00005593b8fc24f0 R15: 00000000ffffffff <4>[67809.992328] </TASK> <4>[67809.992331] ---[ end trace 0000000000000000 ]--- <3>[67809.993936] ================================================================== <3>[67809.993969] BUG: KASAN: null-ptr-deref in _raw_spin_lock+0x78/0xe0 <3>[67809.993990] Write of size 4 at addr 0000000000000000 by task kworker/15:15/99851 <3>[67809.993997] <3>[67809.994036] CPU: 15 PID: 99851 Comm: kworker/15:15 Tainted: G W 6.1.0+ #189 <3>[67809.994046] Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014 <3>[67809.994058] Workqueue: ceph-msgr ceph_con_workfn [libceph] <3>[67809.994361] Call Trace: <3>[67809.994367] <TASK> <3>[67809.994373] dump_stack_lvl+0x34/0x48 <3>[67809.994390] ? _raw_spin_lock+0x78/0xe0 <3>[67809.994401] kasan_report+0xad/0x130 <3>[67809.994430] ? _raw_spin_lock+0x78/0xe0 <3>[67809.994443] kasan_check_range+0x35/0x1c0 <3>[67809.994455] _raw_spin_lock+0x78/0xe0 <3>[67809.994465] ? _raw_spin_lock_bh+0xe0/0xe0 <3>[67809.994476] ? _raw_spin_lock+0x87/0xe0 <3>[67809.994487] ? _raw_spin_lock_bh+0xe0/0xe0 <3>[67809.994498] fscrypt_put_master_key_activeref+0x6c/0x1a0 <3>[67809.994510] put_crypt_info+0xec/0x170 <3>[67809.994522] fscrypt_put_encryption_info+0x25/0x40 <3>[67809.994532] ceph_evict_inode+0x82/0x360 [ceph] <3>[67809.994669] evict+0x189/0x2e0 <3>[67809.994688] ceph_handle_caps+0x931/0xd90 [ceph] <3>[67809.994824] ? ceph_remove_capsnap+0x60/0x60 [ceph] <3>[67809.994961] ? read_partial_message_chunk.part.0+0xf2/0x100 [libceph] <3>[67809.995128] ? mutex_unlock+0x80/0xd0 <3>[67809.995136] ? __mutex_unlock_slowpath.constprop.0+0x210/0x210 <3>[67809.995150] mds_dispatch+0x1ac/0x1e0 [ceph] <3>[67809.995296] ceph_con_process_message+0xd6/0x230 [libceph] <3>[67809.995455] ceph_con_v1_try_read+0x6be/0xc20 [libceph] <3>[67809.995619] ? process_connect+0xba0/0xba0 [libceph] <3>[67809.995787] ? __mutex_lock_slowpath+0x10/0x10 <3>[67809.995796] ? __switch_to+0x2fa/0x680 <3>[67809.995817] ceph_con_workfn+0x238/0x5a0 [libceph] <3>[67809.995975] process_one_work+0x3b4/0x6a0
Updated by Xiubo Li over 1 year ago
There is another case could trigger the crash when I was running the tests:
<6>[ 9404.296174] libceph: mon0 (1)10.72.47.117:40517 session established <6>[ 9404.304244] libceph: client4371 fsid 4188da9e-941b-4117-a64b-7c29f65cd802 <4>[ 9404.306056] ceph: test_dummy_encryption mode enabled <4>[ 9472.879353] fscrypt_destroy_keyring: mk_active_refs = 2 <4>[ 9472.879636] ------------[ cut here ]------------ <4>[ 9472.879641] WARNING: CPU: 26 PID: 57649 at fs/crypto/keyring.c:244 fscrypt_destroy_keyring+0x1ec/0x200 <4>[ 9472.879712] Modules linked in: ceph(E) libceph dns_resolver netfs tcp_diag udp_diag raw_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nft_compat nf_nat_tftp nf_conntrack_tftp bridge stp llc nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill nf_tables nfnetlink sunrpc squashfs loop intel_rapl_msr intel_rapl_common nfit libnvdimm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel sha512_ssse3 joydev i2c_piix4 pcspkr xfs libcrc32c sd_mod t10_pi bochs drm_vram_helper sr_mod drm_kms_helper cdrom crc64_rocksoft crc64 ata_generic syscopyarea sg sysfillrect sysimgblt drm_ttm_helper ttm ata_piix virtio_net net_failover drm virtio_console libata virtio_scsi serio_raw crc32c_intel failover dm_mirror dm_region_hash dm_log dm_mod ipmi_devintf ipmi_msghandler fuse <4>[ 9472.880139] CPU: 26 PID: 57649 Comm: umount Tainted: G E 6.2.0-rc2+ #190 <4>[ 9472.880155] Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014 <4>[ 9472.880165] RIP: 0010:fscrypt_destroy_keyring+0x1ec/0x200 <4>[ 9472.880178] Code: 92 17 94 e8 17 ce af 00 be 04 00 00 00 48 89 ef e8 b9 bd ed ff 48 89 ef e8 b1 ad ed ff 41 8b 47 38 83 f8 01 0f 84 ca fe ff ff <0f> 0b e9 c3 fe ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 90 90 <4>[ 9472.880187] RSP: 0018:ffffc9000db87d50 EFLAGS: 00010202 <4>[ 9472.880198] RAX: 0000000000000002 RBX: 0000000000000000 RCX: dffffc0000000000 <4>[ 9472.880204] RDX: 0000000000000003 RSI: 0000000000000004 RDI: ffff888138337838 <4>[ 9472.880210] RBP: ffff888138337838 R08: ffffffff93096e7f R09: ffff88813833783b <4>[ 9472.880216] R10: ffffed1027066f07 R11: 0000000000000001 R12: 00000000ffffffff <4>[ 9472.880222] R13: ffff8881400b0040 R14: ffff8882026ac000 R15: ffff888138337800 <4>[ 9472.880228] FS: 000014761c4076c0(0000) GS:ffff8887df300000(0000) knlGS:0000000000000000 <4>[ 9472.880289] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 9472.880339] CR2: 0000562d4393f000 CR3: 00000003d33a2001 CR4: 00000000007706e0 <4>[ 9472.880348] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 <4>[ 9472.880353] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 <4>[ 9472.880358] PKRU: 55555554 <4>[ 9472.880366] Call Trace: <4>[ 9472.880371] <TASK> <4>[ 9472.880382] generic_shutdown_super+0x72/0x1d0 <4>[ 9472.880397] kill_anon_super+0x1e/0x40 <4>[ 9472.880406] ceph_kill_sb+0x138/0x190 [ceph] <4>[ 9472.880699] deactivate_locked_super+0x52/0xd0 <4>[ 9472.880728] cleanup_mnt+0x187/0x1f0 <4>[ 9472.880742] task_work_run+0xf6/0x170 <4>[ 9472.880761] ? __pfx_task_work_run+0x10/0x10 <4>[ 9472.880771] ? __x64_sys_umount+0xe3/0x100 <4>[ 9472.880782] exit_to_user_mode_loop+0x121/0x130 <4>[ 9472.880799] exit_to_user_mode_prepare+0xbd/0x100 <4>[ 9472.880810] syscall_exit_to_user_mode+0x12/0x30 <4>[ 9472.880835] do_syscall_64+0x48/0x90 <4>[ 9472.880847] entry_SYSCALL_64_after_hwframe+0x72/0xdc <4>[ 9472.880861] RIP: 0033:0x14761a839dfb <4>[ 9472.880880] Code: ff d0 48 89 c7 b8 3c 00 00 00 0f 05 48 8b 0d 84 50 38 00 f7 d8 64 89 01 48 83 c8 ff c3 66 90 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 5d 50 38 00 f7 d8 64 89 01 48 <4>[ 9472.880888] RSP: 002b:00007fff7763c7b8 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6 <4>[ 9472.880897] RAX: 0000000000000000 RBX: 000055bc7842b310 RCX: 000014761a839dfb <4>[ 9472.880903] RDX: 0000000000000001 RSI: 0000000000000000 RDI: 000055bc78430d20 <4>[ 9472.880908] RBP: 0000000000000000 R08: 000055bc78439f80 R09: 000055bc7842b010 <4>[ 9472.880913] R10: 000014761abbfbc0 R11: 0000000000000246 R12: 000055bc78430d20 <4>[ 9472.880919] R13: 000014761be58184 R14: 000055bc7842b4f0 R15: 00000000ffffffff <4>[ 9472.880931] </TASK> <4>[ 9472.880941] ---[ end trace 0000000000000000 ]--- <3>[ 9472.883746] ================================================================== <3>[ 9472.883784] BUG: KASAN: null-ptr-deref in _raw_spin_lock+0x78/0xe0 <3>[ 9472.883799] Write of size 4 at addr 0000000000000000 by task kworker/28:9/57246 <3>[ 9472.883807] <3>[ 9472.883812] CPU: 28 PID: 57246 Comm: kworker/28:9 Tainted: G W E 6.2.0-rc2+ #190 <3>[ 9472.883822] Hardware name: Red Hat RHEV Hypervisor, BIOS 1.11.0-2.el7 04/01/2014 <3>[ 9472.883833] Workqueue: ceph-msgr ceph_con_workfn [libceph] <3>[ 9472.884203] Call Trace: <3>[ 9472.884208] <TASK> <3>[ 9472.884215] dump_stack_lvl+0x34/0x48 <3>[ 9472.884226] ? _raw_spin_lock+0x78/0xe0 <3>[ 9472.884239] kasan_report+0xad/0x130 <3>[ 9472.884254] ? _raw_spin_lock+0x78/0xe0 <3>[ 9472.884267] kasan_check_range+0x35/0x1c0 <3>[ 9472.884278] _raw_spin_lock+0x78/0xe0 <3>[ 9472.884289] ? __pfx__raw_spin_lock+0x10/0x10 <3>[ 9472.884300] ? _raw_spin_lock+0x87/0xe0 <3>[ 9472.884310] ? __pfx__raw_spin_lock+0x10/0x10 <3>[ 9472.884352] fscrypt_put_master_key_activeref.part.0+0x2e/0x150 <3>[ 9472.884367] put_crypt_info+0x106/0x180 <3>[ 9472.884380] fscrypt_put_encryption_info+0x25/0x40 <3>[ 9472.884392] ceph_evict_inode+0x82/0x360 [ceph] <3>[ 9472.884562] evict+0x189/0x2e0 <3>[ 9472.884575] writepages_finish+0x4b5/0x850 [ceph] <3>[ 9472.884753] __complete_request+0x30/0xb0 [libceph] <3>[ 9472.884965] handle_reply+0x5a4/0x860 [libceph] <3>[ 9472.885181] ? __pfx_handle_reply+0x10/0x10 [libceph] <3>[ 9472.885403] ? read_partial_message_chunk.part.0+0xf2/0x100 [libceph] client_loop: send disconnect: Broken pipe
Will try to fix it.
Updated by Xiubo Li over 1 year ago
Thread A Thread B
ceph unmount -->
--> ceph_kill_sb()
--> kill_anon_super()
--> generic_shutdown_super()
--> sync_filesystem() <== flush the buffers and
queue all the osd requests in
osdc and then ihold('inodeA')
--> ceph_sync_fs() handle_reply() <== the last request finished by ceph
--> finish_request() <== delete the last req from osdc
--> ceph_osdc_sync() <== wait for all the --> __complete_request()
reqs in the osdc to finish
--> evict_inodes() --> req->r_callback()
--> fscrypt_destroy_keyring() <== WARN and
set sb->s_master_keys = NULL;
--> iput('inodeA')
--> dereference the "sb->s_master_keys"
and then crash
The Thread A will fail to wait for the last osd req and it will be woke up while the last req is still running the req->r_callback(), which will release the last reference of inode->i_count. And if the req->r_callback() couldn't be finished just before the evict_inodes() is called then the Thread A will skip evicting the inodeA and let the req->r_callback()->iput() to do the evicting, but the sb->s_master_keys is already set to NULL by the Thread A.
Updated by Xiubo Li over 1 year ago
Updated by Xiubo Li over 1 year ago
With the V3 patches I have passed all the xfstests-dev test case with the test_dummy_encryption option enabled:
[root@lxbceph1 xfstests-dev]# ./check -g quick -E ./ceph.exclude FSTYP -- ceph PLATFORM -- Linux/x86_64 lxbceph1 6.2.0-rc2+ #201 SMP PREEMPT_DYNAMIC Sat Jan 28 17:03:48 CST 2023 MKFS_OPTIONS -- 10.72.47.117:40258:/testB MOUNT_OPTIONS -- -o test_dummy_encryption,name=admin,nowsync,copyfrom,rasize=4096 -o context=system_u:object_r:root_t:s0 10.72.47.117:40258:/testB /mnt/kcephfs.B ceph/001 [expunged] ceph/002 180s ... [not run] mount option "test_dummy_encryption" not allowed in this test ceph/003 198s ... [not run] mount option "test_dummy_encryption" not allowed in this test ceph/004 176s ... 128s ceph/005 170s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/001 360s ... 314s generic/002 181s ... 138s generic/003 [expunged] generic/004 [not run] O_TMPFILE is not supported generic/005 183s ... 134s generic/006 228s ... 182s generic/007 300s ... 263s generic/008 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/009 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/011 228s ... 184s generic/012 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/013 291s ... 332s generic/014 667s ... 792s generic/015 [not run] Filesystem ceph not supported in _scratch_mkfs_sized generic/016 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/018 [not run] defragmentation not supported for fstype "ceph" generic/020 297s ... 264s generic/021 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/022 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/023 227s ... 174s generic/024 [not run] kernel doesn't support renameat2 syscall generic/025 [not run] kernel doesn't support renameat2 syscall generic/026 [not run] ceph does not define maximum ACL count generic/028 176s ... 144s generic/029 173s ... 139s generic/030 177s ... 150s generic/031 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) generic/032 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/033 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/034 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/035 176s ... 135s generic/037 298s ... 268s generic/039 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/040 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/041 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/050 [not run] ceph does not support shutdown generic/052 [not run] ceph does not support shutdown generic/053 176s ... 144s generic/056 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/057 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/058 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/059 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/060 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/061 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/062 205s ... 175s generic/063 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/064 [not run] xfs_io fiemap failed (old kernel/wrong fs?) generic/065 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/066 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/067 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/069 386s ... 351s generic/070 258s ... 177s generic/071 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/073 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/075 259s ... 229s generic/076 [not run] require 10.72.47.117:40258:/testB to be local device generic/078 [not run] kernel doesn't support renameat2 syscall generic/079 [not run] file system doesn't support chattr +ia generic/080 176s ... 136s generic/081 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/082 [not run] disk quotas not supported by this filesystem type: ceph generic/084 178s ... 146s generic/086 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/087 176s ... 136s generic/088 172s ... 139s generic/090 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/091 587s ... 544s generic/092 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/094 [not run] xfs_io fiemap failed (old kernel/wrong fs?) generic/096 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/097 182s ... 142s generic/098 176s ... 138s generic/099 191s ... 154s generic/101 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/103 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/104 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/105 183s ... 136s generic/106 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/107 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/108 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/110 [not run] Reflink not supported by test filesystem type: ceph generic/111 [not run] Reflink not supported by test filesystem type: ceph generic/112 248s ... 235s generic/113 228s ... 362s generic/114 [not run] device block size: 4096 greater than 512 generic/115 [not run] Reflink not supported by test filesystem type: ceph generic/116 [not run] Reflink not supported by test filesystem type: ceph generic/117 227s ... 161s generic/118 [not run] Reflink not supported by test filesystem type: ceph generic/119 [not run] Reflink not supported by test filesystem type: ceph generic/120 199s ... 156s generic/121 [not run] Dedupe not supported by test filesystem type: ceph generic/122 [not run] Dedupe not supported by test filesystem type: ceph generic/123 194s ... 149s generic/124 161s ... 139s generic/126 188s ... 145s generic/128 199s ... 150s generic/129 839s ... 786s generic/130 234s ... 185s generic/131 186s ... 146s generic/134 [not run] Reflink not supported by test filesystem type: ceph generic/135 193s ... 142s generic/136 [not run] Dedupe not supported by test filesystem type: ceph generic/138 [not run] Reflink not supported by test filesystem type: ceph generic/139 [not run] Reflink not supported by test filesystem type: ceph generic/140 [not run] Reflink not supported by test filesystem type: ceph generic/141 185s ... 144s generic/142 [not run] Reflink not supported by test filesystem type: ceph generic/143 [not run] Reflink not supported by test filesystem type: ceph generic/144 [not run] Reflink not supported by test filesystem type: ceph generic/145 [not run] Reflink not supported by test filesystem type: ceph generic/146 [not run] Reflink not supported by test filesystem type: ceph generic/147 [not run] Reflink not supported by test filesystem type: ceph generic/148 [not run] Reflink not supported by test filesystem type: ceph generic/149 [not run] Reflink not supported by test filesystem type: ceph generic/150 [not run] Reflink not supported by test filesystem type: ceph generic/151 [not run] Reflink not supported by test filesystem type: ceph generic/152 [not run] Reflink not supported by test filesystem type: ceph generic/153 [not run] Reflink not supported by test filesystem type: ceph generic/154 [not run] Reflink not supported by test filesystem type: ceph generic/155 [not run] Reflink not supported by test filesystem type: ceph generic/156 [not run] Reflink not supported by test filesystem type: ceph generic/157 [not run] Reflink not supported by test filesystem type: ceph generic/158 [not run] Dedupe not supported by test filesystem type: ceph generic/159 [not run] file system doesn't support chattr +i generic/160 [not run] file system doesn't support chattr +i generic/161 [not run] Reflink not supported by scratch filesystem type: ceph generic/162 [not run] Dedupe not supported by scratch filesystem type: ceph generic/163 [not run] Dedupe not supported by scratch filesystem type: ceph generic/169 191s ... 144s generic/171 [not run] Reflink not supported by scratch filesystem type: ceph generic/172 [not run] Reflink not supported by scratch filesystem type: ceph generic/173 [not run] Reflink not supported by scratch filesystem type: ceph generic/174 [not run] Reflink not supported by scratch filesystem type: ceph generic/177 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/178 [not run] Reflink not supported by test filesystem type: ceph generic/179 [not run] Reflink not supported by test filesystem type: ceph generic/180 [not run] Reflink not supported by test filesystem type: ceph generic/181 [not run] Reflink not supported by test filesystem type: ceph generic/182 [not run] Dedupe not supported by test filesystem type: ceph generic/183 [not run] Reflink not supported by scratch filesystem type: ceph generic/184 186s ... 145s generic/185 [not run] Reflink not supported by scratch filesystem type: ceph generic/188 [not run] Reflink not supported by scratch filesystem type: ceph generic/189 [not run] Reflink not supported by scratch filesystem type: ceph generic/190 [not run] Reflink not supported by scratch filesystem type: ceph generic/191 [not run] Reflink not supported by scratch filesystem type: ceph generic/193 205s ... 168s generic/194 [not run] Reflink not supported by scratch filesystem type: ceph generic/195 [not run] Reflink not supported by scratch filesystem type: ceph generic/196 [not run] Reflink not supported by scratch filesystem type: ceph generic/197 [not run] Reflink not supported by scratch filesystem type: ceph generic/198 208s ... 164s generic/199 [not run] Reflink not supported by scratch filesystem type: ceph generic/200 [not run] Reflink not supported by scratch filesystem type: ceph generic/201 [not run] Reflink not supported by scratch filesystem type: ceph generic/202 [not run] Reflink not supported by scratch filesystem type: ceph generic/203 [not run] Reflink not supported by scratch filesystem type: ceph generic/205 [not run] Reflink not supported by scratch filesystem type: ceph generic/206 [not run] Reflink not supported by scratch filesystem type: ceph generic/207 189s ... 204s generic/210 191s ... 145s generic/211 240s ... 199s generic/212 186s ... 142s generic/213 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/214 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/215 191s ... 152s generic/216 [not run] Reflink not supported by scratch filesystem type: ceph generic/217 [not run] Reflink not supported by scratch filesystem type: ceph generic/218 [not run] Reflink not supported by scratch filesystem type: ceph generic/219 [not run] disk quotas not supported by this filesystem type: ceph generic/220 [not run] Reflink not supported by scratch filesystem type: ceph generic/221 195s ... 144s generic/222 [not run] Reflink not supported by scratch filesystem type: ceph generic/223 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/225 [not run] xfs_io fiemap failed (old kernel/wrong fs?) generic/227 [not run] Reflink not supported by scratch filesystem type: ceph generic/228 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/229 [not run] Reflink not supported by scratch filesystem type: ceph generic/230 [not run] disk quotas not supported by this filesystem type: ceph generic/235 [not run] disk quotas not supported by this filesystem type: ceph generic/236 187s ... 147s generic/237 191s ... 143s generic/238 [not run] Reflink not supported by scratch filesystem type: ceph generic/240 193s ... 153s generic/244 [not run] disk quotas not supported by this filesystem type: ceph generic/245 191s ... 146s generic/246 183s ... 144s generic/247 342s ... 361s generic/248 189s ... 144s generic/249 191s ... 155s generic/250 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/252 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/253 [not run] Reflink not supported by scratch filesystem type: ceph generic/254 [not run] Reflink not supported by scratch filesystem type: ceph generic/255 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/256 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/257 203s ... 155s generic/258 [not run] ceph does not support negative timestamps generic/259 [not run] Reflink not supported by scratch filesystem type: ceph generic/260 [not run] FITRIM not supported on /mnt/kcephfs.B generic/261 [not run] Reflink not supported by scratch filesystem type: ceph generic/262 [not run] Reflink not supported by scratch filesystem type: ceph generic/263 384s ... 338s generic/264 [not run] Reflink not supported by scratch filesystem type: ceph generic/265 [not run] Reflink not supported by scratch filesystem type: ceph generic/266 [not run] Reflink not supported by scratch filesystem type: ceph generic/267 [not run] Reflink not supported by scratch filesystem type: ceph generic/268 [not run] Reflink not supported by scratch filesystem type: ceph generic/271 [not run] Reflink not supported by scratch filesystem type: ceph generic/272 [not run] Reflink not supported by scratch filesystem type: ceph generic/276 [not run] Reflink not supported by scratch filesystem type: ceph generic/277 [not run] file system doesn't support chattr +A generic/278 [not run] Reflink not supported by scratch filesystem type: ceph generic/279 [not run] Reflink not supported by scratch filesystem type: ceph generic/281 [not run] Reflink not supported by scratch filesystem type: ceph generic/282 [not run] Reflink not supported by scratch filesystem type: ceph generic/283 [not run] Reflink not supported by scratch filesystem type: ceph generic/284 [not run] Reflink not supported by scratch filesystem type: ceph generic/286 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/287 [not run] Reflink not supported by scratch filesystem type: ceph generic/288 [not run] FITRIM not supported on /mnt/kcephfs.B generic/289 [not run] Reflink not supported by scratch filesystem type: ceph generic/290 [not run] Reflink not supported by scratch filesystem type: ceph generic/291 [not run] Reflink not supported by scratch filesystem type: ceph generic/292 [not run] Reflink not supported by scratch filesystem type: ceph generic/293 [not run] Reflink not supported by scratch filesystem type: ceph generic/294 192s ... 147s generic/295 [not run] Reflink not supported by scratch filesystem type: ceph generic/296 [not run] Reflink not supported by scratch filesystem type: ceph generic/301 [not run] Reflink not supported by scratch filesystem type: ceph generic/302 [not run] Reflink not supported by scratch filesystem type: ceph generic/303 [not run] Reflink not supported by test filesystem type: ceph generic/304 [not run] Dedupe not supported by test filesystem type: ceph generic/305 [not run] Reflink not supported by scratch filesystem type: ceph generic/306 191s ... 150s generic/307 187s ... 146s generic/308 188s ... 142s generic/309 184s ... 145s generic/312 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/313 191s ... 148s generic/314 190s ... 147s generic/315 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/316 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/317 204s ... 154s generic/318 191s ... 146s generic/319 186s ... 148s generic/321 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/322 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/324 [not run] defragmentation not supported for fstype "ceph" generic/325 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/326 [not run] Reflink not supported by scratch filesystem type: ceph generic/327 [not run] Reflink not supported by scratch filesystem type: ceph generic/328 [not run] Reflink not supported by scratch filesystem type: ceph generic/329 [not run] Reflink not supported by scratch filesystem type: ceph generic/330 [not run] Reflink not supported by scratch filesystem type: ceph generic/331 [not run] Reflink not supported by scratch filesystem type: ceph generic/332 [not run] Reflink not supported by scratch filesystem type: ceph generic/335 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/336 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/337 192s ... 144s generic/338 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/341 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/342 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/343 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/346 332s ... 361s generic/347 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/348 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/353 [not run] Reflink not supported by scratch filesystem type: ceph generic/355 200s ... 157s generic/356 [not run] swapfiles are not supported generic/357 [not run] swapfiles are not supported generic/358 [not run] Reflink not supported by scratch filesystem type: ceph generic/359 [not run] Reflink not supported by scratch filesystem type: ceph generic/360 184s ... 143s generic/361 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/362 [not run] this test requires richacl support on $SCRATCH_DEV generic/363 [not run] this test requires richacl support on $SCRATCH_DEV generic/364 [not run] this test requires richacl support on $SCRATCH_DEV generic/365 [not run] this test requires richacl support on $SCRATCH_DEV generic/366 [not run] this test requires richacl support on $SCRATCH_DEV generic/367 [not run] this test requires richacl support on $SCRATCH_DEV generic/368 [not run] this test requires richacl support on $SCRATCH_DEV generic/369 [not run] this test requires richacl support on $SCRATCH_DEV generic/370 [not run] this test requires richacl support on $SCRATCH_DEV generic/371 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/372 [not run] Reflink not supported by scratch filesystem type: ceph generic/373 [not run] Reflink not supported by scratch filesystem type: ceph generic/374 [not run] Dedupe not supported by scratch filesystem type: ceph generic/375 192s ... 151s generic/376 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/377 184s ... 145s generic/378 195s ... 157s generic/379 [not run] disk quotas not supported by this filesystem type: ceph generic/380 [not run] disk quotas not supported by this filesystem type: ceph generic/381 [not run] disk quotas not supported by this filesystem type: ceph generic/382 [not run] disk quotas not supported by this filesystem type: ceph generic/383 [not run] disk quotas not supported by this filesystem type: ceph generic/384 [not run] disk quotas not supported by this filesystem type: ceph generic/385 [not run] disk quotas not supported by this filesystem type: ceph generic/386 [not run] disk quotas not supported by this filesystem type: ceph generic/389 [not run] O_TMPFILE is not supported generic/391 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/392 [not run] ceph does not support shutdown generic/393 189s ... 148s generic/394 189s ... 148s generic/395 18s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/396 10s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/397 16s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/398 [not run] mount option "test_dummy_encryption" not allowed in this test generic/400 [not run] disk quotas not supported by this filesystem type: ceph generic/401 192s ... 146s generic/402 [not run] filesystem ceph timestamp bounds are unknown generic/403 194s ... 157s generic/404 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/406 199s ... 159s generic/407 [not run] Reflink not supported by test filesystem type: ceph generic/408 [not run] Dedupe not supported by test filesystem type: ceph generic/409 [not run] require 10.72.47.117:40258:/testB to be local device generic/410 [not run] require 10.72.47.117:40258:/testB to be local device generic/411 [not run] require 10.72.47.117:40258:/testB to be local device generic/412 186s ... 151s generic/413 [not run] mount 10.72.47.117:40258:/testB with dax failed generic/414 [not run] Reflink not supported by scratch filesystem type: ceph generic/417 [not run] ceph does not support shutdown generic/419 [not run] mount option "test_dummy_encryption" not allowed in this test generic/420 195s ... [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/421 18s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/422 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/423 193s ... 144s generic/424 [not run] file system doesn't support any of /usr/bin/chattr +a/+c/+d/+i generic/425 [not run] xfs_io fiemap -a failed (old kernel/wrong fs?) generic/426 232s ... 202s generic/427 [not run] Filesystem ceph not supported in _scratch_mkfs_sized generic/428 124s ... 121s generic/430 129s ... 124s generic/431 133s ... 125s generic/432 128s ... 124s generic/433 127s ... 123s generic/434 135s ... 124s generic/436 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/437 130s ... 127s generic/439 133s ... [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/440 14s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/441 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/443 124s ... 121s generic/444 127s ... 126s generic/445 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/446 272s ... [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/448 136s ... 124s generic/449 [not run] Filesystem ceph not supported in _scratch_mkfs_sized generic/450 134s ... 128s generic/451 164s ... 155s generic/452 129s ... 126s generic/453 [not run] ceph does not allow unrestricted byte streams for names generic/454 [not run] ceph does not allow unrestricted byte streams for names generic/456 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/458 [not run] Reflink not supported by scratch filesystem type: ceph generic/460 157s ... 310s generic/462 [not run] mount 10.72.47.117:40258:/testB with dax failed generic/463 [not run] Reflink not supported by test filesystem type: ceph generic/465 154s ... 160s generic/466 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/467 144s ... 136s generic/468 [not run] ceph does not support shutdown generic/469 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/470 [not run] This test requires a valid $LOGWRITES_DEV generic/471 [not run] xfs_io pwrite -V 1 -b 4k -N failed (old kernel/wrong fs?) generic/472 [not run] swapfiles are not supported generic/474 [not run] ceph does not support shutdown generic/477 144s ... 129s generic/478 164s ... 146s generic/479 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/480 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/481 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/483 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/484 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/485 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/486 126s generic/487 [not run] This test requires a valid $SCRATCH_LOGDEV generic/488 [not run] Filesystem ceph not supported in _scratch_mkfs_sized generic/489 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/490 130s ... 122s generic/491 [not run] ceph does not support freezing generic/492 [not run] xfs_io label failed (old kernel/wrong fs?) generic/493 [not run] swapfiles are not supported generic/494 [not run] swapfiles are not supported generic/495 [not run] swapfiles are not supported generic/496 [not run] swapfiles are not supported generic/497 [not run] swapfiles are not supported generic/498 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/499 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/501 [not run] Reflink not supported by scratch filesystem type: ceph generic/502 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/503 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/504 134s ... 125s generic/505 [not run] ceph does not support shutdown generic/506 [not run] disk quotas not supported by this filesystem type: ceph generic/507 [not run] file system doesn't support chattr +ASai generic/508 [not run] lsattr not supported by test filesystem type: ceph generic/509 [not run] O_TMPFILE is not supported generic/510 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/511 [not run] xfs_io falloc -k failed (old kernel/wrong fs?) generic/512 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/513 [not run] Reflink not supported by scratch filesystem type: ceph generic/514 [not run] Reflink not supported by scratch filesystem type: ceph generic/515 [not run] Reflink not supported by scratch filesystem type: ceph generic/516 [not run] Dedupe not supported by test filesystem type: ceph generic/517 [not run] Dedupe not supported by scratch filesystem type: ceph generic/518 [not run] Reflink not supported by scratch filesystem type: ceph generic/519 [not run] FIBMAP not supported by this filesystem generic/520 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/523 135s ... 125s generic/524 196s ... 247s generic/525 [not run] filesystem does not support huge file size generic/526 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/527 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/528 138s ... 132s generic/529 129s ... 131s generic/530 [not run] ceph does not support shutdown generic/531 [expunged] generic/532 128s ... 134s generic/533 139s ... 132s generic/534 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/535 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/536 [not run] ceph does not support shutdown generic/537 [not run] FSTRIM not supported generic/538 [expunged] generic/539 136s ... [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/540 [not run] Reflink not supported by scratch filesystem type: ceph generic/541 [not run] Reflink not supported by scratch filesystem type: ceph generic/542 [not run] Reflink not supported by scratch filesystem type: ceph generic/543 [not run] Reflink not supported by scratch filesystem type: ceph generic/544 [not run] Reflink not supported by scratch filesystem type: ceph generic/545 [not run] file system doesn't support chattr +i generic/546 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/547 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/548 [not run] mount option "test_dummy_encryption" not allowed in this test generic/549 [not run] mount option "test_dummy_encryption" not allowed in this test generic/550 [not run] mount option "test_dummy_encryption" not allowed in this test generic/552 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/553 [not run] xfs_io chattr +i failed (old kernel/wrong fs?) generic/554 [not run] swapfiles are not supported generic/555 [not run] xfs_io chattr +ia failed (old kernel/wrong fs?) generic/556 [not run] ceph does not support casefold feature generic/557 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/563 [not run] cgroup2 not mounted on /sys/fs/cgroup generic/564 140s ... 139s generic/565 132s ... 133s generic/566 [not run] disk quotas not supported by this filesystem type: ceph generic/567 139s ... [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/568 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/569 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/570 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/571 [not run] Require fcntl setlease support generic/572 [not run] fsverity utility required, skipped this test generic/573 [not run] fsverity utility required, skipped this test generic/574 [not run] fsverity utility required, skipped this test generic/575 [not run] fsverity utility required, skipped this test generic/576 [not run] fsverity utility required, skipped this test generic/577 [not run] fsverity utility required, skipped this test generic/578 [not run] Reflink not supported by test filesystem type: ceph generic/580 16s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/581 43s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/582 [not run] mount option "test_dummy_encryption" not allowed in this test generic/583 [not run] mount option "test_dummy_encryption" not allowed in this test generic/584 [not run] mount option "test_dummy_encryption" not allowed in this test generic/586 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/587 [not run] disk quotas not supported by this filesystem type: ceph generic/588 [not run] Reflink not supported by scratch filesystem type: ceph generic/591 130s ... 133s generic/592 [not run] mount option "test_dummy_encryption" not allowed in this test generic/593 17s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/594 [not run] disk quotas not supported by this filesystem type: ceph generic/595 16s ... [not run] mount option "test_dummy_encryption" not allowed in this test generic/596 [not run] file system doesn't support chattr +S generic/597 [not run] fsgqa2 user not defined. generic/598 [not run] fsgqa2 user not defined. generic/599 [not run] ceph does not support shutdown generic/600 [not run] disk quotas not supported by this filesystem type: ceph generic/601 [not run] disk quotas not supported by this filesystem type: ceph generic/602 [not run] mount option "test_dummy_encryption" not allowed in this test generic/603 [not run] disk quotas not supported by this filesystem type: ceph generic/604 172s ... 175s generic/605 [not run] mount 10.72.47.117:40258:/testB with dax=always failed generic/606 [not run] mount 10.72.47.117:40258:/testB with dax=always failed generic/607 [not run] xfs_io chattr +x failed (old kernel/wrong fs?) generic/608 [not run] mount 10.72.47.117:40258:/testB with dax=always failed generic/609 131s ... 131s generic/610 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/611 137s ... 136s generic/612 [not run] Reflink not supported by test filesystem type: ceph generic/613 [not run] mount option "test_dummy_encryption" not allowed in this test generic/614 132s ... [not run] test requires delayed allocation buffered writes generic/618 130s ... 132s generic/620 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/621 [not run] mount option "test_dummy_encryption" not allowed in this test generic/623 [not run] ceph does not support shutdown generic/624 [not run] fsverity utility required, skipped this test generic/625 [not run] fsverity utility required, skipped this test generic/626 [not run] kernel doesn't support renameat2 syscall generic/628 [not run] Reflink not supported by scratch filesystem type: ceph generic/629 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/630 [not run] Dedupe not supported by scratch filesystem type: ceph generic/632 389s ... 422s generic/633 132s ... 129s generic/634 144s ... 134s generic/635 [not run] ceph does not support shutdown generic/636 [not run] swapfiles are not supported generic/637 139s ... 141s generic/638 136s ... 127s generic/639 135s ... 128s generic/640 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/641 [not run] swapfiles are not supported generic/644 [not run] vfstest not support by ceph generic/645 [not run] vfstest not support by ceph generic/646 [not run] ceph does not support shutdown generic/647 130s ... 134s generic/651 [not run] Reflink not supported by scratch filesystem type: ceph generic/652 [not run] Reflink not supported by scratch filesystem type: ceph generic/653 [not run] Reflink not supported by scratch filesystem type: ceph generic/654 [not run] Reflink not supported by scratch filesystem type: ceph generic/655 [not run] Reflink not supported by scratch filesystem type: ceph generic/657 [not run] Reflink not supported by scratch filesystem type: ceph generic/658 [not run] Reflink not supported by scratch filesystem type: ceph generic/659 [not run] Reflink not supported by scratch filesystem type: ceph generic/660 [not run] Reflink not supported by scratch filesystem type: ceph generic/661 [not run] Reflink not supported by scratch filesystem type: ceph generic/662 [not run] Reflink not supported by scratch filesystem type: ceph generic/663 [not run] Reflink not supported by scratch filesystem type: ceph generic/664 [not run] Reflink not supported by scratch filesystem type: ceph generic/665 [not run] Reflink not supported by scratch filesystem type: ceph generic/666 [not run] Reflink not supported by scratch filesystem type: ceph generic/667 [not run] Reflink not supported by scratch filesystem type: ceph generic/668 [not run] Reflink not supported by scratch filesystem type: ceph generic/669 [not run] Reflink not supported by scratch filesystem type: ceph generic/673 [not run] Reflink not supported by scratch filesystem type: ceph generic/674 [not run] Dedupe not supported by scratch filesystem type: ceph generic/675 [not run] Reflink not supported by scratch filesystem type: ceph generic/676 3678s ... 7386s generic/677 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/678 [not run] kernel does not support IO_URING generic/679 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/680 138s ... 136s generic/681 [not run] disk quotas not supported by this filesystem type: ceph generic/682 [not run] disk quotas not supported by this filesystem type: ceph generic/683 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/684 [not run] xfs_io fpunch failed (old kernel/wrong fs?) generic/685 [not run] xfs_io fzero failed (old kernel/wrong fs?) generic/686 [not run] xfs_io finsert failed (old kernel/wrong fs?) generic/687 [not run] xfs_io fcollapse failed (old kernel/wrong fs?) generic/688 [not run] xfs_io falloc failed (old kernel/wrong fs?) generic/689 [not run] vfstest not support by ceph generic/690 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/692 [not run] fsverity utility required, skipped this test generic/693 [not run] mount option "test_dummy_encryption" not allowed in this test generic/695 [not run] require 10.72.47.117:40258:/testB to be valid block disk generic/696 143s ... 145s generic/697 130s ... 139s generic/698 [not run] vfstest not support by ceph generic/699 [not run] vfstest not support by ceph generic/700 [not run] kernel doesn't support renameat2 syscall shared/002 [not run] not suitable for this filesystem type: ceph shared/032 [not run] not suitable for this filesystem type: ceph Ran: ceph/002 ceph/003 ceph/004 ceph/005 generic/001 generic/002 generic/004 generic/005 generic/006 generic/007 generic/008 generic/009 generic/011 generic/012 generic/013 generic/014 generic/015 generic/016 generic/018 generic/020 generic/021 generic/022 generic/023 generic/024 generic/025 generic/026 generic/028 generic/029 generic/030 generic/031 generic/032 generic/033 generic/034 generic/035 generic/037 generic/039 generic/040 generic/041 generic/050 generic/052 generic/053 generic/056 generic/057 generic/058 generic/059 generic/060 generic/061 generic/062 generic/063 generic/064 generic/065 generic/066 generic/067 generic/069 generic/070 generic/071 generic/073 generic/075 generic/076 generic/078 generic/079 generic/080 generic/081 generic/082 generic/084 generic/086 generic/087 generic/088 generic/090 generic/091 generic/092 generic/094 generic/096 generic/097 generic/098 generic/099 generic/101 generic/103 generic/104 generic/105 generic/106 generic/107 generic/108 generic/110 generic/111 generic/112 generic/113 generic/114 generic/115 generic/116 generic/117 generic/118 generic/119 generic/120 generic/121 generic/122 generic/123 generic/124 generic/126 generic/128 generic/129 generic/130 generic/131 generic/134 generic/135 generic/136 generic/138 generic/139 generic/140 generic/141 generic/142 generic/143 generic/144 generic/145 generic/146 generic/147 generic/148 generic/149 generic/150 generic/151 generic/152 generic/153 generic/154 generic/155 generic/156 generic/157 generic/158 generic/159 generic/160 generic/161 generic/162 generic/163 generic/169 generic/171 generic/172 generic/173 generic/174 generic/177 generic/178 generic/179 generic/180 generic/181 generic/182 generic/183 generic/184 generic/185 generic/188 generic/189 generic/190 generic/191 generic/193 generic/194 generic/195 generic/196 generic/197 generic/198 generic/199 generic/200 generic/201 generic/202 generic/203 generic/205 generic/206 generic/207 generic/210 generic/211 generic/212 generic/213 generic/214 generic/215 generic/216 generic/217 generic/218 generic/219 generic/220 generic/221 generic/222 generic/223 generic/225 generic/227 generic/228 generic/229 generic/230 generic/235 generic/236 generic/237 generic/238 generic/240 generic/244 generic/245 generic/246 generic/247 generic/248 generic/249 generic/250 generic/252 generic/253 generic/254 generic/255 generic/256 generic/257 generic/258 generic/259 generic/260 generic/261 generic/262 generic/263 generic/264 generic/265 generic/266 generic/267 generic/268 generic/271 generic/272 generic/276 generic/277 generic/278 generic/279 generic/281 generic/282 generic/283 generic/284 generic/286 generic/287 generic/288 generic/289 generic/290 generic/291 generic/292 generic/293 generic/294 generic/295 generic/296 generic/301 generic/302 generic/303 generic/304 generic/305 generic/306 generic/307 generic/308 generic/309 generic/312 generic/313 generic/314 generic/315 generic/316 generic/317 generic/318 generic/319 generic/321 generic/322 generic/324 generic/325 generic/326 generic/327 generic/328 generic/329 generic/330 generic/331 generic/332 generic/335 generic/336 generic/337 generic/338 generic/341 generic/342 generic/343 generic/346 generic/347 generic/348 generic/353 generic/355 generic/356 generic/357 generic/358 generic/359 generic/360 generic/361 generic/362 generic/363 generic/364 generic/365 generic/366 generic/367 generic/368 generic/369 generic/370 generic/371 generic/372 generic/373 generic/374 generic/375 generic/376 generic/377 generic/378 generic/379 generic/380 generic/381 generic/382 generic/383 generic/384 generic/385 generic/386 generic/389 generic/391 generic/392 generic/393 generic/394 generic/395 generic/396 generic/397 generic/398 generic/400 generic/401 generic/402 generic/403 generic/404 generic/406 generic/407 generic/408 generic/409 generic/410 generic/411 generic/412 generic/413 generic/414 generic/417 generic/419 generic/420 generic/421 generic/422 generic/423 generic/424 generic/425 generic/426 generic/427 generic/428 generic/430 generic/431 generic/432 generic/433 generic/434 generic/436 generic/437 generic/439 generic/440 generic/441 generic/443 generic/444 generic/445 generic/446 generic/448 generic/449 generic/450 generic/451 generic/452 generic/453 generic/454 generic/456 generic/458 generic/460 generic/462 generic/463 generic/465 generic/466 generic/467 generic/468 generic/469 generic/470 generic/471 generic/472 generic/474 generic/477 generic/478 generic/479 generic/480 generic/481 generic/483 generic/484 generic/485 generic/486 generic/487 generic/488 generic/489 generic/490 generic/491 generic/492 generic/493 generic/494 generic/495 generic/496 generic/497 generic/498 generic/499 generic/501 generic/502 generic/503 generic/504 generic/505 generic/506 generic/507 generic/508 generic/509 generic/510 generic/511 generic/512 generic/513 generic/514 generic/515 generic/516 generic/517 generic/518 generic/519 generic/520 generic/523 generic/524 generic/525 generic/526 generic/527 generic/528 generic/529 generic/530 generic/532 generic/533 generic/534 generic/535 generic/536 generic/537 generic/539 generic/540 generic/541 generic/542 generic/543 generic/544 generic/545 generic/546 generic/547 generic/548 generic/549 generic/550 generic/552 generic/553 generic/554 generic/555 generic/556 generic/557 generic/563 generic/564 generic/565 generic/566 generic/567 generic/568 generic/569 generic/570 generic/571 generic/572 generic/573 generic/574 generic/575 generic/576 generic/577 generic/578 generic/580 generic/581 generic/582 generic/583 generic/584 generic/586 generic/587 generic/588 generic/591 generic/592 generic/593 generic/594 generic/595 generic/596 generic/597 generic/598 generic/599 generic/600 generic/601 generic/602 generic/603 generic/604 generic/605 generic/606 generic/607 generic/608 generic/609 generic/610 generic/611 generic/612 generic/613 generic/614 generic/618 generic/620 generic/621 generic/623 generic/624 generic/625 generic/626 generic/628 generic/629 generic/630 generic/632 generic/633 generic/634 generic/635 generic/636 generic/637 generic/638 generic/639 generic/640 generic/641 generic/644 generic/645 generic/646 generic/647 generic/651 generic/652 generic/653 generic/654 generic/655 generic/657 generic/658 generic/659 generic/660 generic/661 generic/662 generic/663 generic/664 generic/665 generic/666 generic/667 generic/668 generic/669 generic/673 generic/674 generic/675 generic/676 generic/677 generic/678 generic/679 generic/680 generic/681 generic/682 generic/683 generic/684 generic/685 generic/686 generic/687 generic/688 generic/689 generic/690 generic/692 generic/693 generic/695 generic/696 generic/697 generic/698 generic/699 generic/700 shared/002 shared/032 Not run: ceph/002 ceph/003 ceph/005 generic/004 generic/008 generic/009 generic/012 generic/015 generic/016 generic/018 generic/021 generic/022 generic/024 generic/025 generic/026 generic/031 generic/032 generic/033 generic/034 generic/039 generic/040 generic/041 generic/050 generic/052 generic/056 generic/057 generic/058 generic/059 generic/060 generic/061 generic/063 generic/064 generic/065 generic/066 generic/067 generic/071 generic/073 generic/076 generic/078 generic/079 generic/081 generic/082 generic/086 generic/090 generic/092 generic/094 generic/096 generic/101 generic/103 generic/104 generic/106 generic/107 generic/108 generic/110 generic/111 generic/114 generic/115 generic/116 generic/118 generic/119 generic/121 generic/122 generic/134 generic/136 generic/138 generic/139 generic/140 generic/142 generic/143 generic/144 generic/145 generic/146 generic/147 generic/148 generic/149 generic/150 generic/151 generic/152 generic/153 generic/154 generic/155 generic/156 generic/157 generic/158 generic/159 generic/160 generic/161 generic/162 generic/163 generic/171 generic/172 generic/173 generic/174 generic/177 generic/178 generic/179 generic/180 generic/181 generic/182 generic/183 generic/185 generic/188 generic/189 generic/190 generic/191 generic/194 generic/195 generic/196 generic/197 generic/199 generic/200 generic/201 generic/202 generic/203 generic/205 generic/206 generic/213 generic/214 generic/216 generic/217 generic/218 generic/219 generic/220 generic/222 generic/223 generic/225 generic/227 generic/228 generic/229 generic/230 generic/235 generic/238 generic/244 generic/250 generic/252 generic/253 generic/254 generic/255 generic/256 generic/258 generic/259 generic/260 generic/261 generic/262 generic/264 generic/265 generic/266 generic/267 generic/268 generic/271 generic/272 generic/276 generic/277 generic/278 generic/279 generic/281 generic/282 generic/283 generic/284 generic/286 generic/287 generic/288 generic/289 generic/290 generic/291 generic/292 generic/293 generic/295 generic/296 generic/301 generic/302 generic/303 generic/304 generic/305 generic/312 generic/315 generic/316 generic/321 generic/322 generic/324 generic/325 generic/326 generic/327 generic/328 generic/329 generic/330 generic/331 generic/332 generic/335 generic/336 generic/338 generic/341 generic/342 generic/343 generic/347 generic/348 generic/353 generic/356 generic/357 generic/358 generic/359 generic/361 generic/362 generic/363 generic/364 generic/365 generic/366 generic/367 generic/368 generic/369 generic/370 generic/371 generic/372 generic/373 generic/374 generic/376 generic/379 generic/380 generic/381 generic/382 generic/383 generic/384 generic/385 generic/386 generic/389 generic/391 generic/392 generic/395 generic/396 generic/397 generic/398 generic/400 generic/402 generic/404 generic/407 generic/408 generic/409 generic/410 generic/411 generic/413 generic/414 generic/417 generic/419 generic/420 generic/421 generic/422 generic/424 generic/425 generic/427 generic/436 generic/439 generic/440 generic/441 generic/445 generic/446 generic/449 generic/453 generic/454 generic/456 generic/458 generic/462 generic/463 generic/466 generic/468 generic/469 generic/470 generic/471 generic/472 generic/474 generic/479 generic/480 generic/481 generic/483 generic/484 generic/485 generic/487 generic/488 generic/489 generic/491 generic/492 generic/493 generic/494 generic/495 generic/496 generic/497 generic/498 generic/499 generic/501 generic/502 generic/503 generic/505 generic/506 generic/507 generic/508 generic/509 generic/510 generic/511 generic/512 generic/513 generic/514 generic/515 generic/516 generic/517 generic/518 generic/519 generic/520 generic/525 generic/526 generic/527 generic/530 generic/534 generic/535 generic/536 generic/537 generic/539 generic/540 generic/541 generic/542 generic/543 generic/544 generic/545 generic/546 generic/547 generic/548 generic/549 generic/550 generic/552 generic/553 generic/554 generic/555 generic/556 generic/557 generic/563 generic/566 generic/567 generic/568 generic/569 generic/570 generic/571 generic/572 generic/573 generic/574 generic/575 generic/576 generic/577 generic/578 generic/580 generic/581 generic/582 generic/583 generic/584 generic/586 generic/587 generic/588 generic/592 generic/593 generic/594 generic/595 generic/596 generic/597 generic/598 generic/599 generic/600 generic/601 generic/602 generic/603 generic/605 generic/606 generic/607 generic/608 generic/610 generic/612 generic/613 generic/614 generic/620 generic/621 generic/623 generic/624 generic/625 generic/626 generic/628 generic/629 generic/630 generic/635 generic/636 generic/640 generic/641 generic/644 generic/645 generic/646 generic/651 generic/652 generic/653 generic/654 generic/655 generic/657 generic/658 generic/659 generic/660 generic/661 generic/662 generic/663 generic/664 generic/665 generic/666 generic/667 generic/668 generic/669 generic/673 generic/674 generic/675 generic/677 generic/678 generic/679 generic/681 generic/682 generic/683 generic/684 generic/685 generic/686 generic/687 generic/688 generic/689 generic/690 generic/692 generic/693 generic/695 generic/698 generic/699 generic/700 shared/002 shared/032 Passed all 565 tests
Updated by Kotresh Hiremath Ravishankar 12 months ago
Actions