Actions
Bug #64035
openfscrypt test trigger "kernel BUG at fs/ceph/addr.c:1183!" with the kernel 6.7
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
<6>[ 773.668394] libceph: mon0 (1)192.168.0.108:40357 session established <6>[ 773.670260] libceph: client34235 fsid 85accf61-baa5-4166-8ad4-a22b1b6a04f1 <4>[ 773.670739] ceph: test_dummy_encryption mode enabled <12>[ 774.251933] run fstests ceph/002 at 2024-01-16 12:27:17 <6>[ 775.508975] fscrypt: AES-256-XTS using implementation "xts-aes-aesni" <12>[ 776.567073] run fstests ceph/003 at 2024-01-16 12:27:19 <12>[ 778.967859] run fstests ceph/004 at 2024-01-16 12:27:22 <4>[ 780.664091] ------------[ cut here ]------------ <4>[ 780.664099] WARNING: CPU: 2 PID: 78 at fs/crypto/crypto.c:123 fscrypt_crypt_data_unit+0x3a9/0x3d0 <4>[ 780.664110] Modules linked in: ceph(E) libceph(E) dns_resolver fscache netfs xsk_diag snd_seq_dummy snd_hrtimer qrtr rfkill intel_rapl_msr intel_rapl_common snd_intel8x0 intel_pmc_core snd_ac97_codec ac97_bus rapl snd_seq snd_seq_device snd_pcm pktcdvd snd_timer snd pcspkr soundcore i2c_piix4 xfs libcrc32c sr_mod sd_mod cdrom t10_pi sg ata_generic vmwgfx drm_ttm_helper ttm ahci crct10dif_pclmul crc32_pclmul virtio_net video drm_kms_helper net_failover crc32c_intel libahci ata_piix wmi failover ghash_clmulni_intel drm e1000 libata serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse <4>[ 780.664213] CPU: 2 PID: 78 Comm: kworker/u23:0 Kdump: loaded Tainted: G E 6.7.0+ #38 <4>[ 780.664219] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 <4>[ 780.664222] Workqueue: writeback wb_workfn (flush-ceph-2) <4>[ 780.664231] RIP: 0010:fscrypt_crypt_data_unit+0x3a9/0x3d0 <4>[ 780.664237] Code: e8 3c 45 d6 00 44 8b a4 24 68 01 00 00 c7 84 24 08 01 00 00 00 00 00 00 e9 e0 fe ff ff 0f 0b 41 bc ea ff ff ff e9 e7 fe ff ff <0f> 0b 41 bc ea ff ff ff e9 da fe ff ff 41 bc f4 ff ff ff e9 cf fe <4>[ 780.664242] RSP: 0018:ffff888101057150 EFLAGS: 00010202 <4>[ 780.664248] RAX: 0000000000000000 RBX: 1ffff1102020ae2d RCX: ffffffff8c3d641e <4>[ 780.664252] RDX: dffffc0000000000 RSI: 0000000000000000 RDI: ffff8882d94613c0 <4>[ 780.664256] RBP: 0000000000000000 R08: ffffffff8be6662f R09: 0000000000000000 <4>[ 780.664259] R10: ffffffff8ef6b467 R11: ffffffff8bc5dff0 R12: 0000000000000001 <4>[ 780.664263] R13: ffff8882d94613c0 R14: ffff888124fc3700 R15: 0000000000000000 <4>[ 780.664269] FS: 0000000000000000(0000) GS:ffff888469400000(0000) knlGS:0000000000000000 <4>[ 780.664274] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 780.664277] CR2: 000055f9b72ba000 CR3: 000000013952a004 CR4: 00000000000706f0 <4>[ 780.664281] Call Trace: <4>[ 780.664285] <TASK> <4>[ 780.664288] ? __warn+0xa1/0x200 <4>[ 780.664296] ? fscrypt_crypt_data_unit+0x3a9/0x3d0 <4>[ 780.664309] ? report_bug+0x1c7/0x1d0 <4>[ 780.664317] ? handle_bug+0x3c/0x70 <4>[ 780.664322] ? exc_invalid_op+0x14/0x50 <4>[ 780.664328] ? asm_exc_invalid_op+0x16/0x20 <4>[ 780.664336] ? ret_from_fork+0x30/0x50 <4>[ 780.664348] ? lockdep_init_map_type+0xdf/0x370 <4>[ 780.664354] ? fscrypt_crypt_data_unit+0xee/0x3d0 <4>[ 780.664360] ? fscrypt_crypt_data_unit+0x3a9/0x3d0 <4>[ 780.664368] ? __pfx_fscrypt_crypt_data_unit+0x10/0x10 <4>[ 780.664373] ? alloc_pages_mpol+0x113/0x300 <4>[ 780.664379] ? kthread+0x1a7/0x1f0 <4>[ 780.664385] ? __pfx_alloc_pages_mpol+0x10/0x10 <4>[ 780.664391] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664396] ? __writeback_single_inode+0x95/0x450 <4>[ 780.664402] ? writeback_sb_inodes+0x395/0x7e0 <4>[ 780.664407] ? wb_writeback+0x2ab/0x530 <4>[ 780.664412] ? fs_reclaim_acquire+0x63/0xf0 <4>[ 780.664419] ? __pfx_mempool_alloc_pages+0x10/0x10 <4>[ 780.664428] ? mempool_alloc+0xe9/0x270 <4>[ 780.664438] ? __kmem_cache_alloc_node+0x235/0x3d0 <4>[ 780.664444] ? trace_hardirqs_on+0x12/0xe0 <4>[ 780.664451] fscrypt_encrypt_pagecache_blocks+0x1c3/0x260 <4>[ 780.664460] ceph_writepages_start+0x891/0x2aa0 [ceph] <4>[ 780.664548] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.664554] ? __list_add_valid_or_report+0x35/0xc0 <4>[ 780.664568] ? blk_insert_flush+0x1b5/0x2f0 <4>[ 780.664579] ? __pfx_ceph_writepages_start+0x10/0x10 [ceph] <4>[ 780.664662] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664669] ? lock_acquire+0xc7/0x3e0 <4>[ 780.664674] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664680] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664687] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664692] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664698] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664704] ? lock_acquire+0xc7/0x3e0 <4>[ 780.664710] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664730] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.664737] ? __pfx_do_raw_spin_trylock+0x10/0x10 <4>[ 780.664747] do_writepages+0x106/0x320 <4>[ 780.664753] ? __pfx_do_writepages+0x10/0x10 <4>[ 780.664758] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664764] ? __pfx_wb_calc_thresh+0x10/0x10 <4>[ 780.664771] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.664777] ? __pfx_do_raw_spin_trylock+0x10/0x10 <4>[ 780.664784] __writeback_single_inode+0x95/0x450 <4>[ 780.664793] writeback_sb_inodes+0x395/0x7e0 <4>[ 780.664801] ? lock_acquire+0xc7/0x3e0 <4>[ 780.664807] ? __pfx_writeback_sb_inodes+0x10/0x10 <4>[ 780.664817] ? down_read_trylock+0x47/0x60 <4>[ 780.664824] __writeback_inodes_wb+0x6a/0x130 <4>[ 780.664831] wb_writeback+0x2ab/0x530 <4>[ 780.664837] ? __pfx_wb_writeback+0x10/0x10 <4>[ 780.664844] ? _find_next_bit+0x37/0xc0 <4>[ 780.664854] wb_do_writeback+0x434/0x4f0 <4>[ 780.664860] ? __pfx_wb_do_writeback+0x10/0x10 <4>[ 780.664865] ? __update_load_avg_cfs_rq+0x5f/0x480 <4>[ 780.664883] wb_workfn+0xe0/0x400 <4>[ 780.664889] ? __pfx_wb_workfn+0x10/0x10 <4>[ 780.664894] ? set_next_entity+0xff/0x320 <4>[ 780.664900] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664905] ? lock_acquire+0xc7/0x3e0 <4>[ 780.664911] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664916] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.664921] ? lock_unpin_lock+0x77/0x240 <4>[ 780.664928] ? rcu_is_watching+0x1f/0x50 <4>[ 780.664933] ? lockdep_hardirqs_on_prepare+0xc/0x60 <4>[ 780.664938] ? trace_hardirqs_on+0x12/0xe0 <4>[ 780.664944] ? finish_task_switch.isra.0+0x1ae/0x560 <4>[ 780.664952] process_one_work+0x46a/0x910 <4>[ 780.664962] ? __pfx_process_one_work+0x10/0x10 <4>[ 780.664968] ? __list_add_valid_or_report+0x35/0xc0 <4>[ 780.664979] worker_thread+0x385/0x670 <4>[ 780.664988] ? __pfx_worker_thread+0x10/0x10 <4>[ 780.664993] kthread+0x1a7/0x1f0 <4>[ 780.664999] ? kthread+0xf9/0x1f0 <4>[ 780.665004] ? __pfx_kthread+0x10/0x10 <4>[ 780.665045] ret_from_fork+0x30/0x50 <4>[ 780.665054] ? __pfx_kthread+0x10/0x10 <4>[ 780.665061] ret_from_fork_asm+0x1b/0x30 <4>[ 780.665072] </TASK> <4>[ 780.665075] irq event stamp: 1818 <4>[ 780.665078] hardirqs last enabled at (1817): [<ffffffff8d1490d4>] _raw_spin_unlock_irq+0x24/0x40 <4>[ 780.665085] hardirqs last disabled at (1818): [<ffffffff8d138f0f>] __schedule+0x6ef/0xde0 <4>[ 780.665091] softirqs last enabled at (1270): [<ffffffff8d14b3f9>] __do_softirq+0x3d9/0x4d5 <4>[ 780.665096] softirqs last disabled at (1261): [<ffffffff8bd7bd82>] __irq_exit_rcu+0xf2/0x110 <4>[ 780.665102] ---[ end trace 0000000000000000 ]--- <3>[ 780.665108] ceph: [85accf61-baa5-4166-8ad4-a22b1b6a04f1 34229]: inode->i_blkbits=12 <4>[ 780.665154] ------------[ cut here ]------------ <2>[ 780.665158] kernel BUG at fs/ceph/addr.c:1183! <4>[ 780.665169] invalid opcode: 0000 [#1] PREEMPT SMP KASAN PTI <4>[ 780.665178] CPU: 2 PID: 78 Comm: kworker/u23:0 Kdump: loaded Tainted: G W E 6.7.0+ #38 <4>[ 780.665187] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 <4>[ 780.665193] Workqueue: writeback wb_workfn (flush-ceph-2) <4>[ 780.665203] RIP: 0010:ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665334] Code: 20 47 95 c1 4c 89 fa 48 c7 c6 20 2f 95 c1 48 c7 c7 10 8e 91 c1 e8 e8 9a e5 ca 48 c7 85 48 fd ff ff ff ff ff ff e9 da de ff ff <0f> 0b 0f 0b 48 8b bd e0 fd ff ff e8 c8 71 9e ca 4c 8b bd f8 fd ff <4>[ 780.665339] RSP: 0018:ffff8881010573a0 EFLAGS: 00010246 <4>[ 780.665344] RAX: 0000000000000047 RBX: ffff888123e3a200 RCX: 0000000000000027 <4>[ 780.665348] RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8884695ffbc8 <4>[ 780.665352] RBP: ffff8881010576f0 R08: ffffffff8c0506ce R09: ffffed108d2bff79 <4>[ 780.665356] R10: ffff8884695ffbcb R11: 0000000000000001 R12: ffff88812d791910 <4>[ 780.665360] R13: ffffea000ba2c7c0 R14: 0000000000000000 R15: ffff8881da544000 <4>[ 780.665366] FS: 0000000000000000(0000) GS:ffff888469400000(0000) knlGS:0000000000000000 <4>[ 780.665370] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 780.665374] CR2: 000055f9b72ba000 CR3: 000000013952a004 CR4: 00000000000706f0 <4>[ 780.665378] Call Trace: <4>[ 780.665382] <TASK> <4>[ 780.665385] ? die+0x33/0x90 <4>[ 780.665392] ? do_trap+0x133/0x170 <4>[ 780.665398] ? ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665486] ? do_error_trap+0xab/0x130 <4>[ 780.665492] ? ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665579] ? ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665689] ? handle_invalid_op+0x2c/0x40 <4>[ 780.665698] ? ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665811] ? exc_invalid_op+0x2b/0x50 <4>[ 780.665818] ? asm_exc_invalid_op+0x16/0x20 <4>[ 780.665830] ? irq_work_claim+0x1e/0x40 <4>[ 780.665840] ? ceph_writepages_start+0x2598/0x2aa0 [ceph] <4>[ 780.665958] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.665969] ? __list_add_valid_or_report+0x35/0xc0 <4>[ 780.665983] ? blk_insert_flush+0x1b5/0x2f0 <4>[ 780.666001] ? __pfx_ceph_writepages_start+0x10/0x10 [ceph] <4>[ 780.666107] ? rcu_is_watching+0x1f/0x50 <4>[ 780.666117] ? lock_acquire+0xc7/0x3e0 <4>[ 780.666126] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.666136] ? rcu_is_watching+0x1f/0x50 <4>[ 780.666149] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.666157] ? rcu_is_watching+0x1f/0x50 <4>[ 780.666163] ? rcu_is_watching+0x1f/0x50 <4>[ 780.666169] ? lock_acquire+0xc7/0x3e0 <4>[ 780.666175] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.666181] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.666187] ? __pfx_do_raw_spin_trylock+0x10/0x10 <4>[ 780.666197] do_writepages+0x106/0x320 <4>[ 780.666203] ? __pfx_do_writepages+0x10/0x10 <4>[ 780.666208] ? __pfx_lock_acquire+0x10/0x10 <4>[ 780.666216] ? __pfx_wb_calc_thresh+0x10/0x10 <4>[ 780.666227] ? do_raw_spin_trylock+0xd0/0x120 <4>[ 780.666236] ? __pfx_do_raw_spin_trylock+0x10/0x10 <4>[ 780.666247] __writeback_single_inode+0x95/0x450 <4>[ 780.666270] writeback_sb_inodes+0x395/0x7e0 <4>[ 780.666285] ? lock_acquire+0xc7/0x3e0 <4>[ 780.666292] ? __pfx_writeback_sb_inodes+0x10/0x10 <4>[ 780.666302] ? down_read_trylock+0x47/0x60 <4>[ 780.666309] __writeback_inodes_wb+0x6a/0x130 client_loop: send disconnect: Broken pipe
It was introduce by the following commit:
commit 5b11888471806edf699316d4dcb9b426caebbef2
Author: Eric Biggers <ebiggers@google.com>
Date: Sun Sep 24 22:54:51 2023 -0700
fscrypt: support crypto data unit size less than filesystem block size
Until now, fscrypt has always used the filesystem block size as the
granularity of file contents encryption. Two scenarios have come up
where a sub-block granularity of contents encryption would be useful:
1. Inline crypto hardware that only supports a crypto data unit size
that is less than the filesystem block size.
2. Support for direct I/O at a granularity less than the filesystem
block size, for example at the block device's logical block size in
order to match the traditional direct I/O alignment requirement.
(1) first came up with older eMMC inline crypto hardware that only
supports a crypto data unit size of 512 bytes. That specific case
ultimately went away because all systems with that hardware continued
using out of tree code and never actually upgraded to the upstream
inline crypto framework. But, now it's coming back in a new way: some
current UFS controllers only support a data unit size of 4096 bytes, and
there is a proposal to increase the filesystem block size to 16K.
(2) was discussed as a "nice to have" feature, though not essential,
when support for direct I/O on encrypted files was being upstreamed.
Still, the fact that this feature has come up several times does suggest
it would be wise to have available. Therefore, this patch implements it
by using one of the reserved bytes in fscrypt_policy_v2 to allow users
to select a sub-block data unit size. Supported data unit sizes are
powers of 2 between 512 and the filesystem block size, inclusively.
Support is implemented for both the FS-layer and inline crypto cases.
This patch focuses on the basic support for sub-block data units. Some
things are out of scope for this patch but may be addressed later:
- Supporting sub-block data units in combination with
FSCRYPT_POLICY_FLAG_IV_INO_LBLK_64, in most cases. Unfortunately this
combination usually causes data unit indices to exceed 32 bits, and
thus fscrypt_supported_policy() correctly disallows it. The users who
potentially need this combination are using f2fs. To support it, f2fs
would need to provide an option to slightly reduce its max file size.
- Supporting sub-block data units in combination with
FSCRYPT_POLICY_FLAG_IV_INO_LBLK_32. This has the same problem
described above, but also it will need special code to make DUN
wraparound still happen on a FS block boundary.
- Supporting use case (2) mentioned above. The encrypted direct I/O
code will need to stop requiring and assuming FS block alignment.
This won't be hard, but it belongs in a separate patch.
- Supporting this feature on filesystems other than ext4 and f2fs.
(Filesystems declare support for it via their fscrypt_operations.)
On UBIFS, sub-block data units don't make sense because UBIFS encrypts
variable-length blocks as a result of compression. CephFS could
support it, but a bit more work would be needed to make the
fscrypt_*_block_inplace functions play nicely with sub-block data
units. I don't think there's a use case for this on CephFS anyway.
Link: https://lore.kernel.org/r/20230925055451.59499-6-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@google.com>
Actions