Project

General

Profile

Bug #55494 ยป trace.txt

Nikola Ciprich, 04/29/2022 07:41 AM

 
Panic#2 Part1
<4>[3198386.207152] rbd: rbd1: write result -2
<3>[3198386.209744] blk_update_request: I/O error, dev rbd1, sector 1320280064 op 0x1:(WRITE) flags 0x104000 phys_seg 1024 prio class 0
<4>[3198386.216627] rbd: rbd1: write result -2
<3>[3198386.219558] blk_update_request: I/O error, dev rbd1, sector 1320288256 op 0x1:(WRITE) flags 0x104000 phys_seg 1024 prio class 0
<4>[3198386.227083] rbd: rbd1: write result -2
<3>[3198386.230352] blk_update_request: I/O error, dev rbd1, sector 1320296448 op 0x1:(WRITE) flags 0x104000 phys_seg 1024 prio class 0
<4>[3198386.238447] rbd: rbd1: write result -2
<3>[3198386.242067] blk_update_request: I/O error, dev rbd1, sector 1320312832 op 0x1:(WRITE) flags 0x104000 phys_seg 1024 prio class 0
<4>[3198386.250877] rbd: rbd1: write result -2
<3>[3198386.254811] blk_update_request: I/O error, dev rbd1, sector 1320304640 op 0x1:(WRITE) flags 0x104000 phys_seg 1024 prio class 0
<4>[3198386.263459] rbd: rbd1: write result -2
<4>[3198386.268129] rbd: rbd1: write result -2
<4>[3198386.272803] rbd: rbd1: write result -2
<4>[3198386.277511] rbd: rbd1: write result -2
<4>[3198386.282053] rbd: rbd1: write result -2
<4>[3198386.286741] rbd: rbd1: write result -2
<4>[3198386.291493] rbd: rbd1: write result -2
<4>[3198386.296500] rbd: rbd1: write result -2
<4>[3198386.300868] rbd: rbd1: write result -2
<4>[3198386.305294] rbd: rbd1: write result -2
<4>[3198386.309726] rbd: rbd1: write result -2
<4>[3198386.314050] rbd: rbd1: write result -2
<4>[3198386.317916] rbd: rbd1: write result -2
<4>[3198386.321879] rbd: rbd1: write result -2
<4>[3198386.325861] rbd: rbd1: write result -2
<4>[3198386.329779] rbd: rbd1: write result -2
<4>[3198386.333917] rbd: rbd1: write result -2
<4>[3198386.337581] rbd: rbd1: write result -2
<4>[3198386.341464] rbd: rbd1: write result -2
<4>[3198386.345286] rbd: rbd1: write result -2
<4>[3198386.349287] rbd: rbd1: write result -2
<4>[3198386.353041] rbd: rbd1: write result -2
<4>[3198386.356900] rbd: rbd1: write result -2
<4>[3198386.360717] rbd: rbd1: write result -2
<4>[3198386.365244] rbd: rbd1: write result -2
<4>[3198386.369624] rbd: rbd1: write result -2
<4>[3198386.374529] rbd: rbd1: write result -2
<4>[3198386.378690] rbd: rbd1: write result -2
<4>[3198386.382728] rbd: rbd1: write result -2
<4>[3198386.386591] rbd: rbd1: write result -2
<4>[3198386.390447] rbd: rbd1: write result -2
<4>[3198386.394016] rbd: rbd1: write result -2
<4>[3198386.397477] rbd: rbd1: write result -2
<4>[3198386.400973] rbd: rbd1: write result -2
<4>[3198386.404304] rbd: rbd1: write result -2
<4>[3198386.407563] rbd: rbd1: write result -2
<4>[3198386.411291] rbd: rbd1: write result -2
<4>[3198386.414708] rbd: rbd1: write result -2
<4>[3198386.417930] rbd: rbd1: write result -2
<4>[3198386.420666] rbd: rbd1: write result -2
<4>[3198386.422967] rbd: rbd1: write result -2
<4>[3198386.425423] rbd: rbd1: write result -2
<4>[3198386.427811] rbd: rbd1: write result -2
<4>[3198386.429896] rbd: rbd1: write result -2
<4>[3198386.432089] rbd: rbd1: write result -2
<4>[3198386.434194] rbd: rbd1: write result -2
<4>[3198386.436868] rbd: rbd1: write result -2
<4>[3198386.439408] rbd: rbd1: write result -2
<4>[3198386.441776] rbd: rbd1: write result -2
<4>[3198386.444201] rbd: rbd1: write result -2
<4>[3198386.446748] rbd: rbd1: write result -2
<4>[3198386.448330] rbd: rbd1: write result -2
<4>[3198386.450476] rbd: rbd1: write result -2
<4>[3198386.452245] rbd: rbd1: write result -2
<4>[3198386.454445] rbd: rbd1: write result -2
<4>[3198386.456814] rbd: rbd1: write result -2
<4>[3198386.459198] rbd: rbd1: write result -2
<4>[3198386.461565] rbd: rbd1: write result -2
<4>[3198386.463975] rbd: rbd1: write result -2
<4>[3198386.466331] rbd: rbd1: write result -2
<4>[3198386.468601] rbd: rbd1: write result -2
<4>[3198386.470343] rbd: rbd1: write result -2
<4>[3198386.472102] rbd: rbd1: write result -2
<4>[3198386.473824] rbd: rbd1: write result -2
<4>[3198386.475545] rbd: rbd1: write result -2
<4>[3198386.477277] rbd: rbd1: write result -2
<4>[3198386.479018] rbd: rbd1: write result -2
<4>[3198386.480703] rbd: rbd1: write result -2
<4>[3198386.483048] rbd: rbd1: write result -2
<4>[3198386.485102] rbd: rbd1: write result -2
<4>[3198386.487327] rbd: rbd1: write result -2
<4>[3198386.489713] rbd: rbd1: write result -2
<4>[3198386.492085] rbd: rbd1: write result -2
<4>[3198386.494471] rbd: rbd1: write result -2
<4>[3198386.496848] rbd: rbd1: write result -2
<4>[3198386.499181] rbd: rbd1: write result -2
<4>[3198386.501541] rbd: rbd1: write result -2
<4>[3198386.503870] rbd: rbd1: write result -2
<4>[3198386.505974] rbd: rbd1: write result -2
<4>[3198386.507731] rbd: rbd1: write result -2
<4>[3198386.509865] rbd: rbd1: write result -2
<4>[3198386.512228] rbd: rbd1: write result -2
<4>[3198386.514598] rbd: rbd1: write result -2
<4>[3198386.516997] rbd: rbd1: write result -2
<4>[3198386.521915] rbd: rbd1: post-acquire action failed: -2
<4>[3198386.523218] rbd: rbd1: write result -2
<4>[3198386.525120] rbd: rbd1: write result -2
<4>[3198386.526986] rbd: rbd1: write result -2
<4>[3198386.528888] rbd: rbd1: write result -2
<4>[3198386.530814] rbd: rbd1: write result -2
<4>[3198386.532660] rbd: rbd1: write result -2
<4>[3198386.534540] rbd: rbd1: write result -2
<4>[3198386.535671] rbd: rbd1: write result -2
<4>[3198386.536806] rbd: rbd1: write result -2
<4>[3198386.538176] rbd: rbd1: write result -2
<4>[3198386.539538] rbd: rbd1: write result -2
<4>[3198386.540871] rbd: rbd1: write result -2
<4>[3198386.542239] rbd: rbd1: write result -2
<4>[3198386.543563] rbd: rbd1: write result -2
<4>[3198386.544698] rbd: rbd1: write result -2
<4>[3198386.545831] rbd: rbd1: write result -2
<4>[3198386.546850] rbd: rbd1: write result -2
<4>[3198386.547850] rbd: rbd1: write result -2
<4>[3198386.548955] rbd: rbd1: write result -2
<4>[3198386.550061] rbd: rbd1: write result -2
<4>[3198386.551188] rbd: rbd1: write result -2
<4>[3198386.552309] rbd: rbd1: write result -2
<4>[3198386.553466] rbd: rbd1: write result -2
<4>[3198386.554645] rbd: rbd1: write result -2
<4>[3198386.556109] rbd: rbd1: write result -2
<4>[3198386.557526] rbd: rbd1: write result -2
<4>[3198386.558916] rbd: rbd1: write result -2
<4>[3198386.560350] rbd: rbd1: write result -2
<4>[3198386.561769] rbd: rbd1: write result -2
<4>[3198386.563176] rbd: rbd1: write result -2
<4>[3198386.564601] rbd: rbd1: write result -2
<4>[3198386.566031] rbd: rbd1: write result -2
<4>[3198386.567239] rbd: rbd1: write result -2
<4>[3198386.568434] rbd: rbd1: write result -2
<4>[3198386.569403] rbd: rbd1: write result -2
<4>[3198386.570386] rbd: rbd1: write result -2
<4>[3198386.571366] rbd: rbd1: write result -2
<4>[3198386.572510] rbd: rbd1: write result -2
<4>[3198386.573634] rbd: rbd1: write result -2
<4>[3198386.574789] rbd: rbd1: write result -2
<4>[3198386.576166] rbd: rbd1: write result -2
<4>[3198386.577530] rbd: rbd1: write result -2
<4>[3198386.581289] rbd: rbd1: post-acquire action failed: -2
<4>[3198386.582462] rbd: rbd1: write result -2
<4>[3198386.583740] rbd: rbd1: write result -2
<4>[3198386.584950] rbd: rbd1: write result -2
<4>[3198386.586163] rbd: rbd1: write result -2
<4>[3198386.587368] rbd: rbd1: write result -2
<4>[3198386.588486] rbd: rbd1: write result -2
<4>[3198386.589600] rbd: rbd1: write result -2
<4>[3198386.590657] rbd: rbd1: write result -2
<4>[3198386.591577] rbd: rbd1: write result -2
<4>[3198386.592679] rbd: rbd1: write result -2
<4>[3198386.593834] rbd: rbd1: write result -2
<4>[3198386.595139] rbd: rbd1: write result -2
<4>[3198386.596477] rbd: rbd1: write result -2
<4>[3198386.597790] rbd: rbd1: write result -2
<4>[3198386.599132] rbd: rbd1: write result -2
<4>[3198386.600458] rbd: rbd1: write result -2
<4>[3198386.601612] rbd: rbd1: write result -2
<4>[3198386.602705] rbd: rbd1: write result -2
<4>[3198386.603814] rbd: rbd1: write result -2
<4>[3198386.604916] rbd: rbd1: write result -2
<4>[3198386.606021] rbd: rbd1: write result -2
<4>[3198386.607133] rbd: rbd1: write result -2
<4>[3198386.608244] rbd: rbd1: write result -2
<4>[3198386.609358] rbd: rbd1: write result -2
<4>[3198386.610719] rbd: rbd1: write result -2
<4>[3198386.612071] rbd: rbd1: write result -2
<4>[3198386.613372] rbd: rbd1: write result -2
<4>[3198386.614640] rbd: rbd1: write result -2
<4>[3198386.615734] rbd: rbd1: write result -2
<4>[3198386.616777] rbd: rbd1: write result -2
<4>[3198386.617847] rbd: rbd1: write result -2
<4>[3198386.619020] rbd: rbd1: write result -2
<4>[3198386.620058] rbd: rbd1: write result -2
<4>[3198386.621085] rbd: rbd1: write result -2
<4>[3198386.622126] rbd: rbd1: write result -2
<4>[3198386.623159] rbd: rbd1: write result -2
<4>[3198386.624193] rbd: rbd1: write result -2
<4>[3198386.625236] rbd: rbd1: write result -2
<4>[3198386.626287] rbd: rbd1: write result -2
<4>[3198386.627319] rbd: rbd1: write result -2
<4>[3198386.628597] rbd: rbd1: write result -2
<4>[3198386.629942] rbd: rbd1: write result -2
<4>[3198386.631276] rbd: rbd1: write result -2
<4>[3198386.632564] rbd: rbd1: write result -2
<4>[3198386.633871] rbd: rbd1: write result -2
<4>[3198386.635183] rbd: rbd1: write result -2
<4>[3198386.636514] rbd: rbd1: write result -2
<4>[3198386.637832] rbd: rbd1: write result -2
<4>[3198386.639100] rbd: rbd1: write result -2
<4>[3198386.640370] rbd: rbd1: write result -2
<4>[3198386.641255] rbd: rbd1: write result -2
<4>[3198386.642312] rbd: rbd1: write result -2
<4>[3198386.643425] rbd: rbd1: write result -2
<4>[3198386.645241] rbd: rbd1: write result -2
<4>[3198386.647022] rbd: rbd1: write result -2
<4>[3198386.648937] rbd: rbd1: write result -2
<4>[3198386.650985] rbd: rbd1: write result -2
<4>[3198386.652769] rbd: rbd1: write result -2
<4>[3198386.654595] rbd: rbd1: write result -2
<4>[3198386.656962] rbd: rbd1: write result -2
<4>[3198386.659373] rbd: rbd1: write result -2
<4>[3198386.661788] rbd: rbd1: write result -2
<4>[3198386.664136] rbd: rbd1: write result -2
<4>[3198386.666538] rbd: rbd1: write result -2
<4>[3198386.669022] rbd: rbd1: write result -2
<4>[3198386.671424] rbd: rbd1: write result -2
<4>[3198386.673813] rbd: rbd1: write result -2
<4>[3198386.676200] rbd: rbd1: write result -2
<4>[3198386.678667] rbd: rbd1: write result -2
<4>[3198386.681130] rbd: rbd1: write result -2
<4>[3198386.683601] rbd: rbd1: write result -2
<4>[3198386.686098] rbd: rbd1: write result -2
<4>[3198386.688544] rbd: rbd1: write result -2
<4>[3198386.690989] rbd: rbd1: write result -2
<4>[3198386.692616] rbd: rbd1: write result -2
<4>[3198386.693994] rbd: rbd1: write result -2
<4>[3198386.695018] rbd: rbd1: write result -2
<4>[3198386.695851] rbd: rbd1: write result -2
<4>[3198386.696675] rbd: rbd1: write result -2
<4>[3198386.697548] rbd: rbd1: write result -2
<4>[3198386.698610] rbd: rbd1: write result -2
<4>[3198386.699718] rbd: rbd1: write result -2
<4>[3198386.700864] rbd: rbd1: write result -2
<4>[3198386.702024] rbd: rbd1: write result -2
<4>[3198386.703399] rbd: rbd1: write result -2
<4>[3198386.704725] rbd: rbd1: write result -2
<4>[3198386.706047] rbd: rbd1: write result -2
<4>[3198386.707346] rbd: rbd1: write result -2
<4>[3198386.708692] rbd: rbd1: write result -2
<4>[3198386.710005] rbd: rbd1: write result -2
<4>[3198386.711335] rbd: rbd1: write result -2
<4>[3198386.712653] rbd: rbd1: write result -2
<4>[3198386.713992] rbd: rbd1: write result -2
<4>[3198386.715270] rbd: rbd1: write result -2
<4>[3198386.716344] rbd: rbd1: write result -2
<4>[3198386.717422] rbd: rbd1: write result -2
<4>[3198386.718499] rbd: rbd1: write result -2
<4>[3198386.719563] rbd: rbd1: write result -2
<4>[3198386.720436] rbd: rbd1: write result -2
<4>[3198386.721324] rbd: rbd1: write result -2
<4>[3198386.722129] rbd: rbd1: write result -2
<4>[3198386.723167] rbd: rbd1: write result -2
<4>[3198386.724384] rbd: rbd1: write result -2
<4>[3198386.725586] rbd: rbd1: write result -2
<4>[3198386.726783] rbd: rbd1: write result -2
<4>[3198386.728004] rbd: rbd1: write result -2
<4>[3198386.729224] rbd: rbd1: write result -2
<4>[3198386.730436] rbd: rbd1: write result -2
<4>[3198386.731626] rbd: rbd1: write result -2
<4>[3198386.732823] rbd: rbd1: write result -2
<4>[3198386.734035] rbd: rbd1: write result -2
<4>[3198386.735269] rbd: rbd1: write result -2
<4>[3198386.736458] rbd: rbd1: write result -2
<4>[3198386.737471] rbd: rbd1: write result -2
<4>[3198386.738437] rbd: rbd1: write result -2
<4>[3198386.739413] rbd: rbd1: write result -2
<4>[3198386.740383] rbd: rbd1: write result -2
<4>[3198386.741328] rbd: rbd1: write result -2
<4>[3198386.742340] rbd: rbd1: write result -2
<4>[3198386.743326] rbd: rbd1: write result -2
<4>[3198386.744270] rbd: rbd1: write result -2
<4>[3198386.745534] rbd: rbd1: write result -2
<4>[3198386.746503] rbd: rbd1: write result -2
<4>[3198386.747447] rbd: rbd1: write result -2
<4>[3198386.748449] rbd: rbd1: write result -2
<4>[3198386.752080] rbd: rbd1: post-acquire action failed: -2
<4>[3198386.753347] rbd: rbd1: write result -2
<3>[3198386.754286]
<3>[3198386.754286] Assertion failure in rbd_try_acquire_lock() at line 4183:
<3>[3198386.754286]
<3>[3198386.754286] rbd_assert(list_empty(&rbd_dev->running_list));
<3>[3198386.754286]
<4>[3198386.758661] ------------[ cut here ]------------
<2>[3198386.759696] kernel BUG at drivers/block/rbd.c:4183!
<4>[3198386.760807] invalid opcode: 0000 [#1] SMP NOPTI
<4>[3198386.761974] CPU: 17 PID: 4146507 Comm: kworker/u64:0 Tainted: G E 5.4.173lb7.01 #1
<4>[3198386.763307] Hardware name: Supermicro AS -2024US-TRT/H12DSU-iN, BIOS 2.1 05/07/2021
<4>[3198386.764723] Workqueue: rbd1-tasks rbd_acquire_lock [rbd]
<4>[3198386.766439] RIP: 0010:rbd_acquire_lock+0xc66/0xca0 [rbd]
<4>[3198386.767936] Code: ea ff ff ff e9 63 fd ff ff 48 c7 c1 88 22 b2 a0 ba 57 10 00 00 48 c7 c6 c0 42 b2 a0 48 c7 c7 e8 1f b2 a0 31 c0 e8 b4 ab cb e0 <0f> 0b 48 c7 c1 60 25 b2 a0 ba 56 10 00 00 48 c7 c6 c0 42 b2 a0 48
<4>[3198386.772498] RSP: 0018:ffffc90030837d58 EFLAGS: 00010246
<4>[3198386.774367] RAX: 000000000000006d RBX: ffff88bf02fe2760 RCX: 0000000000000000
<4>[3198386.776340] RDX: 0000000000000000 RSI: ffff88ff4f45c878 RDI: ffff88ff4f45c878
<4>[3198386.778375] RBP: ffff88befcbb6d80 R08: 00000000000290ef R09: 00000000000290ef
<4>[3198386.780449] R10: 0000000000000040 R11: 000003b400000000 R12: ffff88bf4e438400
<4>[3198386.782573] R13: ffff88bea1072000 R14: 0000000000000000 R15: ffff88977af13a60
<4>[3198386.784747] FS: 0000000000000000(0000) GS:ffff88ff4f440000(0000) knlGS:0000000000000000
<4>[3198386.787034] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[3198386.789381] CR2: 00000000775084a0 CR3: 0000003f24246004 CR4: 0000000000760ee0
<4>[3198386.791846] PKRU: 55555554
<4>[3198386.794335] Call Trace:
<4>[3198386.796858] ? __rbd_notify_op_lock+0xa1/0x100 [rbd]
<4>[3198386.799460] ? free_unref_page_commit+0x85/0xf0
<4>[3198386.802092] process_one_work+0x171/0x380
<4>[3198386.804728] worker_thread+0x49/0x3f0
<4>[3198386.807374] kthread+0xf8/0x130
<4>[3198386.810113] ? max_active_store+0x80/0x80
<4>[3198386.812888] ? kthread_bind+0x10/0x10
<4>[3198386.815682] ret_from_fork+0x22/0x40
<4>[3198386.818502] Modules linked in: loop(E) ip_set(E) netconsole(E) rbd(E) ceph(E) libceph(E) dns_resolver(E) ib_isert(E) iscsi_target_mod(E) openvswitch(E) ib_srpt(E) nf_conncount(E) nsh(E) target_core_mod(E) nfnetlink_log(E) nfnetlink(E) xt_NFLOG(E) xt_limit(E) xt_comment(E) xt_conntrack(E) ib_srp(E) xt_multiport(E) scsi_transport_srp(E) bpfilter(E) nf_nat_ftp(E) nf_conntrack_ftp(E) ip6table_filter(E) ip6_tables(E) iptable_mangle(E) iptable_nat(E) nf_nat(E) nf_conntrack(E) nf_defrag_ipv6(E) libcrc32c(E) nf_defrag_ipv4(E) iptable_filter(E) i40iw(E) rpcrdma(E) rdma_ucm(E) ib_iser(E) ib_umad(E) rdma_cm(E) ib_ipoib(E) iw_cm(E) sunrpc(E) vfat(E) fat(E) libiscsi(E) scsi_transport_iscsi(E) ib_cm(E) wmi_bmof(E) crc32_pclmul(E) aesni_intel(E) glue_helper(E) crypto_simd(E) ghash_clmulni_intel(E) cryptd(E) crct10dif_pclmul(E) pcspkr(E) mlx5_ib(E) ib_uverbs(E) i2c_piix4(E) ib_core(E) i2c_core(E) sg(E) sp5100_tco(E) wmi(E) ipmi_si(E) ipmi_devintf(E) ipmi_msghandler(E) acpi_cpufreq(E) vhost_net(E) tun(E)
<4>[3198386.818545] vhost(E) tap(E) kvm_amd(E) kvm(E) irqbypass(E) ip_tables(E) ext4(E) mbcache(E) jbd2(E) raid1(E) sd_mod(E) crc32c_intel(E) ahci(E) libahci(E) mpt3sas(E) libata(E) xhci_pci(E) i40e(E) raid_class(E) mlx5_core(E) scsi_transport_sas(E) mlxfw(E) xhci_hcd(E) pci_hyperv_intf(E) dm_mirror(E) dm_region_hash(E) dm_log(E) dm_mod(E) dax(E)
<4>[3198386.861162] ---[ end trace d0ee1ebb03c58c8a ]---
<4>[3198387.024133] RIP: 0010:rbd_acquire_lock+0xc66/0xca0 [rbd]
<4>[3198387.029189] Code: ea ff ff ff e9 63 fd ff ff 48 c7 c1 88 22 b2 a0 ba 57 10 00 00 48 c7 c6 c0 42 b2 a0 48 c7 c7 e8 1f b2 a0 31 c0 e8 b4 ab cb e0 <0f> 0b 48 c7 c1 60 25 b2 a0 ba 56 10 00 00 48 c7 c6 c0 42 b2 a0 48
<4>[3198387.039006] RSP: 0018:ffffc90030837d58 EFLAGS: 00010246
<4>[3198387.043933] RAX: 000000000000006d RBX: ffff88bf02fe2760 RCX: 0000000000000000
<4>[3198387.048921] RDX: 0000000000000000 RSI: ffff88ff4f45c878 RDI: ffff88ff4f45c878
<4>[3198387.053887] RBP: ffff88befcbb6d80 R08: 00000000000290ef R09: 00000000000290ef
<4>[3198387.058855] R10: 0000000000000040 R11: 000003b400000000 R12: ffff88bf4e438400
<4>[3198387.063840] R13: ffff88bea1072000 R14: 0000000000000000 R15: ffff88977af13a60
<4>[3198387.068811] FS: 0000000000000000(0000) GS:ffff88ff4f440000(0000) knlGS:0000000000000000
<4>[3198387.073842] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>[3198387.078836] CR2: 00000000775084a0 CR3: 0000003f24246004 CR4: 0000000000760ee0
<4>[3198387.083808] PKRU: 55555554
<0>[3198387.088712] Kernel panic - not syncing: Fatal exception
<0>[3198387.093698] Kernel Offset: disabled
    (1-1/1)