**************** <case 1> **************

Jul 27 15:19:31 R06-NVME-01 ceph-mds[6720]: 2023-07-27T15:19:31.561+0000 7fe50cc96700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 15:19:31 R06-NVME-01 ceph-mds[6720]: 2023-07-27T15:19:31.561+0000 7fe50cc96700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)

Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:  -3021> 2023-07-27T15:19:44.606+0000 7fe512ca2700  5 mds.0.bal try_rebalance    - exporting [pop IRD:[C 2.29e+01] IWR:[C 1.66e-01] RDR:[C 6.64e+01] FET:[C 4.92e-01] STR:[C 0.00e+00] *LOAD:90.7]
                                             90.6669 to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{5=1} v=438 cv=436/4
...
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:  -2957> 2023-07-27T15:19:44.614+0000 7fe50cc96700  3 mds.0.mig export_finish [dir 0x400028b14d3.000* /volumes/_deleting/b7ed80cf-2286-4ebd-9c1a-bdd3636f743c/0ccdd991-fba4-47e4-92e9-e1af96a4d14f/<redacted>/ [2,head] auth{1=1} v=33270 cv=31405/31405 dir_auth=1,0 state=1610613763|complete|frozentree|exporting f(v0 m2023-07-27T15:19:41.817067+0000) n(v0 rc2023-07-27T15:19:41.817067+0000) hs=0+932,ss=0+0 dirty=932 | ptrwaiter=0 request=0 child=1 frozen=1 subtree=1 replicated=1 dirty=1 waiter=0 authpin=0 tempexporting=1 0x55d4ef0fa400]
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:  -2956> 2023-07-27T15:19:44.614+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x400028b2665
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:  -2955> 2023-07-27T15:19:44.614+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x400028b1c69
<more similar lines>
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:   -236> 2023-07-27T15:19:44.634+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x5000ace8e83
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:   -235> 2023-07-27T15:19:44.634+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x5000ace90f1
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:   -234> 2023-07-27T15:19:44.634+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x5000ace91d0
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:   -233> 2023-07-27T15:19:44.634+0000 7fe50cc96700  4 mds.0.purge_queue push: pushing inode 0x5000ace91e0
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:   -232> 2023-07-27T15:19:44.650+0000 7fe50cc96700  4 mds.0.mig export_dir Starting export to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{5=1} v=438 cv=436/436 state=1610874881|complete f(v0 m2023-07-27T15:19:31.561321+0000 1=0+1) n(v26 rc2023-07-27T15:19:31.561321+0000 b14246700612 6733=5440+1293) hs=1+1,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=0 subtree=0 replicated=1 dirty=1 waiter=0 authpin=0 0x55d1adb32880
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]:     -1> 2023-07-27T15:19:44.666+0000 7fe50cc96700 -1 /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: In function 'void Migrator::encode_export_dir(ceph::bufferlist&, CDir*, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&, uint64_t&)' thread 7fe50cc96700 time 2023-07-27T15:19:44.669732+0000
Jul 27 15:19:44 R06-NVME-01 ceph-mds[6720]: /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: 1754: FAILED ceph_assert(dir->get_projected_version() == dir->get_version())

**************** <case 2/1> **************
Jul 27 15:53:34 R07-NVME-03 ceph-mds[2482685]: 2023-07-27T15:53:34.908+0000 7fcbd0c39700  0 mds.0.cache.dir(0x600012ddb0e) _fetched missing object for [dir 0x600012ddb0e /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x5572ee796880]
Jul 27 15:53:34 R07-NVME-03 ceph-mds[2482685]: 2023-07-27T15:53:34.908+0000 7fcbd0c39700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 15:53:34 R07-NVME-03 ceph-mds[2482685]: 2023-07-27T15:53:34.908+0000 7fcbd0c39700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
...
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:  -2065> 2023-07-27T15:53:41.345+0000 7fcbd0c39700  3 mds.0.mig export_finish [dir 0x10006bbbb05 /volumes/_deleting/aa39cc1d-53be-4767-870f-8fb6d5c7af33/c301a722-2091-47a9-8a27-94dd02d5d8e5/<redacted>/ [2,head] auth{1=1} v=1911 cv=0/0 dir_auth=1,0 state=1610875907|complete|frozentree|exporting f(v0 m2023-07-27T15:53:37.174112+0000 1=0+1) n(v29 rc2023-07-27T15:53:41.313911+0000 b6496123653 1371=1061+310) hs=1+16,ss=0+0 dirty=17 | ptrwaiter=0 request=0 child=1 frozen=1 subtree=1 replicated=1 dirty=1 waiter=1 authpin=0 tempexporting=1 0x5572e983d680]
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:  -2064> 2023-07-27T15:53:41.345+0000 7fcbd0c39700  4 mds.0.purge_queue push: pushing inode 0x10006bbc53b
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:  -2063> 2023-07-27T15:53:41.345+0000 7fcbd0c39700  4 mds.0.purge_queue push: pushing inode 0x10006bbc53a
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:  -2062> 2023-07-27T15:53:41.345+0000 7fcbd0c39700  4 mds.0.purge_queue push: pushing inode 0x10006bbc539
<more similar lines>
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:   -997> 2023-07-27T15:53:41.353+0000 7fcbd0c39700  4 mds.0.purge_queue push: pushing inode 0x2000255728d
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:   -996> 2023-07-27T15:53:41.353+0000 7fcbd0c39700  4 mds.0.purge_queue push: pushing inode 0x20002557280
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:   -995> 2023-07-27T15:53:41.357+0000 7fcbd0c39700  4 mds.0.mig export_dir Starting export to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{6=2} v=438 cv=0/0 state=1074003969|complete f(v0 m2023-07-27T15:19:31.561321+0000 1=0+1) n(v26 rc2023-07-27T15:19:31.561321+0000 b14246700612 6733=5440+1293) hs=1+0,ss=0+0 | child=1 replicated=1 waiter=0 authpin=0 0x5572ea76f180]
...
Jul 27 15:53:41 R07-NVME-03 ceph-mds[2482685]:     -1> 2023-07-27T15:53:41.365+0000 7fcbd0c39700 -1 /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: In function 'void Migrator::encode_export_dir(ceph::bufferlist&, CDir*, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&, uint64_t&)' thread 7fcbd0c39700 time 2023-07-27T15:53:41.368317+0000
                                               /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: 1754: FAILED ceph_assert(dir->get_projected_version() == dir->get_version())
                                               
                                                ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
                                                1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7fcbde64c657]
                                                2: /usr/lib64/ceph/libceph-common.so.2(+0x26f7f7) [0x7fcbde64c7f7]
                                                3: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x984) [0x5572e59e6264]
                                                4: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5572e59e5d75]
                                                5: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5572e59e5d75]
                                                6: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5572e59e5d75]
                                                7: (Migrator::export_go_synced(CDir*, unsigned long)+0x3e0) [0x5572e59e6690]
                                                8: (C_M_ExportGo::finish(int)+0x2e) [0x5572e59fcd7e]
                                                9: (MDSContext::complete(int)+0x5b) [0x5572e5aeee8b]
                                                10: (MDSContext::complete(int)+0x5b) [0x5572e5aeee8b]
                                                11: (MDSIOContextBase::complete(int)+0x524) [0x5572e5aef604]
                                                12: (MDSLogContextBase::complete(int)+0x41) [0x5572e5aef991]

**************** <case 2/2 > **************
Jul 27 15:54:34 R07-NVME-03 ceph-mds[2486177]: 2023-07-27T15:54:34.199+0000 7f9d21234700  0 mds.0.cache.dir(0x600012ddb0e) _fetched missing object for [dir 0x600012ddb0e /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x56133e966d00]
Jul 27 15:54:34 R07-NVME-03 ceph-mds[2486177]: 2023-07-27T15:54:34.199+0000 7f9d21234700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 15:54:34 R07-NVME-03 ceph-mds[2486177]: 2023-07-27T15:54:34.199+0000 7f9d21234700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
...
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2610> 2023-07-27T15:54:47.452+0000 7f9d27240700  5 mds.0.bal try_rebalance    - exporting [pop IRD:[C 1.02e+01] IWR:[C 0.00e+00] RDR:[C 2.89e+01] FET:[C 6.37e-01] STR:[C 0.00e+00] *LOAD:40.4]
                                                40.4126 to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{2=3} v=438 cv=0/
...
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2531> 2023-07-27T15:54:47.460+0000 7f9d21234700  3 mds.0.mig export_finish [dir 0x10006b912d2 /volumes/_deleting/2a8120a8-2fc6-4c8f-bd29-706683f513a9/ [2,head] auth{1=6} v=560 cv=514/514 dir_auth=1,0 state=1610875907|complete|frozentree|exporting f(v0 m2023-04-24T07:46:45.878408+0000 2=1+1) n(v38 rc2023-07-27T15:54:47.318771+0000 b1613941277 945=901+44)/n(v38 rc2023-07-27T15:54:47.210916+0000 b1639524630 1034=968+66) hs=2+0,ss=0+0 dirty=1 | ptrwaiter=0 request=0 child=1 frozen=1 subtree=1 replicated=1 dirty=1 waiter=1 authpin=0 tempexporting=1 0x56133ea0e900]
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2530> 2023-07-27T15:54:47.460+0000 7f9d21234700  4 mds.0.purge_queue push: pushing inode 0x3001404555e
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2529> 2023-07-27T15:54:47.460+0000 7f9d21234700  4 mds.0.purge_queue push: pushing inode 0x3001404555f
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2528> 2023-07-27T15:54:47.460+0000 7f9d21234700  4 mds.0.purge_queue push: pushing inode 0x30014045550
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:  -2527> 2023-07-27T15:54:47.460+0000 7f9d21234700  4 mds.0.purge_queue push: pushing inode 0x3001404554f
<more similar lines>
Jul 27 15:54:48 R07-NVME-03 ceph-mds[2486177]:   -101> 2023-07-27T15:54:47.476+0000 7f9d21234700  4 mds.0.purge_queue push: pushing inode 0x30014045c7b
Jul 27 15:54:48 R07-NVME-03 ceph-mds[2486177]:   -100> 2023-07-27T15:54:47.476+0000 7f9d21234700  4 mds.0.mig export_dir Starting export to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{2=3} v=438 cv=0/0 state=1074003969|complete f(v0 m2023-07-27T15:19:31.561321+0000 1=0+1) n(v26 rc2023-07-27T15:19:31.561321+0000 b14246700612 6733=5440+1293) hs=1+0,ss=0+0 | child=1 replicated=1 waiter=0 authpin=0 0x56133c4c2480]
...
Jul 27 15:54:47 R07-NVME-03 ceph-mds[2486177]:     -1> 2023-07-27T15:54:47.484+0000 7f9d21234700 -1 /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: In function 'void Migrator::encode_export_dir(ceph::bufferlist&, CDir*, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&, uint64_t&)' thread 7f9d21234700 time 2023-07-27T15:54:47.485806+0000
                                               /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: 1754: FAILED ceph_assert(dir->get_projected_version() == dir->get_version())
                                               
                                                ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
                                                1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x124) [0x7f9d2ec47657]
                                                2: /usr/lib64/ceph/libceph-common.so.2(+0x26f7f7) [0x7f9d2ec477f7]
                                                3: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x984) [0x5613351e6264]
                                                4: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5613351e5d75]
                                                5: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5613351e5d75]
                                                6: (Migrator::encode_export_dir(ceph::buffer::v15_2_0::list&, CDir*, std::map<client_t, entity_inst_t, std::less<client_t>, std::allocator<std::pair<client_t const, entity_inst_t> > >&, std::map<client_t, client_metadata_t, std::less<client_t>, std::allocator<std::pair<client_t const, client_metadata_t> > >&, unsigned long&)+0x495) [0x5613351e5d75]
                                                7: (Migrator::export_go_synced(CDir*, unsigned long)+0x3e0) [0x5613351e6690]
                                                8: (C_M_ExportGo::finish(int)+0x2e) [0x5613351fcd7e]
                                                9: (MDSContext::complete(int)+0x5b) [0x5613352eee8b]


**************** <case 3 > **************
Jul 27 15:59:44 R08-NVME-03 ceph-mds[2672598]: 2023-07-27T15:59:44.603+0000 7f479b73e700  0 mds.0.cache.dir(0x600012ddb0e) _fetched missing object for [dir 0x600012ddb0e /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x5642cfa12d80]
Jul 27 15:59:44 R08-NVME-03 ceph-mds[2672598]: 2023-07-27T15:59:44.603+0000 7f479b73e700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 15:59:44 R08-NVME-03 ceph-mds[2672598]: 2023-07-27T15:59:44.603+0000 7f479b73e700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0e object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]: /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: In function 'void Migrator::encode_export_dir(ceph::bufferlist&, CDir*, std::map<client_t, entity_inst_t>&, std::map<client_t, client_metadata_t>&, uint64_t&)' thread 7f479b73e700 time 2023-07-27T15:59:58.619365+0000
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]: /home/abuild/rpmbuild/BUILD/ceph-17.2.5/src/mds/Migrator.cc: 1754: FAILED ceph_assert(dir->get_projected_version() == dir->get_version())
...
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]:  -1458> 2023-07-27T15:59:58.600+0000 7f479b73e700  4 mds.0.purge_queue push: pushing inode 0x600007475c0
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]:  -1457> 2023-07-27T15:59:58.600+0000 7f479b73e700  4 mds.0.purge_queue push: pushing inode 0x600007475df
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]:  -1456> 2023-07-27T15:59:58.600+0000 7f479b73e700  4 mds.0.mig export_dir Starting export to mds.2 [dir 0x600012dc072 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/ [2,head] auth{4=6} v=438 cv=0/0 state=1074003969|complete f(v0 m2023-07-27T15:19:31.561321+0000 1=0+1) n(v26 rc2023-07-27T15:19:31.561321+0000 b14246700612 6733=5440+1293) hs=1+0,ss=0+0 | child=1 replicated=1 waiter=0 authpin=0 0x5642d2b23f80]
...
Jul 27 15:59:58 R08-NVME-03 ceph-mds[2672598]:      0> 2023-07-27T15:59:58.616+0000 7f479b73e700 -1 *** Caught signal (Aborted) **


*********** <first scrubbing report - unsuccessful one> ******************
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.304+0000 7f2d12fda700  0 log_channel(cluster) log [INF] : scrub summary: active paths [/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11]
Jul 27 17:21:27 R08-NVME-01 ceph-mds[526045]: 2023-07-27T17:21:27.305+0000 7f2dc5ac5700  0 log_channel(cluster) log [INF] : scrub summary: active paths [/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11]
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 mds.3.cache.dir(0x600012ddc1c) _fetched missing object for [dir 0x600012ddc1c /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55fc64f7c880]
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc1c object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc1c object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc66 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 mds.3.cache.dir(0x600012ddc66) _fetched missing object for [dir 0x600012ddc66 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55fc64f7a900]
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc66 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddc66 (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc66 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v688 ap=1 f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4) n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc3498b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc66:[<0x600012dc073/<redacted> v688>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4)","memory_value.rstat":"n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc66 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v688 ap=1 f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4) n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc3498b600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc66:[<0x600012dc073/<redacted> v688>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4)","memory_value.rstat":"n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddc1c (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc1c [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v781 ap=1 f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3) n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc061f4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc1c:[<0x600012dc073/<redacted> v781>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc1c [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v781 ap=1 f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3) n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc061f4b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc1c:[<0x600012dc073/<redacted> v781>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 mds.3.cache.dir(0x600012ddb45) _fetched missing object for [dir 0x600012ddb45 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55fc469da880]
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb45 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb45 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddb45 (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb45 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v738 ap=1 f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2) n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc43873700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb45:[<0x600012dc073/<redacted> v738>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb45 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v738 ap=1 f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2) n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23) (iversion lock) | dirfrag=0 caps=0 replicated=0 authpin=1 scrubqueue=0 0x55fc43873700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb45:[<0x600012dc073/<redacted> v738>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddb0e (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0e [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v127 ap=1 f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1) n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16) (ifile excl) (iversion lock) | request=0 lock=0 dirfrag=0 caps=0 authpin=1 scrubqueue=0 0x55fc83963b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0e:[<0x600012dd98f/Source v127>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0e [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v127 ap=1 f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1) n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16) (ifile excl) (iversion lock) | request=0 lock=0 dirfrag=0 caps=0 authpin=1 scrubqueue=0 0x55fc83963b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0e:[<0x600012dd98f/Source v127>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddb0d (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0d [c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted> auth v91 ap=1 s=973 n(v0 rc2023-05-24T16:50:54.090267+0000 b973 1=1+0) (iversion lock) | caps=0 authpin=1 scrubqueue=0 0x55fc223de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(14)0x600012ddb0d:[<0x600012dd98f/<redacted> v91>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[5]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0d [c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted> auth v91 ap=1 s=973 n(v0 rc2023-05-24T16:50:54.090267+0000 b973 1=1+0) (iversion lock) | caps=0 authpin=1 scrubqueue=0 0x55fc223de100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-61,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(14)0x600012ddb0d:[<0x600012dd98f/<redacted> v91>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[5]","error_str":"failed to read off disk; see retval"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":-61}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 mds.3.cache.dir(0x600012ddb0b) _fetched missing object for [dir 0x600012ddb0b /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55fbf3352d80]
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0b object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb0b object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddb0b (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0b [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v129 ap=1 f() n(v0 rc2023-05-24T16:50:54.090267+0000 1=0+1) (iversion lock) | dirfrag=1 caps=0 authpin=1 scrubqueue=0 0x55fc355ed600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0b:[<0x600012dd98f/Resources v129>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 rc2023-05-24T16:50:54.090267+0000 1=0+1)","error_str":""},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700 -1 mds.3.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0b [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v129 ap=1 f() n(v0 rc2023-05-24T16:50:54.090267+0000 1=0+1) (iversion lock) | dirfrag=1 caps=0 authpin=1 scrubqueue=0 0x55fc355ed600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0b:[<0x600012dd98f/Resources v129>,<<redacted> v774>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v434>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80467978>,<0x1000000001e/_deleting v52469435>,<0x1/volumes v37846655>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f()","memory_value.rstat":"n(v0 rc2023-05-24T16:50:54.090267+0000 1=0+1)","error_str":""},"return_code":-2}
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.308+0000 7f2d0cfce700  0 log_channel(cluster) log [WRN] : Scrub error on inode 0x600012ddb0c (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>) see mds.nvme-03 log and `damage ls` output for details
<more stuff like above>
Jul 27 17:21:27 R07-NVME-04 ceph-mds[7837]: 2023-07-27T17:21:27.912+0000 7f2d0cfce700  0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11]



*********** <second scrubbing report - no more errors afterwards> ******************

Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.057+0000 7f0b13f88700  0 log_channel(cluster) log [INF] : scrub summary: active paths [/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 mds.1.cache.dir(0x600012ddc66) _fetched missing object for [dir 0x600012ddc66 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55768e597200]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc66 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc66 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc1c object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 mds.1.cache.dir(0x600012ddc1c) _fetched missing object for [dir 0x600012ddc1c /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55768e596900]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc66 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v688 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4) n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=1 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55768ed79700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc66:[<0x600012dc073/<redacted> v688>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4)","memory_value.rstat":"n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddc1c object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc1c [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v781 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3) n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=0 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55768ed78100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc1c:[<0x600012dc073/<redacted> v781>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012ddc66(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0e [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth{7=2} v127 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1) n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16) (inest mix) (ifile sync->lock g=7) (iversion lock) caps={10046157676=pAsXs/pAsXsFs/-@2930},l=10046157676 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=1 replicated=1 waiter=1 authpin=1 scrubqueue=0 0x55768ca3d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0e:[<0x600012dd98f/Source v127>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}ul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012ddc66 (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb45 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc66 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v688 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4) n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=1 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55768ed79700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc66:[<0x600012dc073/<redacted> v688>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.914268+0000 6=2+4)","memory_value.rstat":"n(v5 rc2023-05-24T16:50:54.146267+0000 b2547768 79=68+11)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012ddc1c(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012ddc1c (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddc1c [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v781 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3) n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=0 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55768ed78100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddc1c:[<0x600012dc073/<redacted> v781>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.866268+0000 6=3+3)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.122267+0000 b19454005 74=40+34)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb45 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v738 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2) n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=0 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55769332b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb45:[<0x600012dc073/<redacted> v738>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012dda11 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012ddb0e(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012dda3a object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012ddb0e (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb0e [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth{7=2} v127 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1) n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16) (inest mix) (ifile sync->lock g=7) (iversion lock) caps={10046157676=pAsXs/pAsXsFs/-@2930},l=10046157676 | request=0 lock=0 importingcaps=0 dirfrag=1 caps=1 dirtyparent=1 replicated=1 waiter=1 authpin=1 scrubqueue=0 0x55768ca3d600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb0e:[<0x600012dd98f/Source v127>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.734273+0000 1=0+1)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:53.982268+0000 b245195 55=39+16)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012dda11 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment/ auth v299 pv374 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.294275+0000 6=3+3) n(v4 rc2023-05-24T16:50:54.062267+0000 b16313846 41=36+5) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 replicated=0 waiter=1 authpin=1 scrubqueue=0 0x55768ca3e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012dda11:[<0x600012dd993/Environment v299>,<0x600012dd992/AdvancedLocomotionV4 v140>,<0x600012dd98f/Content v153>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.294275+0000 6=3+3)","memory_value.rstat":"n(v4 rc2023-05-24T16:50:54.062267+0000 b16313846 41=36+5)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 mds.1.cache.dir(0x600012ddb45) _fetched missing object for [dir 0x600012ddb45 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55768dbe9b00]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012dda3a [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets/ auth v309 pv376 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.326275+0000 1=0+1) n(v8 rc2023-05-24T16:50:54.050267+0000 b40063144 209=181+28) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 replicated=0 waiter=1 authpin=1 scrubqueue=0 0x55768ca3c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012dda3a:[<0x600012dd993/CharacterAssets v309>,<0x600012dd992/AdvancedLocomotionV4 v140>,<0x600012dd98f/Content v153>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.326275+0000 1=0+1)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.050267+0000 b40063144 209=181+28)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012ddb45 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012ddb45(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012ddb45 (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012ddb45 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/ auth v738 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2) n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23) (ifile sync->lock) (iversion lock) caps={10046157676=pAsLsXs/pAsLsXsFs/-@2408,10370541489=pAsLsXs/pAsLsXsFs/-@2} | dirfrag=0 caps=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x55769332b080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012ddb45:[<0x600012dc073/<redacted> v738>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:53.366270+0000 5=3+2)","memory_value.rstat":"n(v9 rc2023-05-24T16:50:54.302266+0000 b99375761 215=192+23)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 mds.1.cache.dir(0x600012dda11) _fetched missing object for [dir 0x600012dda11 /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55768b17b200]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012dda11 object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 mds.1.cache.dir(0x600012dda3a) _fetched missing object for [dir 0x600012dda3a /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets/ [2,head] auth v=0 cv=0/0 ap=1+0 state=1073741888|fetching f() n() hs=0+0,ss=0+0 | waiter=1 authpin=1 0x55768b17b680]
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 log_channel(cluster) log [ERR] : dir 0x600012dda3a object missing on disk; some files may be lost (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012dda11(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012dda11 (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012dda11 [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/Environment/ auth v299 pv374 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.294275+0000 6=3+3) n(v4 rc2023-05-24T16:50:54.062267+0000 b16313846 41=36+5) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 replicated=0 waiter=1 authpin=1 scrubqueue=0 0x55768ca3e100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012dda11:[<0x600012dd993/Environment v299>,<0x600012dd992/AdvancedLocomotionV4 v140>,<0x600012dd98f/Content v153>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.294275+0000 6=3+3)","memory_value.rstat":"n(v4 rc2023-05-24T16:50:54.062267+0000 b16313846 41=36+5)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [WRN] : bad backtrace on inode 0x600012dda3a(/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets), rewriting it
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : Scrub repaired inode 0x600012dda3a (/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets)
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x600012dda3a [...c,head] /volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626/<redacted>/CharacterAssets/ auth v309 pv376 ap=3 DIRTYPARENT f(v0 m2023-05-24T16:50:52.326275+0000 1=0+1) n(v8 rc2023-05-24T16:50:54.050267+0000 b40063144 209=181+28) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyparent=1 replicated=0 waiter=1 authpin=1 scrubqueue=0 0x55768ca3c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(4)0x600012dda3a:[<0x600012dd993/CharacterAssets v309>,<0x600012dd992/AdvancedLocomotionV4 v140>,<0x600012dd98f/Content v153>,<<redacted> v851>,<0x600012dc072/db5a8e9a-e491-4ca8-a8ec-0b47f8c19626 v469>,<0x10000000030/8aa153c0-53d8-41c9-be90-270ad4a91c11 v80470270>,<0x1000000001e/_deleting v52474612>,<0x1/volumes v37849970>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2023-05-24T16:50:52.326275+0000 1=0+1)","memory_value.rstat":"n(v8 rc2023-05-24T16:50:54.050267+0000 b40063144 209=181+28)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2}
Jul 27 21:06:25 R08-NVME-05 ceph-mds[1511634]: 2023-07-27T21:06:25.061+0000 7f0b0df7c700  0 log_channel(cluster) log [INF] : scrub summary: idle+waiting paths [/volumes/_deleting/8aa153c0-53d8-41c9-be90-270ad4a91c11]