Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2024-03-29T12:54:10ZCeph
Redmine RADOS - Bug #65227 (New): noscrub cluster flag prevents deep-scrubs from startinghttps://tracker.ceph.com/issues/652272024-03-29T12:54:10ZWes Dillinghamwes@wesdillingham.com
<p>Observed on a 17.2.7 cluster and confirmed on an additional 17.2.7 cluster.</p>
<p>Reproduction steps:<br />- On a cluster that reliably will have scrubs in flight set both the noscrub and nodeep-scrub flags.<br />- As expected ongoing scrubs and deep-scrubs will begin to be cancelled immediately and PGs that were scrubbing will be no longer.<br />- Unset the nodeep-scrub flag only (leave noscrub set) - at this point I would expect deep scrubs to be allowed but shallow scrubs not to be.<br />- Unset the "noscrub" flag as well. At this point deep-scrubs will begin on the cluster.</p>
<p>Expected behavior:<br />"noscrub" flag prevents shallow (non-deep) scrubs from starting but does not control deep scrubs.</p> Ceph - Bug #65226 (Fix Under Review): qa: add a test - peer status show "failed" status for makin...https://tracker.ceph.com/issues/652262024-03-29T10:45:28ZJos CollinCephFS - Bug #65225 (New): ceph_assert on dn->get_projected_linkage()->is_remote https://tracker.ceph.com/issues/652252024-03-29T10:16:03ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<p>In a workload that is heavily hardlinking and moving files, we see ceph-mds assert like the following</p>
<pre>
16.2.9/src/mds/Locker.cc: In function 'bool Locker::acquire_locks(MDRequestRef&, Mutati> /var/log/ceph/ceph-mds.minilevinson-983db664a3.log-20240329.gz
ceph-mds[2693077]: /builddir/build/BUILD/ceph-16.2.9/src/mds/Locker.cc: 404: FAILED ceph_assert(dn->get_projected_linkage()->is_remot>
ceph-mds[2693077]: ceph version 16.2.9-2 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
ceph-mds[2693077]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f40e912ed98]
ceph-mds[2693077]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x276fb2) [0x7f40e912efb2]
ceph-mds[2693077]: 3: (Locker::acquire_locks(boost::intrusive_ptr<MDRequestImpl>&, MutationImpl::LockOpVec&, CInode*, bool)+0x131e) >
ceph-mds[2693077]: 4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0xb43) [0x55eeadce4f73]
ceph-mds[2693077]: 5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xf1b) [0x55eeadd12d3b]
ceph-mds[2693077]: 6: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0x3fc) [0x55eeadd1348c]
ceph-mds[2693077]: 7: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x12b) [0x55eeadd179fb]
ceph-mds[2693077]: 8: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbb4) [0x55eeadc6e224]
ceph-mds[2693077]: 9: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55eeadc70bdb]
ceph-mds[2693077]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55eeadc711d5]
ceph-mds[2693077]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x108) [0x55eeadc60dc8]
ceph-mds[2693077]: 12: (DispatchQueue::entry()+0x126a) [0x7f40e9374b5a]
ceph-mds[2693077]: 13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e9426841]
ceph-mds[2693077]: 14: /lib64/libpthread.so.0(+0x81ca) [0x7f40e810f1ca]
</pre>
<p>The full logs look like the following<br /><pre>
-1 /builddir/build/BUILD/ceph-16.2.9/src/mds/Locker.cc: In function 'bool Locker::acquire_locks(MDRequestRef&, MutationImpl::LockOpV
ec&, CInode*, bool)' thread 7f40e06ef700 time 2024-03-28T19:20:43.701971+0100
/builddir/build/BUILD/ceph-16.2.9/src/mds/Locker.cc: 404: FAILED ceph_assert(dn->get_projected_linkage()->is_remote())
ceph version 16.2.9-2 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x158) [0x7f40e912ed98]
2: /usr/lib64/ceph/libceph-common.so.2(+0x276fb2) [0x7f40e912efb2]
3: (Locker::acquire_locks(boost::intrusive_ptr<MDRequestImpl>&, MutationImpl::LockOpVec&, CInode*, bool)+0x131e) [0x55eeade5669e]
4: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0xb43) [0x55eeadce4f73]
5: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xf1b) [0x55eeadd12d3b]
6: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0x3fc) [0x55eeadd1348c]
7: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x12b) [0x55eeadd179fb]
8: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbb4) [0x55eeadc6e224]
9: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55eeadc70bdb]
10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55eeadc711d5]
11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x108) [0x55eeadc60dc8]
12: (DispatchQueue::entry()+0x126a) [0x7f40e9374b5a]
13: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e9426841]
14: /lib64/libpthread.so.0(+0x81ca) [0x7f40e810f1ca]
15: clone()
0> 2024-03-28T19:20:43.709+0100 7f40e06ef700 -1 *** Caught signal (Aborted) **
in thread 7f40e06ef700 thread_name:ms_dispatch
ceph version 16.2.9-2 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
1: /lib64/libpthread.so.0(+0x12cf0) [0x7f40e8119cf0]
2: gsignal()
3: abort()
4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f40e912ede9]
5: /usr/lib64/ceph/libceph-common.so.2(+0x276fb2) [0x7f40e912efb2]
6: (Locker::acquire_locks(boost::intrusive_ptr<MDRequestImpl>&, MutationImpl::LockOpVec&, CInode*, bool)+0x131e) [0x55eeade5669e]
7: (Server::handle_client_unlink(boost::intrusive_ptr<MDRequestImpl>&)+0xb43) [0x55eeadce4f73]
8: (Server::dispatch_client_request(boost::intrusive_ptr<MDRequestImpl>&)+0xf1b) [0x55eeadd12d3b]
9: (Server::handle_client_request(boost::intrusive_ptr<MClientRequest const> const&)+0x3fc) [0x55eeadd1348c]
10: (Server::dispatch(boost::intrusive_ptr<Message const> const&)+0x12b) [0x55eeadd179fb]
11: (MDSRank::handle_message(boost::intrusive_ptr<Message const> const&)+0xbb4) [0x55eeadc6e224]
12: (MDSRank::_dispatch(boost::intrusive_ptr<Message const> const&, bool)+0x7bb) [0x55eeadc70bdb]
13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr<Message const> const&)+0x55) [0x55eeadc711d5]
14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x108) [0x55eeadc60dc8]
15: (DispatchQueue::entry()+0x126a) [0x7f40e9374b5a]
16: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40e9426841]
17: /lib64/libpthread.so.0(+0x81ca) [0x7f40e810f1ca]
18: clone()
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre></p>
<p>Some interesting log snippets from before <br /><pre>
-434> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2024-03-28T19:18:03.483427+0100)
-433> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client log_queue is 6 last_log 1380 sent 1374 num 6 unsent 6 sending 6
-432> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701758+0100 mds.minilevinson-983db664a3 (mds.0) 1375 : cluster [WRN] 5 slow requests, 5
included below; oldest blocked for > 13.152129 secs
-431> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701765+0100 mds.minilevinson-983db664a3 (mds.0) 1376 : cluster [WRN] slow request 7.203
507 seconds old, received at 2024-03-28T19:18:25.497610+0100: client_request(mds.0:181210 rename #0x100026f338d/sh15637 #0x606/1000295a11a caller_uid=0, caller_gid=0{}) currently sub
mit entry: journal_and_reply
-430> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701771+0100 mds.minilevinson-983db664a3 (mds.0) 1377 : cluster [WRN] slow request 6.519
057 seconds old, received at 2024-03-28T19:18:26.182061+0100: client_request(mds.0:181239 rename #0x100026f338d/sh15811 #0x608/1000295a58f caller_uid=0, caller_gid=0{}) currently sub
mit entry: journal_and_reply
-429> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701774+0100 mds.minilevinson-983db664a3 (mds.0) 1378 : cluster [WRN] slow request 6.372
089 seconds old, received at 2024-03-28T19:18:26.329029+0100: client_request(mds.0:181300 rename #0x1000185ba3c/file117_hardmv_hard #0x603/10002567d82 caller_uid=0, caller_gid=0{}) c
urrently failed to wrlock, waiting
-428> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701776+0100 mds.minilevinson-983db664a3 (mds.0) 1379 : cluster [WRN] slow request 6.371
098 seconds old, received at 2024-03-28T19:18:26.330020+0100: client_request(mds.0:181320 rename #0x1000185ba3c/file373_hardmv_hard #0x608/1000256807c caller_uid=0, caller_gid=0{}) c
urrently failed to wrlock, waiting
-427> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 log_client will send 2024-03-28T19:18:32.701781+0100 mds.minilevinson-983db664a3 (mds.0) 1380 : cluster [WRN] slow request 5.565
985 seconds old, received at 2024-03-28T19:18:27.135132+0100: client_request(mds.0:181349 rename #0x100026f338d/sh15708 #0x606/1000295a111 caller_uid=0, caller_gid=0{}) currently fai
led to wrlock, waiting
-426> 2024-03-28T19:18:33.482+0100 7f40df6ed700 10 monclient: _send_mon_message to mon.minilevinson-f52cbe8096 at v2:188.184.88.255:3300/0
-425> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client handle_log_ack log(last 1380) v1
-424> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701758+0100 mds.minilevinson-983db664a3 (mds.0) 1375 : cluster [WRN] 5 slow requests, 5 in
cluded below; oldest blocked for > 13.152129 secs
-423> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701765+0100 mds.minilevinson-983db664a3 (mds.0) 1376 : cluster [WRN] slow request 7.203507
seconds old, received at 2024-03-28T19:18:25.497610+0100: client_request(mds.0:181210 rename #0x100026f338d/sh15637 #0x606/1000295a11a caller_uid=0, caller_gid=0{}) currently submit
entry: journal_and_reply
-422> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701771+0100 mds.minilevinson-983db664a3 (mds.0) 1377 : cluster [WRN] slow request 6.519057
seconds old, received at 2024-03-28T19:18:26.182061+0100: client_request(mds.0:181239 rename #0x100026f338d/sh15811 #0x608/1000295a58f caller_uid=0, caller_gid=0{}) currently submit
entry: journal_and_reply
-421> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701774+0100 mds.minilevinson-983db664a3 (mds.0) 1378 : cluster [WRN] slow request 6.372089
seconds old, received at 2024-03-28T19:18:26.329029+0100: client_request(mds.0:181300 rename #0x1000185ba3c/file117_hardmv_hard #0x603/10002567d82 caller_uid=0, caller_gid=0{}) curr
ently failed to wrlock, waiting
-420> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701776+0100 mds.minilevinson-983db664a3 (mds.0) 1379 : cluster [WRN] slow request 6.371098
seconds old, received at 2024-03-28T19:18:26.330020+0100: client_request(mds.0:181320 rename #0x1000185ba3c/file373_hardmv_hard #0x608/1000256807c caller_uid=0, caller_gid=0{}) curr
ently failed to wrlock, waiting
-419> 2024-03-28T19:18:33.765+0100 7f40e06ef700 10 log_client logged 2024-03-28T19:18:32.701781+0100 mds.minilevinson-983db664a3 (mds.0) 1380 : cluster [WRN] slow request 5.565985
seconds old, received at 2024-03-28T19:18:27.135132+0100: client_request(mds.0:181349 rename #0x100026f338d/sh15708 #0x606/1000295a111 caller_uid=0, caller_gid=0{}) currently failed
to wrlock, waiting
</pre></p>
<p>We've configured the mds `mds_op_complaint_time` to 5s in order to track some potential slow locks</p> CephFS - Bug #65224 (New): mds: fs subvolume rm failshttps://tracker.ceph.com/issues/652242024-03-29T09:52:40ZMilind Changire
<p>`fs subvolume rm` fails when subvolume dir attempted to move to a different dir where the following code fails in src/mds/Server.cc:Server::handle_client_rename</p>
<pre>
if (src_realm != dest_realm &&
src_realm->get_subvolume_ino() != dest_realm->get_subvolume_ino()) {
respond_to_request(mdr, -CEPHFS_EXDEV);
return;
}
</pre> CephFS - Backport #65223 (New): squid: cephfs-mirror: use snapdiff api for efficient tree traversalhttps://tracker.ceph.com/issues/652232024-03-29T08:23:14ZBackport BotCephFS - Backport #65222 (New): reef: cephfs-mirror: use snapdiff api for efficient tree traversalhttps://tracker.ceph.com/issues/652222024-03-29T08:23:06ZBackport BotDashboard - Backport #65221 (New): reef: ceph-mixin: Add RBD Mirror monitoring alertshttps://tracker.ceph.com/issues/652212024-03-29T08:01:18ZBackport BotDashboard - Backport #65220 (New): squid: ceph-mixin: Add RBD Mirror monitoring alertshttps://tracker.ceph.com/issues/652202024-03-29T08:01:08ZBackport BotDashboard - Bug #65219 (Pending Backport): ceph-mixin: Add RBD Mirror monitoring alertshttps://tracker.ceph.com/issues/652192024-03-29T07:57:35ZAvan ThakkarCeph QA - QA Run #65215 (QA Testing): wip-batrick-testing-20240329.130222https://tracker.ceph.com/issues/652152024-03-28T19:36:30ZPatrick Donnellypdonnell@redhat.comwebsite - Feature #64905 (New): pgcalc from docs.ceph.com seems to be older then the access.redha...https://tracker.ceph.com/issues/649052024-03-14T02:22:12ZMichael Collins
<p>PGCalc is back! <a class="external" href="https://github.com/ceph/ceph.io/issues/265">https://github.com/ceph/ceph.io/issues/265</a></p>
<p>Although something seems a little off about it, the version of pgcalc seen here supports "Jewel and later":<br /><a class="external" href="https://docs.ceph.com/en/reef/rados/operations/pgcalc/">https://docs.ceph.com/en/reef/rados/operations/pgcalc/</a></p>
<p>While the pgcalc seen on the RedHat site appears to be newer, supporting "Luminous and later":<br /><a class="external" href="https://access.redhat.com/labs/cephpgc/manual/">https://access.redhat.com/labs/cephpgc/manual/</a></p>
<p>It makes me wonder what the difference is between these pgcalcs, and could the seemingly newer pgcalc be added to docs.ceph.com?</p> CephFS - Feature #61334 (Pending Backport): cephfs-mirror: use snapdiff api for efficient tree tr...https://tracker.ceph.com/issues/613342023-05-22T10:40:44ZVenky Shankarvshankar@redhat.com
<p>With <a class="external" href="https://github.com/ceph/ceph/pull/43546">https://github.com/ceph/ceph/pull/43546</a> merged, cephfs-mirror can make use the snapdiff api (via readdir_snapdiff) to efficiently traverse the directory tree between two snapshots.</p>
<p>This should hugely improve performance when only a handful of files have changed between two consecutive snapshots.</p> RADOS - Bug #59670 (New): Ceph status shows PG recovering when norecover flag is sethttps://tracker.ceph.com/issues/596702023-05-08T14:07:13ZAishwarya Mathuria
<p>On the Gibba cluster, we observed that ceph -s was showing one PG in recovering state after norecovery flag was set</p>
<pre><code class="text syntaxhl"><span class="CodeRay">[root@gibba001 ~]# ceph -s
cluster:
id: 7e775b16-ea73-11ed-ac35-3cecef3d8fb8
health: HEALTH_WARN
nobackfill,norecover,noscrub,nodeep-scrub flag(s) set
Degraded data redundancy: 2/27732183 objects degraded (0.000%), 1 pg degraded, 1 pg undersized
services:
mon: 5 daemons, quorum gibba001,gibba002,gibba003,gibba006,gibba005 (age 78m)
mgr: gibba006.oxzbun(active, since 71m), standbys: gibba008.fhfdkj
osd: 62 osds: 62 up (since 74m), 62 in (since 74m); 1 remapped pgs
flags nobackfill,norecover,noscrub,nodeep-scrub
rgw: 6 daemons active (6 hosts, 1 zones)
data:
pools: 7 pools, 1217 pgs
objects: 4.62M objects, 203 GiB
usage: 446 GiB used, 10 TiB / 11 TiB avail
pgs: 2/27732183 objects degraded (0.000%)
1216 active+clean
1 active+recovering+undersized+degraded+remapped
</span></code></pre>
<p>PG dump:<br /><pre><code class="text syntaxhl"><span class="CodeRay">
1.0 2 0 2 0 0 1114656 0 0 1334 1334 active+recovering+undersized+degraded+remapped 2023-05-04T08:34:56.217391+0000 109'1334 123:1683 [30,15,0] 30 [15,0] 15 0'0 2023-05-04T08:34:39.648644+0000 0'0 2023-05-04T08:34:39.648644+0000 0 0 periodic scrub scheduled @ 2023-05-05T16:09:53.741099+0000 0 0
dumped all
</span></code></pre></p>
<p>From the cluster logs we can see the norecover flag being set and when the OSDs come up we observed the following logs:</p>
<pre><code class="text syntaxhl"><span class="CodeRay">2023-05-04T12:16:48.219+0000 7f078e80e700 1 osd.29 82 state: booting -> active
2023-05-04T12:16:48.219+0000 7f078e80e700 1 osd.29 82 pausing recovery (NORECOVER flag set)
</span></code></pre>
<p>And after sometime we can see the state of PG 1.0 in the logs:</p>
<pre><code class="text syntaxhl"><span class="CodeRay">2023-05-04T13:24:21.971+0000 7f07909cb700 30 osd.29 pg_epoch: 121 pg[1.0( v 106'1334 lc 80'163 (0'0,106'1334] local-lis/les=0/0 n=2 ec=74/74 lis/c=0/78 les/c/f=0/79/0 sis=84) [29,15,32]/[15,32] r=-1 lpr=84 pi=[78,84)/1 luod=0'0 lua=106'1324 crt=106'1334 mlcod 80'163 *active+remapped* m=2 mbc={}] lock
2023-05-04T13:25:31.984+0000 7f07909cb700 30 osd.29 pg_epoch: 121 pg[1.0( v 106'1334 lc 80'163 (0'0,106'1334] local-lis/les=0/0 n=2 ec=74/74 lis/c=0/78 les/c/f=0/79/0 sis=84) [29,15,32]/[15,32] r=-1 lpr=84 pi=[78,84)/1 luod=0'0 lua=106'1324 crt=106'1334 mlcod 80'163 *active+remapped* m=2 mbc={}] lock
</span></code></pre>
<p>However, ceph status and ceph pg dump still show that PG 1.0 is recovering.</p> bluestore - Bug #53899 (Need More Info): bluefs _allocate allocation failed - BlueFS.cc: 2768: ce...https://tracker.ceph.com/issues/538992022-01-16T17:32:43ZPivert Dubuisson
<p>All OSDs failing to start after OSD near full. Cluster down.<br />3 nodes cluster (pve1, pve2, pve3) - Bluestore on a single LV (lvm) on NVME.</p>
<p>Sequence of events:<br />- 00:30 The ceph RBD was about 90% on a 3 dones cluster with about 140GB on each<br />- 00:47 pve2 Issued a restore VM/CT (in a Proxmox VE) to overwrite a CT of 20GB. From the pve logs:<br /> - Logical volume "vm-105-disk-0" successfully removed<br /> - restoring 'OneDriveSecure1:backup/vzdump-lxc-105-2022_01_15-23_14_19.tar.zst' now..<br />- 00:48:26 pve3 ceph-osd<sup><a href="#fn1681">1681</a></sup>: 2022-01-16T00:48:26.215+0100 7f7bfa5a5700 <del>1 bluefs _allocate allocation failed, needed 0x1687 -> ceph_abort_msg("bluefs enospc")<br /></del> 00:48:33 pve1 ceph-osd<sup><a href="#fn2157">2157</a></sup>: 2022-01-16T00:48:33.904+0100 7f3b78139700 <del>1 bluefs _allocate allocation failed, needed 0x6ff1 -> ceph_abort_msg("bluefs enospc")<br /></del> 01:20 pve2 The restore on pve2 never finished. Server hanging (iowaits). Reboot did not complete.<br />- 01:38 pve2 I had to power off and restart the server.<br />- 01:39:12 pve2 ceph-crash<sup><a href="#fn687">687</a></sup>: INFO:ceph-crash:monitoring path /var/lib/ceph/crash, delay 600s<br /> 01:40:00 pve2 ceph-osd<sup><a href="#fn1681">1681</a></sup>: 2022-01-16T01:40:00.023+0100 7f628f9cbf00 -1 bluefs _allocate allocation failed, needed 0x74240 -> ceph_abort_msg("bluefs enospc")</p>
<p>At this stage, all OSDs are down in a failing to restart endless loop.</p>
<p>All standard logs from the 3 pve hosts are attached.<br />(grep -a ceph /var/log/syslog | xz -z > /root/cephcrash_$HOSTNAME.xz)</p> CephFS - Bug #48562 (New): qa: scrub - object missing on disk; some files may be losthttps://tracker.ceph.com/issues/485622020-12-11T10:40:34ZMilind Changire
<p>2020-12-10T05:14:53.213 INFO:tasks.ceph.mds.b.smithi165.stderr:2020-12-10T05:14:53.212+0000 7f27f1562700 -1 log_channel(cluster) log [ERR] : dir 0x10000000070.110101* object missing on disk; some files may be lost (/client.0/tmp/testdir/dir1/dir2)</p>
<p>teuthology run URL:<br /><a class="external" href="http://pulpito.front.sepia.ceph.com/mchangir-2020-12-10_04:47:36-fs:workload-wip-mchangir-qa-forward-scrub-task-distro-basic-smithi/5697353/">http://pulpito.front.sepia.ceph.com/mchangir-2020-12-10_04:47:36-fs:workload-wip-mchangir-qa-forward-scrub-task-distro-basic-smithi/5697353/</a></p>