Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-03-07T21:37:52Z
Ceph
Redmine
rbd - Bug #64800 (Fix Under Review): unable to remove RBD image when OSD is full and trash object...
https://tracker.ceph.com/issues/64800
2024-03-07T21:37:52Z
Ramana Raja
rraja@redhat.com
<p>In a vstart cluster, created a RBD image, wrote data to it, configured set-full-ratio to make the OSDs full. Failed to remove the image using the CLI<br /><pre>
$ rbd create --size 2048 data/img1
$ sudo ./bin/rbd device map data/img1
$ sudo fio --name=fiotest --filename=/dev/rbd0 --rw=randrw --bs=4M --direct=1 --ioengine=libaio --size=2G
$ sudo ./bin/rbd device unmap /dev/rbd0
$ ceph -s
cluster:
id: 1b6d031f-c462-4236-b88b-7486b1f40603
health: HEALTH_WARN
3 pool(s) have no replicas configured
services:
mon: 1 daemons, quorum a (age 64m)
mgr: x(active, since 64m)
osd: 1 osds: 1 up (since 63m), 1 in (since 64m)
data:
pools: 3 pools, 65 pgs
objects: 277 objects, 1.1 GiB
usage: 2.1 GiB used, 99 GiB / 101 GiB avail
pgs: 65 active+clean
io:
client: 112 KiB/s rd, 90 MiB/s wr, 134 op/s rd, 67 op/s wr
$ # configure OSD setting to make the OSD full
$ ceph osd set-nearfull-ratio 0.017
$ ceph osd set-backfillfull-ratio 0.018
$ ceph osd set-full-ratio 0.020
$ ceph -s
cluster:
id: 1b6d031f-c462-4236-b88b-7486b1f40603
health: HEALTH_ERR
1 full osd(s)
3 pool(s) full
3 pool(s) have no replicas configured
services:
mon: 1 daemons, quorum a (age 73m)
mgr: x(active, since 73m)
osd: 1 osds: 1 up (since 72m), 1 in (since 72m)
data:
pools: 3 pools, 65 pgs
objects: 277 objects, 1.1 GiB
usage: 2.1 GiB used, 99 GiB / 101 GiB avail
pgs: 65 active+clean
$ rbd rm data/img1 --debug_rbd=10
...
2024-03-07T16:34:26.927-0500 7f19c6ffd6c0 10 librbd::ImageCtx: 0x55b8470abab0 ~ImageCtx
2024-03-07T16:34:26.927-0500 7f19ff594580 10 librbd::trash::MoveRequest: 0x7f19c80029f0 trash_add:
2024-03-07T16:34:26.928-0500 7f19e3fff6c0 10 librbd::trash::MoveRequest: 0x7f19c80029f0 handle_trash_add: r=-28
2024-03-07T16:34:26.928-0500 7f19e3fff6c0 -1 librbd::trash::MoveRequest: 0x7f19c80029f0 handle_trash_add: failed to add image to trash: (28) No space left on device
2024-03-07T16:34:26.928-0500 7f19e3fff6c0 10 librbd::trash::MoveRequest: 0x7f19c80029f0 finish: r=-28
2024-03-07T16:34:26.929-0500 7f19ff594580 -1 librbd::api::Trash: move: error setting trash image state: (2) No such file or directory
Removing image: 0% complete...failed.
rbd: delete error: (2) No such file or directory
</pre></p>
<p>This issue is also documented as a FIXME in test/librbd/test_librbd.cc,<br /><pre>
TEST_F(TestLibRBD, RemoveFullTry)
{
...
// FIXME: this is a workaround for rbd_trash object being created
// on the first remove -- pre-create it to avoid bumping into quota
ASSERT_EQ(0, create_image(ioctx, image_name.c_str(), 0, &order));
ASSERT_EQ(0, rbd_remove(ioctx, image_name.c_str()));
remove_full_try(ioctx, image_name, pool_name);
rados_ioctx_destroy(ioctx);
}
</pre><br /><a class="external" href="https://github.com/ceph/ceph/blob/v19.0.0/src/test/librbd/test_librbd.cc#L2193">https://github.com/ceph/ceph/blob/v19.0.0/src/test/librbd/test_librbd.cc#L2193</a></p>
rbd - Bug #64574 (Pending Backport): [test] cross-pollinate diff-continuous and compare-mirror-im...
https://tracker.ceph.com/issues/64574
2024-02-26T17:17:29Z
Ilya Dryomov
<p>In order to expand coverage:</p>
<p>- add compare_mirror_image_alternate_primary.sh workunit to krbd suite to run against testing kernel (based on qa/suites/rbd/mirror/workloads/compare-mirror-image-alternate-primary-krbd.yaml)<br />- add compare_mirror_images.sh workunit to krbd suite to run against testing kernel (based on qa/suites/rbd/mirror/workloads/compare-mirror-images-krbd.yaml)<br />- add diff_continuous.sh workunit to rbd suite to run against distro kernel (based on qa/suites/krbd/thrash/workloads/krbd_diff_continuous.yaml)</p>
rbd - Bug #61617 (Pending Backport): [test][krbd] volume data corruption when using rbd-mirror w/...
https://tracker.ceph.com/issues/61617
2023-06-08T06:35:10Z
Ilya Dryomov
<p>Forked from <a class="external" href="https://tracker.ceph.com/issues/61472">https://tracker.ceph.com/issues/61472</a> to track adding corresponding workunits to krbd suite:</p>
<p>Workload 1: X iterations of a stripped-down reproducer (each iteration creates a new primary image)<br />Workload 2: X iterations of a full reproducer (a single image, primary alternates between clusters)</p>
<p>Could potentially run these in parallel Y times (in separate jobs, for a total of X*Y iterations of a given reproducer per job)</p>
rbd - Bug #59728 (In Progress): [test] single node "ip netns"-based latency injection for rbd-mirror
https://tracker.ceph.com/issues/59728
2023-05-11T10:43:31Z
Ilya Dryomov
<p>Initially just an integration test for <a class="external" href="https://tracker.ceph.com/issues/59393">https://tracker.ceph.com/issues/59393</a>. Might be expanded in the future to vary injected latency, take measurements and assert on results according to the model.</p>
<p>Script to apply latency to NIC+destination IP</p>
<p>NIC=enp1s0</p>
<p>function apply() {<br /> tc qdisc add dev ${NIC} root handle 1: prio priomap 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 <br /> tc qdisc add dev ${NIC} parent 1:2 handle 20: netem delay 0ms<br /> tc filter add dev ${NIC} parent 1:0 protocol ip u32 match ip src `hostname -I` flowid 1:2<br /> tc qdisc add dev ${NIC} parent 1:1 handle 10: netem delay 151ms</p>
<pre><code>#NODES should be IPs of remote cluster you want to<br /> #inject latency to. Include IPs for Mgr, Mon, Osd, etc<br /> NODES=( 8.8.8.8 184.50.33.11 )<br /> for i in ${NODES[@]}<br /> do<br /> tc filter add dev ${NIC} parent 1:0 protocol ip prio 1 u32 match ip dst ${i} flowid 1:1<br /> done<br />}</code></pre>
<p>function cleanup() {<br /> tc qdisc del dev ${NIC} root<br />}</p>
<p>cleanup<br />apply</p>
rbd - Bug #59681 (New): [rbd_support] improve cli_generic.sh tests for recovery from blocklisting
https://tracker.ceph.com/issues/59681
2023-05-08T21:22:58Z
Ramana Raja
rraja@redhat.com
<p>Ilya's suggestions,<br />" <br />for TrashPurgeScheduleHandler and MirrorSnapshotScheduleHandler tests, instead of just checking that both pre- and post-blocklisting schedules show up, it would be good to test that both pre- and post-blocklisting scheduled work actually gets done. For trash purge scheduler, I would suggest creating two pools with an image trashed in each, adding a short (1-2m) schedule for one pool before blocklisting and for another after blocklisting and asserting that trash purge runs after enough time passes. And similarly for mirror snapshot scheduler: a single pool with two images would do as per-image schedules can be added there.</p>
<p>for TaskHandler test, queue some (let's say 5) flattens on different images before blocklisting and a sixth flatten after blocklisting and assert that all six flattens complete."</p>
<p>Source: <a class="external" href="https://github.com/ceph/ceph/pull/49742#issuecomment-1537460028">https://github.com/ceph/ceph/pull/49742#issuecomment-1537460028</a></p>
mgr - Bug #58924 (Fix Under Review): mgr: block register_client on new MgrMap
https://tracker.ceph.com/issues/58924
2023-03-06T18:27:24Z
Patrick Donnelly
pdonnell@redhat.com
<p>A mgr (plugin) may reconnect and use a new RADOS client upon learning its blocklist. The new client may do work racing with a newly active mgr.</p>
<p>The mgr should block return of the register_client method until the new MgrMap is received to ensure the monitors have recorded the new address for the <b>still active</b> mgr.</p>
CephFS - Backport #56590 (New): quincy: qa: iogen workunit: "The following counters failed to be ...
https://tracker.ceph.com/issues/56590
2022-07-18T03:45:52Z
Backport Bot
Linux kernel client - Bug #55052 (New): when mounting with new dev syntax and -o ms_mode=legacy, ...
https://tracker.ceph.com/issues/55052
2022-03-24T20:44:32Z
Jeff Layton
jlayton@redhat.com
<p>When I mount using the new device syntax, with "-o ms_mode=legacy", I don't see the mount option in /proc/mounts.</p>
<p>The problem is that we just don't display ms_mode=legacy at all, which was fine when that was the clear default. /proc/mounts should display ms_mode=legacy when the new mount device syntax is used along with that option. We could also consider hiding "ms_mode=prefer-crc" if the new mount device syntax is used (since that's the new default).</p>
CephFS - Bug #54406 (Triaged): cephadm/mgr-nfs-upgrade: cluster [WRN] overall HEALTH_WARN no acti...
https://tracker.ceph.com/issues/54406
2022-02-24T22:51:30Z
Laura Flores
<p>/a/yuriw-2022-02-21_15:48:20-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6698628<br /><pre><code class="text syntaxhl"><span class="CodeRay">2022-02-21T21:30:00.283 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: cluster
2022-02-21T21:30:00.283 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: 2022-02-21T21:30:00.000151+0000
2022-02-21T21:30:00.283 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: mon.smithi006 (mon.0) 831 : cluster [WRN]
2022-02-21T21:30:00.283 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: Health detail: HEALTH_WARN 1 MDSs report slow requests
2022-02-21T21:30:00.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: cluster
2022-02-21T21:30:00.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: 2022-02-21T
2022-02-21T21:30:00.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: 21:30:00.000204+0000 mon.smithi006 (
2022-02-21T21:30:00.284 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: mon.0) 832 : cluster [WRN]
2022-02-21T21:30:00.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
2022-02-21T21:30:00.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: cluster
2022-02-21T21:30:00.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: 2022-02-21T21:
2022-02-21T21:30:00.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: 30:00.000224+0000 mon.smithi006 (
2022-02-21T21:30:00.285 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 21 21:30:00 smithi006 conmon[29154]: mon.0) 833 : cluster [WRN]
...
2022-02-21T21:30:01.343 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 21 21:30:01 smithi084 conmon[32205]: mon.smithi006 (mon.0) 834 : cluster [WRN] Health check failed: 1 clients failing to respond to capability release (MDS_CLIENT_LATE_RELEASE) ---> this repeats several times
...
2022-02-21T21:30:01.345 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 21 21:30:01 smithi084 conmon[32205]: cluster 2022-02-21T21:30:00.506984+0000 mds.foofs.smithi006.hyukcc (mds.0) 11 : cluster [WRN] 1 slow requests, 0 included below; oldest blocked for > 66.896486 secs --> this part also repeats
...
2022-02-21T21:30:46.843 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 21 21:30:46 smithi084 conmon[32205]: cluster 2022-02-21T21:30:45.675935+0000 mon.smithi006 (mon.0) 17 : cluster [WRN] Health check failed: no active mgr (MGR_DOWN)
... actions leading up to dead job ...
2022-02-22T03:00:00.355 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:00:00 smithi084 conmon[32205]: mon.smithi006 (mon.0) 67 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:00:00.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:00:00 smithi006 conmon[105699]: cluster 2022-02-22T03:00:00.000080
2022-02-22T03:00:00.356 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:00:00 smithi006 conmon[105699]: +0000 mon.smithi006 (mon.0) 67 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:10:00.335 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:10:00 smithi084 conmon[32205]: cluster 2022-02-22T03:10:00.000075+0000 mon.smithi006 (mon.0) 68 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:10:00.353 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:10:00 smithi006 conmon[105699]: cluster 2022-02-22T03:10:00.000075+0000
2022-02-22T03:10:00.354 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:10:00 smithi006 conmon[105699]: mon.smithi006 (mon.0) 68 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:11:02.139 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:11:01 smithi084 conmon[32205]: debug 2022-02-22T03:11:01.990+0000 7f1160fc3700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:11:02.140 INFO:journalctl@ceph.osd.5.smithi084.stdout:Feb 22 03:11:01 smithi084 conmon[42449]: debug 2022-02-22T03:11:01.990+0000 7f02a3d94700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:11:02.141 INFO:journalctl@ceph.osd.6.smithi084.stdout:Feb 22 03:11:01 smithi084 conmon[46647]: debug 2022-02-22T03:11:01.990+0000 7fbf14871700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:11:02.141 INFO:journalctl@ceph.osd.7.smithi084.stdout:Feb 22 03:11:01 smithi084 conmon[50852]: debug 2022-02-22T03:11:01.990+0000 7fd23dae4700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:11:02.142 INFO:journalctl@ceph.osd.4.smithi084.stdout:Feb 22 03:11:01 smithi084 conmon[38345]: debug 2022-02-22T03:11:01.990+0000 7f3c84052700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:20:00.330 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:20:00 smithi084 conmon[32205]: cluster 2022-02-22T03:20:00.000099
2022-02-22T03:20:00.331 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:20:00 smithi084 conmon[32205]: +0000 mon.smithi006 (mon.0) 69 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:20:00.352 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:20:00 smithi006 conmon[105699]: cluster 2022-02-22T
2022-02-22T03:20:00.352 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:20:00 smithi006 conmon[105699]: 03:20:00.000099+0000 mon.smithi006 (mon.0) 69 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:30:00.334 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:30:00 smithi084 conmon[32205]: cluster 2022-02-22T03:30:00.000079+0000 mon.smithi006 (mon.
2022-02-22T03:30:00.351 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:30:00 smithi084 conmon[32205]: 0) 70 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:30:00.363 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:30:00 smithi006 conmon[105699]: cluster 2022-02-22T03:30:00.000079+0000
2022-02-22T03:30:00.363 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:30:00 smithi006 conmon[105699]: mon.smithi006 (mon.0) 70 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:35:02.119 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:35:01 smithi006 conmon[105699]: debug 2022-02-22T03:35:01.816+0000 7f58911f5700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:35:02.121 INFO:journalctl@ceph.osd.0.smithi006.stdout:Feb 22 03:35:01 smithi006 conmon[45979]: debug 2022-02-22T03:35:01.816+0000 7f379f986700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:35:02.121 INFO:journalctl@ceph.osd.1.smithi006.stdout:Feb 22 03:35:01 smithi006 conmon[50175]: debug 2022-02-22T03:35:01.816+0000 7fd9c8e27700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:35:02.122 INFO:journalctl@ceph.osd.2.smithi006.stdout:Feb 22 03:35:01 smithi006 conmon[55725]: debug 2022-02-22T03:35:01.816+0000 7f76cccc9700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:35:02.122 INFO:journalctl@ceph.osd.3.smithi006.stdout:Feb 22 03:35:01 smithi006 conmon[61261]: debug 2022-02-22T03:35:01.816+0000 7fa0c5fad700 -1 received signal: Hangup from Kernel ( Could be generated by pthread_kill(), raise(), abort(), alarm() ) UID: 0
2022-02-22T03:40:00.347 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:40:00 smithi084 conmon[32205]: cluster 2022-02-22T03:40:00.000162+0000 mon.smithi006 (mon.0
2022-02-22T03:40:00.348 INFO:journalctl@ceph.mon.smithi084.smithi084.stdout:Feb 22 03:40:00 smithi084 conmon[32205]: ) 71 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:40:00.371 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:40:00 smithi006 conmon[105699]: cluster 2022-02-22T03:40:00.000162
2022-02-22T03:40:00.371 INFO:journalctl@ceph.mon.smithi006.smithi006.stdout:Feb 22 03:40:00 smithi006 conmon[105699]: +0000 mon.smithi006 (mon.0) 71 : cluster [WRN] overall HEALTH_WARN no active mgr
2022-02-22T03:42:39.393 DEBUG:teuthology.exit:Got signal 15; running 1 handler...
2022-02-22T03:42:39.396 DEBUG:teuthology.task.console_log:Killing console logger for smithi006
2022-02-22T03:42:39.397 DEBUG:teuthology.task.console_log:Killing console logger for smithi084
2022-02-22T03:42:39.398 DEBUG:teuthology.exit:Finished running handlers
</span></code></pre></p>
CephFS - Bug #54108 (Pending Backport): qa: iogen workunit: "The following counters failed to be ...
https://tracker.ceph.com/issues/54108
2022-02-01T18:52:49Z
Patrick Donnelly
pdonnell@redhat.com
<p><a class="external" href="https://pulpito.ceph.com/pdonnell-2022-01-29_01:47:41-fs:workload-wip-pdonnell-testing-20220127.171526-distro-default-smithi/6648139/">https://pulpito.ceph.com/pdonnell-2022-01-29_01:47:41-fs:workload-wip-pdonnell-testing-20220127.171526-distro-default-smithi/6648139/</a></p>
<p><a class="external" href="https://pulpito.ceph.com/pdonnell-2022-01-29_01:47:41-fs:workload-wip-pdonnell-testing-20220127.171526-distro-default-smithi/6648145/">https://pulpito.ceph.com/pdonnell-2022-01-29_01:47:41-fs:workload-wip-pdonnell-testing-20220127.171526-distro-default-smithi/6648145/</a></p>
mgr - Bug #54025 (Pending Backport): mgr/nfs: don't always restart the NFS ganesha service for u...
https://tracker.ceph.com/issues/54025
2022-01-26T20:31:09Z
Ramana Raja
rraja@redhat.com
<p>`ceph nfs export apply` is used to update a CephFS NFS Export. The mgr/nfs module's apply_export() always restarts the NFS service when updating a CephFS NFS export. This temporarily affects the clients of exports that are not updated. The NFS service needs to be restarted only when making certain updates to a CephFS NFS export. For most CephFS NFS export udpates notifying the ganesha's RADOS watch object should be enough.</p>
CephFS - Bug #52531 (Triaged): Quotas smaller than 4MB on subdirs do not have any effect
https://tracker.ceph.com/issues/52531
2021-09-07T16:38:06Z
Ruben Kerkhof
This doesn’t work:<br />root@mon0:~# setfattr -n ceph.quota.max_bytes -v $((4*1024*1024-1)) /mnt/cephfs/<br />root@mon0:~# getfattr -n ceph.quota.max_bytes /mnt/cephfs/<br />getfattr: Removing leading '/' from absolute path names
<ol>
<li>file: mnt/cephfs/<br />ceph.quota.max_bytes="4194303"</li>
</ol>
<p>root@mon0:~# df -h /mnt/cephfs/<br />Filesystem Size Used Avail Use% Mounted on<br />192.168.42.10:6789,192.168.42.11:6789,192.168.42.12:6789:/subdir 28G 0 28G 0% /mnt/cephfs</p>
<p>The 28GB is the size of the parent fs.</p>
But this does:<br />root@mon0:~# setfattr -n ceph.quota.max_bytes -v $((4*1024*1024)) /mnt/cephfs/<br />root@mon0:~# getfattr -n ceph.quota.max_bytes /mnt/cephfs/<br />getfattr: Removing leading '/' from absolute path names
<ol>
<li>file: mnt/cephfs/<br />ceph.quota.max_bytes="4194304"</li>
</ol>
<p>root@mon0:~# df -h /mnt/cephfs/<br />Filesystem Size Used Avail Use% Mounted on<br />192.168.42.10:6789,192.168.42.11:6789,192.168.42.12:6789:/subdir 4.0M 0 4.0M 0% /mnt/cephfs</p>
<p>I did test this in both Octopus and Pacific</p>
CephFS - Bug #52508 (Triaged): nfs-ganesha crash when calls libcephfs, it triggers __ceph_assert_...
https://tracker.ceph.com/issues/52508
2021-09-05T13:36:04Z
le le
<p>centos8<br />cephfs: v16.2.5<br />nfs-ganesha:v3.5 last release</p>
<p>In my code from git,<br />Inode.cc</p>
<p>76: if (!in.dentries.empty())<br />77: out << " parents=" << in.dentries;</p>
<p>I find something strange, my backtrace shows coredump at line 77, but I print the value of "if()" is false , what happened? Or some wrong symbols in backtrace? Or my code doesn't match the libcephfs which backtrace shows?</p>
<p>The in.dentries value as {_front = 0x0, _back = 0x0, _size = 0} , it will be coredump?</p>
<p>some debuginfo:<br /><pre>
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007ff8b60b5db5 in __GI_abort () at abort.c:79
#2 0x00007ff8a3e3db75 in ceph::__ceph_assert_fail(char const*, char const*, int, char const*) () from /usr/lib64/ceph/libceph-common.so.2
#3 0x00007ff8a3e3dd3e in ceph::__ceph_assert_fail(ceph::assert_data const&) () from /usr/lib64/ceph/libceph-common.so.2
#4 0x00007ff8acbad627 in xlist<Dentry*>::const_iterator::operator++ (this=) at /usr/include/c++/8/ostream:559
#5 operator<< (list=..., oss=...) at /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/include/xlist.h:212
#6 operator<< (out=..., in=...) at /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/client/Inode.cc:77
#7 0x00007ff8acb715ae in Client::ll_sync_inode (this=0x55e3a853c7f0, in=in@entry=0x7ff6a9903120, syncdataonly=syncdataonly@entry=false) at /usr/include/c++/8/ostream:559
#8 0x00007ff8acadbe55 in ceph_ll_sync_inode (cmount=cmount@entry=0x55e3a849b2b0, in=in@entry=0x7ff6a9903120, syncdataonly=syncdataonly@entry=0) at /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/libcephfs.cc:1865
#9 0x00007ff8acebba15 in fsal_ceph_ll_setattr (creds=, mask=409, stx=0x7ff87c3b7310, i=, cmount=) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/FSAL/FSAL_CEPH/statx_compat.h:209
#10 ceph_fsal_setattr2 (obj_hdl=0x7ff76c2a75b0, bypass=, state=, attrib_set=0x7ff87c3b75a0) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/FSAL/FSAL_CEPH/handle.c:2410
#11 0x00007ff8b860eb9f in mdcache_setattr2 (obj_hdl=0x7ff76c50d3a8, bypass=, state=0x0, attrs=0x7ff87c3b75a0) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1002
#12 0x00007ff8b852b7b4 in fsal_setattr (obj=0x7ff76c50d3a8, bypass=bypass@entry=false, state=0x0, attr=attr@entry=0x7ff87c3b75a0) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/FSAL/fsal_helper.c:573
#13 0x00007ff8b85e2d17 in nfs4_op_setattr (op=0x7ff800bc0800, data=0x7ff8002c0850, resp=0x7ff8002f2560) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/Protocols/NFS/nfs4_op_setattr.c:212
#14 0x00007ff8b85c678f in process_one_op (data=data@entry=0x7ff8002c0850, status=status@entry=0x7ff87c3b779c) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/Protocols/NFS/nfs4_Compound.c:920
#15 0x00007ff8b85c7927 in nfs4_Compound (arg=0x7ff800cd7d00, req=0x7ff800cd74f0, res=0x7ff80035f410) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/Protocols/NFS/nfs4_Compound.c:1327
#16 0x00007ff8b8547c46 in nfs_rpc_process_request (reqdata=0x7ff800cd74f0) at /usr/src/debug/nfs-ganesha-3.5-3.el8.x86_64/src/MainNFSD/nfs_worker_thread.c:1508
#17 0x00007ff8b82d5800 in svc_request () from /lib64/libntirpc.so.3.5
#18 0x00007ff8b82d2bf9 in svc_rqst_xprt_task_recv () from /lib64/libntirpc.so.3.5
#19 0x00007ff8b82d35d8 in svc_rqst_epoll_loop () from /lib64/libntirpc.so.3.5
#20 0x00007ff8b82de65d in work_pool_thread () from /lib64/libntirpc.so.3.5
#21 0x00007ff8b687114a in start_thread (arg=) at pthread_create.c:479
#22 0x00007ff8b6190dc3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) f 6
#6 operator<< (out=..., in=...) at /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/client/Inode.cc:77
77 in /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/client/Inode.cc
(gdb) p in.dentries
$27 = {_front = 0x0, _back = 0x0, _size = 0}
(gdb) p !(in.dentries._front == 0)
$28 = false
(gdb) f 7
#7 0x00007ff8acb715ae in Client::ll_sync_inode (this=0x55e3a853c7f0, in=in@entry=0x7ff6a9903120, syncdataonly=syncdataonly@entry=false) at /usr/include/c++/8/ostream:559
559 operator<<(basic_ostream<char, _Traits>& __out, const char* __s)
(gdb) p !(in->dentries._front == 0)
$29 = false
(gdb) p in
$30 = (Inode *) 0x7ff6a9903120
(gdb) p *in
$31 = {ceph::common::RefCountedObject = {}, client = 0x55e3a853c7f0, ino = {val = 1099547201712}, snapid = {val = 18446744073709551614}, faked_ino = 0, rdev = 0, ctime = {tv = {tv_sec = 1630293648, tv_nsec = 828338984}}, btime = {tv = {
tv_sec = 1630293648, tv_nsec = 825306256}}, mode = 33188, uid = 1000, gid = 1000, nlink = 1, dir_layout = {dl_dir_hash = 0 '\000', dl_unused1 = 0 '\000', dl_unused2 = 0, dl_unused3 = 0}, layout = {stripe_unit = 4194304, stripe_count = 1,
object_size = 4194304, pool_id = 8, pool_ns = ""}, size = 0, truncate_seq = 1, truncate_size = 18446744073709551615, mtime = {tv = {tv_sec = 1630293648, tv_nsec = 828311544}}, atime = {tv = {tv_sec = 1630293648, tv_nsec = 828311494}},
time_warp_seq = 2, change_attr = 2, max_size = 4194304, dirstat = {<scatter_info_t> = {version = 0}, mtime = {tv = {tv_sec = 0, tv_nsec = 0}}, change_attr = 0, nfiles = 0, nsubdirs = 0}, rstat = {<scatter_info_t> = {version = 0}, rctime = {tv = {
tv_sec = 0, tv_nsec = 0}}, rbytes = 0, rfiles = 1, rsubdirs = 0, rsnaps = 0}, version = 6030, xattr_version = 1, snap_btime = {tv = {tv_sec = 0, tv_nsec = 0}}, snap_metadata = std::map with 0 elements, inline_version = 18446744073709551615,
inline_data = {_buffers = {_root = {next = 0x7ff6a99032a8}, _tail = 0x7ff6a99032a8}, _carriage = 0x7ff8aca94b80 ceph::buffer::v15_2_0::list::always_empty_bptr, _len = 0, _num = 0, static always_empty_bptr = {ceph::buffer::v15_2_0::ptr_hook = {
next = 0x0}, ceph::buffer::v15_2_0::ptr = {_raw = 0x0, _off = 0, _len = 0}, }}, fscrypt = false, flags = 0, quota = {max_bytes = 0, max_files = 0}, dir = 0x0, dirfragtree = {
_splits = {<compact_map_base<frag_t, int, std::map<frag_t, int, std::less<frag_t>, std::allocator<std::pair<frag_t const, int> > > >> = {
map = std::unique_ptr<std::map<frag_t, int, std::less<frag_t>, std::allocator<std::pair<frag_t const, int> > >> = {get() = 0x0}}, }}, dir_release_count = 1, dir_ordered_count = 1, dir_hashed = false, dir_replicated = false,
caps = std::map with 1 element = {[0] = {inode = @0x7ff6a9903120, session = 0x55e3a8559488, cap_id = 1, issued = 16349, implemented = 16349, wanted = 13005, seq = 2, issue_seq = 1, mseq = 0, gen = 0, latest_perms = {m_uid = 1000, m_gid = 1000,
gid_count = 1, gids = 0x7ff7c846f960, alloced_gids = true}, cap_item = {_item = 0x7ff6a8b4e168, _prev = 0x7ff7014917c8, _next = 0x0, _list = 0x55e3a8559518}}}, auth_cap = 0x7ff6a8b4e168, cap_dirtier_uid = 1000, cap_dirtier_gid = 1000,
dirty_caps = 520, flushing_caps = 0, flushing_cap_tids = std::map with 0 elements, shared_gen = 1, cache_gen = 1, snap_caps = 0, snap_cap_refs = 0, hold_caps_until = {tv = {tv_sec = 0, tv_nsec = 0}}, delay_cap_item = {_item = 0x7ff6a9903120,
_prev = 0x0, _next = 0x0, _list = 0x0}, dirty_cap_item = {_item = 0x7ff6a9903120, _prev = 0x7ff6a841f610, _next = 0x0, _list = 0x55e3a853d620}, flushing_cap_item = {_item = 0x7ff6a9903120, _prev = 0x0, _next = 0x0, _list = 0x0},
snaprealm = 0x7ff848011b60, snaprealm_item = {_item = 0x7ff6a9903120, _prev = 0x7ff6a9902e68, _next = 0x0, _list = 0x7ff848011c18}, snapdir_parent = {px = 0x0}, cap_snaps = std::map with 0 elements, open_by_mode = std::map with 1 element = {[2] = 1},
cap_refs = std::map with 0 elements, oset = {parent = 0x7ff6a9903120, ino = {val = 1099547201712}, truncate_seq = 0, truncate_size = 0, poolid = 8, objects = {_front = 0x0, _back = 0x0, _size = 0}, dirty_or_tx = 0, return_enoent = false},
reported_size = 0, wanted_max_size = 0, requested_max_size = 0, ll_ref = 1, dentries = {_front = 0x0, _back = 0x0, _size = 0}, symlink = "", xattrs = std::map with 0 elements, fragmap = std::map with 0 elements, frag_repmap = std::map with 0 elements,
waitfor_caps = empty std::__cxx11::list, waitfor_commit = empty std::__cxx11::list, waitfor_deleg = empty std::__cxx11::list, fcntl_locks = std::unique_ptr<ceph_lock_state_t> = {get() = 0x0}, flock_locks = std::unique_ptr<ceph_lock_state_t> = {
get() = 0x0}, delegations = empty std::__cxx11::list, unsafe_ops = {_front = 0x0, _back = 0x0, _size = 0}, fhs = std::set with 1 element = {[0] = 0x7ff82c445b50}, dir_pin = -1}
(gdb) f 6
#6 operator<< (out=..., in=...) at /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/client/Inode.cc:77
77 /usr/src/debug/ceph-16.2.5-0.el8.x86_64/src/client/Inode.cc: No such file or directory.
(gdb) p in
$32 = (const Inode &) @0x7ff6a9903120: {ceph::common::RefCountedObject = {}, client = 0x55e3a853c7f0, ino = {val = 1099547201712}, snapid = {val = 18446744073709551614}, faked_ino = 0, rdev = 0, ctime = {tv = {tv_sec = 1630293648,
tv_nsec = 828338984}}, btime = {tv = {tv_sec = 1630293648, tv_nsec = 825306256}}, mode = 33188, uid = 1000, gid = 1000, nlink = 1, dir_layout = {dl_dir_hash = 0 '\000', dl_unused1 = 0 '\000', dl_unused2 = 0, dl_unused3 = 0}, layout = {
stripe_unit = 4194304, stripe_count = 1, object_size = 4194304, pool_id = 8, pool_ns = ""}, size = 0, truncate_seq = 1, truncate_size = 18446744073709551615, mtime = {tv = {tv_sec = 1630293648, tv_nsec = 828311544}}, atime = {tv = {
tv_sec = 1630293648, tv_nsec = 828311494}}, time_warp_seq = 2, change_attr = 2, max_size = 4194304, dirstat = {<scatter_info_t> = {version = 0}, mtime = {tv = {tv_sec = 0, tv_nsec = 0}}, change_attr = 0, nfiles = 0, nsubdirs = 0},
rstat = {<scatter_info_t> = {version = 0}, rctime = {tv = {tv_sec = 0, tv_nsec = 0}}, rbytes = 0, rfiles = 1, rsubdirs = 0, rsnaps = 0}, version = 6030, xattr_version = 1, snap_btime = {tv = {tv_sec = 0, tv_nsec = 0}},
snap_metadata = std::map with 0 elements, inline_version = 18446744073709551615, inline_data = {_buffers = {_root = {next = 0x7ff6a99032a8}, _tail = 0x7ff6a99032a8}, _carriage = 0x7ff8aca94b80 ceph::buffer::v15_2_0::list::always_empty_bptr, _len = 0,
_num = 0, static always_empty_bptr = {ceph::buffer::v15_2_0::ptr_hook = {next = 0x0}, ceph::buffer::v15_2_0::ptr = {_raw = 0x0, _off = 0, _len = 0}, }}, fscrypt = false, flags = 0, quota = {max_bytes = 0, max_files = 0},
dir = 0x0, dirfragtree = {_splits = {<compact_map_base<frag_t, int, std::map<frag_t, int, std::less<frag_t>, std::allocator<std::pair<frag_t const, int> > > >> = {
map = std::unique_ptr<std::map<frag_t, int, std::less<frag_t>, std::allocator<std::pair<frag_t const, int> > >> = {get() = 0x0}}, }}, dir_release_count = 1, dir_ordered_count = 1, dir_hashed = false, dir_replicated = false,
caps = std::map with 1 element = {[0] = {inode = @0x7ff6a9903120, session = 0x55e3a8559488, cap_id = 1, issued = 16349, implemented = 16349, wanted = 13005, seq = 2, issue_seq = 1, mseq = 0, gen = 0, latest_perms = {m_uid = 1000, m_gid = 1000,
gid_count = 1, gids = 0x7ff7c846f960, alloced_gids = true}, cap_item = {_item = 0x7ff6a8b4e168, _prev = 0x7ff7014917c8, _next = 0x0, _list = 0x55e3a8559518}}}, auth_cap = 0x7ff6a8b4e168, cap_dirtier_uid = 1000, cap_dirtier_gid = 1000,
dirty_caps = 520, flushing_caps = 0, flushing_cap_tids = std::map with 0 elements, shared_gen = 1, cache_gen = 1, snap_caps = 0, snap_cap_refs = 0, hold_caps_until = {tv = {tv_sec = 0, tv_nsec = 0}}, delay_cap_item = {_item = 0x7ff6a9903120,
_prev = 0x0, _next = 0x0, _list = 0x0}, dirty_cap_item = {_item = 0x7ff6a9903120, _prev = 0x7ff6a841f610, _next = 0x0, _list = 0x55e3a853d620}, flushing_cap_item = {_item = 0x7ff6a9903120, _prev = 0x0, _next = 0x0, _list = 0x0},
snaprealm = 0x7ff848011b60, snaprealm_item = {_item = 0x7ff6a9903120, _prev = 0x7ff6a9902e68, _next = 0x0, _list = 0x7ff848011c18}, snapdir_parent = {px = 0x0}, cap_snaps = std::map with 0 elements, open_by_mode = std::map with 1 element = {[2] = 1},
cap_refs = std::map with 0 elements, oset = {parent = 0x7ff6a9903120, ino = {val = 1099547201712}, truncate_seq = 0, truncate_size = 0, poolid = 8, objects = {_front = 0x0, _back = 0x0, _size = 0}, dirty_or_tx = 0, return_enoent = false},
reported_size = 0, wanted_max_size = 0, requested_max_size = 0, ll_ref = 1, dentries = {_front = 0x0, _back = 0x0, _size = 0}, symlink = "", xattrs = std::map with 0 elements, fragmap = std::map with 0 elements, frag_repmap = std::map with 0 elements,
waitfor_caps = empty std::__cxx11::list, waitfor_commit = empty std::__cxx11::list, waitfor_deleg = empty std::__cxx11::list, fcntl_locks = std::unique_ptr<ceph_lock_state_t> = {get() = 0x0}, flock_locks = std::unique_ptr<ceph_lock_state_t> = {
get() = 0x0}, delegations = empty std::__cxx11::list, unsafe_ops = {_front = 0x0, _back = 0x0, _size = 0}, fhs = std::set with 1 element = {[0] = 0x7ff82c445b50}, dir_pin = -1}
</pre></p>
CephFS - Bug #50695 (Need More Info): nautilus: qa: Test failure: test_kill_mdstable (tasks.cephf...
https://tracker.ceph.com/issues/50695
2021-05-07T21:27:32Z
Ramana Raja
rraja@redhat.com
<p>See this here,<br /><a class="external" href="https://pulpito.ceph.com/yuriw-2021-05-04_15:32:03-multimds-wip-yuri3-testing-2021-04-29-1036-nautilus-distro-basic-smithi/6094466/">https://pulpito.ceph.com/yuriw-2021-05-04_15:32:03-multimds-wip-yuri3-testing-2021-04-29-1036-nautilus-distro-basic-smithi/6094466/</a></p>
<p>Description: multimds/basic/{0-supported-random-distro$/{centos_latest} begin clusters/3-mds conf/{client mds mon osd} inline/no mount/fuse objectstore-ec/bluestore-comp overrides/{basic/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} fuse-default-perm-no} q_check_counter/check_counter tasks/cephfs_test_snapshots}</p>
<pre>
2021-05-05T03:32:14.812 INFO:tasks.cephfs_test_runner:
2021-05-05T03:32:14.813 INFO:tasks.cephfs_test_runner:======================================================================
2021-05-05T03:32:14.813 INFO:tasks.cephfs_test_runner:ERROR: test_kill_mdstable (tasks.cephfs.test_snapshots.TestSnapshots)
2021-05-05T03:32:14.814 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-05-05T03:32:14.815 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-05-05T03:32:14.816 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_36801f537d3dceb7c135151b37ba843b7c595bbe/qa/tasks/cephfs/test_snapshots.py", line 167, in test_kill_mdstable
2021-05-05T03:32:14.816 INFO:tasks.cephfs_test_runner: self.wait_for_daemon_start([rank1['name']])
2021-05-05T03:32:14.817 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_36801f537d3dceb7c135151b37ba843b7c595bbe/qa/tasks/cephfs/cephfs_test_case.py", line 261, in wait_for_daemon_start
2021-05-05T03:32:14.817 INFO:tasks.cephfs_test_runner: timeout=30
2021-05-05T03:32:14.818 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_36801f537d3dceb7c135151b37ba843b7c595bbe/qa/tasks/ceph_test_case.py", line 197, in wait_until_true
2021-05-05T03:32:14.819 INFO:tasks.cephfs_test_runner: raise TestTimeoutError("Timed out after {0}s".format(elapsed))
2021-05-05T03:32:14.819 INFO:tasks.cephfs_test_runner:tasks.ceph_test_case.TestTimeoutError: Timed out after 30s
2021-05-05T03:32:14.820 INFO:tasks.cephfs_test_runner:
2021-05-05T03:32:14.821 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-05-05T03:32:14.821 INFO:tasks.cephfs_test_runner:Ran 1 test in 451.396s
</pre>
CephFS - Bug #43637 (Triaged): nautilus: qa: Health check failed: Reduced data availability: 16 p...
https://tracker.ceph.com/issues/43637
2020-01-16T17:45:46Z
Ramana Raja
rraja@redhat.com
<pre>
2020-01-10T10:41:33.742 INFO:teuthology.orchestra.run.smithi166:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'not responding, replacing' | egrep -v '\(MDS_INSUFFICIENT_STANDBY\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1
2020-01-10T10:41:33.765 INFO:teuthology.orchestra.run.smithi166.stdout:2020-01-10 10:22:15.090135 mon.b (mon.0) 1750 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY)
</pre>
<p>From /a/yuriw-2020-01-09_22:23:54-fs-wip-yuri6-testing-2020-01-09-1744-nautilus-distro-basic-smithi/4650211/</p>
<p>Seen this failure in 12 jobs in fs suite and kcephfs suite,</p>
<p>In fs suite, /a/yuriw-2020-01-09_22:23:54-fs-wip-yuri6-testing-2020-01-09-1744-nautilus-distro-basic-smithi/<br /><pre>
Failure: "2020-01-10 10:22:15.090135 mon.b (mon.0) 1750 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY)" in cluster log
5 jobs: ['4650211', '4650242', '4650177', '4650144', '4650276']
suites intersection: ['clusters/1a3s-mds-2c-client.yaml', 'conf/{client.yaml', 'fs/multifs/{begin.yaml', 'mds.yaml', 'mon-debug.yaml', 'mon.yaml', 'mount/fuse.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'tasks/failover.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
suites union: ['clusters/1a3s-mds-2c-client.yaml', 'conf/{client.yaml', 'fs/multifs/{begin.yaml', 'mds.yaml', 'mon-debug.yaml', 'mon.yaml', 'mount/fuse.yaml', 'objectstore-ec/bluestore-bitmap.yaml', 'objectstore-ec/bluestore-comp-ec-root.yaml', 'objectstore-ec/bluestore-comp.yaml', 'objectstore-ec/bluestore-ec-root.yaml', 'objectstore-ec/filestore-xfs.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'supported-random-distros$/{centos_latest.yaml}', 'supported-random-distros$/{ubuntu_16.04.yaml}', 'supported-random-distros$/{ubuntu_latest.yaml}', 'tasks/failover.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
</pre></p>
<p>In kcephfs suite, /a/yuriw-2020-01-09_22:20:53-kcephfs-wip-yuri6-testing-2020-01-09-1744-nautilus-distro-basic-smithi,<br /><pre>
Failure: "2020-01-10 01:06:47.884132 mon.a (mon.0) 2248 : cluster [WRN] Health check failed: Reduced data availability: 16 pgs inactive (PG_AVAILABILITY)" in cluster log
7 jobs: ['4649992', '4649994', '4650026', '4650028', '4650030', '4650048', '4649990']
suites intersection: ['clusters/1-mds-4-client.yaml', 'conf/{client.yaml', 'kcephfs/recovery/{begin.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{frag_enable.yaml', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
suites union: ['clusters/1-mds-4-client.yaml', 'conf/{client.yaml', 'kcephfs/recovery/{begin.yaml', 'kclient/{mount.yaml', 'log-config.yaml', 'mds.yaml', 'mon.yaml', 'ms-die-on-skipped.yaml}}', 'objectstore-ec/bluestore-bitmap.yaml', 'objectstore-ec/bluestore-comp.yaml', 'objectstore-ec/bluestore-ec-root.yaml', 'objectstore-ec/filestore-xfs.yaml', 'osd-asserts.yaml', 'osd.yaml}', 'overrides/{distro/random/{k-testing.yaml', 'overrides/{distro/rhel/{k-distro.yaml', 'overrides/{frag_enable.yaml', 'rhel_latest.yaml}', 'supported$/{rhel_latest.yaml}}', 'supported$/{ubuntu_latest.yaml}}', 'tasks/damage.yaml}', 'tasks/data-scan.yaml}', 'tasks/failover.yaml}', 'tasks/volume-client.yaml}', 'whitelist_health.yaml', 'whitelist_wrongly_marked_down.yaml}']
</pre></p>