Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2024-03-26T01:42:35ZCeph
Redmine crimson - Bug #65130 (Fix Under Review): crimson: crimson-rados did not detect reintroduction of ...https://tracker.ceph.com/issues/651302024-03-26T01:42:35ZSamuel Justsjust@redhat.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/56376">https://github.com/ceph/ceph/pull/56376</a> would have reintroduced <a class="external" href="https://tracker.ceph.com/issues/61875">https://tracker.ceph.com/issues/61875</a> as it puts the snap mapper keys back into the pg meta object. Oddly, a teuthology run on that branch which seems to have included tests with both snapshots and osd restarts did not show crashes associated with this regression and at least one case that seems like it should have exercised the relevant code passed. A quick glance over PGLog.cc::FuturizedShardStoreReader doesn't show any changes, so it should have crashed in the final else branch of FuturizedShardStoreLogReader::process_entry at e.decode_with_checksum.</p>
<p>Tasks:<br />- Confirm that the crimson-rados suite actually combines snapshots with OSD restarts<br />- Work out why the existing suite didn't fail the above PR<br />- Amend the tests to cover the gap</p> CephFS - Bug #65039 (Triaged): mds: standby-replay segmentation fault in md_log_replayhttps://tracker.ceph.com/issues/650392024-03-21T14:19:46ZPatrick Donnellypdonnell@redhat.com
<pre>
2024-03-21T03:15:55.310 INFO:journalctl@ceph.mds.h.smithi060.stdout:Mar 21 03:15:55 smithi060 ceph-87dd0fc6-e72e-11ee-95c9-87774f69a715-mds-h[71557]: *** Caught signal (Segmentation fault) **
2024-03-21T03:15:55.310 INFO:journalctl@ceph.mds.h.smithi060.stdout:Mar 21 03:15:55 smithi060 ceph-87dd0fc6-e72e-11ee-95c9-87774f69a715-mds-h[71557]: in thread 7f7135d7c700 thread_name:md_log_replay
</pre>
<p>From: /teuthology/pdonnell-2024-03-21_02:37:43-fs:workload-main-distro-default-smithi/7614435/teuthology.log</p>
<p>I logged into the machine and collected a gdb stack trace (attached). Initially I was looking for a deadlock not a segmentation fault. The signal handler for SIGSEGV got deadlocked (predictably) because it was using malloc:</p>
<pre>
Thread 26 (Thread 0x7f7135d7c700 (LWP 72204)):
#0 0x00007f7148e163d0 in base::internal::SpinLockDelay(int volatile*, int, int) () from /lib64/libtcmalloc.so.4
#1 0x00007f7148e162d3 in SpinLock::SlowLock() () from /lib64/libtcmalloc.so.4
#2 0x00007f7148e05a55 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4
#3 0x00007f7148e093e3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long)) () from /lib64/libtcmalloc.so.4
#4 0x00007f71484409b3 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*> () from /usr/lib64/ceph/libceph-common.so.2
#5 0x00007f7148440aa9 in ceph::ClibBackTrace::demangle[abi:cxx11](char const*) () from /usr/lib64/ceph/libceph-common.so.2
#6 0x00007f7148441025 in ceph::ClibBackTrace::print(std::ostream&) const () from /usr/lib64/ceph/libceph-common.so.2
#7 0x000055c9ae7266dd in handle_oneshot_fatal_signal (signum=11) at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/global/signal_handler.cc:331
#8 <signal handler called>
#9 0x00007f7148e05603 in tcmalloc::CentralFreeList::FetchFromOneSpans(int, void**, void**) () from /lib64/libtcmalloc.so.4
#10 0x00007f7148e058ae in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) () from /lib64/libtcmalloc.so.4
#11 0x00007f7148e05971 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) () from /lib64/libtcmalloc.so.4
#12 0x00007f7148e093e3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned int, int, void* (*)(unsigned long)) () from /lib64/libtcmalloc.so.4
#13 0x000055c9ae311e17 in EMetaBlob::fullbit::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) () at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/include/compact_map.h:27
#14 0x000055c9ae31429d in EMetaBlob::dirlump::_decode_bits (this=0x55c9b25c9770) at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/mds/events/EMetaBlob.h:609
#15 0x000055c9ae31c397 in EMetaBlob::replay(MDSRank*, LogSegment*, int, MDPeerUpdate*) () at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/mds/events/EMetaBlob.h:296
#16 0x000055c9ae322551 in EUpdate::replay(MDSRank*) () at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/mds/journal.cc:2252
#17 0x000055c9ae64dd97 in MDLog::_replay_thread (this=0x55c9b18e6000) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:421
#18 0x000055c9ae6543b1 in MDLog::ReplayThread::entry (this=<optimized out>) at /usr/src/debug/ceph-19.0.0-2244.gcab8141b.el8.x86_64/src/mds/MDLog.h:181
#19 0x00007f71471331ca in start_thread () from /lib64/libpthread.so.0
#20 0x00007f71456308d3 in clone () from /lib64/libc.so.6
</pre>
<p>Unfortunately, I didn't get a chance to dig into frame <a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: uclient: Make readdir use the cache (Resolved)" href="https://tracker.ceph.com/issues/13">#13</a> to see why it segfaulted.</p> CephFS - Bug #65020 (Triaged): qa: Scrub error on inode 0x1000000356c (/volumes/qa/sv_0/2f8f6bb4-...https://tracker.ceph.com/issues/650202024-03-21T00:37:10ZPatrick Donnellypdonnell@redhat.com
<p><a class="external" href="https://pulpito.ceph.com/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7612910/">https://pulpito.ceph.com/pdonnell-2024-03-20_18:16:52-fs-wip-batrick-testing-20240320.145742-distro-default-smithi/7612910/</a></p>
<p>and many others.</p>
<p>More fallout after <a class="external" href="https://github.com/ceph/ceph/pull/55455">https://github.com/ceph/ceph/pull/55455</a> was merged.</p> crimson - Bug #64975 (In Progress): crimson: Health check failed: 9 scrub errors (OSD_SCRUB_ERR...https://tracker.ceph.com/issues/649752024-03-18T21:45:04ZSamuel Justsjust@redhat.com
<p>ERROR 2024-03-15 10:04:01,561 [shard 1:main] osd - pg_epoch 198 pg[2.2( empty local-lis/les=11/12 n=0 ec=11/11 lis/c=11/11 les/c/f=12/12/0 sis=11) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean+scrubbing+deep PGScrubber::emit_chunk_result: Scrub errors found. range: start: MIN, end: MAX, result: chunk_result_t(num_scrub_errors: 1, num_deep_scrub_errors: 0, snapset_errors: [[]], object_errors: [[inconsistent_obj_t(error: , object: //snapmapper, version: 0, shards: {osd_shard_t(osd: 0, shard: -1): shard_info_t(error: INFO_MISSING, size: 0, omap_digest_present: true, omap_digest: 4294967295, data_digest_present: true, data_digest: 4294967295, selected_io: false, primary: true), osd_shard_t(osd: 1, shard: -1): shard_info_t(error: INFO_MISSING, size: 0, omap_digest_present: true, omap_digest: 4294967295, data_digest_present: true, data_digest: 4294967295, selected_io: false, primary: false)}, union_shards: INFO_MISSING)]])</p>
<p>The bug is that scrub isn't skipping the snap mapper object.</p> CephFS - Bug #64563 (Triaged): mds: enhance laggy clients detections due to laggy OSDshttps://tracker.ceph.com/issues/645632024-02-26T11:52:01ZDhairya Parmar
<p>Right now the code happily accepts that if there is any laggy OSD and a client got laggy then it must be due to the OSD, the code is note able to differentiate between "this client isn't responding" and "this client is slow to release caps" meaning if the client went off the grid AND we have any laggy OSD, MDS will not evict that client and instead mark it as "laggy due to laggy OSDs" which is completely wrong. There are five instances where clients are added to set `laggy_clients` in `Server.cc` which needs to re-evaluated and make sure that we do consider client eviction in cases like [0] where the last cap renew span is more than the session autoclose duration(i.e. 300 seconds default config) which is long enough to conclude that we have lost the client.</p>
<p>Not only this, there needs to be some sane/practical values for which we consider any osd to be worthy enough to be considered laggy i.e. "laggy enough to make a client(or anything) laggy", current implementation is too naive as it just checks whether any laggy param(osd_xinfo_t.laggy_interval or osd_xinfo_t.laggy_probability) is non-zero. This will make MDS not evict clients even though that slight OSD lagginess might not be that serious at all. In other words we need to make the helper OSDMap::any_osd_laggy a bit more smart and fine-grained.</p>
<p>[0] <a class="external" href="https://github.com/ceph/ceph/blob/main/src/mds/Server.cc#L1184-L1190">https://github.com/ceph/ceph/blob/main/src/mds/Server.cc#L1184-L1190</a></p> CephFS - Bug #64502 (New): pacific/quincy/v18.2.0: client: ceph-fuse fails to unmount after upgra...https://tracker.ceph.com/issues/645022024-02-20T02:15:08ZPatrick Donnellypdonnell@redhat.com
<p>Every ceph-fuse mount for quincy fails to unmount for reef->main:</p>
<p><a class="external" href="https://pulpito.ceph.com/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/">https://pulpito.ceph.com/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/</a></p>
<pre>
2024-02-19T19:17:36.535 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi060.front.sepia.ceph.com...
2024-02-19T19:17:36.535 INFO:teuthology.orchestra.run:Running command with timeout 300
2024-02-19T19:17:36.535 DEBUG:teuthology.orchestra.run.smithi060:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0
2024-02-19T19:17:36.562 INFO:teuthology.orchestra.run:waiting for 300
</pre>
<p>From: /teuthology/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566635/teuthology.log</p>
<pre>
2024-02-19T19:17:36.799+0000 7f9fa7fff640 20 client.14548 tick
2024-02-19T19:17:36.799+0000 7f9fa7fff640 20 client.14548 collect_and_send_metrics
2024-02-19T19:17:36.799+0000 7f9fa7fff640 20 client.14548 collect_and_send_global_metrics
2024-02-19T19:17:36.799+0000 7f9fa7fff640 1 -- 192.168.0.1:0/854663557 --> [v2:172.21.15.60:6826/3594652577,v1:172.21.15.60:6827/3594652577] -- client_metrics [client_metric_type: READ_LATENCY latency: 5.996942, avg_latency: 0.000330, sq_sum: 86627012816408144, count=17901][client_metric_type: WRITE_LATENCY latency: 23.710221, avg_latency: 0.000407, sq_sum: 1890169673992666112, count=56281][client_metric_type: METADATA_LATENCY latency: 238.430933, avg_latency: 0.005247, sq_sum: 13600282437617256448, count=45341][client_metric_type: CAP_INFO cap_hits: 831286 cap_misses: 14792 num_caps: 0][client_metric_type: DENTRY_LEASE dlease_hits: 67 dlease_misses: 154700 num_dentries: 0][client_metric_type: OPENED_FILES opened_files: 0 total_inodes: 1][client_metric_type: PINNED_ICAPS pinned_icaps: 1 total_inodes: 1][client_metric_type: OPENED_INODES opened_inodes: 0 total_inodes: 1][client_metric_type: READ_IO_SIZES total_ops: 22272 total_size: 3731108728][client_metric_type: WRITE_IO_SIZES total_ops: 56281 total_size: 4270138133] v1 -- 0x7f9fa000b9e0 con 0x5637e76f0e80
2024-02-19T19:17:36.799+0000 7f9fa7fff640 20 client.14548 trim_cache size 1 max 16384
2024-02-19T19:17:36.799+0000 7f9fa7fff640 20 client.14548 upkeep thread waiting interval 1.000000000s
...
2024-02-19T20:23:30.865+0000 7f9fc8e36480 2 client.14548 unmounting
</pre>
<p>From: /teuthology/pdonnell-2024-02-19_18:28:45-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7566635/remote/smithi060/log/ceph-client.0.log.gz</p>
<p>During teardown of the cluster the unmount eventual proceeds but it's not clear what was blocking it. I think something was holding the RWRef preventing unmount from proceeding.<br /></pre></p> Linux kernel client - Bug #64471 (New): kernel: upgrades from quincy/v18.2.[01]/reef to main|squi...https://tracker.ceph.com/issues/644712024-02-16T15:14:45ZPatrick Donnellypdonnell@redhat.com
<pre>
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout:{
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275",
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "in_progress": true,
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "which": "Upgrading all daemon types on all hosts",
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "services_complete": [],
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "progress": "",
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:f78a58c0ffd401d1493058a1022c35f011d65275 image",
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout: "is_paused": false
2024-02-16T02:34:47.515 INFO:teuthology.orchestra.run.smithi032.stdout:}
...
2024-02-16T02:34:52.075 INFO:tasks.workunit.client.0.smithi032.stderr:+ pushd fsstress
2024-02-16T02:34:52.076 INFO:tasks.workunit.client.0.smithi032.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp
2024-02-16T02:34:52.076 INFO:tasks.workunit.client.0.smithi032.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz
2024-02-16T02:35:10.364 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:09 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:09.988+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6804 osd.3 since back 2024-02-16T02:34:45.611069+0000 front 2024-02-16T02:34:45.611050+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
2024-02-16T02:35:10.364 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:09 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:09.988+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6812 osd.4 since back 2024-02-16T02:34:45.611191+0000 front 2024-02-16T02:34:45.611160+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
2024-02-16T02:35:10.364 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:09 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:09.988+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6820 osd.5 since back 2024-02-16T02:34:45.611126+0000 front 2024-02-16T02:34:45.611214+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
2024-02-16T02:35:11.316 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:10 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:10.982+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6804 osd.3 since back 2024-02-16T02:34:45.611069+0000 front 2024-02-16T02:34:45.611050+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
2024-02-16T02:35:11.316 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:10 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:10.982+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6812 osd.4 since back 2024-02-16T02:34:45.611191+0000 front 2024-02-16T02:34:45.611160+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
2024-02-16T02:35:11.317 INFO:journalctl@ceph.osd.2.smithi032.stdout:Feb 16 02:35:10 smithi032 ceph-a2883a70-cc72-11ee-95ba-87774f69a715-osd-2[62304]: 2024-02-16T02:35:10.982+0000 7f5e5529a700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.196:6820 osd.5 since back 2024-02-16T02:34:45.611126+0000 front 2024-02-16T02:34:45.611214+0000 (oldest deadline 2024-02-16T02:35:09.711083+0000)
</pre>
<p>From: /teuthology/pdonnell-2024-02-16_01:25:08-fs:upgrade:mds_upgrade_sequence-wip-batrick-testing-20240215.160715-distro-default-smithi/7561891/teuthology.log</p>
<p>and many others in that run. It's quite reproducible. I don't think it happens with quincy -> main.</p>
<p>This might be related to <a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: lots of "heartbeat_check: no reply from X.X.X.X" in OSD logs (Resolved)" href="https://tracker.ceph.com/issues/55258">#55258</a>.</p> Ceph - Bug #64279 (New): "Error ENOTSUP: Warning: due to ceph-mgr restart" in octopus-x/pacific s...https://tracker.ceph.com/issues/642792024-01-31T22:53:19ZYuri Weinsteinyweinste@redhat.com
<p>This is for 16.2.15</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/">https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/</a><br />Jobs: 7540432 7540434 7540435<br />Logs: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/7540432/">https://pulpito.ceph.com/yuriw-2024-01-31_16:13:03-upgrade:octopus-x-pacific-release-distro-default-smithi/7540432/</a></p>
<pre>
2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Error ENOTSUP: Warning: due to ceph-mgr restart, some PG states may not be up to date
2024-01-31T17:36:17.722 INFO:teuthology.orchestra.run.smithi159.stderr:Module 'orchestrator' is not enabled/loaded (required by command 'orch ps'): use `ceph mgr module enable orchestrator` to enable it
</pre>
<pre>
024-01-31T17:36:17.933 INFO:journalctl@ceph.mon.a.smithi159.stdout:Jan 31 17:36:17 smithi159 ceph-157b0ed4-c05e-11ee-95b3-87774f69a715-mon-a[70364]: cluster 2024-01-31T17:36:17.656902+0000 mon.a (mon.0) 62 : cluster [INF] Manager daemon y is now available
2024-01-31T17:36:18.034 DEBUG:teuthology.orchestra.run:got remote process result: 95
2024-01-31T17:36:18.035 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 56, in task
p.spawn(_run_spawned, ctx, confg, taskname)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/parallel.py", line 64, in _run_spawned
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/task/sequential.py", line 47, in task
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 1058, in shell
_shell(ctx, cluster_name, remote,
File "/home/teuthworker/src/github.com_ceph_ceph-c_88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b/qa/tasks/cephadm.py", line 34, in _shell
return remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 95: "sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:octopus shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 157b0ed4-c05e-11ee-95b3-87774f69a715 -e sha1=88fb4c6adb4bae6b1cdf34fca7eae2dacb06cc7b -- bash -c 'ceph orch ps'"
2024-01-31T17:36:18.263 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=05855bfb45164e1fa69621bf999495c7
</pre> Ceph - Bug #64256 (New): "Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 resu...https://tracker.ceph.com/issues/642562024-01-30T20:17:42ZYuri Weinsteinyweinste@redhat.com
<p>This is for 16.2.15 release</p>
<p>Run: <a class="external" href="https://pulpito.ceph.com/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/">https://pulpito.ceph.com/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/</a><br />Jobs: '7538534', '7538533', '7538535', '7538528'</p>
<p>Logs: <a class="external" href="http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/teuthology.log">http://qa-proxy.ceph.com/teuthology/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/teuthology.log</a></p>
<pre>
2024-01-30T16:09:48.249 INFO:teuthology.task.ansible:Archiving ansible failure log at: /home/teuthworker/archive/yuriw-2024-01-30_15:54:26-upgrade:pacific-x-quincy-distro-default-smithi/7538528/ansible_failures.yaml
2024-01-30T16:09:48.251 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 508, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 342, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 374, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 402, in _handle_failure
raise AnsibleFailedError(failures)
teuthology.exceptions.AnsibleFailedError: smithi174.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []smithi006.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []
2024-01-30T16:09:48.630 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3cf9a763f1224702b05b0ad425b4fe45
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/run_tasks.py", line 109, in run_tasks
manager.__enter__()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/__init__.py", line 123, in __enter__
self.begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 508, in begin
super(CephLab, self).begin()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 342, in begin
self.execute_playbook()
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 374, in execute_playbook
self._handle_failure(command, status)
File "/home/teuthworker/src/git.ceph.com_teuthology_6128cc3ecb49b7d62475e3595041c19b5326ca7c/teuthology/task/ansible.py", line 402, in _handle_failure
raise AnsibleFailedError(failures)
teuthology.exceptions.AnsibleFailedError: smithi174.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []smithi006.front.sepia.ceph.com: _ansible_no_log: null changed: false invocation: module_args: allow_downgrade: false allowerasing: false autoremove: false bugfix: false cacheonly: false conf_file: null disable_excludes: null disable_gpg_check: false disable_plugin: [] disablerepo: [] download_dir: null download_only: false enable_plugin: [] enablerepo: [] exclude: [] install_repoquery: true install_weak_deps: true installroot: / list: null lock_timeout: 30 name: - krb5-workstation nobest: false releasever: null security: false skip_broken: false sslverify: true state: present update_cache: false update_only: false validate_certs: true msg: 'Failed to download metadata for repo ''codeready-builder-for-rhel-8-x86_64-rpms'': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried' rc: 1 results: []
2024-01-30T16:09:48.632 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab
2024-01-30T16:09:48.645 INFO:teuthology.task.ansible:Skipping ansible cleanup...
</pre> Ceph - Bug #64254 (New): PR check builds fail because clang isn't availablehttps://tracker.ceph.com/issues/642542024-01-30T19:40:35ZCasey Bodleycbodley@redhat.com
<p>from <a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/128387/consoleFull">https://jenkins.ceph.com/job/ceph-pull-requests/128387/consoleFull</a></p>
<p>during <code>src/script/run-make.sh</code>, <code>install-deps.sh</code> fails to install clang:<br /><pre>
The following packages have unmet dependencies:
libclang-common-14-dev : Depends: libllvm14 (= 1:14.0.0-1ubuntu1.1) but 1:14.0.6~++20230131082223+f28c006a5895-1~exp1~20230131082249.127 is to be installed
libclang-cpp14 : Depends: libllvm14 (= 1:14.0.0-1ubuntu1.1) but 1:14.0.6~++20230131082223+f28c006a5895-1~exp1~20230131082249.127 is to be installed
E: Unable to correct problems, you have held broken packages.
</pre></p>
<p><code>discover_compiler()</code> looks for the latest clang version, but doesn't find any so defaults to gcc:<br /><pre>
CI_DEBUG: Finding compiler for ci-build
CI_DEBUG: Our cmake_opts are: -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER=gcc
</pre></p>
<p>the build ends up failing due to warnings from the cpp_redis submodule:<br /><pre>
[243/2719] Building CXX object src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o
FAILED: src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o
/usr/bin/ccache /usr/bin/g++ -DBOOST_ASIO_DISABLE_THREAD_KEYWORD_EXTENSION -DBOOST_ASIO_HAS_IO_URING -DBOOST_ASIO_NO_TS_EXECUTORS -DHAVE_CONFIG_H -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -D_REENTRANT -D_THREAD_SAFE -D__CEPH__ -D__STDC_FORMAT_MACROS -D__linux__ -I/home/jenkins-build/build/workspace/ceph-pull-requests/build/src/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/src -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/deps/include -I/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/tacopie/includes -isystem /opt/ceph/include -isystem /home/jenkins-build/build/workspace/ceph-pull-requests/build/include -std=c++11 -W -Wall -Wextra -O3 -Og -Werror -fPIC -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free -DBOOST_PHOENIX_STL_TUPLE_H_ -Wall -fno-strict-aliasing -fsigned-char -Wtype-limits -Wignored-qualifiers -Wpointer-arith -Werror=format-security -Winit-self -Wno-unknown-pragmas -Wnon-virtual-dtor -Wno-ignored-qualifiers -ftemplate-depth-1024 -Wpessimizing-move -Wredundant-move -Wstrict-null-sentinel -Woverloaded-virtual -DCEPH_DEBUG_MUTEX -fstack-protector-strong -D_GLIBCXX_ASSERTIONS -fdiagnostics-color=auto -std=c++20 -MD -MT src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o -MF src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o.d -o src/cpp_redis/CMakeFiles/cpp_redis.dir/sources/core/client.cpp.o -c /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/client.hpp:35,
from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:23:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp: In lambda function:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp:109:58: error: implicit capture of ‘this’ via ‘[=]’ is deprecated in C++20 [-Werror=deprecated]
109 | if (!m_sync_condvar.wait_for(lock_callback, timeout, [=] {
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/includes/cpp_redis/core/sentinel.hpp:109:58: note: add explicit ‘this’ or ‘*this’ capture
In file included from /home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:23:
(snip)
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp: In lambda function:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:4031:19: error: implicit capture of ‘this’ via ‘[=]’ is deprecated in C++20 [-Werror=deprecated]
4031 | return exec_cmd([=](const reply_callback_t& cb) -> client& { return zunionstore(destination, numkeys, keys, weights, method, cb); });
| ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/cpp_redis/sources/core/client.cpp:4031:19: note: add explicit ‘this’ or ‘*this’ capture
cc1plus: all warnings being treated as errors
</pre></p>
<p>in earlier successful builds (ex. <a class="external" href="https://jenkins.ceph.com/job/ceph-pull-requests/128316/consoleFull">https://jenkins.ceph.com/job/ceph-pull-requests/128316/consoleFull</a>):<br /><pre>
clang is already the newest version (1:14.0-55~exp2).
...
CI_DEBUG: Finding compiler for ci-build
CI_DEBUG: Our cmake_opts are: -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14
</pre><br />and cpp_redis compiles/links without warnings/errors<br /><pre>
[317/2719] Linking CXX static library lib/libcpp_redis.a
</pre></p> Linux kernel client - Bug #64172 (Fix Under Review): Test failure: test_multiple_path_r (tasks.ce...https://tracker.ceph.com/issues/641722024-01-25T05:55:58ZVenky Shankarvshankar@redhat.com
<p>/a/vshankar-2024-01-22_07:03:31-fs-wip-vshankar-testing-20240119.075157-1-testing-default-smithi/7525717</p>
<p>The test setup involves "read" cap on a file system path (directory), remount the directory as file system root and read the created files.</p>
<p>MDS logs: ./remote/smithi157/log/ceph-mds.c.log.gz</p>
<pre>
2024-01-22T08:27:55.205+0000 7f81a7600640 1 -- [v2:172.21.15.157:6835/1231338113,v1:172.21.15.157:6837/1231338113] <== client.17628 v1:192.168.0.1:0/312855551 9 ==== client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6 ==== 176+0+0 (unknown 772642831 0 0) 0x55ab1d52bb00 con 0x55ab1d52e400
2024-01-22T08:27:55.205+0000 7f81a7600640 4 mds.0.server handle_client_request client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 20 mds.0.356 get_session have 0x55ab1d202f00 client.17628 v1:192.168.0.1:0/312855551 state open
2024-01-22T08:27:55.205+0000 7f81a7600640 15 mds.0.server oldest_client_tid=5
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.cache request_start request(client.17628:5 nref=2 cr=0x55ab1d52bb00)
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.server dispatch_client_request client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 20 Session check_access path
2024-01-22T08:27:55.205+0000 7f81a7600640 10 MDSAuthCap is_capable inode(path / owner 0:0 mode 041777) by caller 0:0 mask 0 new 0:0 cap: MDSAuthCaps[allow r fsname=cephfs path="/dir1/dir12", allow r fsname=cephfs path="/dir2/dir22"]
2024-01-22T08:27:55.205+0000 7f81a7600640 7 mds.0.server reply_client_request -13 ((13) Permission denied) client_request(client.17628:5 lookupino #0x1 2024-01-22T08:27:55.205939+0000 caller_uid=0, caller_gid=0{0,}) v6
2024-01-22T08:27:55.205+0000 7f81a7600640 10 mds.0.server apply_allocated_inos 0x0 / [] / 0x0
2024-01-22T08:27:55.205+0000 7f81a7600640 20 mds.0.server lat 0.000095
2024-01-22T08:27:55.205+0000 7f81a7600640 10 mds.0.356 send_message_client client.17628 v1:192.168.0.1:0/312855551 client_reply(???:5 = -13 (13) Permission denied) v1
2024-01-22T08:27:55.205+0000 7f81a7600640 1 -- [v2:172.21.15.157:6835/1231338113,v1:172.21.15.157:6837/1231338113] --> v1:192.168.0.1:0/312855551 -- client_reply(???:5 = -13 (13) Permission denied) v1 -- 0x55ab1d5b4700 con 0x55ab1d52e400
</pre> Ceph - Bug #63617 (New): ceph-common: CommonSafeTimer<std::mutex>::timer_thread(): python3.12 kil...https://tracker.ceph.com/issues/636172023-11-23T18:40:45ZKaleb KEITHLEY
<p><a class="external" href="https://bugzilla.redhat.com/show_bug.cgi?id=2251165">https://bugzilla.redhat.com/show_bug.cgi?id=2251165</a></p>
<p>Description of problem:</p>
<p>Version-Release number of selected component:<br />ceph-common-2:18.2.1-1.fc39</p>
<p>Additional info:<br />reporter: libreport-2.17.11<br />cmdline: /usr/bin/python3.12 /usr/bin/ceph -s<br />backtrace_rating: 4<br />runlevel: N 5<br />executable: /usr/bin/python3.12<br />journald_cursor: s=9f8a7a66b4194fdcbd75dcd3edf4da87;i=173e8c976;b=a08b8db920744522980a5387af245706;m=2743cc1c;t=60accf74a277f;x=cef1ac3a8dc81a9d<br />comment: <br />cgroup: 0::/user.slice/user-1000.slice/user/app.slice/app-org.kde.konsole-44b42a69b68946748c9899bd38ac8c6d.scope<br />kernel: 6.6.2-200.fc39.x86_64<br />uid: 1000<br />rootdir: /<br />crash_function: CommonSafeTimer<std::mutex>::timer_thread<br />type: CCpp<br />package: ceph-common-2:18.2.1-1.fc39<br />reason: python3.12 killed by SIGSEGV</p>
<p>Truncated backtrace:<br />Thread no. 1 (3 frames)<br /> #0 CommonSafeTimer<std::mutex>::timer_thread at /usr/src/debug/ceph-18.2.1-1.fc39.x86_64/src/common/Timer.cc:103<br /> <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a> CommonSafeTimerThread<std::mutex>::entry at /usr/src/debug/ceph-18.2.1-1.fc39.x86_64/src/common/Timer.cc:33<br /> <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: leaked dentry ref on umount (Closed)" href="https://tracker.ceph.com/issues/3">#3</a> clone3 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78</p> sepia - Bug #63567 (New): apt-key add is deprecated, must stop using ansible apt_key modulehttps://tracker.ceph.com/issues/635672023-11-17T05:37:41ZDan Mickdmick@redhat.com
<p>ansible's apt_key uses apt-key add, which has been deprecated for some time and is now obsolete. We must change our apt key management strategy in ansible code.</p>
<p>Perhaps the best answer is to use the new deb822_repository module:</p>
<p><a class="external" href="https://www.jeffgeerling.com/blog/2022/aptkey-deprecated-debianubuntu-how-fix-ansible">https://www.jeffgeerling.com/blog/2022/aptkey-deprecated-debianubuntu-how-fix-ansible</a></p>
<p>More info at <a class="external" href="https://unix.stackexchange.com/questions/332672/how-to-add-a-third-party-repo-and-key-in-debian/582853#582853">https://unix.stackexchange.com/questions/332672/how-to-add-a-third-party-repo-and-key-in-debian/582853#582853</a></p>
<p>and</p>
<p><a class="external" href="https://askubuntu.com/questions/1286545/what-commands-exactly-should-replace-the-deprecated-apt-key">https://askubuntu.com/questions/1286545/what-commands-exactly-should-replace-the-deprecated-apt-key</a> (which refers to the above)</p>
<p><a class="external" href="https://wiki.debian.org/DebianRepository/UseThirdParty">https://wiki.debian.org/DebianRepository/UseThirdParty</a> is much-cited as the authoritative source.</p> CephFS - Bug #63233 (New): mon|client|mds: valgrind reports possible leaks in the MDShttps://tracker.ceph.com/issues/632332023-10-18T12:24:10ZVenky Shankarvshankar@redhat.com
<p>/a/vshankar-2023-10-14_01:51:22-fs-wip-vshankar-testing-20231013.093215-testing-default-smithi/7427332</p>
<pre>
2023-10-16T04:24:05.833 DEBUG:teuthology.orchestra.run.smithi031:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.842 DEBUG:teuthology.orchestra.run.smithi062:> sudo zgrep '<kind>' /var/log/ceph/valgrind/* /dev/null | sort | uniq
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.b.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.886 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.d.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mds.f.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.b.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.887 INFO:teuthology.orchestra.run.smithi062.stdout:/var/log/ceph/valgrind/mon.c.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.a.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mds.c.log: <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.978 INFO:teuthology.orchestra.run.smithi031.stdout:/var/log/ceph/valgrind/mon.a.log: <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.979 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.a.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.c.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.980 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.a.log
2023-10-16T04:24:05.980 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.b.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.d.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mds.f.log kind <kind>Leak_PossiblyLost</kind>
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.981 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.b.log
2023-10-16T04:24:05.981 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind <kind>Leak_StillReachable</kind>
2023-10-16T04:24:05.982 ERROR:tasks.ceph:saw valgrind issue <kind>Leak_StillReachable</kind> in /var/log/ceph/valgrind/mon.c.log
</pre> CephFS - Bug #63089 (New): qa: tasks/mirror times outhttps://tracker.ceph.com/issues/630892023-10-04T07:06:31ZVenky Shankarvshankar@redhat.com
<p>/a/vshankar-2023-09-28_07:23:59-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7405363</p>
<pre>
2023-09-28T11:15:33.524 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs mirror enable cephfs
2023-09-28T11:15:33.549 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.549+0000 7f1d69c56040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member
2023-09-28T11:15:33.604 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.605+0000 7f1d69c56040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member
2023-09-28T11:15:33.657 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.657+0000 7f1d69c56040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member
2023-09-28T11:15:33.721 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.721+0000 7f1d69c56040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member
2023-09-28T11:15:33.794 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.794+0000 7f1d69c56040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
2023-09-28T11:15:33.935 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.935+0000 7f1d69c56040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member
2023-09-28T11:15:34.002 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.002+0000 7f1d69c56040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member
2023-09-28T11:15:34.056 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.056+0000 7f1d69c56040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
2023-09-28T11:15:34.118 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.118+0000 7f1d69c56040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member
2023-09-28T11:15:34.172 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.172+0000 7f1d69c56040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member
2023-09-28T11:15:34.534 INFO:teuthology.orchestra.run:Running command with timeout 30
2023-09-28T11:15:34.534 DEBUG:teuthology.orchestra.run.smithi105:mirror status for fs: cephfs> ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@56
2023-09-28T11:15:34.572 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.572+0000 7f1d69c56040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member
2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:no valid command found; 1 closest matches:
2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:fs mirror status cephfs@54
2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:admin_socket: invalid command
2023-09-28T11:15:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 22
2023-09-28T11:15:34.730 WARNING:tasks.cephfs.test_mirroring:mirror daemon command with label "mirror status for fs: cephfs" failed: Command failed (mirror status for fs: cephfs) on smithi105 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@56'
</pre>