Project

General

Profile

Activity

From 02/21/2023 to 03/22/2023

03/22/2023

09:53 PM Bug #56393: failed to complete snap trimming before timeout
https://pulpito.ceph.com/yuriw-2023-03-15_23:30:58-rados-wip-yuri4-testing-2023-03-15-1418-distro-default-smithi/7209... Laura Flores
07:09 PM Backport #58639 (Resolved): quincy: Mon fail to send pending metadata through MMgrUpdate after an...
Prashant D
03:59 PM Bug #57782: [mon] high cpu usage by fn_monstore thread
We hit the same issue while creating pools:... Lucian Petrut

03/21/2023

10:41 PM Bug #59124: "Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" during quincy p2p upgrade...
Perhaps something needs to be whitelisted here. Laura Flores
10:30 PM Bug #59124 (New): "Health check failed: 1/3 mons down, quorum b,c (MON_DOWN)" during quincy p2p u...
/a/yuriw-2023-03-14_21:36:24-upgrade:quincy-p2p-quincy-release-distro-default-smithi/7208396... Laura Flores
10:15 PM Bug #56393: failed to complete snap trimming before timeout
/a/yuriw-2023-03-15_21:14:59-upgrade:pacific-x-quincy-release-distro-default-smithi/7209137 Laura Flores
10:03 PM Bug #56393: failed to complete snap trimming before timeout
/a/yuriw-2023-03-14_21:33:13-upgrade:octopus-x-quincy-release-distro-default-smithi/7208213... Laura Flores
06:04 PM Bug #59049: WaitReplicas::react(const DigestUpdate&): Unexpected DigestUpdate event
IIRC you were looking into jammy things. If you came to another observations, might be worth adding them here as well. Radoslaw Zarzynski
05:51 PM Bug #57977: osd:tick checking mon for new map
Sure, feel free to send anytime! Would love to review. Radoslaw Zarzynski
05:47 PM Bug #59099: PG move causes data duplication
Notes from the scrub:
1. there are important bounds on the inflation which makes priority high but not urgent,
2....
Radoslaw Zarzynski
05:41 PM Bug #59057: rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_t...
Thanks for info, Laura! Lowering the priority. Radoslaw Zarzynski
05:31 PM Bug #58940: src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
@Patrick, if you can reproduce the issue would you mind setting debug_osd to 30? Aishwarya Mathuria
05:09 PM Bug #58940: src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
@Venky, Rishabh, and Patrick, can you provide a good commit and a bad commit? Laura Flores
05:07 PM Bug #58940: src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
A reoccurrence: https://pulpito.ceph.com/pdonnell-2023-03-21_13:11:41-fs:workload-wip-pdonnell-testing-20230316.21463... Radoslaw Zarzynski
03:08 PM Bug #59114 (New): osd: Unsupported rollback case
The current implementation will not behave correctly for a rados operation consisting of a mutation followed by a rol... Matan Breizman

03/17/2023

07:19 PM Backport #59101 (In Progress): quincy: msg/async: mismatch between in size/types of public_addr a...
Radoslaw Zarzynski
07:03 PM Backport #59101 (Resolved): quincy: msg/async: mismatch between in size/types of public_addr and ...
https://github.com/ceph/ceph/pull/50575 Backport Bot
07:03 PM Backport #59102 (In Progress): reef: msg/async: mismatch between in size/types of public_addr and...
https://github.com/ceph/ceph/pull/52226 Backport Bot
06:57 PM Bug #59100 (Pending Backport): msg/async: mismatch between in size/types of public_addr and publi...
Radoslaw Zarzynski
06:35 PM Bug #59100 (Pending Backport): msg/async: mismatch between in size/types of public_addr and publi...
Radoslaw Zarzynski
03:00 PM Bug #59099: PG move causes data duplication
Additional observations made during testing.
a) Never did expansion occur more then 2x.
b) Expansion is always in...
Adam Kupczyk
02:55 PM Bug #59099: PG move causes data duplication
I made additional test.
Modified size in script from 50M to 1M and looked what operation is requested on BS side.
...
Adam Kupczyk
01:55 PM Bug #59099: PG move causes data duplication
> In some cases OBJ.head + OBJ.1 will take 2M.
The first thing would be to clarify when exactly.
Radoslaw Zarzynski
01:51 PM Bug #59099 (New): PG move causes data duplication
Lets imagine we have a pool TEST.
In the PG we have object OBJ of size 1M.
We create snap SNAP-1 and write some 4...
Adam Kupczyk
09:03 AM Bug #57977: osd:tick checking mon for new map
yite gu wrote:
> Radoslaw Zarzynski wrote:
> > That's a very good question. How about providing logs from both moni...
yite gu
08:59 AM Bug #57977: osd:tick checking mon for new map
Radoslaw Zarzynski wrote:
> That's a very good question. How about providing logs from both monitors and the problem...
yite gu

03/15/2023

09:45 PM Backport #59085 (In Progress): pacific: cache tier set proxy faild
Radoslaw Zarzynski
06:37 PM Backport #59085 (Resolved): pacific: cache tier set proxy faild
https://github.com/ceph/ceph/pull/50552 Backport Bot
09:42 PM Backport #59084 (In Progress): quincy: cache tier set proxy faild
Radoslaw Zarzynski
06:36 PM Backport #59084 (In Progress): quincy: cache tier set proxy faild
https://github.com/ceph/ceph/pull/50551 Backport Bot
06:46 PM Bug #47838: mon/test_mon_osdmap_prune.sh: first_pinned != trim_to
Is there any update on that? Radoslaw Zarzynski
06:40 PM Bug #49961: scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed
Might be a test issue. Radoslaw Zarzynski
06:36 PM Bug #58970 (In Progress): Proxy not available as a cache mode in quincy
Fixed the @Backports@ field in the https://tracker.ceph.com/issues/54576 and moved it back to the @Pending Backport@ ... Radoslaw Zarzynski
06:31 PM Bug #54576 (Pending Backport): cache tier set proxy faild
Radoslaw Zarzynski
05:04 PM Bug #59080 (Resolved): mclock-config.sh: TEST_profile_disallow_builtin_params_modify fails when $...
/a/yuriw-2023-03-10_22:37:27-rados-wip-yuri7-testing-2023-03-10-0830-distro-default-smithi/7202810... Laura Flores
02:44 PM Bug #59079 (New): AssertionError: timeout expired in wait_for_all_osds_up
/a/yuriw-2023-03-10_22:37:27-rados-wip-yuri7-testing-2023-03-10-0830-distro-default-smithi/7202891... Laura Flores
08:49 AM Bug #50637 (Fix Under Review): OSD slow ops warning stuck after OSD fail
Prashant D

03/14/2023

03:10 AM Backport #57208 (In Progress): pacific: lazy_omap_stats_test: "ceph osd deep-scrub all" hangs
https://github.com/ceph/ceph/pull/50518 Brad Hubbard
01:40 AM Bug #49727: lazy_omap_stats_test: "ceph osd deep-scrub all" hangs
Laura Flores wrote:
> /a/yuriw-2023-03-10_22:46:37-rados-reef-distro-default-smithi/7203287
This one is different...
Brad Hubbard

03/13/2023

11:01 PM Backport #57209 (Resolved): quincy: lazy_omap_stats_test: "ceph osd deep-scrub all" hangs
Brad Hubbard
09:12 PM Bug #59057: rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_t...
@Radek it is not fallout from RocksDB. I found an example here from February:
/a/yuriw-2023-02-16_22:44:43-rados-w...
Laura Flores
09:00 PM Bug #59057: rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env_librados_t...
Might be a fallout from the recent rocksdb upgrade. However, we started it pretty recently, after a bunch of successf... Radoslaw Zarzynski
07:54 PM Bug #59057 (Resolved): rados/test_envlibrados_for_rocksdb.sh: No rule to make target 'rocksdb_env...
/a/yuriw-2023-03-10_14:59:04-rados-wip-yuri4-testing-2023-03-09-1458-reef-distro-default-smithi/7201180... Laura Flores
08:57 PM Backport #59005 (Resolved): reef: osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill....
Laura Flores
04:47 PM Backport #59005 (Resolved): reef: osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill....
Backport Bot
08:50 PM Bug #57977: osd:tick checking mon for new map
That's a very good question. How about providing logs from both monitors and the problematic OSD with extra verbosity... Radoslaw Zarzynski
08:22 PM Bug #58925: rocksdb "Leak_StillReachable" memory leak in mons
https://github.com/ceph/ceph/pull/50424 merged Yuri Weinstein
08:21 PM Bug #58496: osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty())
https://github.com/ceph/ceph/pull/50464 merged Yuri Weinstein
07:57 PM Bug #59049: WaitReplicas::react(const DigestUpdate&): Unexpected DigestUpdate event
/a/yuriw-2023-03-10_15:57:59-rados-wip-yuri4-testing-2023-03-09-1458-reef-distro-default-smithi/7201721
This was s...
Laura Flores
05:49 PM Bug #59049 (Pending Backport): WaitReplicas::react(const DigestUpdate&): Unexpected DigestUpdate ...
/a/yuriw-2023-03-10_22:46:37-rados-reef-distro-default-smithi/7203358... Laura Flores
05:22 PM Bug #49727: lazy_omap_stats_test: "ceph osd deep-scrub all" hangs
/a/yuriw-2023-03-10_22:46:37-rados-reef-distro-default-smithi/7203287 Laura Flores
05:19 PM Bug #59047 (New): mon/MgrStatMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:18 PM Bug #59046 (New): mon/LogMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:18 PM Bug #59045 (New): mon/KVMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:18 PM Bug #59044 (New): mon/HealthMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:17 PM Bug #59043 (New): mon/ConfigMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:17 PM Bug #59042 (New): mon/AuthMonitor: do not erroneously propose on error in ::prepare_update
Patrick Donnelly
05:15 PM Bug #58974 (Fix Under Review): mon/MonmapMonitor: do not propose on error in prepare_update
Patrick Donnelly
04:30 PM Bug #58974 (Pending Backport): mon/MonmapMonitor: do not propose on error in prepare_update
See discussion: https://github.com/ceph/ceph/pull/50404#discussion_r1133791746 Patrick Donnelly
04:42 PM Backport #58979 (Resolved): reef: rocksdb "Leak_StillReachable" memory leak in mons
Backport Bot
04:22 PM Bug #58972 (Fix Under Review): mon/OSDMonitor: do not propose on error in prepare_update
Patrick Donnelly
04:19 PM Bug #58972 (Pending Backport): mon/OSDMonitor: do not propose on error in prepare_update
See discussion: https://github.com/ceph/ceph/pull/50404#discussion_r1133791746 Patrick Donnelly
04:16 PM Bug #58970 (In Progress): Proxy not available as a cache mode in quincy
https://tracker.ceph.com/issues/54576 tracks that proxy as a cache mode was erroneously removed as a cache_mode from ... Jamison Lofthouse

03/10/2023

07:29 PM Bug #58496: osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty())
/a/yuriw-2023-03-08_23:00:31-rados-wip-yuri11-testing-2023-03-08-1220-distro-default-smithi/7199112 Laura Flores
07:27 PM Bug #52316: qa/tasks/mon_thrash.py: _do_thrash AssertionError len(s['quorum']) == len(mons)
/a/yuriw-2023-03-08_23:00:31-rados-wip-yuri11-testing-2023-03-08-1220-distro-default-smithi/7198899 Laura Flores
04:37 PM Bug #54369: mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]]
/a/yuriw-2023-03-10_15:00:22-rados-wip-yuri5-testing-2023-03-09-0941-quincy-distro-default-smithi/7201010 Laura Flores
01:33 PM Bug #50637: OSD slow ops warning stuck after OSD fail
Prashant D wrote:
> I am able to reproduce this issue on latest main branch. Kindly allow me sometime to investigate...
Christian Rohmann
06:53 AM Bug #50637: OSD slow ops warning stuck after OSD fail
I am able to reproduce this issue on latest main branch. Kindly allow me sometime to investigate it further and fix t... Prashant D
12:46 AM Bug #6297: ceph osd tell * will break when FD limit reached, messenger should close pipes as nece...
Thanks for the confirmation Frank. I'm revisiting this. Brad Hubbard

03/09/2023

11:12 PM Backport #57696: quincy: ceph log last command fail to log by verbosity level
Backport Bot wrote:
> https://github.com/ceph/ceph/pull/50407
merged
Yuri Weinstein
10:49 PM Backport #58169: quincy: extra debugs for: [mon] high cpu usage by fn_monstore thread
Backport Bot wrote:
> https://github.com/ceph/ceph/pull/50406
merged
Yuri Weinstein
10:48 PM Backport #58334: quincy: mon/monclient: update "unable to obtain rotating service keys when osd i...
Backport Bot wrote:
> https://github.com/ceph/ceph/pull/50405
merged
Yuri Weinstein
10:46 PM Backport #58639: quincy: Mon fail to send pending metadata through MMgrUpdate after an upgrade re...
Neha Ojha wrote:
> https://github.com/ceph/ceph/pull/49989
merged
Yuri Weinstein
10:45 PM Backport #55543: quincy: should use TCMalloc for better performance
Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47927
merged
Yuri Weinstein
08:55 PM Bug #58496 (Pending Backport): osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.emp...
Laura Flores
08:27 PM Bug #49961: scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed
/a/yuriw-2023-03-08_15:52:33-rados-wip-yuri8-testing-2023-03-07-1527-distro-default-smithi/7197485... Laura Flores
05:57 PM Bug #50042: rados/test.sh: api_watch_notify failures
/a/yuriw-2023-03-07_23:24:03-rados-wip-yuri6-testing-2023-03-07-1336-quincy-distro-default-smithi/7196726... Laura Flores
04:15 PM Bug #58940: src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
Saw this bug on following teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-ris... Rishabh Dave
03:44 PM Bug #58940: src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
Logs from osd.1 before it crashed: ... Aishwarya Mathuria
09:42 AM Bug #58940 (Resolved): src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op")
This looks similar to https://tracker.ceph.com/issues/48793, but since that tracker is from a couple of years ago, I ... Venky Shankar
08:14 AM Bug #6297: ceph osd tell * will break when FD limit reached, messenger should close pipes as nece...
Hi Brad, yes I can. I tried with 1300 and it works fine. I added "ulimit -n 2048" to the script as a work-around.
...
Frank Schilder
03:05 AM Bug #6297: ceph osd tell * will break when FD limit reached, messenger should close pipes as nece...
Hi Frank,
Can you confirm that increasing the file limit to some level just above 1260 (allowing for some miscella...
Brad Hubbard

03/08/2023

07:56 PM Bug #56896: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_...
Hi, just reporting, that I just hit this problem on few OSDs in 17.2.5 Nikola Ciprich
02:04 PM Bug #58052: Empty Pool (zero objects) shows usage.
Some notes from from the last Ceph RADOS Team meeting:
1. It's very interesting trait that, although the @USED@ va...
Radoslaw Zarzynski
03:48 AM Bug #57977: osd:tick checking mon for new map
Radoslaw Zarzynski wrote:
> The unwanted nonce match causes that @OSDMonitor::preprocess_boot()@ returns @true@, and...
yite gu

03/07/2023

05:48 PM Bug #56386: Writes to a cephfs after metadata pool snapshot causes inconsistent objects
Sorry to warm this up again but our cluster is still in an unhealthy state and we are trying to find ways to recover ... Pascal Ehlert
03:57 PM Bug #58925 (Pending Backport): rocksdb "Leak_StillReachable" memory leak in mons
Laura Flores
12:53 PM Bug #58915 (Fix Under Review): map eXX had wrong heartbeat addr
Radoslaw Zarzynski
12:08 PM Bug #58915 (In Progress): map eXX had wrong heartbeat addr
Radoslaw Zarzynski
12:07 PM Bug #58915: map eXX had wrong heartbeat addr
> Why the return of @set_addr_unknowns@ turned out to be @false@?
Because it was already learnt!...
Radoslaw Zarzynski
11:58 AM Bug #58915: map eXX had wrong heartbeat addr
What's interesting is that the log contains:... Radoslaw Zarzynski
11:53 AM Bug #58915: map eXX had wrong heartbeat addr
The direct reason why the OSDMap got wrong hb address is that the OSD had sent so:... Radoslaw Zarzynski
11:49 AM Bug #58915: map eXX had wrong heartbeat addr
The heartbeat msgr instances started with the @INADDR_ANY@ addresses.... Radoslaw Zarzynski
10:57 AM Bug #58915: map eXX had wrong heartbeat addr
Verified whether the testing branch had the @public_bind_addr@ support in OSD:... Radoslaw Zarzynski
12:25 PM Bug #6297: ceph osd tell * will break when FD limit reached, messenger should close pipes as nece...
Just run into this problem as well. I'm scraping OSD perf dumps to a file in a script and I get... Frank Schilder
07:45 AM Bug #58884 (Fix Under Review): ceph: osd blocklist does not accept v2/v1: prefix for addr
Nitzan Mordechai
12:46 AM Bug #58049 (Resolved): mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_tracker...
Kamoltat (Junior) Sirivadhna
12:44 AM Backport #58381 (Resolved): quincy: mon:stretch-cluster: mishandled removed_ranks -> inconsistent...
Kamoltat (Junior) Sirivadhna
12:43 AM Backport #58381: quincy: mon:stretch-cluster: mishandled removed_ranks -> inconsistent peer_track...
https://github.com/ceph/ceph/pull/49433 Kamoltat (Junior) Sirivadhna
12:41 AM Backport #58380 (Resolved): pacific: mon:stretch-cluster: mishandled removed_ranks -> inconsisten...
https://github.com/ceph/ceph/pull/49312 Kamoltat (Junior) Sirivadhna

03/06/2023

10:55 PM Bug #49961 (New): scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed
/a/yuriw-2023-03-01_19:28:10-rados-wip-yuri3-testing-2023-03-01-0812-quincy-distro-default-smithi/7190326]... Laura Flores
10:00 PM Bug #58925 (Fix Under Review): rocksdb "Leak_StillReachable" memory leak in mons
Laura Flores
08:37 PM Bug #58925: rocksdb "Leak_StillReachable" memory leak in mons
Steps to reproduce:... Laura Flores
07:50 PM Bug #58925: rocksdb "Leak_StillReachable" memory leak in mons
Caused by https://github.com/ceph/ceph/pull/49006. Laura Flores
07:33 PM Bug #58925: rocksdb "Leak_StillReachable" memory leak in mons
Laura Flores wrote:
> [...]
This example was from /a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7...
Laura Flores
07:32 PM Bug #58925 (Resolved): rocksdb "Leak_StillReachable" memory leak in mons
... Laura Flores
07:33 PM Backport #57117 (In Progress): quincy: mon: race condition between `mgr fail` and MgrMonitor::pre...
Radoslaw Zarzynski
07:25 PM Backport #57696 (In Progress): quincy: ceph log last command fail to log by verbosity level
Radoslaw Zarzynski
07:22 PM Backport #58169 (In Progress): quincy: extra debugs for: [mon] high cpu usage by fn_monstore thread
Radoslaw Zarzynski
07:17 PM Backport #58334 (In Progress): quincy: mon/monclient: update "unable to obtain rotating service k...
Radoslaw Zarzynski
07:16 PM Bug #58496: osd/PeeringState: FAILED ceph_assert(!acting_recovery_backfill.empty())
/a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7193142 Laura Flores
07:14 PM Bug #47838: mon/test_mon_osdmap_prune.sh: first_pinned != trim_to
/a/yuriw-2023-03-03_17:39:09-rados-reef-distro-default-smithi/7193126... Laura Flores
06:23 PM Bug #58739: "Leak_IndirectlyLost" valgrind report on mon.a
The quincy backport of the auth key rotation (https://github.com/ceph/ceph/pull/48093) got merged on 8 Feb. However, ... Radoslaw Zarzynski
06:09 PM Bug #58915: map eXX had wrong heartbeat addr
I wonder whether this is a fallout from the public_bind changes (for the overlapping IP problem) but it looks the bra... Radoslaw Zarzynski
06:01 PM Bug #55141 (In Progress): thrashers/fastread: assertion failure: rollback_info_trimmed_to == head
Radoslaw Zarzynski
02:36 PM Bug #55141: thrashers/fastread: assertion failure: rollback_info_trimmed_to == head
Since this is EC pool, the NO_SHARD is confusing, we are not maintaining rollback_info_trimmed_to on replicas, lookin... Nitzan Mordechai
01:22 PM Bug #55141: thrashers/fastread: assertion failure: rollback_info_trimmed_to == head
I'm probably missing something here, but i'll try to summarize my finds... Nitzan Mordechai
05:53 PM Backport #56602 (Resolved): quincy: ceph report missing osdmap_clean_epochs if answered by peon
Radoslaw Zarzynski
05:49 PM Feature #54280 (Resolved): support truncation sequences in sparse reads
Radoslaw Zarzynski
05:48 PM Bug #54509 (Resolved): FAILED ceph_assert due to issue manifest API to the original object
Radoslaw Zarzynski
05:47 PM Bug #54558 (Resolved): malformed json in a Ceph RESTful API call can stop all ceph-mon services
Radoslaw Zarzynski
05:46 PM Backport #55296 (Resolved): pacific: malformed json in a Ceph RESTful API call can stop all ceph-...
Radoslaw Zarzynski
05:46 PM Backport #55298 (Resolved): octopus: malformed json in a Ceph RESTful API call can stop all ceph-...
Radoslaw Zarzynski
05:45 PM Backport #55297 (Resolved): quincy: malformed json in a Ceph RESTful API call can stop all ceph-m...
Radoslaw Zarzynski
05:43 PM Bug #54994 (Resolved): osd: add scrub duration for scrubs after recovery
Radoslaw Zarzynski
05:43 PM Backport #55282 (Resolved): quincy: osd: add scrub duration for scrubs after recovery
Radoslaw Zarzynski
05:42 PM Bug #55088 (Resolved): Manager is failing to keep updated metadata in daemon_state for upgraded M...
Radoslaw Zarzynski
05:42 PM Backport #55305 (Resolved): quincy: Manager is failing to keep updated metadata in daemon_state f...
Radoslaw Zarzynski
05:39 PM Backport #55542 (Rejected): octopus: should use TCMalloc for better performance
Neha Ojha
05:32 PM Bug #57017: mon-stretched_cluster: degraded stretched mode lead to Monitor crash
The quincy backport is important and needed. Radoslaw Zarzynski
05:30 PM Bug #57533 (Resolved): Able to modify the mclock reservation, weight and limit parameters when bu...
Radoslaw Zarzynski
05:29 PM Backport #58708 (Resolved): quincy: Able to modify the mclock reservation, weight and limit param...
Radoslaw Zarzynski
05:28 PM Fix #57577 (Resolved): osd: Improve osd bench accuracy by using buffers with random patterns
Radoslaw Zarzynski
05:28 PM Backport #58214 (Resolved): quincy: osd: Improve osd bench accuracy by using buffers with random ...
Radoslaw Zarzynski
05:26 PM Backport #58638 (Resolved): pacific: Mon fail to send pending metadata through MMgrUpdate after a...
Radoslaw Zarzynski
05:24 PM Bug #57859 (Resolved): bail from handle_command() if _generate_command_map() fails
Radoslaw Zarzynski
05:24 PM Backport #58007 (Resolved): pacific: bail from handle_command() if _generate_command_map() fails
Radoslaw Zarzynski
05:24 PM Bug #57698 (Resolved): osd/scrub: "scrub a chunk" requests are sent to the wrong set of replicas
Neha Ojha
05:24 PM Backport #58006 (Resolved): quincy: bail from handle_command() if _generate_command_map() fails
Radoslaw Zarzynski
05:22 PM Fix #57963 (Resolved): osd: Misleading information displayed for the running configuration of osd...
Radoslaw Zarzynski
05:22 PM Backport #58186 (Resolved): quincy: osd: Misleading information displayed for the running configu...
Neha Ojha
05:14 PM Backport #58872 (Rejected): octopus: ClsLock.TestExclusiveEphemeralStealEphemeral failed
Neha Ojha
05:13 PM Bug #44092 (Resolved): mon: config commands do not accept whitespace style config name
Radoslaw Zarzynski
05:09 PM Backport #57346 (Resolved): quincy: expected valgrind issues and found none
Radoslaw Zarzynski
05:00 PM Bug #56101 (Resolved): Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in function safe_timer
Neha Ojha
04:52 PM Backport #58586 (Resolved): quincy: Gibba Cluster: 17.2.0 to 17.2.1 RC upgrade OSD crash in funct...
Neha Ojha
03:34 PM Bug #57977: osd:tick checking mon for new map
The unwanted nonce match causes that @OSDMonitor::preprocess_boot()@ returns @true@, and thus prevents @OSDMonitor::p... Radoslaw Zarzynski

03/03/2023

10:20 PM Bug #54750: crash: PeeringState::Crashed::Crashed(boost::statechart::state<PeeringState::Crashed,...
/a/yuriw-2023-02-22_20:55:15-rados-wip-yuri4-testing-2023-02-22-0817-quincy-distro-default-smithi/7184685... Laura Flores
10:03 PM Bug #58915 (Pending Backport): map eXX had wrong heartbeat addr
Occurred during "Unwinding manager ceph" task.
/a/yuriw-2023-02-22_20:55:15-rados-wip-yuri4-testing-2023-02-22-081...
Laura Flores
05:59 PM Bug #51904 (Resolved): test_pool_min_size:AssertionError:wait_for_clean:failed before timeout exp...
Neha Ojha
05:58 PM Backport #57026 (Resolved): pacific: test_pool_min_size:AssertionError:wait_for_clean:failed befo...
Neha Ojha

03/02/2023

08:52 PM Bug #43887: ceph_test_rados_delete_pools_parallel failure
Kamoltat (Junior) Sirivadhna wrote:
> Encountered this error in: yuriw-2023-03-02_00:09:05-rados-wip-yuri11-testing-...
Laura Flores
08:18 PM Bug #43887: ceph_test_rados_delete_pools_parallel failure
Encountered this error in: yuriw-2023-03-02_00:09:05-rados-wip-yuri11-testing-2023-03-01-1424-distro-default-smithi/7... Kamoltat (Junior) Sirivadhna
07:40 PM Bug #58739: "Leak_IndirectlyLost" valgrind report on mon.a
HIT in /a/yuriw-2023-03-02_00:09:05-rados-wip-yuri11-testing-2023-03-01-1424-distro-default-smithi/7191392/remote/smi... Kamoltat (Junior) Sirivadhna
05:48 PM Bug #58739: "Leak_IndirectlyLost" valgrind report on mon.a
https://github.com/ceph/ceph/pull/48641 is already merge. If we don't see new replications over some time (a few mont... Radoslaw Zarzynski
07:31 PM Bug #55141: thrashers/fastread: assertion failure: rollback_info_trimmed_to == head
/a/yuriw-2023-03-02_00:09:05-rados-wip-yuri11-testing-2023-03-01-1424-distro-default-smithi/7191380 Kamoltat (Junior) Sirivadhna
05:52 PM Bug #50637: OSD slow ops warning stuck after OSD fail
Bump up + ping. Radoslaw Zarzynski
02:18 PM Bug #57977: osd:tick checking mon for new map
Radoslaw Zarzynski wrote:
> Thanks, yite gu!
> The fix is: https://github.com/ceph/ceph/pull/50344/commits/fb868d4e...
yite gu
12:01 PM Bug #57977 (Fix Under Review): osd:tick checking mon for new map
Thanks, yite gu!
The fix is: https://github.com/ceph/ceph/pull/50344/commits/fb868d4e71d3871cbd17cfbd4a536470e5c023f...
Radoslaw Zarzynski
01:06 PM Bug #58884 (In Progress): ceph: osd blocklist does not accept v2/v1: prefix for addr
looks like addr type is CephEntityAddr, which mean it will accept "CephEntityAddr: CephIPAddr + optional '/nonce'"
<...
Nitzan Mordechai

03/01/2023

08:46 PM Bug #58894 (Fix Under Review): [pg-autoscaler][mgr] does not throw warn to increase PG count on p...
Kamoltat (Junior) Sirivadhna
08:32 PM Bug #58894 (Resolved): [pg-autoscaler][mgr] does not throw warn to increase PG count on pools wit...
Here pool test 1-3 should be spitting health warnings like: PG TOO FEW PLEASE SCALE.... Kamoltat (Junior) Sirivadhna
07:41 PM Bug #58893: test_map_discontinuity: AssertionError: wait_for_clean: failed before timeout expired
Marking this as related to #51076 since there was a case of `test_map_discontinuity` logged there. Laura Flores
07:40 PM Bug #58893 (New): test_map_discontinuity: AssertionError: wait_for_clean: failed before timeout e...
/a/yuriw-2023-02-24_17:50:19-rados-main-distro-default-smithi/7186711... Laura Flores
03:34 PM Bug #57977: osd:tick checking mon for new map
... yite gu
12:44 PM Bug #57977: osd:tick checking mon for new map
osd.0 hanppen restart, but Since then, it has not join the cluster alway. I upload osd boot log. yite gu
11:27 AM Bug #57977: osd:tick checking mon for new map
Radoslaw Zarzynski wrote:
> Thanks for the update! Yeah, it might stuck there. To confirm we would logs with increas...
yite gu
02:44 PM Bug #58288 (Fix Under Review): quincy: mon: pg_num_check() according to crush rule
Revert is merged https://github.com/ceph/ceph/pull/49465.
PR#50327 pushed as the actual fix.
Matan Breizman

02/28/2023

11:14 PM Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed wi...
Bunch of tests from LibRadosIoEC failing from "rados_mon_command osd pool create failed with error -22"
/a/yuriw-2...
Laura Flores
10:57 PM Bug #58052: Empty Pool (zero objects) shows usage.
Well, I need to move on, so I am deleting the pools. I may try to re-create this in a lab later. If I do, I will tr... Brian Woods
07:26 PM Feature #58885 (New): [pg-autoscaler] include warning and explanation in ceph -s when there's ove...
Currently, we only warn the user about overlapping roots in mgr log.
Since there have been cases where the user file...
Kamoltat (Junior) Sirivadhna
05:39 PM Bug #58884 (Resolved): ceph: osd blocklist does not accept v2/v1: prefix for addr
... Patrick Donnelly
03:10 PM Bug #57105 (Resolved): quincy: ceph osd pool set <pool> size math error
Matan Breizman
03:10 PM Bug #54188 (Resolved): Setting too many PGs leads error handling overflow
Matan Breizman
02:47 PM Bug #58141 (Resolved): mon/MonCommands: Support dump_historic_slow_ops
Matan Breizman
02:47 PM Backport #58143 (Resolved): quincy: mon/MonCommands: Support dump_historic_slow_ops
Matan Breizman
02:46 PM Backport #58144 (Resolved): pacific: mon/MonCommands: Support dump_historic_slow_ops
Matan Breizman
02:44 PM Bug #49689 (Resolved): osd/PeeringState.cc: ceph_abort_msg("past_interval start interval mismatch...
Matan Breizman
02:44 PM Bug #55549 (Resolved): OSDs crashing
Matan Breizman
10:53 AM Backport #58872 (In Progress): octopus: ClsLock.TestExclusiveEphemeralStealEphemeral failed
Nitzan Mordechai
10:03 AM Backport #58872 (Rejected): octopus: ClsLock.TestExclusiveEphemeralStealEphemeral failed
https://github.com/ceph/ceph/pull/50303 Backport Bot
10:47 AM Backport #58869 (In Progress): quincy: rados/test.sh: api_watch_notify failures
Nitzan Mordechai
10:02 AM Backport #58869 (Resolved): quincy: rados/test.sh: api_watch_notify failures
https://github.com/ceph/ceph/pull/49938 Backport Bot
10:47 AM Backport #58868 (In Progress): pacific: rados/test.sh: api_watch_notify failures
Nitzan Mordechai
10:02 AM Backport #58868 (Resolved): pacific: rados/test.sh: api_watch_notify failures
https://github.com/ceph/ceph/pull/49943 Backport Bot
10:03 AM Backport #58871 (New): quincy: ClsLock.TestExclusiveEphemeralStealEphemeral failed
Backport Bot
10:02 AM Backport #58870 (Rejected): pacific: ClsLock.TestExclusiveEphemeralStealEphemeral failed
Backport Bot
10:02 AM Bug #38357 (Pending Backport): ClsLock.TestExclusiveEphemeralStealEphemeral failed
Nitzan Mordechai
10:01 AM Bug #50042 (Pending Backport): rados/test.sh: api_watch_notify failures
Nitzan Mordechai

02/27/2023

07:00 PM Bug #58379: no active mgr after ~1 hour
Review-in-progress. Radoslaw Zarzynski
06:55 PM Bug #58837: mgr/test_progress.py: test_osd_healthy_recovery fails after timeout
Hi Junior! Would find some time for it? Radoslaw Zarzynski
06:52 PM Bug #44400 (Won't Fix): Marking OSD out causes primary-affinity 0 to be ignored when up_set has n...
The discussion's outcome is that the fix could likely make more harm (for sure: bring more complexity) than the the s... Radoslaw Zarzynski
06:50 PM Bug #57977 (In Progress): osd:tick checking mon for new map
Radoslaw Zarzynski
06:45 PM Bug #49428: ceph_test_rados_api_snapshots fails with "rados_mon_command osd pool create failed wi...
Seems like a similar failure:
/a/yuriw-2023-02-16_22:44:43-rados-wip-yuri-testing-2023-02-16-0839-distro-default-s...
Laura Flores
06:18 PM Bug #58797: scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpectedly low ...
/a/lflores-2023-02-20_21:22:20-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi/7181477... Laura Flores
04:11 PM Bug #58797 (Resolved): scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Unexpe...
Laura Flores
06:09 PM Bug #49961: scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed
/a/yuriw-2023-02-16_22:44:43-rados-wip-yuri-testing-2023-02-16-0839-distro-default-smithi/7177204... Laura Flores

02/23/2023

08:44 PM Bug #51729: Upmap verification fails for multi-level crush rule
Hi Chris, yes, I will post another update soon with my findings. Laura Flores
07:07 PM Bug #58797 (Fix Under Review): scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR...
Ronen Friedman
06:53 PM Bug #58837: mgr/test_progress.py: test_osd_healthy_recovery fails after timeout
Seen in the mgr logs: 2 pgs stuck in recovery... Laura Flores
06:31 PM Bug #58837 (New): mgr/test_progress.py: test_osd_healthy_recovery fails after timeout
/a/yuriw-2023-02-22_20:55:15-rados-wip-yuri4-testing-2023-02-22-0817-quincy-distro-default-smithi/7184746... Laura Flores

02/22/2023

09:00 PM Backport #58708: quincy: Able to modify the mclock reservation, weight and limit parameters when ...
Backport Bot wrote:
> https://github.com/ceph/ceph/pull/50097
merged
Yuri Weinstein
01:11 PM Bug #21592 (Fix Under Review): LibRadosCWriteOps.CmpExt got 0 instead of -4095-1
Nitzan Mordechai

02/21/2023

10:30 PM Bug #51729: Upmap verification fails for multi-level crush rule
Is there any news on this? Thanks. Chris Durham
04:17 PM Bug #58797 (In Progress): scrub/osd-scrub-dump.sh: TEST_recover_unexpected fails from "ERROR: Une...
This is an unintended side effect of https://github.com/ceph/ceph/pull/44749. I will create a fix.
Explanation:
...
Ronen Friedman
12:12 PM Feature #55169 (In Progress): crush: should validate rule outputs osds
Shreyansh Sancheti
12:11 PM Backport #58816 (In Progress): quincy: ceph versions : mds : remove empty list entries from ceph ...
Backport Bot
12:10 PM Backport #58815 (New): quincy: Set single compression algorithm as a default value in ms_osd_comp...
Backport Bot
12:09 PM Bug #57585 (Pending Backport): ceph versions : mds : remove empty list entries from ceph versions
Shreyansh Sancheti
12:07 PM Bug #58410 (Pending Backport): Set single compression algorithm as a default value in ms_osd_comp...
Shreyansh Sancheti
 

Also available in: Atom