Activity

From 02/13/2022 to 03/14/2022

03/14/2022

08:24 PM Bug #54556 (Won't Fix): Pools are wrongly reported to have non-power-of-two pg_num after update: We just updated our cluster from 14.2.1 to 14.2.22. Now (in addition two a few more) a new warning appears which we h... Martin H.
05:21 PM Bug #54552 (Fix Under Review): ceph windows test hanging quincy backport PRs: Kamoltat (Junior) Sirivadhna
03:42 PM Bug #54552 (Resolved): ceph windows test hanging quincy backport PRs: ... Kamoltat (Junior) Sirivadhna
02:21 PM Backport #54526 (In Progress): pacific: cephadm upgrade pacific to quincy autoscaler is scaling p...: Kamoltat (Junior) Sirivadhna
02:20 PM Backport #54526: pacific: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> ...: https://github.com/ceph/ceph/pull/45364 Kamoltat (Junior) Sirivadhna
02:21 PM Backport #54527 (In Progress): quincy: cephadm upgrade pacific to quincy autoscaler is scaling pg...: Kamoltat (Junior) Sirivadhna
02:21 PM Backport #54527: quincy: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 3...: https://github.com/ceph/ceph/pull/45363 Kamoltat (Junior) Sirivadhna
02:10 PM Bug #46847: Loss of placement information on OSD reboot: Oh also ceph pg repeer has not totally worked. I have a single object remaining unfound. ... Malcolm Haak
04:02 AM Bug #46847: Loss of placement information on OSD reboot: Neha Ojha wrote:
> Frank Schilder wrote:
> > Could somebody please set the status back to open and Affected Version... Malcolm Haak
01:05 PM Bug #54548 (Won't Fix): mon hang when run ceph -s command after execute "ceph osd in osd.<x>" com...: 1. run command "ceph osd in osd.<x>"
2. run command "ceph -s", I want to see progress, but "ceph -s" hang at this ti... yite gu

03/13/2022

09:04 AM Bug #51307 (Fix Under Review): LibRadosWatchNotify.Watch2Delete fails: https://github.com/ceph/ceph/pull/45366 Nitzan Mordechai
08:34 AM Bug #51307: LibRadosWatchNotify.Watch2Delete fails: In that case it was not injection socket failure, it was:
2022-02-16T09:56:22.598+0000 15af4700 1 -- [v2:172.21.1... Nitzan Mordechai

03/11/2022

01:02 PM Bug #53729: ceph-osd takes all memory before oom on boot: Dan van der Ster wrote:
> Guillaume Fenollar wrote:
> > Dan van der Ster wrote:
> > > Could you revert that and tr... Guillaume Fenollar
08:22 AM Bug #53729: ceph-osd takes all memory before oom on boot: Guillaume Fenollar wrote:
> Dan van der Ster wrote:
> > Could you revert that and try running
> >
> > ceph-osd -... Dan van der Ster
07:09 AM Bug #53729: ceph-osd takes all memory before oom on boot: Dan van der Ster wrote:
> Could you revert that and try running
>
> ceph-osd --debug_ms=1 --debug_osd=20 --debug_... Guillaume Fenollar
06:47 AM Bug #53729: ceph-osd takes all memory before oom on boot: Guillaume Fenollar wrote:
> See that it reaches 14GB of RAM in 90 seconds approx and starts writing while crashing (... Dan van der Ster
03:09 AM Bug #53729: ceph-osd takes all memory before oom on boot: Dan van der Ster wrote:
> > Can you somehow annotate the usage over time in the log?
>
> Could you please also se... Guillaume Fenollar
03:02 AM Bug #53729: ceph-osd takes all memory before oom on boot: Mykola Golub wrote:
> Mykola Golub wrote:
>
> > pool 2 'ssd' replicated size 3 min_size 2 crush_rule 0 object_has... Neha Ojha
09:54 AM Bug #52026: osd: pgs went back into snaptrim state after osd restart: We are having the same issue with ceph 15.2.13. We take RBD snapshots that gets deleted after 3 days.
The problem ge... Jack Y
03:19 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: ... jianwei zhang

03/10/2022

11:58 PM Bug #54516: mon/config.sh: unrecognized config option 'debug asok': This was the first occurrence of this test failure according to the Sentry history (March 5th 2022), and it has since... Laura Flores
03:10 PM Bug #54516 (Won't Fix): mon/config.sh: unrecognized config option 'debug asok': /a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721689... Kamoltat (Junior) Sirivadhna
11:46 PM Bug #54521: daemon: Error while waiting for process to exit: This looks a lot like a valgrind failure, but there were unfortunately no osd logs collected.... Laura Flores
03:35 PM Bug #54521 (Need More Info): daemon: Error while waiting for process to exit: This causes dead job: hit max job timeout
/a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-dis... Kamoltat (Junior) Sirivadhna
11:30 PM Bug #54529: mon/mon-bind.sh: Failure due to cores found: "Failure due to cores found" means that there is a coredump, and indeed there is a crash. Did we merge something rece... Neha Ojha
11:17 PM Bug #54529 (Duplicate): mon/mon-bind.sh: Failure due to cores found: Looks like this failed due to external connection issues, but I'll log it for documentation.
/a/teuthology-2022-01... Laura Flores
11:30 PM Bug #54517: scrub/osd-scrub-snaps.sh: TEST FAILED WITH 1 ERRORS: Ronen this looks a lot like https://tracker.ceph.com/issues/54458, just with a slightly different output. Can you che... Laura Flores
03:18 PM Bug #54517 (Duplicate): scrub/osd-scrub-snaps.sh: TEST FAILED WITH 1 ERRORS: /a/teuthology-archive/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721751... Kamoltat (Junior) Sirivadhna
10:52 PM Bug #54296: OSDs using too much memory: Hi Ruben, Did you make any more progress on this?
I'm going through all the osd pglog memory usage tickets, and it... Dan van der Ster
10:21 PM Bug #53729: ceph-osd takes all memory before oom on boot: > Can you somehow annotate the usage over time in the log?
Could you please also set debug_prioritycache=5 -- this... Dan van der Ster
09:35 PM Bug #53729: ceph-osd takes all memory before oom on boot: Guillaume Fenollar wrote:
> Neha Ojha wrote:
> > Can anyone provide osd logs with debug_osd=20,debug_ms=1 for OSDs ... Dan van der Ster
05:23 AM Bug #53729: ceph-osd takes all memory before oom on boot: Mykola Golub wrote:
> pool 2 'ssd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1024 pgp_... Mykola Golub
05:16 AM Bug #53729: ceph-osd takes all memory before oom on boot: Neha Ojha wrote:
> Can anyone provide osd logs with debug_osd=20,debug_ms=1 for OSDs that are hitting OOM?
I just... Guillaume Fenollar
10:00 PM Backport #54527 (Resolved): quincy: cephadm upgrade pacific to quincy autoscaler is scaling pgs f...: Backport Bot
10:00 PM Backport #54526 (Resolved): pacific: cephadm upgrade pacific to quincy autoscaler is scaling pgs ...: Backport Bot
09:57 PM Bug #54263 (Pending Backport): cephadm upgrade pacific to quincy autoscaler is scaling pgs from 3...: Kamoltat (Junior) Sirivadhna
09:15 PM Feature #54525 (New): osd/mon: log memory usage during tick: The MDS has a nice feature that it prints out the rss and other memory stats every couple seconds at debug level 2.
... Dan van der Ster
06:13 PM Bug #54507 (Duplicate): workunit test cls/test_cls_rgw: Manager failed: thrashosds: Laura Flores
03:28 PM Bug #51846: rados/test.sh: LibRadosList.ListObjectsCursor did not complete.: /a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721371
/a/yuriw-2022-03-... Kamoltat (Junior) Sirivadhna
03:00 PM Bug #54515 (New): mon/health-mute.sh: TEST_mute: return 1 (HEALTH WARN 3 mgr modules have failed ...: /a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721547... Kamoltat (Junior) Sirivadhna
02:48 PM Bug #45423: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite: /a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721464 Kamoltat (Junior) Sirivadhna
10:50 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: Neha Ojha wrote:
> jianwei zhang wrote:
> > 1711'7107 : s0/1/2/3/4/5都有所以都能写下去
> > 1715'7108 : s0/2/3/5 满足k=4，所以... jianwei zhang
01:57 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: Neha Ojha wrote:
> jianwei zhang wrote:
> > 1711'7107 : s0/1/2/3/4/5都有所以都能写下去
> > 1715'7108 : s0/2/3/5 满足k=4，所以... jianwei zhang
04:59 AM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...: /a/yuriw-2022-03-04_21:56:41-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6721329 Kamoltat (Junior) Sirivadhna
04:32 AM Bug #54511 (Resolved): test_pool_min_size: AssertionError: not clean before minsize thrashing starts: /a/yuriw-2022-03-04_00:56:58-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6719015... Kamoltat (Junior) Sirivadhna
04:15 AM Bug #53767: qa/workunits/cls/test_cls_2pc_queue.sh: killing an osd during thrashing causes timeout: /a/yuriw-2022-03-04_00:56:58-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6718855 Kamoltat (Junior) Sirivadhna
01:48 AM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...: https://tracker.ceph.com/issues/54509 Myoungwon Oh
01:47 AM Bug #54509: FAILED ceph_assert due to issue manifest API to the original object: https://github.com/ceph/ceph/pull/45137 Myoungwon Oh
01:47 AM Bug #54509 (Resolved): FAILED ceph_assert due to issue manifest API to the original object: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55f1f3750606]
2: ceph-osd(+0x5b... Myoungwon Oh

03/09/2022

09:44 PM Bug #54507 (Duplicate): workunit test cls/test_cls_rgw: Manager failed: thrashosds: /a/yuriw-2022-03-04_00:56:58-rados-wip-yuri4-testing-2022-03-03-1448-distro-default-smithi/6718934... Kamoltat (Junior) Sirivadhna
08:23 PM Bug #52535: monitor crashes after an OSD got destroyed: OSDMap.cc: 5686: FAILED ceph_assert(num_d...: Hello Radoslaw,
thank you for your response!
About two weeks ago I did first remove and then add 6 OSDs. I did no... Sebastian Mazza
07:35 PM Bug #52535: monitor crashes after an OSD got destroyed: OSDMap.cc: 5686: FAILED ceph_assert(num_d...: Neha has made an interesting observation about the occurrences among different versions.
http://telemetry.front.se... Radoslaw Zarzynski
07:32 PM Bug #52535: monitor crashes after an OSD got destroyed: OSDMap.cc: 5686: FAILED ceph_assert(num_d...: Hello Sebastian!
Was there any change about the OSD count? I mean particularly OSD removal. Radoslaw Zarzynski
01:10 AM Bug #52535: monitor crashes after an OSD got destroyed: OSDMap.cc: 5686: FAILED ceph_assert(num_d...: I Faced the same problem with ceph version 16.2.6. It occurred after shutting down all 3 physical servers of the clus... Sebastian Mazza
08:16 PM Backport #54506 (In Progress): quincy: doc/rados/operations/placement-groups/#automated-scaling: ...: https://github.com/ceph/ceph/pull/45321 Kamoltat (Junior) Sirivadhna
07:50 PM Backport #54506 (Resolved): quincy: doc/rados/operations/placement-groups/#automated-scaling: --b...: Backport Bot
08:15 PM Backport #54505 (In Progress): pacific: doc/rados/operations/placement-groups/#automated-scaling:...: https://github.com/ceph/ceph/pull/45328 Kamoltat (Junior) Sirivadhna
07:50 PM Backport #54505 (Resolved): pacific: doc/rados/operations/placement-groups/#automated-scaling: --...: Backport Bot
07:59 PM Bug #47299: Assertion in pg_missing_set: p->second.need <= v || p->second.is_delete(): If this is easily reproducible could you please provide us with logs of replicas for the failing PG? It can be figure... Radoslaw Zarzynski
07:55 PM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: More for my own reference, but it's clear that the rw_manager problem occurs here in the PrimaryLogPG code when prepp... Laura Flores
07:47 PM Bug #54485 (Pending Backport): doc/rados/operations/placement-groups/#automated-scaling: --bulk i...: Neha Ojha
07:46 PM Bug #51307 (In Progress): LibRadosWatchNotify.Watch2Delete fails: Radoslaw Zarzynski
07:36 PM Bug #53729: ceph-osd takes all memory before oom on boot: Neha Ojha wrote:
> Can you share the output of "ceph osd dump"? I suspect that though you may have disabled the au... Mykola Golub
07:31 PM Bug #53729: ceph-osd takes all memory before oom on boot: Neha Ojha wrote:
> Can anyone provide osd logs with debug_osd=20,debug_ms=1 for OSDs that are hitting OOM?
I uplo... Mykola Golub
06:58 PM Bug #53729: ceph-osd takes all memory before oom on boot: Can anyone provide osd logs with debug_osd=20,debug_ms=1 for OSDs that are hitting OOM? Neha Ojha
06:45 PM Bug #53729: ceph-osd takes all memory before oom on boot: Mykola Golub wrote:
> We seem to observe a similar issue (16.2.7). On a pool with autoscale disabled pg num was chan... Neha Ojha
05:56 PM Bug #53729: ceph-osd takes all memory before oom on boot: We seem to observe a similar issue (16.2.7). On a pool with autoscale disabled pg num was changed from 256 to 1024. A... Mykola Golub
07:13 PM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: jianwei zhang wrote:
> 1711'7107 : s0/1/2/3/4/5都有所以都能写下去
> 1715'7108 : s0/2/3/5 满足k=4，所以能写下去
> 1715'7109 : s0/2... Neha Ojha
06:43 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: ceph v15.2.13 tag jianwei zhang
05:42 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: [root@node1 ceph]# zcat ceph.client.log-20220308.gz|grep 202000000034931.0000001a
2022-03-08T03:12:25.531+0800 7f484... jianwei zhang
05:37 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: 1711'7107 : s0/1/2/3/4/5都有所以都能写下去
1715'7108 : s0/2/3/5 满足k=4，所以能写下去
1715'7109 : s0/2/3/5 满足k=4，所以能写下去
1715'71... jianwei zhang
05:35 AM Bug #53924: EC PG stuckrecovery_unfound+undersized+degraded+remapped+peered: I had a similar problem with pg recovery_unfound ... jianwei zhang
07:06 PM Bug #50042: rados/test.sh: api_watch_notify failures: Let's use this tracker to track all the watch notify failures. For other api test failures, let's open new trackers. ... Neha Ojha
03:36 PM Bug #50042: rados/test.sh: api_watch_notify failures: Found a case of https://tracker.ceph.com/issues/45423 in master, which had a fix that was merged. Seems like it's pop... Laura Flores
05:06 PM Backport #54468 (In Progress): octopus: Setting osd_pg_max_concurrent_snap_trims to 0 prematurely...: Laura Flores
04:56 PM Backport #54466 (In Progress): pacific: Setting osd_pg_max_concurrent_snap_trims to 0 prematurely...: Laura Flores
04:42 PM Backport #54467 (In Progress): quincy: Setting osd_pg_max_concurrent_snap_trims to 0 prematurely ...: Laura Flores
04:39 PM Backport #53659 (Resolved): pacific: mon: "FAILED ceph_assert(session_map.sessions.empty())" when...: https://github.com/ceph/ceph/pull/44543 has been merged. Laura Flores
04:36 PM Backport #53978 (Resolved): quincy: [RFE] Limit slow request details to mgr log: Laura Flores
04:36 PM Backport #53388 (Resolved): pacific: pg-temp entries are not cleared for PGs that no longer exist: Laura Flores
04:36 PM Backport #51150 (Resolved): pacific: When read failed, ret can not take as data len, in FillInVer...: Laura Flores
04:35 PM Backport #53486 (Resolved): pacific: LibRadosTwoPoolsPP.ManifestSnapRefcount Failure.: Laura Flores
04:35 PM Backport #53702 (Resolved): pacific: qa/tasks/backfill_toofull.py: AssertionError: 2.0 not in bac...: Laura Flores
04:33 PM Backport #53942 (Resolved): pacific: mon: all mon daemon always crash after rm pool: Laura Flores
04:33 PM Backport #53535 (Resolved): pacific: mon: mgrstatmonitor spams mgr with service_map: Laura Flores
04:32 PM Backport #53718 (Resolved): pacific: mon: frequent cpu_tp had timed out messages: Laura Flores
04:28 PM Backport #53480 (Resolved): pacific: Segmentation fault under Pacific 16.2.1 when using a custom ...: Laura Flores
04:12 PM Backport #52077 (In Progress): octopus: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite: Laura Flores
04:11 PM Backport #52078 (In Progress): pacific: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite: Laura Flores
03:33 PM Bug #45423: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite: Seems like this may have come back:
/a/dgalloway-2022-03-09_02:34:58-rados-wip-45272-distro-basic-smithi/6727572 Laura Flores

03/08/2022

06:52 PM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...: Myoungwon Oh wrote:
> The error message looks like similar before, but the cause is difference from the prior case.
... Neha Ojha
10:00 AM Bug #54489 (New): mon: ops get stuck in "resend forwarded message to leader": I hited this bug "BUG #22114":https://tracker.ceph.com/issues/22114#change-211414 in octopus.
"description": "log(2 ... Jiaxing Fan
09:02 AM Bug #51307: LibRadosWatchNotify.Watch2Delete fails: Laura Flores wrote:
> /a/yuriw-2022-02-16_00:25:26-rados-wip-yuri-testing-2022-02-15-1431-distro-default-smithi/6687... Nitzan Mordechai

03/07/2022

03:00 PM Bug #54485 (Fix Under Review): doc/rados/operations/placement-groups/#automated-scaling: --bulk i...: Kamoltat (Junior) Sirivadhna
02:45 PM Bug #54485 (Resolved): doc/rados/operations/placement-groups/#automated-scaling: --bulk invalid c...: Command for creating a pool
was: `ceph osd create test_pool --bulk`
should be: `ceph osd pool create test_pool ... Kamoltat (Junior) Sirivadhna

03/06/2022

08:42 PM Bug #47299: Assertion in pg_missing_set: p->second.need <= v || p->second.is_delete(): Just got this again during a recovery after doing maintenance on another node this OSD crashed.
-1> 2022-03-06T... Tobias Urdin

03/01/2022

11:22 PM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: I'm guessing that the problem involves pgs that are stuck in the `active+recovering+undersized+remapped` state (or `a... Laura Flores
05:26 PM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: /a/yuriw-2022-02-15_16:22:25-rados-wip-yuri6-testing-2022-02-14-1456-distro-default-smithi/6685233... Laura Flores
08:11 PM Bug #54438: test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid].data, ...: /a/benhanokh-2021-08-04_06:12:22-rados-wip_gbenhano_ncbz-distro-basic-smithi/6310791/ Neha Ojha
05:40 PM Bug #54438 (New): test/objectstore/store_test.cc: FAILED ceph_assert(bl_eq(state->contents[noid]....: /a/yuriw-2022-02-15_16:22:25-rados-wip-yuri6-testing-2022-02-14-1456-distro-default-smithi/6685291... Laura Flores
06:13 PM Bug #52319: LibRadosWatchNotify.WatchNotify2 fails: I linked a related issue that looks very similar to this failure, except with a slightly different LibRadosWatchNotif... Laura Flores
06:11 PM Bug #54439 (New): LibRadosWatchNotify.WatchNotify2Multi fails: /a/yuriw-2022-02-28_21:23:00-rados-wip-yuri-testing-2022-02-28-0823-quincy-distro-default-smithi/6711961... Laura Flores
05:21 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): /a/yuriw-2022-02-15_16:22:25-rados-wip-yuri6-testing-2022-02-14-1456-distro-default-smithi/6685226 Laura Flores

02/28/2022

09:28 PM Backport #54082 (Resolved): pacific: mon: osd pool create <pool-name> with --bulk flag: Kamoltat (Junior) Sirivadhna
06:53 PM Bug #50842: pacific: recovery does not complete because of rw_manager lock not being released: I recovered logs from a scenario that looks very similar.
See the full result of `zcat /a/yuriw-2022-02-17_22:49:5... Laura Flores
11:34 AM Bug #54423 (New): osd/scrub: bogus DigestUpdate events are created, logged and (hopefully) rejected: A mishandling of the counter of "the digest-updates we are waiting for, before finishing
with this scrubbed chunk" c... Ronen Friedman

02/25/2022

10:57 PM Backport #53480: pacific: Segmentation fault under Pacific 16.2.1 when using a custom crush locat...: Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44897
merged Yuri Weinstein
10:56 PM Backport #54082: pacific: mon: osd pool create <pool-name> with --bulk flag: Kamoltat Sirivadhna wrote:
> pull request: https://github.com/ceph/ceph/pull/44847
merged Yuri Weinstein
09:59 PM Backport #54412 (Rejected): pacific:osd:add pg_num_max value: https://github.com/ceph/ceph/pull/45173 Kamoltat (Junior) Sirivadhna
05:55 PM Bug #50042: rados/test.sh: api_watch_notify failures: /a/yuriw-2022-02-24_22:04:22-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6704772... Laura Flores
05:54 AM Bug #54364 (Resolved): The built-in osd bench test shows inflated results.: Sridhar Seshasayee
05:54 AM Backport #54393 (Resolved): quincy: The built-in osd bench test shows inflated results.: Sridhar Seshasayee

02/24/2022

10:45 PM Backport #54386: octopus: [RFE] Limit slow request details to mgr log: please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/45154
ceph-backport.sh versi... Ponnuvel P
07:43 PM Bug #52136: Valgrind reports memory "Leak_DefinitelyLost" errors.: /a/sseshasa-2022-02-24_11:27:07-rados-wip-45118-45121-quincy-testing-distro-default-smithi/6704275/remote/smithi174/l... Laura Flores
07:19 PM Bug #53294: rados/test.sh hangs while running LibRadosTwoPoolsPP.TierFlushDuringFlush: /a/sseshasa-2022-02-24_11:27:07-rados-wip-45118-45121-quincy-testing-distro-default-smithi/6704402... Laura Flores
06:36 PM Bug #54368 (Duplicate): ModuleNotFoundError: No module named 'tasks.cephadm': Neha Ojha
05:51 PM Backport #53644 (In Progress): pacific: Disable health warning when autoscaler is on: Christopher Hoffman
03:33 PM Backport #53551 (Resolved): pacific: [RFE] Provide warning when the 'require-osd-release' flag do...: Sridhar Seshasayee
08:56 AM Bug #54396: Setting osd_pg_max_concurrent_snap_trims to 0 prematurely clears the snaptrim queue: More context:... Dan van der Ster
08:44 AM Bug #54396 (Fix Under Review): Setting osd_pg_max_concurrent_snap_trims to 0 prematurely clears t...: Dan van der Ster
08:41 AM Bug #54396 (Resolved): Setting osd_pg_max_concurrent_snap_trims to 0 prematurely clears the snapt...: See https://www.spinics.net/lists/ceph-users/msg71061.html... Dan van der Ster
08:38 AM Backport #54393 (Resolved): quincy: The built-in osd bench test shows inflated results.: https://github.com/ceph/ceph/pull/45141 Sridhar Seshasayee
08:37 AM Bug #54364 (Pending Backport): The built-in osd bench test shows inflated results.: Sridhar Seshasayee
02:45 AM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...: The error message looks like similar before, but the cause is difference from the prior case.
Anyway, I posted the f... Myoungwon Oh

02/23/2022

05:32 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): Happened in a dead job.
/a/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6... Laura Flores
05:16 PM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...: Happened again. Could this be a new occurrence?
/a/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800... Laura Flores
05:00 PM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...: /a/yuriw-2022-02-21_15:40:41-rados-wip-yuri4-testing-2022-02-18-0800-distro-default-smithi/6698327 Laura Flores
03:15 PM Backport #54386 (Resolved): octopus: [RFE] Limit slow request details to mgr log: Backport Bot

02/22/2022

09:10 PM Bug #54210 (Fix Under Review): pacific: mon/pg_autoscaler.sh: echo failed on "bash -c 'ceph osd p...: Kamoltat (Junior) Sirivadhna
09:09 PM Bug #54210: pacific: mon/pg_autoscaler.sh: echo failed on "bash -c 'ceph osd pool get a pg_num | ...: After going through sentry, I've realized that the only occurrence of this bug in master happens before the merge of
... Kamoltat (Junior) Sirivadhna
08:39 PM Backport #54233 (In Progress): octopus: devices: mon devices appear empty when scraping SMART met...: Neha Ojha
08:39 PM Backport #54232 (In Progress): pacific: devices: mon devices appear empty when scraping SMART met...: Neha Ojha
08:14 PM Bug #54369 (New): mon/test_mon_osdmap_prune.sh: jq .osdmap_first_committed [[ 11 -eq 20 ]]: /a/yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6692990... Kamoltat (Junior) Sirivadhna
07:31 PM Bug #54368 (Duplicate): ModuleNotFoundError: No module named 'tasks.cephadm': /a/yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6692894... Kamoltat (Junior) Sirivadhna
07:19 PM Bug #47589: radosbench times out "reached maximum tries (800) after waiting for 4800 seconds": /a/yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6692841 Kamoltat (Junior) Sirivadhna
06:29 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): /a/yuriw-2022-02-21_18:20:15-rados-wip-yuri11-testing-2022-02-21-0831-quincy-distro-default-smithi/6699270
Happene... Laura Flores
05:09 PM Bug #50659: Segmentation fault under Pacific 16.2.1 when using a custom crush location hook: Chris Durham wrote:
> This issue bit us in our upgrade to 16.2.7 from 15.2.15. We have a manual cluster (non-cephadm... Chris Durham
04:11 PM Bug #50659: Segmentation fault under Pacific 16.2.1 when using a custom crush location hook: This issue bit us in our upgrade to 16.2.7 from 15.2.15. We have a manual cluster (non-cephadm). We followed the proc... Chris Durham
12:57 PM Bug #54364 (Resolved): The built-in osd bench test shows inflated results.: The built-in osd bench shows inflated results with up to 3x-4x the expected values.
Example:
Before:
{
"b... Sridhar Seshasayee
07:13 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: We were able to narrow it down further. We can trigger the problem reliably by doing this:
- 2 clusters, multisite... Dieter Roels

02/21/2022

05:09 PM Feature #44107 (Resolved): mon: produce stable election results when netsplits and other errors h...: Oh, this has been done for ages. Greg Farnum
12:47 PM Bug #51463: blocked requests while stopping/starting OSDs: I think we hit the same issue while upgrading our nautilus cluster to pacific.
While I did not hit this when testing... Maximilian Stinsky
12:42 PM Backport #53339: pacific: src/osd/scrub_machine.cc: FAILED ceph_assert(state_cast<const NotActive...: Hey I got here through the following mailing list post: https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thre... Maximilian Stinsky
01:27 AM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: /a/yuriw-2022-02-17_22:49:55-rados-wip-yuri3-testing-2022-02-17-1256-distro-default-smithi/6692376... Laura Flores

02/20/2022

09:25 AM Bug #52901: osd/scrub: setting then clearing noscrub may lock a PG in 'scrubbing' state: Will this be backported to a stable release? André Cruz
09:23 AM Bug #54172: ceph version 16.2.7 PG scrubs not progressing: I'm also seeing the same issue on 16.2.7, but it's been going on for almost two weeks. Already set and unset noscrub/... André Cruz
09:18 AM Backport #53339: pacific: src/osd/scrub_machine.cc: FAILED ceph_assert(state_cast<const NotActive...: Could this be the reason I'm seeing a spam of "handle_scrub_reserve_grant: received unsolicited reservation grant" me... André Cruz

02/18/2022

01:32 PM Bug #52026: osd: pgs went back into snaptrim state after osd restart: Fortunately (or perhaps not so fortunately), in the process of dealing with this issue we performed a full restart of... David Prude
11:49 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: Dieter Roels wrote:
> Hi CHristian. Are your rgws collocated with the osds of the metadata pools?
> We now notice i... Christian Rohmann
11:31 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: Hi CHristian. Are your rgws collocated with the osds of the metadata pools?
We now notice in our clusters that the... Dieter Roels

02/17/2022

09:43 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): /a/yuriw-2022-02-16_15:53:49-rados-wip-yuri11-testing-2022-02-15-1643-distro-default-smithi/6688846 Laura Flores
07:43 PM Bug #54210: pacific: mon/pg_autoscaler.sh: echo failed on "bash -c 'ceph osd pool get a pg_num | ...: yep this is a bug, thanks for letting me know, patch coming up. Kamoltat (Junior) Sirivadhna
06:54 PM Backport #54290 (Resolved): quincy: pybind/mgr/progress: disable pg recovery event by default: Kamoltat (Junior) Sirivadhna
05:46 PM Bug #54316 (Resolved): mon/MonCommands.h: target_size_ratio range is incorrect: Currently if we give `target_size_ratio` a value more than 1.0 using the command: `ceph osd pool create <pool-name> -... Kamoltat (Junior) Sirivadhna
05:00 PM Bug #54263: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 32768 for ceph...: Update:
From the monitor sides of things of pool creation, target_size_ratio cannot be more than 1.0 or less than ... Kamoltat (Junior) Sirivadhna
04:52 PM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: /a/yuriw-2022-02-16_00:25:26-rados-wip-yuri-testing-2022-02-15-1431-distro-default-smithi/6687342
Same issue with ... Laura Flores
04:30 PM Bug #51307: LibRadosWatchNotify.Watch2Delete fails: /a/yuriw-2022-02-16_00:25:26-rados-wip-yuri-testing-2022-02-15-1431-distro-default-smithi/6687338 Laura Flores
11:54 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: We just observed 12 more scrub errors spread across 7 pgs and all on our primary (used for user access, read/write) z... Christian Rohmann
09:47 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: Neha Ojha wrote:
> yite gu wrote:
> > Christian Rohmann wrote:
> > > yite gu wrote:
> > > > This is inconsistent ... yite gu
09:51 AM Bug #54296: OSDs using too much memory: Hi Dan,
Thanks for your response.
I only adjusted osd_max_pg_log_entries and left osd_min_pg_log_entries alone. A... Ruben Kerkhof
09:03 AM Bug #54296: OSDs using too much memory: Ruben Kerkhof wrote:
> One thing I tried was to set osd_max_pg_log_entries to 500 instead of the default of 10000, b... Dan van der Ster
09:33 AM Bug #54172: ceph version 16.2.7 PG scrubs not progressing: I would just like to add that scrubs started all of the sudden and the cluster is HEALTH_OK again. Daan van Gorkum

02/16/2022

10:32 PM Bug #52657: MOSDPGLog::encode_payload(uint64_t): Assertion `HAVE_FEATURE(features, SERVER_NAUTILUS)': /ceph/teuthology-archive/yuriw-2022-02-15_22:35:42-rados-wip-yuri8-testing-2022-02-15-1214-distro-default-smithi/6686... Neha Ojha
09:20 PM Backport #53718: pacific: mon: frequent cpu_tp had timed out messages: Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44545
merged Yuri Weinstein
08:24 PM Backport #54290: quincy: pybind/mgr/progress: disable pg recovery event by default: https://github.com/ceph/ceph/pull/45043 merged Yuri Weinstein
07:59 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): /a/yuriw-2022-02-15_22:40:39-rados-wip-yuri7-testing-2022-02-15-1102-quincy-distro-default-smithi/6686655/remote/smit... Neha Ojha
07:07 PM Backport #53535: pacific: mon: mgrstatmonitor spams mgr with service_map: Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44721
merged Yuri Weinstein
07:06 PM Backport #53942: pacific: mon: all mon daemon always crash after rm pool: Backport Bot wrote:
> https://github.com/ceph/ceph/pull/44698
merged Yuri Weinstein
07:00 PM Feature #54280: support truncation sequences in sparse reads: Neha mentioned taking a look at the history, so I did a bit of git archeology today. The limitation dates back to the... Jeff Layton
06:58 PM Bug #53751: "N monitors have not enabled msgr2" is always shown for new clusters: Hello. Could you please provide the output from @ceph health detail@? We suspect the warning might got replaced with ... Radoslaw Zarzynski
06:46 PM Bug #54255: utc time is used when ceph crash ls: Yaarit, was this choice intentional? Neha Ojha
06:44 PM Bug #51338 (Duplicate): osd/scrub_machine.cc: FAILED ceph_assert(state_cast<const NotActive*&g...: Neha Ojha
06:44 PM Bug #51338: osd/scrub_machine.cc: FAILED ceph_assert(state_cast<const NotActive*>()): André Cruz wrote:
> I'm also encountering this issue on Pacific (16.2.7):
>
> [...]
>
> Any pointers?
I thi... Neha Ojha
06:31 PM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: yite gu wrote:
> Christian Rohmann wrote:
> > yite gu wrote:
> > > This is inconsistent pg 7.2 from your upload fi... Neha Ojha
06:24 PM Bug #53663 (New): Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: Neha Ojha
09:50 AM Bug #53663: Random scrub errors (omap_digest_mismatch) on pgs of RADOSGW metadata pools: Dieter Roels wrote:
> After the repair the inconsistencies do not re-appear. However, we can reproduce the issue in ... Dieter Roels
06:18 PM Bug #46847: Loss of placement information on OSD reboot: Frank Schilder wrote:
> Could somebody please set the status back to open and Affected Versions to all?
The ticke... Neha Ojha
06:13 PM Bug #53729 (Need More Info): ceph-osd takes all memory before oom on boot: Neha Ojha
12:21 PM Bug #54296: OSDs using too much memory: Hi Igor,
See attachment.
One thing I tried was to set osd_max_pg_log_entries to 500 instead of the default of 1... Ruben Kerkhof
12:15 PM Bug #54296: OSDs using too much memory: Hi Ruben,
please share full dump_mempools output. Igor Fedotov
10:34 AM Bug #54296 (Resolved): OSDs using too much memory: One of our customers upgraded from Nautilus to Octopus, and now a lot of his OSDs are using way more ram than allowed... Ruben Kerkhof

02/15/2022

11:21 PM Bug #54263: cephadm upgrade pacific to quincy autoscaler is scaling pgs from 32 -> 32768 for ceph...: In summary,
the root cause of the problem is after the upgrade to quincy, cephfs meta data pool was somehow given a ... Kamoltat (Junior) Sirivadhna
10:57 PM Bug #53855 (Fix Under Review): rados/test.sh hangs while running LibRadosTwoPoolsPP.ManifestFlush...: Laura Flores
02:07 AM Bug #53855: rados/test.sh hangs while running LibRadosTwoPoolsPP.ManifestFlushDupCount: https://github.com/ceph/ceph/pull/45035 Myoungwon Oh
07:27 PM Bug #51904: test_pool_min_size:AssertionError:wait_for_clean:failed before timeout expired due to...: /a/yuriw-2022-02-08_17:00:23-rados-wip-yuri5-testing-2022-02-08-0733-pacific-distro-default-smithi/6670539
last pg... Laura Flores
07:15 PM Bug #50222: osd: 5.2s0 deep-scrub : stat mismatch: Looks similar, but different test.
/a/yuriw-2022-02-09_22:52:18-rados-wip-yuri5-testing-2022-02-09-1322-pacific-di... Laura Flores
06:55 PM Bug #45702: PGLog::read_log_and_missing: ceph_assert(miter == missing.get_items().end() || (miter...: /a/yuriw-2022-02-09_22:52:18-rados-wip-yuri5-testing-2022-02-09-1322-pacific-distro-default-smithi/6672070 Laura Flores
06:47 PM Bug #53327: osd: osd_fast_shutdown_notify_mon not quite right and enable osd_fast_shutdown_notify...: Hi Nitzan,
I checked your patch on the current pacific branch.
unfortunately I still get slow ops (slow >= 5 seco... Manuel Lausch
06:46 PM Bug #48997: rados/singleton/all/recovery-preemption: defer backfill|defer recovery not found in logs: /a/yuriw-2022-02-09_22:52:18-rados-wip-yuri5-testing-2022-02-09-1322-pacific-distro-default-smithi/6672005 Laura Flores
03:45 PM Backport #54290 (Resolved): quincy: pybind/mgr/progress: disable pg recovery event by default: Kamoltat (Junior) Sirivadhna
03:42 PM Bug #47273 (Fix Under Review): ceph report missing osdmap_clean_epochs if answered by peon: Dan van der Ster
03:08 AM Bug #52421: test tracker: Crash signature (v1) and Crash signature (v2) are of invalid format, and are breaking the telemetry crashes bot, remo... Yaarit Hatuka

02/14/2022

11:46 PM Bug #52124: Invalid read of size 8 in handle_recovery_delete(): /a/yuriw-2022-02-08_17:00:23-rados-wip-yuri5-testing-2022-02-08-0733-pacific-distro-default-smithi/6670360 Laura Flores
11:29 PM Bug #51234: LibRadosService.StatusFormat failed, Expected: (0) != (retry), actual: 0 vs 0: Pacific:
/a/yuriw-2022-02-09_22:52:18-rados-wip-yuri5-testing-2022-02-09-1322-pacific-distro-default-smithi/6672177 Laura Flores
08:21 PM Feature #54280 (Resolved): support truncation sequences in sparse reads: I've been working on sparse read support in the kclient, and got something working today, only to notice that after t... Jeff Layton
03:39 PM Bug #51076: "wait_for_recovery: failed before timeout expired" during thrashosd test with EC back...: /a/yuriw-2022-02-11_22:59:19-rados-wip-yuri4-testing-2022-02-11-0858-distro-default-smithi/6677733
Last pg map bef... Laura Flores
10:06 AM Bug #46847: Loss of placement information on OSD reboot: Could somebody please set the status back to open and Affected Versions to all? Frank Schilder

« Previous

Also available in: Atom

Project

General

Profile

Ceph » RADOS

Activity

Activity

03/14/2022

03/13/2022

03/11/2022

03/10/2022

03/09/2022

03/08/2022

03/07/2022

03/06/2022

03/04/2022

03/03/2022

03/02/2022

03/01/2022

02/28/2022

02/25/2022

02/24/2022

02/23/2022

02/22/2022

02/21/2022

02/20/2022

02/18/2022

02/17/2022

02/16/2022

02/15/2022

02/14/2022