Activity
From 06/24/2021 to 07/23/2021
07/23/2021
- 10:50 PM Bug #51816: monitor segfault on startup in container
- I tested yesterday with the latest master build available on shaman : "ceph version 17.0.0-6285-gc011af69 (c011af6903...
- 09:21 PM Bug #51816: monitor segfault on startup in container
- This is related to https://github.com/ceph/ceph/pull/42014. I know there have been a few follow-on fixes for this PR,...
- 10:10 PM Bug #51729: Upmap verification fails for multi-level crush rule
- Binary osd map attached.
- 09:38 PM Bug #51729 (Need More Info): Upmap verification fails for multi-level crush rule
- Can you share copy of your binary osdmap?
- 09:59 PM Backport #51663: pacific: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmak...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42344
merged - 09:58 PM Backport #51582: octopus: osd does not proactively remove leftover PGs
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42239
merged - 09:57 PM Backport #51584: pacific: osd does not proactively remove leftover PGs
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42238
merged - 09:54 PM Bug #51000: LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- Myoungwon Oh wrote:
> Ok, I'll take a look.
thanks! - 09:54 PM Bug #51000 (Fix Under Review): LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- 09:54 PM Backport #49377: pacific: building libcrc32
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/39902
merged - 09:43 PM Bug #51083 (Need More Info): Raw space filling up faster than used space
- Moving to need more info, please let us know if it happens again.
- 09:41 PM Bug #45423: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite
- Hi Myoungwon Oh, this is tiering test has been causing a lot of failures, do you think you can help investigate?
- 09:34 PM Bug #48908 (Need More Info): EC Pool OSD crashes
- Can you capture a coredump for the crash? This appears to be memory leak of some sort, what is the memory/RSS usage l...
- 09:23 PM Bug #51799 (Fix Under Review): osd: snaptrim logs to derr at every tick
- 06:42 PM Backport #51662: octopus: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmak...
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42421
merged - 06:30 PM Backport #51553: pacific: rebuild-mondb hangs
- Kefu Chai wrote:
> https://github.com/ceph/ceph/pull/42411
merged - 02:40 PM Backport #51830 (Resolved): pacific: set a non-zero default value for osd_client_message_cap
- https://github.com/ceph/ceph/pull/42615
- 02:36 PM Bug #49894 (Pending Backport): set a non-zero default value for osd_client_message_cap
- 11:45 AM Bug #48750: ceph config set using osd/host mask not working
- Do the other masks (non-host) masks work for you?
I have the same problem in octopus. class masks work, as do crus...
07/22/2021
- 10:18 PM Bug #51816 (Resolved): monitor segfault on startup in container
- The ceph-container project runs a demo container to validate the container build which starts few daemons and tests i...
- 08:53 PM Bug #49689: osd/PeeringState.cc: ceph_abort_msg("past_interval start interval mismatch") start
- /a/yuriw-2021-07-21_19:04:13-rados-wip-yuri4-testing-2021-07-21-0859-pacific-distro-basic-smithi/6285380
- 04:25 PM Backport #51496: octopus: mgr spamming with repeated set pgp_num_actual while merging
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42420
merged - 12:41 PM Bug #51799 (Resolved): osd: snaptrim logs to derr at every tick
- Hi,
We reduced `osd_snap_trim_sleep` to 0.1 to speed up snap trim process but now the OSD spam the log on every ti... - 05:45 AM Bug #48841 (Resolved): test_turn_off_module: wait_until_equal timed out
- https://github.com/ceph/ceph/pull/41907 fixed this
- 04:04 AM Bug #51725 (Fix Under Review): make bufferlist::c_str() skip rebuild when it isn't necessary
07/21/2021
- 09:32 PM Backport #51452: octopus: Add simultaneous scrubs to rados/thrash
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42422
merged - 09:28 PM Backport #51314: octopus: osd:scrub skip some pg
- Mykola Golub wrote:
> https://github.com/ceph/ceph/pull/41972
merged - 04:38 PM Backport #51315: nautilus: osd:scrub skip some pg
- Mykola Golub wrote:
> https://github.com/ceph/ceph/pull/41973
merged
- 02:00 PM Backport #51766 (Resolved): pacific: scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag fa...
- https://github.com/ceph/ceph/pull/42398
- 01:58 PM Bug #51581 (Pending Backport): scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- 01:55 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- David Galloway wrote:
> Deepika Upadhyay wrote:
> > David Galloway wrote:
> > > Oh, I see. Okay. I will fix.
> ... - 03:56 AM Bug #48908: EC Pool OSD crashes
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=45667d498bde59f350e2c1c48...
07/20/2021
- 06:47 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- Deepika Upadhyay wrote:
> David Galloway wrote:
> > Oh, I see. Okay. I will fix.
>
> Awesome, thanks! :)
Sh... - 04:15 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- David Galloway wrote:
> Oh, I see. Okay. I will fix.
Awesome, thanks! :) - 04:14 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- Oh, I see. Okay. I will fix.
- 04:13 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- Deepika Upadhyay wrote:
> http://qa-proxy.ceph.com/teuthology/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021... - 03:29 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- http://qa-proxy.ceph.com/teuthology/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021-07-16-0906-octopus-distro-...
- 04:35 PM Bug #49894 (Fix Under Review): set a non-zero default value for osd_client_message_cap
- 03:53 PM Bug #50106: scrub/osd-scrub-repair.sh: corrupt_scrub_erasure: return 1
- /a/yuriw-2021-07-16_18:39:18-rados-wip-yuri-testing-master-7.16.21-distro-basic-smithi/6276655
- 03:38 PM Backport #50911 (Rejected): nautilus: PGs always go into active+clean+scrubbing+deep+repair in th...
- nautilus is EOL
- 03:36 PM Backport #51452 (In Progress): octopus: Add simultaneous scrubs to rados/thrash
- 03:33 PM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...
- http://qa-proxy.ceph.com/teuthology/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021-07-16-0906-octopus-distro-...
- 03:32 PM Backport #51662 (In Progress): octopus: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol look...
- 03:31 PM Backport #51496 (In Progress): octopus: mgr spamming with repeated set pgp_num_actual while merging
- 03:15 PM Backport #51741 (Rejected): octopus: CommandCrashedError: Command crashed: 'mkdir -p -- /home/ubu...
- 03:14 PM Bug #50393: CommandCrashedError: Command crashed: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client...
- /ceph/teuthology-archive/yuriw-2021-07-16_17:39:36-rados-wip-yuri7-testing-2021-07-16-0906-octopus-distro-basic-smith...
- 07:45 AM Backport #51553 (In Progress): pacific: rebuild-mondb hangs
- 06:03 AM Bug #51581 (Fix Under Review): scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
07/19/2021
- 05:12 PM Bug #51729 (In Progress): Upmap verification fails for multi-level crush rule
- We have a 6+3 EC crush rule that looks like this:...
- 02:25 PM Bug #51725 (In Progress): make bufferlist::c_str() skip rebuild when it isn't necessary
- WIP branch: https://github.com/ceph/ceph/compare/master...rzarzynski:wip-bl-optimize-emptybptrs.
- 11:00 AM Bug #51725 (Resolved): make bufferlist::c_str() skip rebuild when it isn't necessary
- > The ceph_msg_header2 on the other hand, the bufferlist comes from the segment data, which is also contiguous, but s...
- 02:20 PM Bug #50659: Segmentation fault under Pacific 16.2.1 when using a custom crush location hook
- I saw that 16.2.5 was released. Though I didn't expect it to address this issue, I tested with it anyway just to be s...
- 01:45 PM Backport #50900: pacific: PGs always go into active+clean+scrubbing+deep+repair in the LRC
- I created a backport PR.
https://github.com/ceph/ceph/pull/42398 - 01:33 PM Backport #50900: pacific: PGs always go into active+clean+scrubbing+deep+repair in the LRC
- please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/42398
ceph-backport.sh versi...
07/18/2021
07/16/2021
- 06:08 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- Neha Ojha wrote:
> David Galloway wrote:
> > https://github.com/ceph/ceph-cm-ansible/pull/638
> >
> > I'd love f... - 01:23 PM Bug #45423: api_tier_pp: [ FAILED ] LibRadosTwoPoolsPP.HitSetWrite
- /a/sseshasa-2021-07-14_10:37:09-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6270208
- 01:18 PM Bug #47025: rados/test.sh: api_watch_notify_pp LibRadosWatchNotifyECPP.WatchNotify failed
- Observed on master:
/a/sseshasa-2021-07-14_10:37:09-rados-wip-sseshasa-testing-2021-07-14-1320-distro-basic-smithi/6... - 10:38 AM Bug #51083: Raw space filling up faster than used space
- So apparently "arbitrary" changes to the CRUSH map are able to free up the space again:
A few days ago, we activat...
07/15/2021
- 11:37 PM Bug #51652: heartbeat timeouts on filestore OSDs while deleting objects in upgrade:pacific-p2p-pa...
- ...
- 08:17 PM Bug #43584 (Fix Under Review): MON_DOWN during mon_join process
- 03:57 PM Bug #51688 (Pending Backport): "stuck peering for" warning is misleading
- When OSDs restart or crush maps change it is common to see a HEALTH_WARN claiming that PGs have been stuck peering si...
- 03:34 PM Backport #51556: pacific: mon: return -EINVAL when handling unknown option in 'ceph osd pool get'
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42229
merged - 02:47 PM Backport #51498: pacific: mgr spamming with repeated set pgp_num_actual while merging
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42223
merged - 02:33 PM Bug #45702: PGLog::read_log_and_missing: ceph_assert(miter == missing.get_items().end() || (miter...
- Observed the assert on master:
/a/sseshasa-2021-07-14_10:37:09-rados-wip-sseshasa-testing-2021-07-14-1320-distro-bas... - 07:47 AM Bug #51641: qa/standalone/scrub/osd-scrub-test.sh: TEST_scrub_test() failed.
- At first glance:
- on OSD.1, PG 1.0 status was updated to 'clean' at 07:08.8
- AFAIK the update might take up to 5 ...
07/14/2021
- 09:22 PM Backport #51663 (In Progress): pacific: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol look...
- 05:32 PM Bug #51638 (Resolved): rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel...
- David Galloway wrote:
> https://github.com/ceph/ceph-cm-ansible/pull/638
>
> I'd love for that ... - 03:33 PM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- https://github.com/ceph/ceph-cm-ansible/pull/638
I'd love for that ... - 04:25 AM Bug #51638: rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel with cento...
- Discussed this with Neha and we took a bit of a look on a node where the test failed.
In order to enable the Power... - 04:50 PM Backport #51603 (Resolved): pacific: qa/standalone: Add missing teardowns at the end of a subset ...
- 02:24 PM Backport #51570: pacific: CommandCrashedError: Command crashed: 'mkdir -p -- /home/ubuntu/cephtes...
- Neha Ojha wrote:
> https://github.com/ceph/ceph/pull/42221
merged - 11:47 AM Bug #51641: qa/standalone/scrub/osd-scrub-test.sh: TEST_scrub_test() failed.
- I looked into the osd logs for more info. The repair was initiated on osd.1.
osd.1 got the request for shallow rep...
07/13/2021
- 09:15 PM Backport #51663 (Resolved): pacific: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup ...
- https://github.com/ceph/ceph/pull/42344
- 09:15 PM Backport #51662 (Resolved): octopus: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup ...
- https://github.com/ceph/ceph/pull/42421
- 09:12 PM Bug #51101 (Pending Backport): rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error:...
- Based on /a/teuthology-2021-06-23_03:30:02-rados-octopus-distro-basic-gibba/6185359
- 01:43 AM Bug #51101 (Fix Under Review): rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error:...
- 01:26 AM Bug #51101: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmake: undefined s...
- archive_write_add_filter_zstd() was added on libarchive-3.3.3. see https://bugzilla.redhat.com/show_bug.cgi?id=182792...
- 03:38 PM Bug #51644 (Fix Under Review): don't assert on bogus CEPH_OSD_ZERO request
- 02:22 PM Bug #51644: don't assert on bogus CEPH_OSD_ZERO request
- Building an image to test the fix now.
- 01:54 PM Bug #51644 (Resolved): don't assert on bogus CEPH_OSD_ZERO request
- I was testing some changes to the kclient, and was able to crash the OSD with a stack trace like this:...
- 03:02 PM Bug #51652: heartbeat timeouts on filestore OSDs while deleting objects in upgrade:pacific-p2p-pa...
- re: https://github.com/ceph/ceph/pull/42264
Rerun: https://pulpito.ceph.com/yuriw-2021-07-13_14:59:01-upgrade:paci... - 03:02 PM Bug #51652 (New): heartbeat timeouts on filestore OSDs while deleting objects in upgrade:pacific-...
- Run: https://pulpito.ceph.com/teuthology-2021-07-10_14:15:22-upgrade:pacific-p2p-pacific-distro-basic-smithi/
Job: 6... - 11:17 AM Backport #50900: pacific: PGs always go into active+clean+scrubbing+deep+repair in the LRC
- Could you tell me the status of this backport patch? I want this backport because I hit a problem related to this pro...
- 10:04 AM Bug #51000 (In Progress): LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- https://github.com/ceph/ceph/pull/42302
- 08:34 AM Bug #51641 (New): qa/standalone/scrub/osd-scrub-test.sh: TEST_scrub_test() failed.
- /a/sseshasa-2021-07-13_04:55:53-rados:standalone-wip-sseshasa-testing-2021-07-12-1850-distro-basic-smithi/6267736
...
07/12/2021
- 11:11 PM Bug #51101: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmake: undefined s...
- Neha Ojha wrote:
> fails differently with centos 8.stream
Tracked in https://tracker.ceph.com/issues/51638
>
... - 05:56 PM Bug #51101: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmake: undefined s...
- fails differently with centos 8.stream...
- 05:38 PM Bug #51101: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmake: undefined s...
- /a/yuriw-2021-07-12_16:33:44-rados-wip-yuriw-master-7.8.21-distro-basic-smithi/6265227/
This seems to be an issue ... - 09:25 PM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...
- spotted again at ksirivad-2021-07-11_01:45:00-rados-wip-pg-autoscaler-overlap-distro-basic-smithi/6262857/
- 01:20 PM Bug #51627 (Fix Under Review): FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missi...
- 08:16 AM Bug #51627: FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missing(soi...
- https://github.com/ceph/ceph/pull/42279
- 02:31 AM Bug #51627 (Resolved): FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_...
- spotted again,...
- 09:24 PM Bug #49525: found snap mapper error on pg 3.2s1 oid 3:4abe9991:::smithi10121515-14:e4 snaps missi...
- spotted again /a/ksirivad-2021-07-11_01:45:00-rados-wip-pg-autoscaler-overlap-distro-basic-smithi/6262966/
- 08:51 PM Bug #51638 (Resolved): rados/test_envlibrados_for_rocksdb.sh: No match for argument: snappy-devel...
- ...
- 08:00 PM Backport #50748 (Resolved): pacific: max_misplaced was replaced by target_max_misplaced_ratio
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/42250
m... - 07:54 PM Backport #50790 (Resolved): octopus: osd: write_trunc omitted to clear data digest
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41620
m... - 02:31 AM Bug #50192 (Pending Backport): FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missi...
- 02:22 AM Bug #50192 (New): FAILED ceph_assert(attrs || !recovery_state.get_pg_log().get_missing().is_missi...
07/11/2021
- 04:02 PM Bug #51581: scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- The bug is triggered when scrubbing is not initiated on the first tick-timer after being requested. That happens if t...
- 03:55 PM Bug #51581 (In Progress): scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- 09:13 AM Bug #51626 (New): OSD uses all host memory (80g) on startup due to pg_split
- After upgrading from 15.2.4 to 15.2.13 _some_ OSDs fails to start.
The OSDs which are failing to start seem to be... - 08:41 AM Support #51609: OSD refuses to start (OOMK) due to pg split
- Tor Martin Ølberg wrote:
> After an upgrade to 15.2.13 from 15.2.4 my small home lab cluster ran into issues with OS...
07/09/2021
- 10:32 PM Bug #45761: mon_thrasher: "Error ENXIO: mon unavailable" during sync_force command leads to "fail...
- reducing priority based on https://tracker.ceph.com/issues/45761#note-26
- 10:19 PM Bug #50659 (New): Segmentation fault under Pacific 16.2.1 when using a custom crush location hook
- 01:04 PM Support #51609 (New): OSD refuses to start (OOMK) due to pg split
- After an upgrade to 15.2.13 from 15.2.4 my small home lab cluster ran into issues with OSDs failing on all four hosts...
- 11:35 AM Backport #51605 (Resolved): pacific: bufferlist::splice() may cause stack corruption in bufferlis...
- https://github.com/ceph/ceph/pull/42976
- 11:35 AM Backport #51604 (Resolved): octopus: bufferlist::splice() may cause stack corruption in bufferlis...
- https://github.com/ceph/ceph/pull/42975
- 11:32 AM Bug #51419 (Pending Backport): bufferlist::splice() may cause stack corruption in bufferlist::reb...
- 09:18 AM Backport #51603 (In Progress): pacific: qa/standalone: Add missing teardowns at the end of a subs...
- 08:25 AM Backport #51603 (Resolved): pacific: qa/standalone: Add missing teardowns at the end of a subset ...
- https://github.com/ceph/ceph/pull/42258
- 08:24 AM Fix #51580 (Pending Backport): qa/standalone: Add missing teardowns at the end of a subset of osd...
07/08/2021
- 11:48 PM Backport #50748 (In Progress): pacific: max_misplaced was replaced by target_max_misplaced_ratio
- https://github.com/ceph/ceph/pull/42250
- 09:46 PM Bug #50346 (Resolved): OSD crash FAILED ceph_assert(!is_scrubbing())
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 09:36 PM Backport #51453 (Resolved): pacific: Add simultaneous scrubs to rados/thrash
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/42120
m... - 09:33 PM Backport #50791 (Resolved): pacific: osd: write_trunc omitted to clear data digest
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/42019
m... - 06:37 PM Backport #51316 (Duplicate): nautilus: osd:scrub skip some pg
- 05:16 PM Bug #51581: scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- Looks like an issue with the test that was added in d6eb3e3a3c29a02d6c7c088ef7c8c668a872d16e. Ronen, can you please t...
- 04:53 PM Bug #51581: scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- /a/sage-2021-06-12_13:06:29-rados-master-distro-basic-smithi/6168272
- 02:43 PM Bug #50659: Segmentation fault under Pacific 16.2.1 when using a custom crush location hook
- I just wanted to note that I see the status is listed as "Need More Info", but I think I have provided everything I h...
- 05:58 AM Backport #51583 (In Progress): nautilus: osd does not proactively remove leftover PGs
- 01:11 AM Backport #51583 (Resolved): nautilus: osd does not proactively remove leftover PGs
- https://github.com/ceph/ceph/pull/42240
- 05:48 AM Backport #51582 (In Progress): octopus: osd does not proactively remove leftover PGs
- 01:11 AM Backport #51582 (Resolved): octopus: osd does not proactively remove leftover PGs
- https://github.com/ceph/ceph/pull/42239
- 05:47 AM Backport #51584 (In Progress): pacific: osd does not proactively remove leftover PGs
- 01:11 AM Backport #51584 (Resolved): pacific: osd does not proactively remove leftover PGs
- https://github.com/ceph/ceph/pull/42238
- 05:39 AM Fix #51580 (Fix Under Review): qa/standalone: Add missing teardowns at the end of a subset of osd...
- 01:06 AM Bug #38931 (Pending Backport): osd does not proactively remove leftover PGs
- 12:27 AM Bug #51000: LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- Ok, I'll take a look.
07/07/2021
- 10:24 PM Bug #51581 (Resolved): scrub/osd-scrub-repair.sh: TEST_auto_repair_bluestore_tag failed
- ...
- 10:18 PM Bug #50245: TEST_recovery_scrub_2: Not enough recovery started simultaneously
- /a/sseshasa-2021-07-07_19:22:19-rados:standalone-master-distro-basic-smithi/6258022
- 10:16 PM Bug #49961: scrub/osd-recovery-scrub.sh: TEST_recovery_scrub_1 failed
- /a/sseshasa-2021-07-07_19:22:19-rados:standalone-master-distro-basic-smithi/6258018
/a/sseshasa-2021-07-14_10:37:09-... - 09:49 PM Fix #51580 (Resolved): qa/standalone: Add missing teardowns at the end of a subset of osd and scr...
- A subset of osd and scrub standalone tests are not properly cleaning up after
completion.
# osd/osd-force-cre... - 05:44 PM Backport #51556 (In Progress): pacific: mon: return -EINVAL when handling unknown option in 'ceph...
- 08:20 AM Backport #51556 (Resolved): pacific: mon: return -EINVAL when handling unknown option in 'ceph os...
- https://github.com/ceph/ceph/pull/42229
- 05:17 PM Backport #51568 (In Progress): pacific: pool last_epoch_clean floor is stuck after pg merging
- 01:26 PM Backport #51568 (Resolved): pacific: pool last_epoch_clean floor is stuck after pg merging
- https://github.com/ceph/ceph/pull/42224
- 05:13 PM Backport #51498 (In Progress): pacific: mgr spamming with repeated set pgp_num_actual while merging
- 05:11 PM Backport #51371 (Resolved): pacific: OSD crash FAILED ceph_assert(!is_scrubbing())
- https://github.com/ceph/ceph/pull/41944
- 03:53 PM Bug #51000: LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- Myoungwon Oh, can you please help take a look at this.
- 03:47 PM Bug #51576 (New): qa/tasks/radosbench.py times out
- ...
- 02:56 PM Backport #51549 (In Progress): pacific: cephadm bootstrap on arm64 fails to start ceph/ceph-grafa...
- https://github.com/ceph/ceph/pull/42211
- 07:58 AM Backport #51549 (Resolved): pacific: cephadm bootstrap on arm64 fails to start ceph/ceph-grafana ...
- https://github.com/ceph/ceph/pull/42211
- 02:36 PM Backport #51570 (In Progress): pacific: CommandCrashedError: Command crashed: 'mkdir -p -- /home/...
- https://github.com/ceph/ceph/pull/42221
- 02:35 PM Backport #51570 (Resolved): pacific: CommandCrashedError: Command crashed: 'mkdir -p -- /home/ubu...
- https://github.com/ceph/ceph/pull/42221
- 02:30 PM Bug #50393 (Pending Backport): CommandCrashedError: Command crashed: 'mkdir -p -- /home/ubuntu/ce...
- ...
- 01:26 PM Backport #51569 (Resolved): octopus: pool last_epoch_clean floor is stuck after pg merging
- https://github.com/ceph/ceph/pull/42837
- 01:23 PM Bug #48212 (Pending Backport): poollast_epoch_clean floor is stuck after pg merging
- 11:02 AM Bug #50441 (Resolved): cephadm bootstrap on arm64 fails to start ceph/ceph-grafana service
- 10:16 AM Bug #50441: cephadm bootstrap on arm64 fails to start ceph/ceph-grafana service
- moved temp to RADOS so that we can use backport scripts
- 10:15 AM Bug #50441 (Pending Backport): cephadm bootstrap on arm64 fails to start ceph/ceph-grafana service
- 09:33 AM Bug #50441 (Resolved): cephadm bootstrap on arm64 fails to start ceph/ceph-grafana service
- 09:36 AM Bug #42884: OSDMapTest.CleanPGUpmaps failure
- https://jenkins.ceph.com/job/ceph-pull-requests/78813/consoleFull#-4535647526733401c-e9d0-4737-9832-6594c5da0afa
- 08:30 AM Bug #45457 (Pending Backport): CEPH Graylog Logging Missing "host" Field
- 08:20 AM Backport #51555 (Resolved): octopus: mon: return -EINVAL when handling unknown option in 'ceph os...
- https://github.com/ceph/ceph/pull/43266
- 08:03 AM Backport #51553 (Resolved): pacific: rebuild-mondb hangs
- https://github.com/ceph/ceph/pull/42411
- 08:02 AM Backport #51552 (Resolved): octopus: rebuild-mondb hangs
- https://github.com/ceph/ceph/pull/43263
- 07:59 AM Backport #51551 (Rejected): octopus: cephadm bootstrap on arm64 fails to start ceph/ceph-grafana ...
07/06/2021
- 05:38 PM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...
- /a/yuriw-2021-07-02_17:35:44-rados-pacific-distro-basic-smithi/6249971
- 05:35 PM Bug #38357: ClsLock.TestExclusiveEphemeralStealEphemeral failed
- /a/yuriw-2021-07-02_17:35:44-rados-pacific-distro-basic-smithi/6250131
- 05:32 PM Bug #51000: LibRadosTwoPoolsPP.ManifestSnapRefcount failure
- similar failure seen in pacific...
- 09:27 AM Bug #23565 (Fix Under Review): Inactive PGs don't seem to cause HEALTH_ERR
07/05/2021
- 01:27 PM Bug #46847: Loss of placement information on OSD reboot
- Last week we had a power outage affecting all OSD machines in a 14.2.20 cluster. A small percentage of PGs didn't act...
- 01:18 PM Bug #51527 (Resolved): Ceph osd crashed due to segfault
- Hi everyone,
We have 9 osd nodes with 12 deamons for each node.
Ceph used for s3 objects and rbd images.
ceph ... - 12:06 PM Bug #48965: qa/standalone/osd/osd-force-create-pg.sh: TEST_reuse_id: return 1
- /a/sseshasa-2021-07-05_10:18:42-rados:standalone-wip-test-stdalone-mclk-1-distro-basic-smithi/6253062
- 11:49 AM Bug #45761 (Need More Info): mon_thrasher: "Error ENXIO: mon unavailable" during sync_force comma...
- Stopped Reproducing, please reopen if you hit another instance
- 11:47 AM Bug #48609 (Closed): osd/PGLog: don’t fast-forward can_rollback_to during merge_log if the log is...
- Root cause resolved
- 11:46 AM Backport #51522 (Resolved): pacific: osd: Delay sending info to new backfill peer resetting last_...
- https://github.com/ceph/ceph/pull/41136
- 11:35 AM Backport #51522 (Resolved): pacific: osd: Delay sending info to new backfill peer resetting last_...
- 11:45 AM Backport #51523: octopus: osd: Delay sending info to new backfill peer resetting last_backfill un...
- https://github.com/ceph/ceph/pull/40593/
- 11:35 AM Backport #51523 (Resolved): octopus: osd: Delay sending info to new backfill peer resetting last_...
- 11:36 AM Backport #51525 (Rejected): octopus: osd: Delay sending info to new backfill peer resetting last_...
- 11:35 AM Bug #48611: osd: Delay sending info to new backfill peer resetting last_backfill until backfill a...
- since nautilus has reached EOL removed it
- 11:34 AM Bug #48611 (Pending Backport): osd: Delay sending info to new backfill peer resetting last_backfi...
- 04:19 AM Bug #45457 (Fix Under Review): CEPH Graylog Logging Missing "host" Field
07/03/2021
- 06:16 AM Bug #51338: osd/scrub_machine.cc: FAILED ceph_assert(state_cast<const NotActive*>())
- Another OSD crash after scrub assert bug, log attached. corrupted rocskdb.
07/02/2021
- 06:55 PM Bug #50866: osd: stat mismatch on objects
- /ceph/teuthology-archive/pdonnell-2021-07-02_10:08:50-fs-wip-pdonnell-testing-20210701.192056-distro-basic-smithi/624...
- 05:00 PM Backport #51498 (Resolved): pacific: mgr spamming with repeated set pgp_num_actual while merging
- https://github.com/ceph/ceph/pull/42223
- 05:00 PM Backport #51497 (Rejected): nautilus: mgr spamming with repeated set pgp_num_actual while merging
- https://github.com/ceph/ceph/pull/43218
- 05:00 PM Backport #51496 (Resolved): octopus: mgr spamming with repeated set pgp_num_actual while merging
- https://github.com/ceph/ceph/pull/42420
- 04:59 PM Bug #51433 (Pending Backport): mgr spamming with repeated set pgp_num_actual while merging
07/01/2021
- 09:39 PM Bug #51307: LibRadosWatchNotify.Watch2Delete fails
- Seems very similar to https://tracker.ceph.com/issues/50042#note-2
- 09:05 PM Bug #48417 (Duplicate): unfound EC objects in sepia's LRC after upgrade
- 06:57 PM Backport #51453: pacific: Add simultaneous scrubs to rados/thrash
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42120
merged - 04:42 PM Bug #48212 (Fix Under Review): poollast_epoch_clean floor is stuck after pg merging
- 02:13 PM Bug #48212: poollast_epoch_clean floor is stuck after pg merging
- Dan van der Ster wrote:
> I suspect the cause is that there's a leftover epoch value for the now-deleted PG in `epoc... - 01:28 PM Bug #48212: poollast_epoch_clean floor is stuck after pg merging
- I suspect the cause is that there's a leftover epoch value for the now-deleted PG in `epoch_by_pg` in `void LastEpoch...
- 04:38 PM Bug #38931 (Fix Under Review): osd does not proactively remove leftover PGs
- Our customer reported a similar case, providing an easy way to reproduce the issue: if when purging a pg the osd is m...
- 11:03 AM Fix #51464 (Fix Under Review): osd: Add mechanism to avoid running osd benchmark on osd init when...
- 08:54 AM Fix #51464 (Resolved): osd: Add mechanism to avoid running osd benchmark on osd init when using m...
- The current behavior is to let the osd benchmark run on each osd
init, which is not necessary. If the underlying dev... - 07:44 AM Bug #51463 (Resolved): blocked requests while stopping/starting OSDs
- Hi,
we run into a lot of slow requests. (IO blocked for several seconds) while stopping or starting one or more OS... - 07:13 AM Bug #51419: bufferlist::splice() may cause stack corruption in bufferlist::rebuild_aligned_size_a...
- Initially triggered with fio when testing rbd persistent write-back cache in ssd mode:...
06/30/2021
- 09:50 PM Bug #49894 (In Progress): set a non-zero default value for osd_client_message_cap
- Neha Ojha wrote:
> Neha Ojha wrote:
> > The current default of 0 doesn't help and we've tried setting it to 5000 fo... - 06:53 PM Backport #50790: octopus: osd: write_trunc omitted to clear data digest
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41620
merged - 06:50 PM Backport #50791: pacific: osd: write_trunc omitted to clear data digest
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/42019
merged - 06:49 PM Bug #50346: OSD crash FAILED ceph_assert(!is_scrubbing())
- https://github.com/ceph/ceph/pull/41944 merged
- 06:44 PM Bug #51457 (New): qa/standalone/scrub/osd-scrub-test.sh: TEST_interval_changes: date check failed
- ...
- 04:44 PM Backport #51453 (In Progress): pacific: Add simultaneous scrubs to rados/thrash
- 04:30 PM Backport #51453 (Resolved): pacific: Add simultaneous scrubs to rados/thrash
- https://github.com/ceph/ceph/pull/42120
- 04:39 PM Bug #45868: rados_api_tests: LibRadosWatchNotify.AioWatchNotify2 fails
- /a/yuriw-2021-06-29_19:12:08-rados-wip-yuri2-testing-2021-06-28-0858-pacific-distro-basic-smithi/6243653
- 04:35 PM Bug #51454 (New): Simultaneous OSD's crash with tp_osd_tp on rocksdb::MergingIterator::Next()
- Ceph v14.2.15
Main use case is RGW.
Bucket indexes on SSD OSDs.
Majority of SSD OSD under bucket intexes are FileS... - 04:30 PM Backport #51452 (Resolved): octopus: Add simultaneous scrubs to rados/thrash
- https://github.com/ceph/ceph/pull/42422
- 04:28 PM Bug #51451 (Resolved): Add simultaneous scrubs to rados/thrash
- Motivated by https://tracker.ceph.com/issues/50346.
- 09:18 AM Bug #51419 (Fix Under Review): bufferlist::splice() may cause stack corruption in bufferlist::reb...
- 06:55 AM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...
- http://qa-proxy.ceph.com/teuthology/ideepika-2021-06-30_04:28:06-rados-wip-yuri7-testing-2021-06-28-1224-octopus-dist...
06/29/2021
- 09:16 PM Bug #51433 (Fix Under Review): mgr spamming with repeated set pgp_num_actual while merging
- 08:33 PM Bug #51433 (Resolved): mgr spamming with repeated set pgp_num_actual while merging
- While merging PGs our osdmaps are churning through ~2000 epochs per hour.
The osdmap diffs are empty:... - 08:03 PM Bug #49525: found snap mapper error on pg 3.2s1 oid 3:4abe9991:::smithi10121515-14:e4 snaps missi...
- The sequence looks a lit different this time.
/a/rfriedma-2021-06-26_19:32:15-rados-wip-ronenf-scrubs-config-distr... - 05:03 PM Bug #39150: mon: "FAILED ceph_assert(session_map.sessions.empty())" when out of quorum
- /a/yuriw-2021-06-28_17:32:48-rados-wip-yuri2-testing-2021-06-28-0858-pacific-distro-basic-smithi/6239590
- 04:59 PM Bug #45721: CommandFailedError: Command failed (workunit test rados/test_python.sh) FAIL: test_ra...
- /a/yuriw-2021-06-28_17:32:48-rados-wip-yuri2-testing-2021-06-28-0858-pacific-distro-basic-smithi/6239575
- 08:13 AM Bug #48613 (Resolved): Reproduce https://tracker.ceph.com/issues/48417
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:12 AM Bug #49139 (Resolved): rados/perf: cosbench workloads hang forever
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:11 AM Bug #49988 (Resolved): Global Recovery Event never completes
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:11 AM Bug #50230 (Resolved): mon: spawn loop after mon reinstalled
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:10 AM Bug #50466 (Resolved): _delete_some additional unexpected onode list
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:10 AM Bug #50477 (Resolved): mon/MonClient: reset authenticate_err in _reopen_session()
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:08 AM Bug #50964 (Resolved): mon: slow ops due to osd_failure
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:03 AM Backport #51237 (Resolved): nautilus: rebuild-mondb hangs
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41874
m... - 08:03 AM Bug #50245: TEST_recovery_scrub_2: Not enough recovery started simultaneously
- /a//kchai-2021-06-27_13:33:07-rados-wip-kefu-testing-2021-06-27-1907-distro-basic-smithi/6238237
- 08:00 AM Backport #50987 (Resolved): octopus: unaligned access to member variables of crush_work_bucket
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41622
m... - 08:00 AM Backport #50796 (Resolved): octopus: mon: spawn loop after mon reinstalled
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41621
m... - 07:56 AM Backport #51269 (Resolved): octopus: rados/perf: cosbench workloads hang forever
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41922
m... - 07:56 AM Backport #50990: octopus: mon: slow ops due to osd_failure
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41618
m... - 07:53 AM Backport #50705 (Resolved): octopus: _delete_some additional unexpected onode list
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41623
m... - 07:53 AM Backport #50152 (Resolved): octopus: Reproduce https://tracker.ceph.com/issues/48417
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41609
m... - 07:52 AM Backport #50750 (Resolved): octopus: max_misplaced was replaced by target_max_misplaced_ratio
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41624
m... - 07:38 AM Backport #51313 (Resolved): pacific: osd:scrub skip some pg
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41971
m... - 07:37 AM Backport #50505 (Resolved): pacific: mon/MonClient: reset authenticate_err in _reopen_session()
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41019
m... - 07:37 AM Backport #50986 (Resolved): pacific: unaligned access to member variables of crush_work_bucket
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41983
m... - 07:36 AM Backport #50989 (Resolved): pacific: mon: slow ops due to osd_failure
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41982
m... - 07:32 AM Backport #50797 (Resolved): pacific: mon: spawn loop after mon reinstalled
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41768
m... - 07:28 AM Backport #51215: pacific: Global Recovery Event never completes
- Nathan, would you be so kind as to add a link to this issue in https://github.com/ceph/ceph/pull/41872 ?
- 07:27 AM Backport #51215 (Resolved): pacific: Global Recovery Event never completes
- 07:19 AM Backport #50706 (Resolved): pacific: _delete_some additional unexpected onode list
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/41680
m... - 05:40 AM Bug #51419 (Resolved): bufferlist::splice() may cause stack corruption in bufferlist::rebuild_ali...
- *** stack smashing detected ***: terminated2073 IOPS][eta 02h:59m:36s]
--Type <RET> for more, q to quit, c to contin...
06/28/2021
- 07:29 PM Backport #50987: octopus: unaligned access to member variables of crush_work_bucket
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41622
merged - 07:29 PM Backport #50796: octopus: mon: spawn loop after mon reinstalled
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41621
merged - 04:25 PM Backport #50505: pacific: mon/MonClient: reset authenticate_err in _reopen_session()
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41019
mergedReviewed-by: Kefu Chai <kchai@redhat.com> - 06:54 AM Bug #50089: mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reducing number of monitors i...
- I see a similar crash on quincy, suspect its seen when I try to add mons from 1 to 3 .
/]# ceph crash info 2021-06...
06/26/2021
- 02:27 PM Backport #50986: pacific: unaligned access to member variables of crush_work_bucket
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41983
merged - 02:26 PM Backport #50989: pacific: mon: slow ops due to osd_failure
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/41982
merged
06/25/2021
- 09:43 PM Bug #51101: rados/test_envlibrados_for_rocksdb.sh: cmake: symbol lookup error: cmake: undefined s...
- /a/yuriw-2021-06-24_16:54:31-rados-wip-yuri-testing-2021-06-24-0708-pacific-distro-basic-smithi/6190738
- 03:50 PM Backport #51371 (Resolved): pacific: OSD crash FAILED ceph_assert(!is_scrubbing())
- https://github.com/ceph/ceph/pull/41944
- 03:48 PM Bug #50346 (Pending Backport): OSD crash FAILED ceph_assert(!is_scrubbing())
- 06:48 AM Backport #50990 (Resolved): octopus: mon: slow ops due to osd_failure
06/24/2021
Also available in: Atom