Activity
From 01/06/2021 to 02/04/2021
02/04/2021
- 06:43 PM Bug #45903 (Resolved): BlueFS replay log grows without end
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 04:36 PM Bug #49138: blk/kernel/KernelDevice.cc: void KernelDevice::_aio_thread() Unexpected IO error
- ...
- 03:16 PM Bug #49170: BlueFS might end-up with huge WAL files when upgrading OMAPs
- Here is a sample of such a huge (7+GB) WAL file:
0x0: op_file_update file(ino 431989 size 0x1cf0c7888 mtime 202... - 03:13 PM Bug #49170: BlueFS might end-up with huge WAL files when upgrading OMAPs
- And I believe we've heard about some more cases when BlueFS got corrupted after upgrade to Octopus....
- 03:08 PM Bug #49170: BlueFS might end-up with huge WAL files when upgrading OMAPs
- Huge WAL files aren't good by themselves but BlueFS has a bug in handling >4GB writes which presumably cause data cor...
- 03:06 PM Bug #49170 (Resolved): BlueFS might end-up with huge WAL files when upgrading OMAPs
- When performing OMAP on-disk format upgrade BlueStore's fsck might flood BlueFS with converting transactions which re...
- 02:42 PM Bug #49168: Bluefs improperly handles huge (>4GB) writes which causes data corruption
- The above looks like 32-bit value overflow and indeed BlueFS::_flush() uses FileWriter::get_buffer_length() which is ...
- 02:37 PM Bug #49168 (Resolved): Bluefs improperly handles huge (>4GB) writes which causes data corruption
- Here is the symptomatic log snippet, please note the length(9136e44b) in _flush() call and offset/length in subsequen...
- 09:56 AM Bug #49110: BlueFS.cc: 1542: FAILED assert(r == 0)
- My first try with ...
02/03/2021
- 06:31 PM Bug #49138: blk/kernel/KernelDevice.cc: void KernelDevice::_aio_thread() Unexpected IO error
- Neha Ojha wrote:
> I would have thought so too, but it is weird that I am seeing it on multiple smithi machines now.... - 06:28 PM Bug #49138: blk/kernel/KernelDevice.cc: void KernelDevice::_aio_thread() Unexpected IO error
- I would have thought so too, but it is weird that I am seeing it on multiple smithi machines now.
/a/yuriw-2021-01... - 05:56 PM Bug #49138: blk/kernel/KernelDevice.cc: void KernelDevice::_aio_thread() Unexpected IO error
- From my experience this highly likely means real H/W problems... Suggest to check with dmesg and/or smartctl..
- 04:54 PM Bug #49138 (New): blk/kernel/KernelDevice.cc: void KernelDevice::_aio_thread() Unexpected IO error
- ...
- 05:37 PM Backport #48281: octopus: osd: fix bluestore bitmap allocator
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38430
m... - 12:39 PM Backport #48281 (Resolved): octopus: osd: fix bluestore bitmap allocator
- 12:43 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Adam has submitted a PR which might be helpful in detecting transient read errors:
https://github.com/ceph/ceph/pu... - 12:09 PM Bug #40300: ceph-osd segfault: "rocksdb: Corruption: file is too short"
- Nautilus backport:
https://github.com/ceph/ceph/pull/39254
Octopus backport:
https://github.com/ceph/ceph/pull/3...
02/02/2021
- 09:40 PM Bug #49110: BlueFS.cc: 1542: FAILED assert(r == 0)
- Could you please set debug-bluefs to 20, start OSD again and share the relevant OSD log... Or at least last 20000 lin...
- 06:47 PM Bug #49110 (Won't Fix): BlueFS.cc: 1542: FAILED assert(r == 0)
- All the SSD based OSDs in my ceph cluster crashed.
The initial error was:... - 04:12 PM Backport #48281: octopus: osd: fix bluestore bitmap allocator
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/38430
merged - 03:24 PM Backport #49098 (In Progress): octopus: FAILED ceph_assert(o->pinned) in BlueStore::Collection::s...
- 11:11 AM Backport #49098 (Resolved): octopus: FAILED ceph_assert(o->pinned) in BlueStore::Collection::spli...
- https://github.com/ceph/ceph/pull/39230
- 03:24 PM Backport #49097: pacific: FAILED ceph_assert(o->pinned) in BlueStore::Collection::split_cache(Blu...
- https://github.com/ceph/ceph/pull/39228
- 03:22 PM Backport #49097 (In Progress): pacific: FAILED ceph_assert(o->pinned) in BlueStore::Collection::s...
- 11:11 AM Backport #49097 (Resolved): pacific: FAILED ceph_assert(o->pinned) in BlueStore::Collection::spli...
- https://github.com/ceph/ceph/pull/39228
- 03:23 PM Backport #49100 (In Progress): pacific: crash in BlueStore::Onode::put()
- https://github.com/ceph/ceph/pull/39228
- 11:12 AM Backport #49100 (Resolved): pacific: crash in BlueStore::Onode::put()
- https://github.com/ceph/ceph/pull/39228
- 03:22 PM Backport #49099 (In Progress): octopus: crash in BlueStore::Onode::put()
- https://github.com/ceph/ceph/pull/39230
- 11:12 AM Backport #49099 (Resolved): octopus: crash in BlueStore::Onode::put()
- https://github.com/ceph/ceph/pull/39230
- 11:12 AM Bug #48781 (Pending Backport): crash in BlueStore::Onode::put()
- 09:15 AM Bug #48781 (Fix Under Review): crash in BlueStore::Onode::put()
- @Tom - thanks a lot.
I presume the root cause for the bug is an improper (too early) nref decrement in Onode::put me... - 11:10 AM Bug #48966 (Pending Backport): FAILED ceph_assert(o->pinned) in BlueStore::Collection::split_cach...
- 10:33 AM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Igor Fedotov wrote:
> @Christian - thanks for the update. Could you please keep monitoring these counters on a per-d... - 10:20 AM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- @Christian - thanks for the update. Could you please keep monitoring these counters on a per-day basis for a while?
... - 09:57 AM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Igor Fedotov wrote:
> Hi @Christian,
> sorry for the long analysis. But again nothing very interesting...
Too ba...
02/01/2021
- 07:37 PM Backport #46194 (Resolved): nautilus: BlueFS replay log grows without end
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37948
m... - 05:26 PM Backport #46194: nautilus: BlueFS replay log grows without end
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/37948
merged - 12:03 PM Bug #48781: crash in BlueStore::Onode::put()
- Extra logs
- 11:59 AM Bug #48781: crash in BlueStore::Onode::put()
- Here you go (output from cephadm logs)
This crash is the first one now after 1 week. - 11:26 AM Bug #48781: crash in BlueStore::Onode::put()
- Tom Myny wrote:
> Here is a dump of our latest crash
@Tom, may I have additional 10000 lines of the log preceding... - 10:41 AM Bug #48781: crash in BlueStore::Onode::put()
- Here is a dump of our latest crash
01/29/2021
- 07:41 AM Bug #46780 (Triaged): BlueFS Spillover without db being full
- Seena, this fixed in 14.2.11, and default in 14.2.12
01/28/2021
- 03:27 PM Bug #48256 (Can't reproduce): Many4KWritesNoCSumTest fails on nautilus [ FAILED ] ObjectStore/S...
- 03:24 PM Bug #48218 (Can't reproduce): ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixCompressionAlgor...
- 02:43 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Christian Rohmann wrote:
> I was able to dump all of the output of the osds from journald now, properly timestamped ... - 01:11 PM Bug #48781 (Need More Info): crash in BlueStore::Onode::put()
- 01:10 PM Backport #49039 (Resolved): octopus: Cannot allocate memory appears when using io_uring osd
- https://github.com/ceph/ceph/pull/39899
- 01:10 PM Backport #49038 (Resolved): pacific: Cannot allocate memory appears when using io_uring osd
- https://github.com/ceph/ceph/pull/39898
- 01:09 PM Bug #47661 (Pending Backport): Cannot allocate memory appears when using io_uring osd
- 12:52 PM Bug #48776 (Resolved): ObjectStore/StoreTest hangs
- 12:51 PM Backport #48950 (Resolved): pacific: ObjectStore/StoreTest hangs
- https://github.com/ceph/ceph/pull/38989
01/27/2021
- 11:51 PM Bug #48776: ObjectStore/StoreTest hangs
- Neha Ojha wrote:
> pacific backport - https://github.com/ceph/ceph/pull/38989
merged - 08:11 PM Bug #20870 (Resolved): OSD compression: incorrect display of the used disk space
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:09 PM Bug #46411 (Rejected): mimic: Disks associated to osds have small write io even on an idle ceph c...
- mimic EOL
- 08:08 PM Bug #38150 (Resolved): KernelDevice exclusive lock broken
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:06 PM Feature #40704 (Resolved): BlueStore tool to check fragmentation
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:06 PM Bug #41188 (Resolved): incorrect RW_IO_MAX
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:05 PM Bug #41901 (Resolved): bluestore: unused calculation is broken
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:05 PM Bug #42091 (Resolved): bluefs: sync_metadata leaks dirty files if log_t is empty
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:01 PM Bug #45788 (Resolved): ObjectStore/StoreTestSpecificAUSize.ExcessiveFragmentation/2 failed
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:00 PM Bug #46552 (Resolved): Rescue procedure for extremely large bluefs log
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 08:00 PM Bug #47475 (Resolved): Compressed blobs lack checksums
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 07:58 PM Backport #43086 (Rejected): mimic: bluefs: sync_metadata leaks dirty files if log_t is empty
- mimic EOL
- 07:57 PM Backport #47895 (Rejected): mimic: Compressed blobs lack checksums
- 07:57 PM Backport #46192 (Rejected): mimic: BlueFS replay log grows without end
- 07:57 PM Backport #46713 (Rejected): mimic: Rescue procedure for extremely large bluefs log
- 07:57 PM Backport #46010 (Rejected): mimic: ObjectStore/StoreTestSpecificAUSize.ExcessiveFragmentation/2 f...
- 07:57 PM Backport #45062 (Rejected): mimic: bluestore: unused calculation is broken
- 07:57 PM Backport #41280 (Rejected): mimic: BlueStore tool to check fragmentation
- 07:57 PM Backport #41461 (Rejected): mimic: incorrect RW_IO_MAX
- 07:57 PM Backport #38161 (Rejected): mimic: KernelDevice exclusive lock broken
- 07:57 PM Backport #36641 (Rejected): mimic: Unable to recover from ENOSPC in BlueFS
- 07:57 PM Backport #37564 (Rejected): mimic: OSD compression: incorrect display of the used disk space
- 07:27 PM Backport #47893 (Rejected): luminous: Compressed blobs lack checksums
- luminous EOL
- 07:26 PM Backport #36640 (Rejected): luminous: Unable to recover from ENOSPC in BlueFS
- luminous EOL
- 07:26 PM Backport #38160 (Rejected): luminous: KernelDevice exclusive lock broken
- luminous EOL
- 07:25 PM Backport #41462 (Rejected): luminous: incorrect RW_IO_MAX
- luminous EOL
- 06:25 PM Bug #47551 (Resolved): Some structs aren't bound to mempools properly
- 06:25 PM Backport #47670 (Rejected): mimic: Some structs aren't bound to mempools properly
- mimic EOL
01/26/2021
- 10:24 AM Bug #47661: Cannot allocate memory appears when using io_uring osd
- Jiang Yu wrote:
> The kernel panic problem can be solved by upgrading to 5.4.0-49.
> But ceph osd will crash abnorm...
01/23/2021
- 05:48 PM Bug #48966 (Fix Under Review): FAILED ceph_assert(o->pinned) in BlueStore::Collection::split_cach...
01/22/2021
- 08:24 PM Bug #48966 (Resolved): FAILED ceph_assert(o->pinned) in BlueStore::Collection::split_cache(BlueSt...
- ...
01/21/2021
- 10:23 PM Bug #48036: bluefs corrupted in a OSD
- I've tried to reproduce this problem of multiple ceph-osds sharing the same device (with cephadm) and the second ceph...
- 11:20 AM Backport #48950 (Resolved): pacific: ObjectStore/StoreTest hangs
- https://github.com/ceph/ceph/pull/38989
01/20/2021
- 05:16 PM Bug #48776: ObjectStore/StoreTest hangs
- pacific backport - https://github.com/ceph/ceph/pull/38989
- 01:56 AM Bug #48776 (Pending Backport): ObjectStore/StoreTest hangs
- 08:12 AM Bug #48036 (Closed): bluefs corrupted in a OSD
- 01:32 AM Bug #48036: bluefs corrupted in a OSD
- This issue is fixed in Rook.
https://github.com/rook/rook/pull/6793
01/19/2021
- 06:35 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- I was able to dump all of the output of the osds from journald now, properly timestamped - see the attached file jour...
- 02:01 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- This doesn't have preceding rocksdb log output. Hence makes a little sense.
Sorry it looks like a dead end from ro... - 10:16 AM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- I attached stack traces in file crash_stacktraces.log.
Unfortunately since this was shipped from syslog -> elastic... - 02:09 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- William Law wrote:
> Hi Igor -
>
> Thank you for your help and two options; we will discuss.
>
> Given this, i...
01/18/2021
- 10:27 PM Backport #48478: octopus: bluefs _allocate failed to allocate bdev 1 and 2,cause ceph_assert(r == 0)
- Fabio Martins wrote:
> Has this been merged into Octopus?
Not yet, this is gonna to happen after merge to master - 09:38 PM Backport #48478: octopus: bluefs _allocate failed to allocate bdev 1 and 2,cause ceph_assert(r == 0)
- Has this been merged into Octopus?
- 04:30 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Hi Igor -
Thank you for your help and two options; we will discuss.
Given this, is there any way we could (rela... - 03:35 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- William Law wrote:
> Hi Igor – thanks for the clarification. We actually did that but for whatever reason, we upload... - 02:45 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Actual OSDs logs prior to the assertion are what I'd really like to see. The attached crash info isn't enough - it's ...
- 02:29 PM Backport #48479: nautilus: bluefs _allocate failed to allocate bdev 1 and 2,cause ceph_assert(r =...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38475
m...
01/17/2021
- 03:35 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- I did a...
01/16/2021
- 05:06 PM Bug #48776 (Fix Under Review): ObjectStore/StoreTest hangs
- 01:35 AM Bug #48776: ObjectStore/StoreTest hangs
- IMO this PR is a culprit: https://github.com/ceph/ceph/pull/30027
Will try to fix after weekend. - 12:22 AM Bug #48776: ObjectStore/StoreTest hangs
- Here's a test that's hung, logs in ubuntu@smithi013....
- 05:01 PM Bug #48876 (Duplicate): osd crash in bluestore code
- Despite different symptoms the root cause is pretty the same - osr locking regression caused by https://github.com/ce...
- 12:23 AM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- https://github.com/ceph/ceph/pull/38929 this will let us see this in teuthology logs at least
- 12:08 AM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- Igor, you can log on to ubuntu@smithi092 right now and check out osd.4.
The test isn't doing much special, it's th... - 12:05 AM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- This looks related to my recent PR introducing per-pg omap naming scheme: https://github.com/ceph/ceph/pull/38651
...
01/15/2021
- 11:58 PM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- Cause appears to be a bluestore issue - manually running the objectstore command with --log-to-stderr we can see it's...
- 10:47 PM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- This is happening due to an exception running ceph-objectstore-tool:...
- 11:40 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Christian Rohmann wrote:
> The fsck using
>
> [...]
>
> just returned
>
> [...]
>
>
> So this is of n... - 10:00 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- The fsck using...
- 03:00 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Clarification: All ODSs that crashed came back up, I manually (!) took a single one offline to do fsck. Until then th...
- 02:59 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
Igor Fedotov wrote:
> So at this point I need some clarification on what you'd like to do first:
> 1) Recover OSD...- 02:55 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Igor Fedotov wrote:
> Christian Rohmann wrote:
> > We offlined an OSD and ran an _fsck --deep_, which resulted in
... - 02:41 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- So at this point I need some clarification on what you'd like to do first:
1) Recover OSDs. It's still unclear wheth... - 02:28 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Christian Rohmann wrote:
> Could there be an issue with a recent RocksDB update as observed by the Gentoo folks: htt... - 02:20 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Christian Rohmann wrote:
> We offlined an OSD and ran an _fsck --deep_, which resulted in
>
> [...]
>
> error... - 05:55 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Hi Igor – thanks for the clarification. We actually did that but for whatever reason, we uploaded the wrong file, sor...
- 09:31 AM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Hi William,
this should be 0xedf022~21 indeed.
But extract-337-modified doesn't look correct in any case.
There ... - 02:54 PM Bug #48389 (Rejected): _do_read bdev-read failed
- 02:53 PM Bug #48781: crash in BlueStore::Onode::put()
- Could you please share yet another 10000 lines of log preceding ones from crash.zip?
- 01:22 PM Bug #48876: osd crash in bluestore code
- Unfortunately, I don't have the rest of the log after all. I'm OOTO for a few days, but should be back on Monday. I'l...
- 01:20 PM Bug #48876: osd crash in bluestore code
- @Jeff - would you please share yet another 10000 lines of log prior to the one you've already attached.
01/14/2021
- 11:11 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Hi Igor -
Thanks again for your kind and patient assistance. I think we got mixed up a little and hope you can he... - 03:12 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- William Law wrote:
> OK great! How do we do that? And we have at least 4 other OSDs this happened to; should it be ... - 02:13 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- OK great! How do we do that? And we have at least 4 other OSDs this happened to; should it be the same or how do we ...
- 01:00 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- So intermediate summary on the issue.
Bluefs log contains a transaction (seq = 3790994) with improperly ordered oper... - 05:48 PM Backport #48479 (Resolved): nautilus: bluefs _allocate failed to allocate bdev 1 and 2,cause ceph...
- 04:15 PM Backport #48479: nautilus: bluefs _allocate failed to allocate bdev 1 and 2,cause ceph_assert(r =...
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/38475
merged - 04:48 PM Bug #46800 (Duplicate): Octopus OSD died and fails to start with FAILED ceph_assert(is_valid_io(o...
- 04:45 PM Bug #48276 (Duplicate): OSD Crash with ceph_assert(is_valid_io(off, len))
- 04:07 PM Bug #47751 (Resolved): Hybrid allocator might segfault when fallback allocator is present
- While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are ...
- 03:43 PM Bug #48876 (Duplicate): osd crash in bluestore code
- OSD crash seen when doing some cephfs testing with some experimental MDS and client patches. Build was based on top o...
- 07:37 AM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- I ran the fsck with bluestore debug level 20 again, in case you might need more details. The whole file is about 18 G...
01/13/2021
- 05:08 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Could there be an issue with a recent RocksDB update as observed by the Gentoo folks: https://bugs.gentoo.org/764221 ...
- 02:48 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Igor Fedotov wrote:
> Wondering if you had experienced any recent OSD crashes prior to this failure?
Not recently...
01/12/2021
- 05:06 PM Bug #48729 (Triaged): Bluestore memory leak on srub operations
- It looks like high RAM usage is caused by improper onode cache trimming inside BlueStore. Which in turn might be caus...
- 10:55 AM Bug #48729: Bluestore memory leak on srub operations
- @Igor
here you are:
https://cf2.cloudferro.com:8080/swift/v1/AUTH_5b9ea421deb745bfb4dab930cebe153f/ceph-sharings/... - 02:02 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Thank you thank you. They are attached.
Best,
Will - 11:43 AM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- @Will - to make block.db extract just use:
dd if=block.db ibs=1 skip=15589376 count=32768 of=dump.out - 01:19 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- Wondering if you had experienced any recent OSD crashes prior to this failure?
You might also want to Check for HW... - 12:43 PM Bug #48849: BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- BTW, I looked through other reported issues and found https://tracker.ceph.com/issues/48002 or https://tracker.ceph.c...
- 12:41 PM Bug #48849 (Need More Info): BlueStore.cc: 11380: FAILED ceph_assert(r == 0)
- We experienced a few OSD crashes all with the same signature in the logs:
--- cut ---
2021-01-08 06:13:54.946 7f3... - 11:43 AM Backport #48194 (Resolved): octopus: bufferlist c_str() sometimes clears assignment to mempool
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38429
m... - 11:43 AM Backport #48094 (Resolved): octopus: Hybrid allocator might segfault when fallback allocator is p...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38428
m... - 11:41 AM Backport #48093: nautilus: Hybrid allocator might segfault when fallback allocator is present
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/38637
m... - 11:40 AM Backport #47672: nautilus: Hybrid allocator might cause duplicate admin socket command registrati...
- This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/37793
m... - 10:23 AM Bug #42928: ceph-bluestore-tool bluefs-bdev-new-db does not update lv tags
- to answer my question - head -n 2 /dev/vg/lv will give the block device uuid
- 09:44 AM Bug #42928: ceph-bluestore-tool bluefs-bdev-new-db does not update lv tags
- Any way to determine the correct DB->Block arrangement after they are lost? I have a host that has hit this bug and a...
- 01:19 AM Bug #48776: ObjectStore/StoreTest hangs
- ...
01/11/2021
- 09:19 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- HI Igor -
I feel like I did something wrong as hexdump returned nothing... My apologies we haven't slept much
@ro... - 08:33 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- @Will, would you please share the hex dump of block.db file starting offset 0xede000 length 0x8000.
Latest startup... - 05:00 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- Igor, thank you! It's attached.
Will - 04:24 PM Bug #48827: Ceph Bluestore OSDs fail to start on WAL corruption
- @William - would you please share OSD startup log with debug-bluefs set to 20?
- 04:03 PM Bug #48827 (Duplicate): Ceph Bluestore OSDs fail to start on WAL corruption
- Hi -
I posted a note to the Ceph user list also, but we've run into this bug and it unfortunately hit 5 OSDs at th... - 07:59 PM Bug #48729: Bluestore memory leak on srub operations
- Presuming mem utilization is still that high could you please temporary set debug_bluestore to 20 for the osd in ques...
- 10:25 AM Bug #48729: Bluestore memory leak on srub operations
- Unfortunately, That's not the case. After 4 days some of the osds took >10GB of ram.
In example:... - 07:55 PM Backport #48194: octopus: bufferlist c_str() sometimes clears assignment to mempool
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/38429
merged - 07:55 PM Backport #48094: octopus: Hybrid allocator might segfault when fallback allocator is present
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/38428
merged - 04:50 PM Bug #47443 (Resolved): Hybrid allocator might cause duplicate admin socket command registration.
- 04:49 PM Backport #47672 (Resolved): nautilus: Hybrid allocator might cause duplicate admin socket command...
- 04:43 PM Backport #47672: nautilus: Hybrid allocator might cause duplicate admin socket command registrati...
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/37793
merged - 04:48 PM Backport #48093 (Resolved): nautilus: Hybrid allocator might segfault when fallback allocator is ...
- 04:44 PM Backport #48093: nautilus: Hybrid allocator might segfault when fallback allocator is present
- Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/38637
merged - 04:13 AM Bug #48819 (New): fsck error: found stray (per-pg) omap data on omap_head
- /a/kchai-2021-01-10_13:20:22-rados-master-distro-basic-smithi/
01/08/2021
- 01:08 PM Bug #48781: crash in BlueStore::Onode::put()
- and on the last host:
Jan 7 07:34:17 ceph2 kernel: [107054.315343] tp_osd_tp[20519]: segfault at 0 ip 00007efd3db... - 01:04 PM Bug #48781: crash in BlueStore::Onode::put()
- On another system we see the following to:
Jan 7 10:02:32 ceph1 kernel: [114774.759038] tp_osd_tp[17449]: segfaul... - 01:02 PM Bug #48781: crash in BlueStore::Onode::put()
- We also see the following in our OS logs:
[119268.259883] tp_osd_tp[32332]: segfault at 0 ip 00007f8ccce40733 sp 0...
01/07/2021
- 09:34 PM Bug #48776: ObjectStore/StoreTest hangs
- ...
- 12:38 AM Bug #48776: ObjectStore/StoreTest hangs
- /a/teuthology-2021-01-05_07:01:02-rados-master-distro-basic-smithi/5755704
- 12:38 AM Bug #48776 (Resolved): ObjectStore/StoreTest hangs
- ...
- 02:45 PM Bug #48781: crash in BlueStore::Onode::put()
- Download file in attachment with extra logs
- 02:21 PM Bug #48781: crash in BlueStore::Onode::put()
- Here is some extra information regarding this problem:
{
"backtrace": [
"(()+0x12b20) [0x7f0afc7a8b2... - 09:26 AM Bug #48781 (Resolved): crash in BlueStore::Onode::put()
- Following the earlier issue reported in #48778, I now see frequent OSD crashes. I'm not sure both are related.
<pr...
01/06/2021
- 11:14 PM Bug #45765: BlueStore::_collection_list causes huge latency growth pg deletion
- No problem, and thanks for confirming!
- 11:12 PM Bug #45765: BlueStore::_collection_list causes huge latency growth pg deletion
- Joshua Baergen wrote:
> Interesting, thanks. Is that 14.2.17 change this one: https://tracker.ceph.com/issues/47044 ... - 11:10 PM Bug #45765: BlueStore::_collection_list causes huge latency growth pg deletion
- Interesting, thanks. Is that 14.2.17 change this one: https://tracker.ceph.com/issues/47044 ?
FWIW, what I'm seein... - 11:07 PM Bug #45765: BlueStore::_collection_list causes huge latency growth pg deletion
- Joshua Baergen wrote:
> Hey Dan/Eric, did either of you see a big increase in the number of writes hitting your disk... - 11:00 PM Bug #45765: BlueStore::_collection_list causes huge latency growth pg deletion
- Hey Dan/Eric, did either of you see a big increase in the number of writes hitting your disks when buffered mode was ...
Also available in: Atom