Project

General

Profile

Activity

From 10/15/2021 to 11/13/2021

11/13/2021

10:39 AM Bug #53261 (Duplicate): pacific: OMAP upgrade to PER-PG format result in skipped first key.
This is a regression introduced by fix to omap upgrade: https://github.com/ceph/ceph/pull/43687
The problem is that ...
Adam Kupczyk
10:36 AM Bug #53260 (Resolved): OMAP upgrade to PER-PG format result in skipped first key.
This is a regression introduced by fix to omap upgrade: https://github.com/ceph/ceph/pull/43687
The problem is that ...
Adam Kupczyk

11/11/2021

11:55 AM Backport #51764 (In Progress): octopus: Missed shared block repair doesn't fix related issues
https://github.com/ceph/ceph/pull/43887 Igor Fedotov
10:32 AM Backport #52767 (In Progress): octopus: bluestore repair might cause invalid write
https://github.com/ceph/ceph/pull/43885 Igor Fedotov
09:37 AM Backport #53195 (In Progress): pacific: fsck/repair uses invalid prefix when removing undecodable...
https://github.com/ceph/ceph/pull/43882 Igor Fedotov
09:36 AM Backport #53196 (In Progress): octopus: fsck/repair uses invalid prefix when removing undecodable...
https://github.com/ceph/ceph/pull/43883 Igor Fedotov
08:45 AM Backport #53100 (Resolved): octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_...
Igor Fedotov
08:44 AM Backport #53102 (Resolved): octopus: os/bluestore/AvlAllocator: specialize _block_picker() and c...
Igor Fedotov
08:43 AM Backport #53104 (Resolved): octopus: os/bluestore: Improve _block_picker function
Igor Fedotov
08:43 AM Fix #48272 (Resolved): osd: fix bluestore avl allocator
Igor Fedotov
08:43 AM Backport #48477 (Resolved): octopus: osd: fix bluestore avl allocator
Igor Fedotov

11/10/2021

11:35 PM Backport #53100: octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
Mauricio Oliveira wrote:
> https://github.com/ceph/ceph/pull/43747
merged
Yuri Weinstein
11:35 PM Backport #53102: octopus: os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
Mauricio Oliveira wrote:
> https://github.com/ceph/ceph/pull/43747
merged
Yuri Weinstein
11:35 PM Backport #53104: octopus: os/bluestore: Improve _block_picker function
Mauricio Oliveira wrote:
> https://github.com/ceph/ceph/pull/43747
merged
Yuri Weinstein
11:35 PM Backport #48477: octopus: osd: fix bluestore avl allocator
Mauricio Oliveira wrote:
> https://github.com/ceph/ceph/pull/43747
merged
Yuri Weinstein
11:48 AM Bug #53185 (Fix Under Review): FSCK removes allocation file when called in DEEP mode causing next...
Gabriel BenHanokh

11/09/2021

01:07 PM Bug #53184: failed to start new osd due to SIGSEGV in BlueStore::read()
> To check the relevant content in DB could you please run ceph-kvstore-tool bluestore-kv <path-to-osd> list C >out a... Satoru Takeuchi
11:03 AM Bug #51034 (Closed): osd: failed to initialize OSD in Rook
Presumably fixed by: https://github.com/ceph/ceph/pull/42424 Igor Fedotov
02:59 AM Bug #51034: osd: failed to initialize OSD in Rook
Please close this ticket, thank you very much for handling this problem! Satoru Takeuchi
02:58 AM Bug #51034: osd: failed to initialize OSD in Rook
I confirmed this problem disappeared in ceph v16.2.6 and rook v1.7.5 environment. However, another problem, which was... Satoru Takeuchi

11/08/2021

11:29 PM Bug #53002 (Fix Under Review): crash BlueStore::Onode::put from BlueStore::TransContext::~TransCo...
Igor Fedotov
11:29 PM Bug #53002 (Pending Backport): crash BlueStore::Onode::put from BlueStore::TransContext::~TransCo...
Igor Fedotov
11:26 PM Bug #49815 (Resolved): BlueRocksEnv::GetChildren may pass trailing slashes to BlueFS readdir
Igor Fedotov
06:00 PM Backport #53196 (Resolved): octopus: fsck/repair uses invalid prefix when removing undecodable Sh...
https://github.com/ceph/ceph/pull/43883 Backport Bot
06:00 PM Backport #53195 (Resolved): pacific: fsck/repair uses invalid prefix when removing undecodable Sh...
Backport Bot
05:57 PM Bug #53011 (Pending Backport): fsck/repair uses invalid prefix when removing undecodable Shared Blob
Neha Ojha
05:50 PM Bug #53011: fsck/repair uses invalid prefix when removing undecodable Shared Blob
https://github.com/ceph/ceph/pull/43621 merged Yuri Weinstein
11:18 AM Bug #53184: failed to start new osd due to SIGSEGV in BlueStore::read()
Looks like list of OSD's collections is empty. I don't know the root cause but I'm getting pretty the same effects on... Igor Fedotov
07:33 AM Bug #53184 (Closed): failed to start new osd due to SIGSEGV in BlueStore::read()
A new OSD failed to start due to SIGSEGV. Here is the backtrace.
```
debug -3> 2021-11-08T07:06:17.324+0000 7...
Satoru Takeuchi
11:02 AM Bug #53139: OSD might wrongly attempt to use "slow" device when single device is backing the store
A. Saber Shenouda wrote:
> In which case this bug can be reproduced exactly?
I failed to reproduce that locally h...
Igor Fedotov
09:47 AM Bug #53185 (Resolved): FSCK removes allocation file when called in DEEP mode causing next mount t...
FSCK removes allocation file when called in DEEP mode causing next mount to do unnecessary full recovery.
Gabriel BenHanokh

11/05/2021

12:59 PM Bug #53139: OSD might wrongly attempt to use "slow" device when single device is backing the store
In which case this bug can be reproduced exactly? A. Saber Shenouda
10:03 AM Bug #53139: OSD might wrongly attempt to use "slow" device when single device is backing the store
Adam Kupczyk wrote:
> It looks to me more like a problem with allocator code itself.
> Checking douts from _allocat...
Igor Fedotov
09:44 AM Bug #53139 (Fix Under Review): OSD might wrongly attempt to use "slow" device when single device ...
Igor Fedotov

11/04/2021

09:32 PM Bug #53139 (In Progress): OSD might wrongly attempt to use "slow" device when single device is ba...
Igor Fedotov
08:07 PM Bug #53062 (Resolved): OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Igor Fedotov
08:07 PM Backport #53124 (Resolved): pacific: OMAP upgrade to PER-PG format result in ill-formatted OMAP k...
Igor Fedotov
06:37 PM Backport #53124: pacific: OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/43793
merged
Yuri Weinstein
12:15 PM Bug #50788 (Duplicate): crash in BlueStore::Onode::put()
Igor Fedotov

11/03/2021

03:22 PM Bug #53139: OSD might wrongly attempt to use "slow" device when single device is backing the store
It looks to me more like a problem with allocator code itself.
Checking douts from _allocate() (or rather lack of th...
Adam Kupczyk
01:49 PM Bug #53139 (Resolved): OSD might wrongly attempt to use "slow" device when single device is backi...
This looks like a regression introduced by https://github.com/ceph/ceph/pull/42992
Providing RocksDB with an additio...
Igor Fedotov
02:47 PM Backport #53124 (In Progress): pacific: OMAP upgrade to PER-PG format result in ill-formatted OMA...
https://github.com/ceph/ceph/pull/43793 Igor Fedotov
01:52 PM Backport #53124: pacific: OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Neha Ojha wrote:
> Igor, can you please take care of the backport?
yeah, in progress atm
Igor Fedotov
01:50 PM Backport #53124: pacific: OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Igor, can you please take care of the backport? Neha Ojha

11/02/2021

09:50 PM Backport #51648: nautilus: Bluestore repair might erroneously remove SharedBlob entries.
This update was made using the script "backport-resolve-issue".
backport PR https://github.com/ceph/ceph/pull/43365
m...
Loïc Dachary
03:54 PM Bug #53129 (Resolved): BlueFS truncate() and poweroff can create corrupted files
It is possible to create condition in which a BlueFS contains file that is corrupted.
It can happen when BlueFS repl...
Adam Kupczyk
12:12 PM Bug #53002 (In Progress): crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
Igor Fedotov

11/01/2021

07:15 PM Backport #53124 (Resolved): pacific: OMAP upgrade to PER-PG format result in ill-formatted OMAP k...
https://github.com/ceph/ceph/pull/43793 Backport Bot
07:13 PM Bug #53062 (Pending Backport): OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Neha Ojha
01:54 PM Backport #52934 (In Progress): pacific: os/bluestore: _do_write_small fix head_pad
https://github.com/ceph/ceph/pull/43756 Igor Fedotov
01:53 PM Backport #52935 (In Progress): octopus: os/bluestore: _do_write_small fix head_pad
https://github.com/ceph/ceph/pull/43757 Igor Fedotov
01:45 PM Backport #48477 (In Progress): octopus: osd: fix bluestore avl allocator
Igor Fedotov
01:44 PM Backport #53104 (In Progress): octopus: os/bluestore: Improve _block_picker function
Igor Fedotov
01:44 PM Backport #53102 (In Progress): octopus: os/bluestore/AvlAllocator: specialize _block_picker() an...
Igor Fedotov

10/29/2021

11:58 PM Backport #53100: octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
https://github.com/ceph/ceph/pull/43747 Mauricio Oliveira
10:51 PM Backport #53100 (In Progress): octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_...
Neha Ojha
10:36 PM Backport #53100: octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/43747
ceph-backport.sh versi...
Mauricio Oliveira
08:25 PM Backport #53100 (Resolved): octopus: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_...
https://github.com/ceph/ceph/pull/43747 Backport Bot
11:57 PM Backport #53102: octopus: os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
https://github.com/ceph/ceph/pull/43747 Mauricio Oliveira
08:25 PM Backport #53102 (Resolved): octopus: os/bluestore/AvlAllocator: specialize _block_picker() and c...
https://github.com/ceph/ceph/pull/43747 Backport Bot
11:57 PM Backport #53104: octopus: os/bluestore: Improve _block_picker function
https://github.com/ceph/ceph/pull/43747 Mauricio Oliveira
08:25 PM Backport #53104 (Resolved): octopus: os/bluestore: Improve _block_picker function
https://github.com/ceph/ceph/pull/43747 Backport Bot
11:57 PM Backport #48477: octopus: osd: fix bluestore avl allocator
https://github.com/ceph/ceph/pull/43747 Mauricio Oliveira
10:17 PM Backport #48477 (New): octopus: osd: fix bluestore avl allocator
Neha Ojha
03:01 PM Backport #48477: octopus: osd: fix bluestore avl allocator
Please set:
- Status to New
Per Igor's feedback in https://tracker.ceph.com/issues/52804#note-16
Mauricio Oliveira
11:19 PM Fix #48272 (Pending Backport): osd: fix bluestore avl allocator
Igor Fedotov
03:01 PM Fix #48272: osd: fix bluestore avl allocator
Please set:
- Backport to Octopus
- Status to Pending Backport
Per Igor's feedback in https://tracker.ceph.com/i...
Mauricio Oliveira
11:16 PM Backport #53105 (In Progress): pacific: os/bluestore: Improve _block_picker function
Igor Fedotov
09:21 PM Backport #53105: pacific: os/bluestore: Improve _block_picker function
https://github.com/ceph/ceph/pull/43745 Mauricio Oliveira
08:25 PM Backport #53105 (Resolved): pacific: os/bluestore: Improve _block_picker function
https://github.com/ceph/ceph/pull/43745 Backport Bot
11:14 PM Backport #53103 (In Progress): pacific: os/bluestore/AvlAllocator: specialize _block_picker() an...
Igor Fedotov
09:21 PM Backport #53103: pacific: os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
https://github.com/ceph/ceph/pull/43745 Mauricio Oliveira
08:25 PM Backport #53103 (Resolved): pacific: os/bluestore/AvlAllocator: specialize _block_picker() and c...
https://github.com/ceph/ceph/pull/43745 Backport Bot
10:38 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Octopus backport PR: https://github.com/ceph/ceph/pull/43747 Mauricio Oliveira
09:59 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Attaching chart with tail latency improvements (Octopus) Mauricio Oliveira
09:58 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Backported PRs:
https://github.com/ceph/ceph/pull/38148 (Octopus-only)
https://github.com/ceph/ceph/pull/41398 (O...
Mauricio Oliveira
09:18 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Pacific backport PR: https://github.com/ceph/ceph/pull/43745
Attaching chart with tail latency improvements.
Mauricio Oliveira
01:00 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Igor Fedotov wrote:
> If possible it would be great to have Pacific backports ASAP as new minor release is coming in...
Mauricio Oliveira
12:57 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Mauricio Oliveira wrote:
> @Igor
>
> Right, the key point is to backport the patches to Pacific/Octopus. I'm work...
Igor Fedotov
09:12 PM Backport #53101 (In Progress): pacific: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_...
Neha Ojha
09:09 PM Backport #53101: pacific: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
please link this Backport tracker issue with GitHub PR https://github.com/ceph/ceph/pull/43745
ceph-backport.sh versi...
Mauricio Oliveira
08:25 PM Backport #53101 (Resolved): pacific: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_...
https://github.com/ceph/ceph/pull/43745 Backport Bot
08:21 PM Bug #53085 (Pending Backport): os/bluestore: Improve _block_picker function
Neha Ojha
02:51 PM Bug #53085: os/bluestore: Improve _block_picker function
Master PR is merged.
Please set Status to Pending Backport.
Mauricio Oliveira
02:50 PM Bug #53085 (Resolved): os/bluestore: Improve _block_picker function
master tracker issue for https://github.com/ceph/ceph/pull/41398 Mauricio Oliveira
08:21 PM Bug #53086 (Pending Backport): os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
Neha Ojha
02:54 PM Bug #53086: os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
Master PR is merged.
Please set Status to Pending Backport.
Mauricio Oliveira
02:54 PM Bug #53086 (Resolved): os/bluestore/AvlAllocator: specialize _block_picker() and cleanups
Master tracker issue for PR https://github.com/ceph/ceph/pull/41825 Mauricio Oliveira
08:20 PM Bug #53087 (Pending Backport): os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* ...
Neha Ojha
02:58 PM Bug #53087: os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
Master PR is merged.
Please set Status to Pending Backport.
Mauricio Oliveira
02:58 PM Bug #53087 (Resolved): os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options
Master tracker issue for PR https://github.com/ceph/ceph/pull/41615 Mauricio Oliveira
12:52 PM Backport #51763 (In Progress): pacific: Missed shared block repair doesn't fix related issues
https://github.com/ceph/ceph/pull/43731 Igor Fedotov
12:51 PM Backport #52768 (In Progress): pacific: bluestore repair might cause invalid write
https://github.com/ceph/ceph/pull/43731 Igor Fedotov

10/28/2021

08:13 PM Bug #52399 (Resolved): src/os/bluestore/HybridAllocator.cc: FAILED ceph_assert(false)
Sage Weil

10/27/2021

08:10 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
@Igor
Right, the key point is to backport the patches to Pacific/Octopus. I'm working on it, if that is OK w/ you....
Mauricio Oliveira
12:12 PM Bug #53064: pgmeta onode isn't tagged with FLAG_PGMETA_OMAP if created in pre-mimic are
Sorry, that's https://tracker.ceph.com/issues/53062 which is revealed by the issue, not https://tracker.ceph.com/issu... Igor Fedotov
11:58 AM Bug #53064 (New): pgmeta onode isn't tagged with FLAG_PGMETA_OMAP if created in pre-mimic are
Legacy pgmeta onodes might have the flag unset after upgrading from Luminous (and before) to recent Ceph releases. Wh... Igor Fedotov
11:42 AM Bug #53062 (Fix Under Review): OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Igor Fedotov
11:09 AM Bug #53062 (Resolved): OMAP upgrade to PER-PG format result in ill-formatted OMAP keys.
Looks like the code appends the full legacy key to the new prefix rather that use the user-provided OMAP name from th... Igor Fedotov

10/26/2021

11:29 AM Bug #44924 (Fix Under Review): High memory usage in fsck/repair
Igor Fedotov

10/25/2021

03:53 PM Bug #52399: src/os/bluestore/HybridAllocator.cc: FAILED ceph_assert(false)
We need both of these:
https://github.com/ceph/ceph/pull/43645
https://github.com/ceph/ceph/pull/43583
Neha Ojha

10/22/2021

12:13 AM Bug #51619 (Resolved): Bluestore repair might erroneously remove SharedBlob entries.
Igor Fedotov

10/21/2021

11:58 PM Backport #51648 (Resolved): nautilus: Bluestore repair might erroneously remove SharedBlob entries.
Igor Fedotov
10:36 PM Backport #51648: nautilus: Bluestore repair might erroneously remove SharedBlob entries.
Igor Fedotov wrote:
> https://github.com/ceph/ceph/pull/43365
merged
Yuri Weinstein
07:29 PM Bug #53011 (Fix Under Review): fsck/repair uses invalid prefix when removing undecodable Shared Blob
Igor Fedotov
03:38 PM Bug #53011 (Resolved): fsck/repair uses invalid prefix when removing undecodable Shared Blob
Igor Fedotov
12:16 PM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
In frame 7 I can print the Onode. Some of the vals look quite strange (but I don't know if that's normal):... Dan van der Ster
09:45 AM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
More context: the cluster was upgraded from 14.2.20 to 15.2.14 two weeks ago. We've never seen this before today; it ... Dan van der Ster
09:43 AM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
Igor Fedotov wrote:
> Dan van der Ster wrote:
> > We've just seen this crash in the wild running 15.2.14. Maybe a d...
Dan van der Ster
09:39 AM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
Dan van der Ster wrote:
> We've just seen this crash in the wild running 15.2.14. Maybe a dup of #50788?
I'm pret...
Igor Fedotov
08:43 AM Bug #53002 (Duplicate): crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
We've just seen this crash in the wild running 15.2.14. Maybe a dup of #50788?... Dan van der Ster

10/20/2021

02:54 PM Bug #52399 (Fix Under Review): src/os/bluestore/HybridAllocator.cc: FAILED ceph_assert(false)
Igor Fedotov
02:51 PM Bug #52399: src/os/bluestore/HybridAllocator.cc: FAILED ceph_assert(false)
PR is ready Gabriel BenHanokh

10/19/2021

08:54 PM Bug #22066: bluestore osd asserts repeatedly with ceph-12.2.1/src/include/buffer.h: 882: FAILED a...
Eric Nelson wrote:
> No problem, for the time being some of these have been converted to bluestore osds with wal/db ...
Ist Gab
01:39 PM Bug #52939: lockdep cycle under BlueFS::_compact_log_async_LD_NF_D()
steps to reproduce:... Casey Bodley
01:34 PM Bug #52939: lockdep cycle under BlueFS::_compact_log_async_LD_NF_D()
i tested with the fix from https://github.com/ceph/ceph/pull/43589 (commit 4b23ecfa2967d1df37562c67df028b5ce1700afb),... Casey Bodley
12:48 PM Bug #52939 (Fix Under Review): lockdep cycle under BlueFS::_compact_log_async_LD_NF_D()
Adam Kupczyk

10/18/2021

11:11 AM Bug #52804 (Triaged): pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Igor Fedotov
10:50 AM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Additionally it would be interesting to learn allocations which produce that "latency tail". Would you add some print... Igor Fedotov
10:46 AM Bug #52804 (New): pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
@Mauricio - great findings, thanks a lot!
So the major point is that we need to backport all the mentioned patches ...
Igor Fedotov

10/15/2021

10:38 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Part 4)
Fake unit test:
@ ceph.git/src/test/objectstore/Allocator_test.cc...
Mauricio Oliveira
10:34 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Part 3)
With these commits for the AVL allocator backported to v15.2.12 (at least one is already in the latest 15....
Mauricio Oliveira
10:33 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Part 2)
This does _not_ seem to be a regression from this commit introduced in v15.2.9:
c25def8 octopus: os/blues...
Mauricio Oliveira
10:33 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Part 1)
The numbers for the Bitmap and AVL allocators show a long tail on AVL only:
- average of 3 runs
- bitmap...
Mauricio Oliveira
10:32 PM Bug #52804: pacific: Hybrid Allocator exhibits high tail latency for writes in Octopus
Good progress this week!
We could reproduce the issue with the allocator's state dump and the allocation requests ...
Mauricio Oliveira
 

Also available in: Atom