Project

General

Profile

Activity

From 11/22/2018 to 12/21/2018

12/21/2018

11:08 PM Bug #37652 (Resolved): bluestore: "fsck warning: legacy statfs record found, suggest to run store...
Igor Fedotov
07:59 PM Bug #37733 (Fix Under Review): os/bluestore: fixup access a destroy cond cause deadlock or undefi...
https://github.com/ceph/ceph/pull/25659 Sage Weil
06:55 AM Bug #37733 (Resolved): os/bluestore: fixup access a destroy cond cause deadlock or undefine behav...
1. osd has been mark down because of on heartbeat
2. gdb attach, found thread hung by __lock_lock_wait...
bing lin
03:13 PM Bug #36455 (Fix Under Review): BlueStore: ENODATA not fully handled
Radoslaw Zarzynski
03:12 PM Bug #36455: BlueStore: ENODATA not fully handled
https://github.com/ceph/ceph/pull/25670 Radoslaw Zarzynski

12/20/2018

03:50 PM Bug #24639: [segfault] segfault in BlueFS::read
I don't have the osd and disk anymore and can confirm that the disk itself was in a terrible state (many hardware rel... Compile Nix
03:34 PM Bug #24639 (Need More Info): [segfault] segfault in BlueFS::read
can you do ceph-bluestore-tool fsck --path ... --log-file log --log-level 20 and attach the output? Sage Weil
03:40 PM Bug #22534 (Resolved): Debian's bluestore *rocksdb* does not support neither fast CRC nor compres...
Sage Weil
03:38 PM Bug #23165 (Resolved): OSD used for Metadata / MDS storage constantly entering heartbeat timeout
This sounds like the cephfs directories weren't well fragmented, leading to very large omap objects. Sage Weil
03:37 PM Bug #23206 (Need More Info): ceph-osd daemon crashes - *** Caught signal (Aborted) **
Sage Weil
03:37 PM Bug #23372 (Can't reproduce): osd: segfault
Sage Weil
03:36 PM Bug #23819 (Won't Fix): how to make compactions smooth
the compaction is a function of rocksdb and there isn't a lot to be done about it at the moment... Sage Weil
03:35 PM Bug #24561 (Fix Under Review): if disableWAL is set, submit_transacton_sync will met error.
Sage Weil
03:34 PM Bug #23390 (Resolved): Identifying NVMe via PCI serial isn't sufficient (Bluestore/SPDK)
fixed in master. now the spdk backend is using PCI device's selector instead. see https://github.com/ceph/ceph/pull/2... Kefu Chai
03:32 PM Bug #24901 (Need More Info): Client reads fail due to bad CRC under high memory pressure on OSDs
Sage Weil
03:30 PM Bug #24906 (Need More Info): fio with bluestore crushed
Sage Weil
03:30 PM Bug #24906: fio with bluestore crushed
Is this still broken? Sage Weil
03:26 PM Bug #36108 (Duplicate): Assertion due to ENOENT result on clonerange2
Sage Weil

12/18/2018

11:15 PM Bug #37652 (Fix Under Review): bluestore: "fsck warning: legacy statfs record found, suggest to r...
Patrick Donnelly
10:33 PM Bug #37652: bluestore: "fsck warning: legacy statfs record found, suggest to run store repair to ...
Reassigning to Igor, sorry. Patrick Donnelly

12/17/2018

04:42 PM Bug #22534: Debian's bluestore *rocksdb* does not support neither fast CRC nor compression
Bumping up RocksDB in Luminous: https://github.com/ceph/ceph/pull/25592. Radoslaw Zarzynski

12/13/2018

07:41 PM Bug #37652 (Resolved): bluestore: "fsck warning: legacy statfs record found, suggest to run store...
... Patrick Donnelly

12/07/2018

09:02 AM Backport #37565 (Need More Info): luminous: OSD compression: incorrect display of the used disk s...
Kefu writes in the parent issue:
https://github.com/ceph/ceph/pull/19454
i am not sure if we are able to backpo...
Nathan Cutler
09:01 AM Backport #37565 (Rejected): luminous: OSD compression: incorrect display of the used disk space
Nathan Cutler
09:01 AM Backport #37564 (Rejected): mimic: OSD compression: incorrect display of the used disk space
Nathan Cutler
02:58 AM Bug #20870 (Pending Backport): OSD compression: incorrect display of the used disk space
https://github.com/ceph/ceph/pull/19454
i am not sure if we are able to backport PR 19454 to luminous, as it looks...
Kefu Chai
02:36 AM Feature #22159 (Resolved): allow tracking of bluestore compression ratio by pool
Kefu Chai

12/05/2018

03:51 PM Bug #22534: Debian's bluestore *rocksdb* does not support neither fast CRC nor compression
fix on rocksdb side:
- https://github.com/ceph/rocksdb/pull/41
Kefu Chai

12/04/2018

12:10 PM Bug #20236 (Can't reproduce): bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTe...
Igor Fedotov
12:09 PM Bug #20236: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure
Sage Weil wrote:
> I haven't seen this in a while.. have you?
Me too. Just gave this another try for both master ...
Igor Fedotov

12/03/2018

11:32 PM Backport #37495 (In Progress): luminous: bluefs-bdev-expand aborts
Igor Fedotov
11:17 PM Backport #37495: luminous: bluefs-bdev-expand aborts
https://github.com/ceph/ceph/pull/25384 Igor Fedotov
08:36 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
Igor Fedotov wrote:
> May be benchmark this drive using FIO?
> And try to simulate the use pattern: mixed read + w...
Gavin Baker
08:35 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
Igor Fedotov wrote:
> BTW - do these drives/controllers have write caching enabled? May be try to disable if so? AFA...
Gavin Baker

12/01/2018

06:38 AM Backport #37494 (In Progress): mimic: bluefs-bdev-expand aborts
Nathan Cutler
06:37 AM Backport #37494 (Resolved): mimic: bluefs-bdev-expand aborts
https://github.com/ceph/ceph/pull/25348 Nathan Cutler
06:37 AM Backport #37495 (Resolved): luminous: bluefs-bdev-expand aborts
https://github.com/ceph/ceph/pull/25384 Nathan Cutler
06:37 AM Bug #37360 (Pending Backport): bluefs-bdev-expand aborts
Nathan Cutler

11/30/2018

06:49 PM Bug #37360: bluefs-bdev-expand aborts
mimic fix (which is completely different from Nautilus one as we don't backport main device expansion feature): https... Igor Fedotov
01:30 PM Bug #20236: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure
I haven't seen this in a while.. have you? Sage Weil
01:29 PM Bug #26896 (Can't reproduce): store_test.cc: FAILED ObjectStore/StoreTest.Rename/2
Sage Weil

11/29/2018

08:22 PM Bug #23463 (Can't reproduce): src/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
Sage Weil
08:21 PM Bug #25006 (Can't reproduce): bad csum during upgrade test
http://pulpito.ceph.com/sage-2018-11-29_15:08:26-upgrade:luminous-x-mimic-distro-basic-smithi/
Sage Weil
07:30 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
BTW - do these drives/controllers have write caching enabled? May be try to disable if so? AFAIR there were some talk... Igor Fedotov
07:19 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
May be benchmark this drive using FIO?
And try to simulate the use pattern: mixed read + write + fdatasync.
Igor Fedotov
05:25 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
[2041833.966145] INFO: task bstore_kv_sync:79243 blocked for more than 120 seconds.
[2041833.966148] "echo 0 > /proc...
Gavin Baker
05:24 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
I also less frequently get these dmesg errors. Not sure if they are related.
[2041833.966150] bstore_kv_sync D ff...
Gavin Baker
05:03 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
No not SMR, these drives are Seagate Exos 10TB Enterprise sata drives. We are seeing this behavior on multiple types ... Gavin Baker
03:31 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
The code is just timing fdatasync(2), so the problem is almost certainly going to be below ceph (kernel or hardware)
...
Sage Weil
03:29 PM Bug #36364 (Need More Info): Bluestore OSD IO Hangs near Flush (flush in 90.330556)
This flush time is suspiciously close to 90s (flush in 90.330556)...
These aren't SMR drives, right?
Sage Weil
03:54 PM Bug #36268 (Fix Under Review): Unable to recover from ENOSPC in BlueFS
https://github.com/ceph/ceph/pull/25132 Igor Fedotov
03:45 PM Bug #23120 (Can't reproduce): OSDs continously crash during recovery
Sage Weil
03:44 PM Bug #25207 (Can't reproduce): ceph-volume lvm create gives segmentation fault
Sage Weil
03:38 PM Bug #36284 (Duplicate): Bluestore might be hanging OSD
Sage Weil
03:35 PM Bug #36303 (Duplicate): luminous: 12.2.8 - FAILED assert(0 == "put on missing extent (nothing bef...
Josh Durgin
03:34 PM Bug #36331: FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 (zeros)
... Sage Weil
03:26 PM Bug #36455: BlueStore: ENODATA not fully handled
Sage Weil
03:19 PM Bug #36567 (Duplicate): Segmentation fault in BlueStore::Blob::discard_unallocated
Sage Weil
03:18 PM Bug #37090 (Can't reproduce): BlueStore.cc: 3099: FAILED assert(0 == "uh oh, missing shared_blob")
I have a feeling this is caused by http://tracker.ceph.com/issues/36526, the fix for which is in 12.2.10.
Sage Weil
03:15 PM Bug #37282 (Need More Info): rocksdb: submit_transaction_sync error: Corruption: block checksum m...
Josh Durgin
02:58 PM Bug #37282: rocksdb: submit_transaction_sync error: Corruption: block checksum mismatch code = 2
Somewhat similar issue, may be useful as recovery guidance:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018...
Igor Fedotov
03:11 PM Bug #25001 (Can't reproduce): Crashing OSDs after going from 12.2.5 -> 12.2.6 -> 13.2.0
I believe this is related to the SharedBLob refcounting bugs. See 7031addfe6fcd070df8c4c7b175f374bda77a671 and ff883... Sage Weil
03:06 PM Bug #25050 (Need More Info): osd: OSD Failed to Start In function 'int BlueStore::_do_alloc_write
Josh Durgin
02:55 PM Bug #37360 (Fix Under Review): bluefs-bdev-expand aborts
Igor Fedotov
02:55 PM Bug #37360: bluefs-bdev-expand aborts
https://github.com/ceph/ceph/pull/25308 Igor Fedotov
09:11 AM Bug #32731 (Resolved): fsck: cid is improperly matched to oid
Nathan Cutler
09:11 AM Backport #36145 (Resolved): luminous: fsck: cid is improperly matched to oid
Nathan Cutler
01:07 AM Backport #36145: luminous: fsck: cid is improperly matched to oid
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/24705
merged
Yuri Weinstein
09:09 AM Backport #36638 (Resolved): luminous: rename does not old ref to replacement onode at old name
Nathan Cutler
01:04 AM Backport #36638: luminous: rename does not old ref to replacement onode at old name
Patrick Donnelly wrote:
> https://github.com/ceph/ceph/pull/24989
merged
Yuri Weinstein
06:14 AM Bug #24439 (Resolved): os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes >= b->length) ...
Nathan Cutler
06:14 AM Backport #26943 (Resolved): luminous: os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes...
Nathan Cutler
01:04 AM Backport #26943: luminous: os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes >= b->leng...
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/24992
merged
Yuri Weinstein
04:28 AM Backport #36639 (In Progress): mimic: rename does not old ref to replacement onode at old name
https://github.com/ceph/ceph/pull/25313 Prashant D
01:08 AM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
https://github.com/ceph/ceph/pull/24649 mergedhttps://github.com/ceph/ceph/pull/24649 Yuri Weinstein

11/26/2018

11:41 PM Backport #36754 (Resolved): mimic: _aio_log_start inflight overlap of 0x10000~1000 with [65536~4096]
Nathan Cutler
08:49 PM Backport #36754: mimic: _aio_log_start inflight overlap of 0x10000~1000 with [65536~4096]
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25062
merged
Yuri Weinstein

11/22/2018

11:25 AM Bug #37360: bluefs-bdev-expand aborts
Got it. Thanks, Mark!
So as I said before main device resize isn't supported at the moment.
Will probably start a...
Igor Fedotov
11:10 AM Bug #37360: bluefs-bdev-expand aborts
I decided to enlarge OSD backing store device to be able to store more data on this OSD without re-creating it.
Se...
Марк Коренберг
10:17 AM Bug #37360: bluefs-bdev-expand aborts
Actually there are 2 aspects for this ticket:
1) the tool improperly handles OSD deployments that lack DB and/or WAL...
Igor Fedotov
09:34 AM Bug #37360 (In Progress): bluefs-bdev-expand aborts
Igor Fedotov
09:04 AM Bug #37360: bluefs-bdev-expand aborts
Problem is still triggered every time. Марк Коренберг
09:04 AM Bug #37360: bluefs-bdev-expand aborts
... Марк Коренберг
09:03 AM Bug #37360: bluefs-bdev-expand aborts
... Марк Коренберг
08:46 AM Bug #37360: bluefs-bdev-expand aborts
Wondering if bluefs-bdev-sizes command works fine? What's about fsck? Igor Fedotov
 

Also available in: Atom