Activity
From 11/21/2018 to 12/20/2018
12/20/2018
- 03:50 PM Bug #24639: [segfault] segfault in BlueFS::read
- I don't have the osd and disk anymore and can confirm that the disk itself was in a terrible state (many hardware rel...
- 03:34 PM Bug #24639 (Need More Info): [segfault] segfault in BlueFS::read
- can you do ceph-bluestore-tool fsck --path ... --log-file log --log-level 20 and attach the output?
- 03:40 PM Bug #22534 (Resolved): Debian's bluestore *rocksdb* does not support neither fast CRC nor compres...
- 03:38 PM Bug #23165 (Resolved): OSD used for Metadata / MDS storage constantly entering heartbeat timeout
- This sounds like the cephfs directories weren't well fragmented, leading to very large omap objects.
- 03:37 PM Bug #23206 (Need More Info): ceph-osd daemon crashes - *** Caught signal (Aborted) **
- 03:37 PM Bug #23372 (Can't reproduce): osd: segfault
- 03:36 PM Bug #23819 (Won't Fix): how to make compactions smooth
- the compaction is a function of rocksdb and there isn't a lot to be done about it at the moment...
- 03:35 PM Bug #24561 (Fix Under Review): if disableWAL is set, submit_transacton_sync will met error.
- 03:34 PM Bug #23390 (Resolved): Identifying NVMe via PCI serial isn't sufficient (Bluestore/SPDK)
- fixed in master. now the spdk backend is using PCI device's selector instead. see https://github.com/ceph/ceph/pull/2...
- 03:32 PM Bug #24901 (Need More Info): Client reads fail due to bad CRC under high memory pressure on OSDs
- 03:30 PM Bug #24906 (Need More Info): fio with bluestore crushed
- 03:30 PM Bug #24906: fio with bluestore crushed
- Is this still broken?
- 03:26 PM Bug #36108 (Duplicate): Assertion due to ENOENT result on clonerange2
12/18/2018
- 11:15 PM Bug #37652 (Fix Under Review): bluestore: "fsck warning: legacy statfs record found, suggest to r...
- 10:33 PM Bug #37652: bluestore: "fsck warning: legacy statfs record found, suggest to run store repair to ...
- Reassigning to Igor, sorry.
12/17/2018
- 04:42 PM Bug #22534: Debian's bluestore *rocksdb* does not support neither fast CRC nor compression
- Bumping up RocksDB in Luminous: https://github.com/ceph/ceph/pull/25592.
12/13/2018
- 07:41 PM Bug #37652 (Resolved): bluestore: "fsck warning: legacy statfs record found, suggest to run store...
- ...
12/07/2018
- 09:02 AM Backport #37565 (Need More Info): luminous: OSD compression: incorrect display of the used disk s...
- Kefu writes in the parent issue:
https://github.com/ceph/ceph/pull/19454
i am not sure if we are able to backpo... - 09:01 AM Backport #37565 (Rejected): luminous: OSD compression: incorrect display of the used disk space
- 09:01 AM Backport #37564 (Rejected): mimic: OSD compression: incorrect display of the used disk space
- 02:58 AM Bug #20870 (Pending Backport): OSD compression: incorrect display of the used disk space
- https://github.com/ceph/ceph/pull/19454
i am not sure if we are able to backport PR 19454 to luminous, as it looks... - 02:36 AM Feature #22159 (Resolved): allow tracking of bluestore compression ratio by pool
12/05/2018
- 03:51 PM Bug #22534: Debian's bluestore *rocksdb* does not support neither fast CRC nor compression
- fix on rocksdb side:
- https://github.com/ceph/rocksdb/pull/41
12/04/2018
- 12:10 PM Bug #20236 (Can't reproduce): bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTe...
- 12:09 PM Bug #20236: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure
- Sage Weil wrote:
> I haven't seen this in a while.. have you?
Me too. Just gave this another try for both master ...
12/03/2018
- 11:32 PM Backport #37495 (In Progress): luminous: bluefs-bdev-expand aborts
- 11:17 PM Backport #37495: luminous: bluefs-bdev-expand aborts
- https://github.com/ceph/ceph/pull/25384
- 08:36 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- Igor Fedotov wrote:
> May be benchmark this drive using FIO?
> And try to simulate the use pattern: mixed read + w... - 08:35 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- Igor Fedotov wrote:
> BTW - do these drives/controllers have write caching enabled? May be try to disable if so? AFA...
12/01/2018
- 06:38 AM Backport #37494 (In Progress): mimic: bluefs-bdev-expand aborts
- 06:37 AM Backport #37494 (Resolved): mimic: bluefs-bdev-expand aborts
- https://github.com/ceph/ceph/pull/25348
- 06:37 AM Backport #37495 (Resolved): luminous: bluefs-bdev-expand aborts
- https://github.com/ceph/ceph/pull/25384
- 06:37 AM Bug #37360 (Pending Backport): bluefs-bdev-expand aborts
11/30/2018
- 06:49 PM Bug #37360: bluefs-bdev-expand aborts
- mimic fix (which is completely different from Nautilus one as we don't backport main device expansion feature): https...
- 01:30 PM Bug #20236: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure
- I haven't seen this in a while.. have you?
- 01:29 PM Bug #26896 (Can't reproduce): store_test.cc: FAILED ObjectStore/StoreTest.Rename/2
11/29/2018
- 08:22 PM Bug #23463 (Can't reproduce): src/os/bluestore/StupidAllocator.cc: 336: FAILED assert(rm.empty())
- 08:21 PM Bug #25006 (Can't reproduce): bad csum during upgrade test
- http://pulpito.ceph.com/sage-2018-11-29_15:08:26-upgrade:luminous-x-mimic-distro-basic-smithi/
- 07:30 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- BTW - do these drives/controllers have write caching enabled? May be try to disable if so? AFAIR there were some talk...
- 07:19 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- May be benchmark this drive using FIO?
And try to simulate the use pattern: mixed read + write + fdatasync.
- 05:25 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- [2041833.966145] INFO: task bstore_kv_sync:79243 blocked for more than 120 seconds.
[2041833.966148] "echo 0 > /proc... - 05:24 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- I also less frequently get these dmesg errors. Not sure if they are related.
[2041833.966150] bstore_kv_sync D ff... - 05:03 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- No not SMR, these drives are Seagate Exos 10TB Enterprise sata drives. We are seeing this behavior on multiple types ...
- 03:31 PM Bug #36364: Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- The code is just timing fdatasync(2), so the problem is almost certainly going to be below ceph (kernel or hardware)
... - 03:29 PM Bug #36364 (Need More Info): Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- This flush time is suspiciously close to 90s (flush in 90.330556)...
These aren't SMR drives, right? - 03:54 PM Bug #36268 (Fix Under Review): Unable to recover from ENOSPC in BlueFS
- https://github.com/ceph/ceph/pull/25132
- 03:45 PM Bug #23120 (Can't reproduce): OSDs continously crash during recovery
- 03:44 PM Bug #25207 (Can't reproduce): ceph-volume lvm create gives segmentation fault
- 03:38 PM Bug #36284 (Duplicate): Bluestore might be hanging OSD
- 03:35 PM Bug #36303 (Duplicate): luminous: 12.2.8 - FAILED assert(0 == "put on missing extent (nothing bef...
- 03:34 PM Bug #36331: FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 (zeros)
- ...
- 03:26 PM Bug #36455: BlueStore: ENODATA not fully handled
- 03:19 PM Bug #36567 (Duplicate): Segmentation fault in BlueStore::Blob::discard_unallocated
- 03:18 PM Bug #37090 (Can't reproduce): BlueStore.cc: 3099: FAILED assert(0 == "uh oh, missing shared_blob")
- I have a feeling this is caused by http://tracker.ceph.com/issues/36526, the fix for which is in 12.2.10.
- 03:15 PM Bug #37282 (Need More Info): rocksdb: submit_transaction_sync error: Corruption: block checksum m...
- 02:58 PM Bug #37282: rocksdb: submit_transaction_sync error: Corruption: block checksum mismatch code = 2
- Somewhat similar issue, may be useful as recovery guidance:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018... - 03:11 PM Bug #25001 (Can't reproduce): Crashing OSDs after going from 12.2.5 -> 12.2.6 -> 13.2.0
- I believe this is related to the SharedBLob refcounting bugs. See 7031addfe6fcd070df8c4c7b175f374bda77a671 and ff883...
- 03:06 PM Bug #25050 (Need More Info): osd: OSD Failed to Start In function 'int BlueStore::_do_alloc_write
- 02:55 PM Bug #37360 (Fix Under Review): bluefs-bdev-expand aborts
- 02:55 PM Bug #37360: bluefs-bdev-expand aborts
- https://github.com/ceph/ceph/pull/25308
- 09:11 AM Bug #32731 (Resolved): fsck: cid is improperly matched to oid
- 09:11 AM Backport #36145 (Resolved): luminous: fsck: cid is improperly matched to oid
- 01:07 AM Backport #36145: luminous: fsck: cid is improperly matched to oid
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/24705
merged - 09:09 AM Backport #36638 (Resolved): luminous: rename does not old ref to replacement onode at old name
- 01:04 AM Backport #36638: luminous: rename does not old ref to replacement onode at old name
- Patrick Donnelly wrote:
> https://github.com/ceph/ceph/pull/24989
merged - 06:14 AM Bug #24439 (Resolved): os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes >= b->length) ...
- 06:14 AM Backport #26943 (Resolved): luminous: os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes...
- 01:04 AM Backport #26943: luminous: os/bluestore/BlueStore.cc: 1025: FAILED assert(buffer_bytes >= b->leng...
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/24992
merged - 04:28 AM Backport #36639 (In Progress): mimic: rename does not old ref to replacement onode at old name
- https://github.com/ceph/ceph/pull/25313
- 01:08 AM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- https://github.com/ceph/ceph/pull/24649 mergedhttps://github.com/ceph/ceph/pull/24649
11/26/2018
- 11:41 PM Backport #36754 (Resolved): mimic: _aio_log_start inflight overlap of 0x10000~1000 with [65536~4096]
- 08:49 PM Backport #36754: mimic: _aio_log_start inflight overlap of 0x10000~1000 with [65536~4096]
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25062
merged
11/22/2018
- 11:25 AM Bug #37360: bluefs-bdev-expand aborts
- Got it. Thanks, Mark!
So as I said before main device resize isn't supported at the moment.
Will probably start a... - 11:10 AM Bug #37360: bluefs-bdev-expand aborts
- I decided to enlarge OSD backing store device to be able to store more data on this OSD without re-creating it.
Se... - 10:17 AM Bug #37360: bluefs-bdev-expand aborts
- Actually there are 2 aspects for this ticket:
1) the tool improperly handles OSD deployments that lack DB and/or WAL... - 09:34 AM Bug #37360 (In Progress): bluefs-bdev-expand aborts
- 09:04 AM Bug #37360: bluefs-bdev-expand aborts
- Problem is still triggered every time.
- 09:04 AM Bug #37360: bluefs-bdev-expand aborts
- ...
- 09:03 AM Bug #37360: bluefs-bdev-expand aborts
- ...
- 08:46 AM Bug #37360: bluefs-bdev-expand aborts
- Wondering if bluefs-bdev-sizes command works fine? What's about fsck?
11/21/2018
- 09:35 PM Bug #37360 (Resolved): bluefs-bdev-expand aborts
- root@node1:~# ceph-bluestore-tool bluefs-bdev-expand --path /var/lib/ceph/osd/ceph-16
infering bluefs devices from b...
Also available in: Atom