Activity
From 02/05/2019 to 03/06/2019
03/06/2019
- 09:55 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- So that's why write_big operations may be also deferred just like write_small's. OK, thank you very much, it's clear now
- 08:29 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- It's not deferring because at the layer that deferring happens, we're talking about blobs (not writes), and the blogs...
- 04:08 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- Forgot to mention, this was Ceph 14.1.0
- 09:25 AM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- I've just tried to set
[osd]
bluestore_prefer_deferred_size_hdd = 4194304
On a test HDD plugged in my laptop. ... - 06:54 PM Bug #38557: pkg dependency issues upgrading from 12.2.y to 14.x.y
- accidentally opened against bluestore. You may close it.
See https://tracker.ceph.com/issues/38612 instead.
03/05/2019
- 05:45 PM Backport #38587 (Resolved): mimic: OSD crashes in get_str_map while creating with ceph-volume
- https://github.com/ceph/ceph/pull/26810
- 05:45 PM Backport #38586 (Resolved): luminous: OSD crashes in get_str_map while creating with ceph-volume
- https://github.com/ceph/ceph/pull/26900
- 03:18 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- I've just verified deferred writes behavior for 4M writes using objectstore FIO plugin.
Indeed bluestore splits writ... - 08:10 AM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- Sage Weil wrote:
> > all writes of size 4MB with bluestore_prefer_deferred_size_hdd < 524288 go HDD directly. >= 524... - 11:46 AM Bug #38363: Failure in assert when calling: ceph-volume lvm prepare --bluestore --data /dev/sdg
- I finally found the extended debug log in /var/log/ceph/ceph-osd.0.log. I attached the log output file (44k) to this ...
03/04/2019
- 11:23 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
- > all writes of size 4MB with bluestore_prefer_deferred_size_hdd < 524288 go HDD directly. >= 524288 through SSD (I m...
- 05:55 PM Bug #38574 (Resolved): mimic: Unable to recover from ENOSPC in BlueFS
- This the same issue as https://tracker.ceph.com/issues/36268.
We have alternate fix for mimic, which will be backpor... - 03:36 PM Bug #38329 (Pending Backport): OSD crashes in get_str_map while creating with ceph-volume
- 03:23 PM Bug #36268 (Resolved): Unable to recover from ENOSPC in BlueFS
- Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735
03/03/2019
- 08:07 PM Bug #38559 (Resolved): 50-100% iops lost due to bluefs_preextend_wal_files = false
- Hi.
I was investigating why RocksDB performance is so bad considering random 4K iops. I was looking at strace and ... - 01:30 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- We have upgraded to 12.2.11. During reboots the following would pass by:
[16:20:59] @ bitrot: osd.17 [ERR] 7... - 11:55 AM Bug #38557 (Closed): pkg dependency issues upgrading from 12.2.y to 14.x.y
- Description of problem:
With respect to https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
03/02/2019
- 02:28 PM Bug #38554 (Duplicate): ObjectStore/StoreTestSpecificAUSize.TooManyBlobsTest/2 fail, Expected: (r...
- ...
03/01/2019
- 04:44 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
- FYI, I think I hit another case with this with this in the last two weeks.
A RGW only case where if you would list... - 03:27 PM Bug #36455 (Resolved): BlueStore: ENODATA not fully handled
- 03:27 PM Backport #37825 (Resolved): luminous: BlueStore: ENODATA not fully handled
- 03:10 PM Backport #36641 (New): mimic: Unable to recover from ENOSPC in BlueFS
- 03:10 PM Backport #36640 (New): luminous: Unable to recover from ENOSPC in BlueFS
- 03:09 PM Bug #36268 (Pending Backport): Unable to recover from ENOSPC in BlueFS
- Sage, did you mean to cancel the mimic and luminous backports when you changed the status to Resolved?
- 10:46 AM Bug #25077 (New): Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- 10:46 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- Stefan Kooman wrote:
> @Igor Fedotov:
>
> We are using ceph balancer to get PGs balanced accross the cluster. The... - 07:56 AM Bug #38363: Failure in assert when calling: ceph-volume lvm prepare --bluestore --data /dev/sdg
- I tried but the output of ceph-volume remains the same....
I added this to /etc/ceph/ceph.conf on my testing nod...
02/28/2019
- 07:30 PM Backport #37825: luminous: BlueStore: ENODATA not fully handled
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25855
merged - 05:15 PM Bug #37282: rocksdb: submit_transaction_sync error: Corruption: block checksum mismatch code = 2
- We're not sure how to proceed without being able to reprdocue the crash, and we have never seen this.
1. Would it... - 04:41 PM Bug #38329 (Fix Under Review): OSD crashes in get_str_map while creating with ceph-volume
- reproduce this and got a core.
I think the problem is an empty string passed to trim() in str_map.cc. Fix here: h... - 03:37 PM Bug #23206 (Rejected): ceph-osd daemon crashes - *** Caught signal (Aborted) **
- not enough info
- 03:35 PM Bug #24639 (Can't reproduce): [segfault] segfault in BlueFS::read
- sounds like a hardware problem then!
- 03:34 PM Bug #25098: Bluestore OSD failed to start with `bluefs_types.h: 54: FAILED assert(pos <= end)`
- Current status:
We want a more concrete source of truth for whether the db and/or wal partitions should exist--som... - 03:31 PM Bug #34526 (Duplicate): OSD crash in KernelDevice::direct_read_unaligned while scrubbing
- 09:55 AM Bug #34526: OSD crash in KernelDevice::direct_read_unaligned while scrubbing
- IMO this is BlueStore (or more precisely BlueFS and/or RocksDB) related.
And I think it's duplicate of #36482
O... - 03:30 PM Bug #36268 (Resolved): Unable to recover from ENOSPC in BlueFS
- 03:30 PM Bug #36331 (Need More Info): FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 ...
- this was an ubuntu 18.04 kernel. maybe this was the pread vs swap zeroed pages kerenl bug?
i think we need anothe... - 03:27 PM Bug #36364 (Can't reproduce): Bluestore OSD IO Hangs near Flush (flush in 90.330556)
- 03:27 PM Bug #38049 (Resolved): random osds failing in thread_name:bstore_kv_final
- 03:23 PM Bug #38250 (Need More Info): assert failure crash prevents ceph-osd from running
- Is the errno EIO in this case?
On read error we do crash and fail the OSD. There is generally no recovery path fo... - 03:18 PM Bug #38272 (In Progress): "no available blob id" assertion might occur
- 03:16 PM Bug #38363 (Need More Info): Failure in assert when calling: ceph-volume lvm prepare --bluestore ...
- Can you reproduce this with debug_bluestore=20, debug_bluefs=20, debug_bdev=20?
Thanks! - 03:14 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
- - it looks like implementing readahead in bluefs would help
- we think newer rocksdb does its own readahead
- 02:41 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
- We've got another occurrence for this issue too.
Omap listing for specific onode consistently takes ~2 mins while d... - 02:19 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
- I think this is the same issue:
https://marc.info/?l=ceph-devel&m=155134206210976&w=2 - 03:04 PM Bug #37914 (Can't reproduce): bluestore: segmentation fault
- no logs or core. hoping it was teh hypercombined bufferlist memory corruption issue
- 09:57 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- @Igor Fedotov:
We are using ceph balancer to get PGs balanced accross the cluster. The day after the crashes, the ...
02/27/2019
- 11:18 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- Check, I'll collect the needed information. Note, during the restarts of the storage servers the *same* OSDs crashed ...
- 07:09 AM Feature #38494: Bluestore: issue discards on everything non-discarded during deep-scrubs
- Included link is just related PR.
- 07:07 AM Feature #38494: Bluestore: issue discards on everything non-discarded during deep-scrubs
- text formatting of previous message is wrong. I did not want to stroke-out the text.
- 07:07 AM Feature #38494 (New): Bluestore: issue discards on everything non-discarded during deep-scrubs
- Yes, we have bdev_enable_discard and bdev_async_discard, but they are not documented.
Ubuntu issues ...
02/26/2019
- 09:25 PM Bug #38489 (Resolved): bluestore_prefer_deferred_size_hdd units are not clear
- I have done an experiment. I made a pool with one PG of size 1. Next I run this command:
rados bench -p qwe -b 4M ... - 02:43 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- Stefan, to make sure (at our best) this is exactly the same bug, could you please check PG states with ceph-objectsto...
02/22/2019
- 04:05 PM Bug #37733 (Resolved): os/bluestore: fixup access a destroy cond cause deadlock or undefine behav...
- 04:05 PM Backport #38142 (Resolved): luminous: os/bluestore: fixup access a destroy cond cause deadlock or...
- 03:32 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
- Added related-to link to #38144 where the GCC 9 FTBFS is being discussed. A patch has been proposed there, but it inc...
02/21/2019
- 09:52 PM Backport #38142: luminous: os/bluestore: fixup access a destroy cond cause deadlock or undefine b...
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26261
merged - 03:49 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
- (original reporter here)
I have following customisation in ceph.conf:... - 01:00 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
- We *think* we have hit this same issue "in the field" on a Luminous 12.2.8 cluster:
2019-02-20 18:42:45.261357 7fd...
02/20/2019
- 11:15 AM Bug #38395 (Fix Under Review): luminous: write following remove might access previous onode
- 10:35 AM Bug #38395: luminous: write following remove might access previous onode
- 10:25 AM Bug #38395 (Resolved): luminous: write following remove might access previous onode
- So the sequence is as follows:
T1:
remove A
T2:
touch A
write A
In Luminous there is a chance that A is rem...
02/18/2019
- 01:41 PM Bug #38363 (Need More Info): Failure in assert when calling: ceph-volume lvm prepare --bluestore ...
- I run Ubuntu 18.04 and and ceph version 13.2.4-1bionic from this repo: https://download.ceph.com/debian-mimic.
Whe...
02/16/2019
- 11:00 AM Backport #37990 (Resolved): mimic: Compression not working, and when applied OSD disks are failin...
02/15/2019
- 10:38 PM Bug #37839: Compression not working, and when applied OSD disks are failing randomly
- merged https://github.com/ceph/ceph/pull/26342
https://github.com/ceph/ceph/pull/26544
- 08:29 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
- - have any options been customized?
- what version is this? 14.0.1-2.fc30 is a random dev checkpoint commit from ma... - 08:12 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
- Changing back to the Ceph tracker, this is not a crash in ceph-volume or specific to ceph-volume that I can see
- 12:57 PM Bug #38329 (Resolved): OSD crashes in get_str_map while creating with ceph-volume
- see https://bugzilla.redhat.com/show_bug.cgi?id=1661583...
- 03:10 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
- We have not experienced any further crashes in over a week (compared to multiple crashes per hour before), so it look...
02/13/2019
- 12:39 PM Bug #38230 (Resolved): segv in onode lookup
- https://github.com/ceph/ceph/pull/26391
02/12/2019
- 03:35 PM Bug #38272: "no available blob id" assertion might occur
- onode dump shortly before the assertion:
2019-02-12 18:23:47.546 7fca6fab1b40 0 bluestore(bluestore.test_temp_dir) ... - 03:29 PM Bug #38272: "no available blob id" assertion might occur
- Backtrace from UT:
-1> 2019-02-12 18:23:48.346 7fca6fab1b40 -1 /home/if/ceph/src/os/bluestore/BlueStore.cc: In fun... - 03:26 PM Bug #38272: "no available blob id" assertion might occur
- Stack trace from the customer log:
2019-02-06 00:04:25.934977 7ff3e3bca700 -1 /home/abuild/rpmbuild/BUILD/ceph-12.2.... - 03:25 PM Bug #38272 (Resolved): "no available blob id" assertion might occur
- We observed that on-site but unfortunately OSD were removed and are unavailable for inspection.
However I managed to... - 03:01 PM Backport #38143 (Resolved): mimic: os/bluestore: fixup access a destroy cond cause deadlock or un...
- 12:00 AM Backport #38143: mimic: os/bluestore: fixup access a destroy cond cause deadlock or undefine beha...
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26260
merged - 02:53 PM Backport #38188 (In Progress): luminous: deep fsck fails on inspecting very large onodes
- 02:50 PM Backport #38187 (Resolved): mimic: deep fsck fails on inspecting very large onodes
02/11/2019
- 09:08 PM Backport #38187: mimic: deep fsck fails on inspecting very large onodes
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26291
merged - 04:34 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
- From the first log, this looks like #36541. My guess is the crashes you were seeing after were continued problems fr...
- 10:06 AM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- I still seem to be experiencing these errors, albeit at a much reduced rate since upgrading to 13.2.3. I could wake u...
02/09/2019
- 05:14 PM Bug #38250 (Rejected): assert failure crash prevents ceph-osd from running
- One of my OSDs keeps crashing shortly after startup, which is preventing it from joining the cluster. The core issue...
02/08/2019
- 03:40 PM Bug #38230: segv in onode lookup
- ...
02/07/2019
- 10:41 PM Bug #38230: segv in onode lookup
- i'm guessing this is the same heap corruption we've been seeing, but logging it anyway
- 10:40 PM Bug #38230 (Resolved): segv in onode lookup
- ...
02/06/2019
- 03:56 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
- After two days of running fine I had set the bluestore and bluefs log level back to default, so I dont know how helpf...
- 03:27 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
- Lawrence, would you share the log for current crashes please?
Existing failures with fsck are expected as the patc... - 01:24 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
- We have since patched our ceph with https://github.com/ceph/ceph/pull/24686 which fixes Issue #36541. Since then the ...
- 02:16 PM Bug #38176: Unable to recover from ENOSPC in BlueFS, WAL
- Fixed link to bug replication script.
https://drive.google.com/file/d/10Lvcf6_Lj2c2sydcfU170lbb-IQClvH- - 09:05 AM Bug #37360 (Resolved): bluefs-bdev-expand aborts
- 08:54 AM Backport #38188: luminous: deep fsck fails on inspecting very large onodes
- No need for that additional cherry-pick, just add new option using the method applicable for luminous
02/05/2019
- 11:56 PM Backport #38188 (Need More Info): luminous: deep fsck fails on inspecting very large onodes
- We need to cherry-pick additional commits to get this backport PR, Option::TYPE_SIZE and Option::FLAG_RUNTIME not de...
- 05:02 PM Backport #38188 (Resolved): luminous: deep fsck fails on inspecting very large onodes
- https://github.com/ceph/ceph/pull/26387
- 11:31 PM Backport #38187 (In Progress): mimic: deep fsck fails on inspecting very large onodes
- https://github.com/ceph/ceph/pull/26291
- 05:01 PM Backport #38187 (Resolved): mimic: deep fsck fails on inspecting very large onodes
- https://github.com/ceph/ceph/pull/26291
- 09:39 PM Backport #37494 (Resolved): mimic: bluefs-bdev-expand aborts
- 09:17 PM Backport #37494: mimic: bluefs-bdev-expand aborts
- Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25348
merged - 11:43 AM Bug #38176 (Fix Under Review): Unable to recover from ENOSPC in BlueFS, WAL
- 11:28 AM Bug #38176 (Won't Fix): Unable to recover from ENOSPC in BlueFS, WAL
- It is possible to insert so much OMAP data into objects that it will overflow storage and cause ENOSPC when rocksdb t...
Also available in: Atom