Project

General

Profile

Activity

From 02/05/2019 to 03/06/2019

03/06/2019

09:55 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
So that's why write_big operations may be also deferred just like write_small's. OK, thank you very much, it's clear now Vitaliy Filippov
08:29 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
It's not deferring because at the layer that deferring happens, we're talking about blobs (not writes), and the blogs... Sage Weil
04:08 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
Forgot to mention, this was Ceph 14.1.0 Vitaliy Filippov
09:25 AM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
I've just tried to set
[osd]
bluestore_prefer_deferred_size_hdd = 4194304
On a test HDD plugged in my laptop. ...
Vitaliy Filippov
06:54 PM Bug #38557: pkg dependency issues upgrading from 12.2.y to 14.x.y
accidentally opened against bluestore. You may close it.
See https://tracker.ceph.com/issues/38612 instead.
Kaleb KEITHLEY

03/05/2019

05:45 PM Backport #38587 (Resolved): mimic: OSD crashes in get_str_map while creating with ceph-volume
https://github.com/ceph/ceph/pull/26810 Nathan Cutler
05:45 PM Backport #38586 (Resolved): luminous: OSD crashes in get_str_map while creating with ceph-volume
https://github.com/ceph/ceph/pull/26900 Nathan Cutler
03:18 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
I've just verified deferred writes behavior for 4M writes using objectstore FIO plugin.
Indeed bluestore splits writ...
Igor Fedotov
08:10 AM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
Sage Weil wrote:
> > all writes of size 4MB with bluestore_prefer_deferred_size_hdd < 524288 go HDD directly. >= 524...
Марк Коренберг
11:46 AM Bug #38363: Failure in assert when calling: ceph-volume lvm prepare --bluestore --data /dev/sdg
I finally found the extended debug log in /var/log/ceph/ceph-osd.0.log. I attached the log output file (44k) to this ... Rainer Krienke

03/04/2019

11:23 PM Bug #38489: bluestore_prefer_deferred_size_hdd units are not clear
> all writes of size 4MB with bluestore_prefer_deferred_size_hdd < 524288 go HDD directly. >= 524288 through SSD (I m... Sage Weil
05:55 PM Bug #38574 (Resolved): mimic: Unable to recover from ENOSPC in BlueFS
This the same issue as https://tracker.ceph.com/issues/36268.
We have alternate fix for mimic, which will be backpor...
Neha Ojha
03:36 PM Bug #38329 (Pending Backport): OSD crashes in get_str_map while creating with ceph-volume
Sage Weil
03:23 PM Bug #36268 (Resolved): Unable to recover from ENOSPC in BlueFS
Alternative fix for mimic and luminous: https://github.com/ceph/ceph/pull/26735 Sage Weil

03/03/2019

08:07 PM Bug #38559 (Resolved): 50-100% iops lost due to bluefs_preextend_wal_files = false
Hi.
I was investigating why RocksDB performance is so bad considering random 4K iops. I was looking at strace and ...
Vitaliy Filippov
01:30 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
We have upgraded to 12.2.11. During reboots the following would pass by:
[16:20:59] @ bitrot: osd.17 [ERR] 7...
Stefan Kooman
11:55 AM Bug #38557 (Closed): pkg dependency issues upgrading from 12.2.y to 14.x.y
Description of problem:
With respect to https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
Kaleb KEITHLEY

03/02/2019

02:28 PM Bug #38554 (Duplicate): ObjectStore/StoreTestSpecificAUSize.TooManyBlobsTest/2 fail, Expected: (r...
... Sage Weil

03/01/2019

04:44 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
FYI, I think I hit another case with this with this in the last two weeks.
A RGW only case where if you would list...
Wido den Hollander
03:27 PM Bug #36455 (Resolved): BlueStore: ENODATA not fully handled
Nathan Cutler
03:27 PM Backport #37825 (Resolved): luminous: BlueStore: ENODATA not fully handled
Nathan Cutler
03:10 PM Backport #36641 (New): mimic: Unable to recover from ENOSPC in BlueFS
Nathan Cutler
03:10 PM Backport #36640 (New): luminous: Unable to recover from ENOSPC in BlueFS
Nathan Cutler
03:09 PM Bug #36268 (Pending Backport): Unable to recover from ENOSPC in BlueFS
Sage, did you mean to cancel the mimic and luminous backports when you changed the status to Resolved? Nathan Cutler
10:46 AM Bug #25077 (New): Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
Igor Fedotov
10:46 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
Stefan Kooman wrote:
> @Igor Fedotov:
>
> We are using ceph balancer to get PGs balanced accross the cluster. The...
Igor Fedotov
07:56 AM Bug #38363: Failure in assert when calling: ceph-volume lvm prepare --bluestore --data /dev/sdg
I tried but the output of ceph-volume remains the same....
I added this to /etc/ceph/ceph.conf on my testing nod...
Rainer Krienke

02/28/2019

07:30 PM Backport #37825: luminous: BlueStore: ENODATA not fully handled
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25855
merged
Yuri Weinstein
05:15 PM Bug #37282: rocksdb: submit_transaction_sync error: Corruption: block checksum mismatch code = 2
We're not sure how to proceed without being able to reprdocue the crash, and we have never seen this.
1. Would it...
Sage Weil
04:41 PM Bug #38329 (Fix Under Review): OSD crashes in get_str_map while creating with ceph-volume
reproduce this and got a core.
I think the problem is an empty string passed to trim() in str_map.cc. Fix here: h...
Sage Weil
03:37 PM Bug #23206 (Rejected): ceph-osd daemon crashes - *** Caught signal (Aborted) **
not enough info Sage Weil
03:35 PM Bug #24639 (Can't reproduce): [segfault] segfault in BlueFS::read
sounds like a hardware problem then! Sage Weil
03:34 PM Bug #25098: Bluestore OSD failed to start with `bluefs_types.h: 54: FAILED assert(pos <= end)`
Current status:
We want a more concrete source of truth for whether the db and/or wal partitions should exist--som...
Sage Weil
03:31 PM Bug #34526 (Duplicate): OSD crash in KernelDevice::direct_read_unaligned while scrubbing
Sage Weil
09:55 AM Bug #34526: OSD crash in KernelDevice::direct_read_unaligned while scrubbing
IMO this is BlueStore (or more precisely BlueFS and/or RocksDB) related.
And I think it's duplicate of #36482
O...
Igor Fedotov
03:30 PM Bug #36268 (Resolved): Unable to recover from ENOSPC in BlueFS
Sage Weil
03:30 PM Bug #36331 (Need More Info): FAILED ObjectStore/StoreTestSpecificAUSize.SyntheticMatrixNoCsum/2 ...
this was an ubuntu 18.04 kernel. maybe this was the pread vs swap zeroed pages kerenl bug?
i think we need anothe...
Sage Weil
03:27 PM Bug #36364 (Can't reproduce): Bluestore OSD IO Hangs near Flush (flush in 90.330556)
Sage Weil
03:27 PM Bug #38049 (Resolved): random osds failing in thread_name:bstore_kv_final
Sage Weil
03:23 PM Bug #38250 (Need More Info): assert failure crash prevents ceph-osd from running
Is the errno EIO in this case?
On read error we do crash and fail the OSD. There is generally no recovery path fo...
Sage Weil
03:18 PM Bug #38272 (In Progress): "no available blob id" assertion might occur
Sage Weil
03:16 PM Bug #38363 (Need More Info): Failure in assert when calling: ceph-volume lvm prepare --bluestore ...
Can you reproduce this with debug_bluestore=20, debug_bluefs=20, debug_bdev=20?
Thanks!
Sage Weil
03:14 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
- it looks like implementing readahead in bluefs would help
- we think newer rocksdb does its own readahead
Sage Weil
02:41 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
We've got another occurrence for this issue too.
Omap listing for specific onode consistently takes ~2 mins while d...
Igor Fedotov
02:19 PM Bug #36482: High amount of Read I/O on BlueFS/DB when listing omap keys
I think this is the same issue:
https://marc.info/?l=ceph-devel&m=155134206210976&w=2
Igor Fedotov
03:04 PM Bug #37914 (Can't reproduce): bluestore: segmentation fault
no logs or core. hoping it was teh hypercombined bufferlist memory corruption issue Sage Weil
09:57 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
@Igor Fedotov:
We are using ceph balancer to get PGs balanced accross the cluster. The day after the crashes, the ...
Stefan Kooman

02/27/2019

11:18 AM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
Check, I'll collect the needed information. Note, during the restarts of the storage servers the *same* OSDs crashed ... Stefan Kooman
07:09 AM Feature #38494: Bluestore: issue discards on everything non-discarded during deep-scrubs
Included link is just related PR. Марк Коренберг
07:07 AM Feature #38494: Bluestore: issue discards on everything non-discarded during deep-scrubs
text formatting of previous message is wrong. I did not want to stroke-out the text. Марк Коренберг
07:07 AM Feature #38494 (New): Bluestore: issue discards on everything non-discarded during deep-scrubs
Yes, we have bdev_enable_discard and bdev_async_discard, but they are not documented.
Ubuntu issues ...
Марк Коренберг

02/26/2019

09:25 PM Bug #38489 (Resolved): bluestore_prefer_deferred_size_hdd units are not clear
I have done an experiment. I made a pool with one PG of size 1. Next I run this command:
rados bench -p qwe -b 4M ...
Марк Коренберг
02:43 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
Stefan, to make sure (at our best) this is exactly the same bug, could you please check PG states with ceph-objectsto... Igor Fedotov

02/22/2019

04:05 PM Bug #37733 (Resolved): os/bluestore: fixup access a destroy cond cause deadlock or undefine behav...
Nathan Cutler
04:05 PM Backport #38142 (Resolved): luminous: os/bluestore: fixup access a destroy cond cause deadlock or...
Nathan Cutler
03:32 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
Added related-to link to #38144 where the GCC 9 FTBFS is being discussed. A patch has been proposed there, but it inc... Nathan Cutler

02/21/2019

09:52 PM Backport #38142: luminous: os/bluestore: fixup access a destroy cond cause deadlock or undefine b...
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26261
merged
Yuri Weinstein
03:49 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
(original reporter here)
I have following customisation in ceph.conf:...
Tomasz Torcz
01:00 PM Bug #25077: Occasional assertion in ObjectStore/StoreTest.HashCollisionTest/2
We *think* we have hit this same issue "in the field" on a Luminous 12.2.8 cluster:
2019-02-20 18:42:45.261357 7fd...
Stefan Kooman

02/20/2019

11:15 AM Bug #38395 (Fix Under Review): luminous: write following remove might access previous onode
Igor Fedotov
10:35 AM Bug #38395: luminous: write following remove might access previous onode
Igor Fedotov
10:25 AM Bug #38395 (Resolved): luminous: write following remove might access previous onode
So the sequence is as follows:
T1:
remove A
T2:
touch A
write A
In Luminous there is a chance that A is rem...
Igor Fedotov

02/18/2019

01:41 PM Bug #38363 (Need More Info): Failure in assert when calling: ceph-volume lvm prepare --bluestore ...
I run Ubuntu 18.04 and and ceph version 13.2.4-1bionic from this repo: https://download.ceph.com/debian-mimic.
Whe...
Rainer Krienke

02/16/2019

11:00 AM Backport #37990 (Resolved): mimic: Compression not working, and when applied OSD disks are failin...
Nathan Cutler

02/15/2019

10:38 PM Bug #37839: Compression not working, and when applied OSD disks are failing randomly
merged https://github.com/ceph/ceph/pull/26342
https://github.com/ceph/ceph/pull/26544
Yuri Weinstein
08:29 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
- have any options been customized?
- what version is this? 14.0.1-2.fc30 is a random dev checkpoint commit from ma...
Sage Weil
08:12 PM Bug #38329: OSD crashes in get_str_map while creating with ceph-volume
Changing back to the Ceph tracker, this is not a crash in ceph-volume or specific to ceph-volume that I can see Alfredo Deza
12:57 PM Bug #38329 (Resolved): OSD crashes in get_str_map while creating with ceph-volume
see https://bugzilla.redhat.com/show_bug.cgi?id=1661583... Kaleb KEITHLEY
03:10 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
We have not experienced any further crashes in over a week (compared to multiple crashes per hour before), so it look... Lawrence Smith

02/13/2019

12:39 PM Bug #38230 (Resolved): segv in onode lookup
https://github.com/ceph/ceph/pull/26391 Sage Weil

02/12/2019

03:35 PM Bug #38272: "no available blob id" assertion might occur
onode dump shortly before the assertion:
2019-02-12 18:23:47.546 7fca6fab1b40 0 bluestore(bluestore.test_temp_dir) ...
Igor Fedotov
03:29 PM Bug #38272: "no available blob id" assertion might occur
Backtrace from UT:
-1> 2019-02-12 18:23:48.346 7fca6fab1b40 -1 /home/if/ceph/src/os/bluestore/BlueStore.cc: In fun...
Igor Fedotov
03:26 PM Bug #38272: "no available blob id" assertion might occur
Stack trace from the customer log:
2019-02-06 00:04:25.934977 7ff3e3bca700 -1 /home/abuild/rpmbuild/BUILD/ceph-12.2....
Igor Fedotov
03:25 PM Bug #38272 (Resolved): "no available blob id" assertion might occur
We observed that on-site but unfortunately OSD were removed and are unavailable for inspection.
However I managed to...
Igor Fedotov
03:01 PM Backport #38143 (Resolved): mimic: os/bluestore: fixup access a destroy cond cause deadlock or un...
Nathan Cutler
12:00 AM Backport #38143: mimic: os/bluestore: fixup access a destroy cond cause deadlock or undefine beha...
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26260
merged
Yuri Weinstein
02:53 PM Backport #38188 (In Progress): luminous: deep fsck fails on inspecting very large onodes
Nathan Cutler
02:50 PM Backport #38187 (Resolved): mimic: deep fsck fails on inspecting very large onodes
Nathan Cutler

02/11/2019

09:08 PM Backport #38187: mimic: deep fsck fails on inspecting very large onodes
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/26291
merged
Yuri Weinstein
04:34 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
From the first log, this looks like #36541. My guess is the crashes you were seeing after were continued problems fr... Sage Weil
10:06 AM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
I still seem to be experiencing these errors, albeit at a much reduced rate since upgrading to 13.2.3. I could wake u... Nick Fisk

02/09/2019

05:14 PM Bug #38250 (Rejected): assert failure crash prevents ceph-osd from running
One of my OSDs keeps crashing shortly after startup, which is preventing it from joining the cluster. The core issue... Adam DC949

02/08/2019

03:40 PM Bug #38230: segv in onode lookup
... Sage Weil

02/07/2019

10:41 PM Bug #38230: segv in onode lookup
i'm guessing this is the same heap corruption we've been seeing, but logging it anyway Sage Weil
10:40 PM Bug #38230 (Resolved): segv in onode lookup
... Sage Weil

02/06/2019

03:56 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
After two days of running fine I had set the bluestore and bluefs log level back to default, so I dont know how helpf... Lawrence Smith
03:27 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
Lawrence, would you share the log for current crashes please?
Existing failures with fsck are expected as the patc...
Igor Fedotov
01:24 PM Bug #38049: random osds failing in thread_name:bstore_kv_final
We have since patched our ceph with https://github.com/ceph/ceph/pull/24686 which fixes Issue #36541. Since then the ... Lawrence Smith
02:16 PM Bug #38176: Unable to recover from ENOSPC in BlueFS, WAL
Fixed link to bug replication script.
https://drive.google.com/file/d/10Lvcf6_Lj2c2sydcfU170lbb-IQClvH-
Adam Kupczyk
09:05 AM Bug #37360 (Resolved): bluefs-bdev-expand aborts
Igor Fedotov
08:54 AM Backport #38188: luminous: deep fsck fails on inspecting very large onodes
No need for that additional cherry-pick, just add new option using the method applicable for luminous Igor Fedotov

02/05/2019

11:56 PM Backport #38188 (Need More Info): luminous: deep fsck fails on inspecting very large onodes
We need to cherry-pick additional commits to get this backport PR, Option::TYPE_SIZE and Option::FLAG_RUNTIME not de... Prashant D
05:02 PM Backport #38188 (Resolved): luminous: deep fsck fails on inspecting very large onodes
https://github.com/ceph/ceph/pull/26387 Nathan Cutler
11:31 PM Backport #38187 (In Progress): mimic: deep fsck fails on inspecting very large onodes
https://github.com/ceph/ceph/pull/26291 Prashant D
05:01 PM Backport #38187 (Resolved): mimic: deep fsck fails on inspecting very large onodes
https://github.com/ceph/ceph/pull/26291 Nathan Cutler
09:39 PM Backport #37494 (Resolved): mimic: bluefs-bdev-expand aborts
Igor Fedotov
09:17 PM Backport #37494: mimic: bluefs-bdev-expand aborts
Nathan Cutler wrote:
> https://github.com/ceph/ceph/pull/25348
merged
Yuri Weinstein
11:43 AM Bug #38176 (Fix Under Review): Unable to recover from ENOSPC in BlueFS, WAL
Kefu Chai
11:28 AM Bug #38176 (Won't Fix): Unable to recover from ENOSPC in BlueFS, WAL
It is possible to insert so much OMAP data into objects that it will overflow storage and cause ENOSPC when rocksdb t... Adam Kupczyk
 

Also available in: Atom