Project

General

Profile

Activity

From 08/24/2017 to 09/22/2017

09/22/2017

04:09 PM Bug #19511: bluestore overwhelms aio queue
To have more information what is happening while performing a test:
# rbd bench --io-type write --io-pattern rand ...
Dmitry Smirnov

09/21/2017

08:24 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
I believe the problem here is that the rocksdb is more tightly bound to tcmalloc. Not exactly sure what the issues o... Sage Weil
07:34 PM Bug #20997 (Need More Info): bluestore_types.h: 739: FAILED assert(p != extents.end())
Jason, is this still happening?
Logs have been archived so I have no more information here besides the backtrace.
Sage Weil
07:33 PM Bug #19511 (Need More Info): bluestore overwhelms aio queue
Sage Weil

09/20/2017

04:01 PM Bug #21480 (Resolved): bluestore: flush_commit is racy
observed hang on 'osd bench' command:
/a/yuriw-2017-09-19_19:54:13-rados-wip-yuri-testing3-2017-09-19-1710-distro-ba...
Sage Weil
03:38 PM Bug #19511: bluestore overwhelms aio queue
Jan- I can't find that commit (1264bae1a8) in the repo. can you verify which version you're running? And/or try with... Sage Weil
03:28 PM Bug #21259 (Resolved): bluestore: segv in BlueStore::TwoQCache::_trim
Josh Durgin

09/19/2017

08:07 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
I upgraded cluster to 12.2.0-178-gba746cd (ba746cd14ddd70a4f24a734f83ff9d276dd327d1) last week (to mitigate aio submi... Henrik Korkuc

09/15/2017

02:40 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
Sorry, I missed your earlier message. Repushed the branch; should build now. It also includes the other bluestore fi... Sage Weil

09/12/2017

10:17 PM Bug #19511: bluestore overwhelms aio queue
Just re-ran the same fio experiment (multiple clients hit a ssd/nvme backed cluster via cephfs) on ceph version 12.2.... Jan Fajerski
04:57 PM Bug #19511: bluestore overwhelms aio queue
Yes, I got a similar behaviour yesterday again after an upgrade to version 12.2.0.
I haven't collected enough inform...
Dmitry Smirnov

09/11/2017

09:35 AM Bug #19511: bluestore overwhelms aio queue
Jan Fajerski
09:08 AM Bug #19511: bluestore overwhelms aio queue
We are still able to trigger this behaviour. On a cluster where 3 OSD's with block storage on SSD's share an NVME for... Jan Fajerski

09/10/2017

06:56 PM Bug #21312: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
/a/sage-2017-09-10_02:44:16-rados-wip-yuri-testing4-2017-09-08-1631-distro-basic-smithi/1614995 Sage Weil
05:54 AM Bug #21332 (Duplicate): OSD Caught signal with StupidAllocator on 12.2.0
This was originally posted to to the ceph email list: https://www.spinics.net/lists/ceph-users/msg38646.html
I am ...
Eric Eastman

09/09/2017

05:25 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
Seems that I don't have a WAL bdev, do I? Maybe I created the bluefs using a too-early release? WANG Guoqin
01:09 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
A quick response of the result, before me taking some time to investigate,... WANG Guoqin

09/08/2017

05:54 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
btw I am running on debian Jessie, it looks like shaman does not build for it, so I am going to build it myself, I al... Henrik Korkuc
06:54 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
it looks like build failed... https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAIL... Henrik Korkuc
03:37 PM Bug #21312 (Resolved): occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
... Sage Weil
01:40 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I can sit tight for a while if necessary. That said, if there is a straightforward and reliable way to fix it now (e... Edward Huyer
08:00 AM Feature #21306 (Rejected): Reduce RBD filestore/bluestore fragmentation throught fallocate
It is well know fact that RBD images make XFS filesystem heavily fragmented on Filestore OSDs.
So I propose a way ...
Марк Коренберг
02:59 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
I've pushed a wip-bluefs-luminous branch (see shaman.ceph.com for builds) that adds 2 new commands to ceph-bluestore-... Sage Weil

09/07/2017

10:01 PM Bug #21087 (In Progress): BlueFS Becomes Totally Inaccessible when Failed to Allocate
rocksdb itself is doing some writes during mount that are then failing. first step is probably to make fsck operate ... Sage Weil
11:01 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
More specifically, for almost any of the above ops to run, @BlueStore::_mount()@ must be called in advance, therefore... WANG Guoqin
09:13 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
I've pushed a branch, wip-21259 to shaman. Packages should appear at shaman.ceph.com in an hour or so. They add som... Sage Weil
07:02 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
I thought that all needed information is between "begin dump of recent events" and "end dump of recent events" so I t... Henrik Korkuc
07:02 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I'm fine with it. Thanks for looking into this, Sage. Charles Alva
03:36 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Ok, unless you've seen *new* inconsistencies appear since 12.2.0, I think this is from #20983, fixed by d5ba7061ee588... Sage Weil
09:49 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Brad Hubbard wrote:
> Ed, Charles, Nicolas,
>
> So all three of you are using Proxmox? Could you let us know if y...
Charles Alva
07:27 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I use ceph with proxmox 5 packages : ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7) luminous (rc) wit... Nicolas Drufin

09/06/2017

11:22 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Ed, Charles, Nicolas,
So all three of you are using Proxmox? Could you let us know if you are currently using, or ...
Brad Hubbard
02:43 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a...
Edward Huyer
07:26 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a...
Nicolas Drufin
01:44 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a...
Charles Alva
08:44 PM Bug #21257: bluestore: BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())
Another: /ceph/teuthology-archive/pdonnell-2017-09-06_15:35:31-multimds-wip-pdonnell-testing-20170906-testing-basic-s... Patrick Donnelly
03:36 AM Bug #21257 (Duplicate): bluestore: BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())
... Patrick Donnelly
08:42 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
I have also had this segfault on 5/244 osd's in the past 24 hours. It's not been a major problem yet as systemd kick... Mike Lowe
03:11 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
Crash from the logs is:... Josh Durgin
06:25 AM Bug #21259 (Resolved): bluestore: segv in BlueStore::TwoQCache::_trim
Sometimes OSD segfaults:... Henrik Korkuc
01:48 AM Bug #21255 (Closed): stop bluestore nvme osd, sgdisk it hang, sync operation hang
stop and remove one bluestore osd, and remake it.
sgdisk hang:
root 6849 14788 0 09:25 pts/0 00:00:00 /usr...
Yong Wang

09/05/2017

10:13 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, and also describe you... Sage Weil
12:31 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Since I have destroy osd and recreate with filestore but @active+clean+inconsistent@ persist. In @ceph-osd.<osd-num>.... Nicolas Drufin
09:59 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
You can found it here : ceph-post-file: 258cf93d-a195-4996-af73-dd88aaf858cf
Thanks for your attention.
Nicolas Drufin
09:47 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
http://docs.ceph.com/docs/master/man/8/ceph-post-file/
Let us know the identifier (tag) when it completes (and be ...
Brad Hubbard
06:53 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I have run the command but log file is 500Mo size, how can I send you easily ? Nicolas Drufin

09/04/2017

11:08 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Nicolas,
Could you run the command Sage posted in comment #14?
Changing the allocator will not repair an existi...
Brad Hubbard
12:40 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I have the same problem, my ceph-cluster as 3 pgs active+clean+inconsistent. When I try ceph pg repair, pg status add... Nicolas Drufin

09/03/2017

10:41 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Excellent Edward, thanks for the update. Brad Hubbard

09/01/2017

03:30 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I haven't seen any new inconsistencies since I switched to the "stupid" bluefs allocator Sunday night.
I'll see if...
Edward Huyer

08/29/2017

10:04 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Sage Weil wrote:
> The fact that the same range on the object had a bad csum on all 3 replicas suggests a (reproduci...
Charles Alva
07:43 PM Bug #21040 (Need More Info): bluestore: multiple objects (clones?) referencing same blocks (on al...
The fact that the same range on the object had a bad csum on all 3 replicas suggests a (reproducible) bluestore bug.
...
Sage Weil

08/28/2017

02:39 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Brad Hubbard wrote:
> In order to test whether the bitmap allocator is the culprit here you can add the following to...
Charles Alva
02:21 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
In order to test whether the bitmap allocator is the culprit here you can add the following to ceph.conf and restart ... Brad Hubbard
12:56 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Charles Alva wrote:
> Is the data still safe? Or is it really corrupted?
Still looking into the answer to this.
Brad Hubbard
12:16 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Brad Hubbard wrote:
> It looks like this could be a problem with the allocator. I'm looking into whether we can swit...
Charles Alva
12:08 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
It looks like this could be a problem with the allocator. I'm looking into whether we can switch allocators as a test. Brad Hubbard

08/27/2017

02:07 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
I am also getting Bluestore OSD segfaults with jemalloc enabled at runtime (jemalloc preload enabled in /etc/defaults... Mikko Tanner

08/25/2017

01:09 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Also, another inconsistent pg this morning.... Edward Huyer
12:51 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
No trim for me, unless RHEL7 is doing something dumb without my knowledge. All my OSDs are 100% spinning rust. The ... Edward Huyer
10:49 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Brad Hubbard wrote:
> I notice trim mentioned. Are both/any of you somehow manually running trim on these devices?
...
Charles Alva

08/24/2017

09:38 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
I notice trim mentioned. Are both/any of you somehow manually running trim on these devices? Brad Hubbard
01:27 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
Got another inconsistent that may or may not be related. It just cropped up overnight. "rados list-inconsistent-obj"... Edward Huyer
 

Also available in: Atom