Activity
From 08/24/2017 to 09/22/2017
09/22/2017
- 04:09 PM Bug #19511: bluestore overwhelms aio queue
- To have more information what is happening while performing a test:
# rbd bench --io-type write --io-pattern rand ...
09/21/2017
- 08:24 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
- I believe the problem here is that the rocksdb is more tightly bound to tcmalloc. Not exactly sure what the issues o...
- 07:34 PM Bug #20997 (Need More Info): bluestore_types.h: 739: FAILED assert(p != extents.end())
- Jason, is this still happening?
Logs have been archived so I have no more information here besides the backtrace. - 07:33 PM Bug #19511 (Need More Info): bluestore overwhelms aio queue
09/20/2017
- 04:01 PM Bug #21480 (Resolved): bluestore: flush_commit is racy
- observed hang on 'osd bench' command:
/a/yuriw-2017-09-19_19:54:13-rados-wip-yuri-testing3-2017-09-19-1710-distro-ba... - 03:38 PM Bug #19511: bluestore overwhelms aio queue
- Jan- I can't find that commit (1264bae1a8) in the repo. can you verify which version you're running? And/or try with...
- 03:28 PM Bug #21259 (Resolved): bluestore: segv in BlueStore::TwoQCache::_trim
09/19/2017
- 08:07 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- I upgraded cluster to 12.2.0-178-gba746cd (ba746cd14ddd70a4f24a734f83ff9d276dd327d1) last week (to mitigate aio submi...
09/15/2017
- 02:40 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- Sorry, I missed your earlier message. Repushed the branch; should build now. It also includes the other bluestore fi...
09/12/2017
- 10:17 PM Bug #19511: bluestore overwhelms aio queue
- Just re-ran the same fio experiment (multiple clients hit a ssd/nvme backed cluster via cephfs) on ceph version 12.2....
- 04:57 PM Bug #19511: bluestore overwhelms aio queue
- Yes, I got a similar behaviour yesterday again after an upgrade to version 12.2.0.
I haven't collected enough inform...
09/11/2017
- 09:35 AM Bug #19511: bluestore overwhelms aio queue
- 09:08 AM Bug #19511: bluestore overwhelms aio queue
- We are still able to trigger this behaviour. On a cluster where 3 OSD's with block storage on SSD's share an NVME for...
09/10/2017
- 06:56 PM Bug #21312: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
- /a/sage-2017-09-10_02:44:16-rados-wip-yuri-testing4-2017-09-08-1631-distro-basic-smithi/1614995
- 05:54 AM Bug #21332 (Duplicate): OSD Caught signal with StupidAllocator on 12.2.0
- This was originally posted to to the ceph email list: https://www.spinics.net/lists/ceph-users/msg38646.html
I am ...
09/09/2017
- 05:25 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
- Seems that I don't have a WAL bdev, do I? Maybe I created the bluefs using a too-early release?
- 01:09 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
- A quick response of the result, before me taking some time to investigate,...
09/08/2017
- 05:54 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- btw I am running on debian Jessie, it looks like shaman does not build for it, so I am going to build it myself, I al...
- 06:54 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- it looks like build failed... https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAIL...
- 03:37 PM Bug #21312 (Resolved): occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
- ...
- 01:40 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I can sit tight for a while if necessary. That said, if there is a straightforward and reliable way to fix it now (e...
- 08:00 AM Feature #21306 (Rejected): Reduce RBD filestore/bluestore fragmentation throught fallocate
- It is well know fact that RBD images make XFS filesystem heavily fragmented on Filestore OSDs.
So I propose a way ... - 02:59 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
- I've pushed a wip-bluefs-luminous branch (see shaman.ceph.com for builds) that adds 2 new commands to ceph-bluestore-...
09/07/2017
- 10:01 PM Bug #21087 (In Progress): BlueFS Becomes Totally Inaccessible when Failed to Allocate
- rocksdb itself is doing some writes during mount that are then failing. first step is probably to make fsck operate ...
- 11:01 AM Bug #21087: BlueFS Becomes Totally Inaccessible when Failed to Allocate
- More specifically, for almost any of the above ops to run, @BlueStore::_mount()@ must be called in advance, therefore...
- 09:13 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- I've pushed a branch, wip-21259 to shaman. Packages should appear at shaman.ceph.com in an hour or so. They add som...
- 07:02 AM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- I thought that all needed information is between "begin dump of recent events" and "end dump of recent events" so I t...
- 07:02 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I'm fine with it. Thanks for looking into this, Sage.
- 03:36 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Ok, unless you've seen *new* inconsistencies appear since 12.2.0, I think this is from #20983, fixed by d5ba7061ee588...
- 09:49 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Brad Hubbard wrote:
> Ed, Charles, Nicolas,
>
> So all three of you are using Proxmox? Could you let us know if y... - 07:27 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I use ceph with proxmox 5 packages : ceph version 12.1.2 (cd7bc3b11cdbe6fa94324b7322fb2a4716a052a7) luminous (rc) wit...
09/06/2017
- 11:22 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Ed, Charles, Nicolas,
So all three of you are using Proxmox? Could you let us know if you are currently using, or ... - 02:43 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a... - 07:26 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a... - 01:44 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Sage Weil wrote:
> Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, a... - 08:44 PM Bug #21257: bluestore: BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())
- Another: /ceph/teuthology-archive/pdonnell-2017-09-06_15:35:31-multimds-wip-pdonnell-testing-20170906-testing-basic-s...
- 03:36 AM Bug #21257 (Duplicate): bluestore: BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())
- ...
- 08:42 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- I have also had this segfault on 5/244 osd's in the past 24 hours. It's not been a major problem yet as systemd kick...
- 03:11 PM Bug #21259: bluestore: segv in BlueStore::TwoQCache::_trim
- Crash from the logs is:...
- 06:25 AM Bug #21259 (Resolved): bluestore: segv in BlueStore::TwoQCache::_trim
- Sometimes OSD segfaults:...
- 01:48 AM Bug #21255 (Closed): stop bluestore nvme osd, sgdisk it hang, sync operation hang
- stop and remove one bluestore osd, and remake it.
sgdisk hang:
root 6849 14788 0 09:25 pts/0 00:00:00 /usr...
09/05/2017
- 10:13 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Ed, Charles, Nicolas: can you please share which version bluestore was initially deployed with, and also describe you...
- 12:31 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Since I have destroy osd and recreate with filestore but @active+clean+inconsistent@ persist. In @ceph-osd.<osd-num>....
- 09:59 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- You can found it here : ceph-post-file: 258cf93d-a195-4996-af73-dd88aaf858cf
Thanks for your attention. - 09:47 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- http://docs.ceph.com/docs/master/man/8/ceph-post-file/
Let us know the identifier (tag) when it completes (and be ... - 06:53 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I have run the command but log file is 500Mo size, how can I send you easily ?
09/04/2017
- 11:08 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Nicolas,
Could you run the command Sage posted in comment #14?
Changing the allocator will not repair an existi... - 12:40 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I have the same problem, my ceph-cluster as 3 pgs active+clean+inconsistent. When I try ceph pg repair, pg status add...
09/03/2017
- 10:41 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Excellent Edward, thanks for the update.
09/01/2017
- 03:30 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I haven't seen any new inconsistencies since I switched to the "stupid" bluefs allocator Sunday night.
I'll see if...
08/29/2017
- 10:04 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Sage Weil wrote:
> The fact that the same range on the object had a bad csum on all 3 replicas suggests a (reproduci... - 07:43 PM Bug #21040 (Need More Info): bluestore: multiple objects (clones?) referencing same blocks (on al...
- The fact that the same range on the object had a bad csum on all 3 replicas suggests a (reproducible) bluestore bug.
...
08/28/2017
- 02:39 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Brad Hubbard wrote:
> In order to test whether the bitmap allocator is the culprit here you can add the following to... - 02:21 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- In order to test whether the bitmap allocator is the culprit here you can add the following to ceph.conf and restart ...
- 12:56 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Charles Alva wrote:
> Is the data still safe? Or is it really corrupted?
Still looking into the answer to this.
- 12:16 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Brad Hubbard wrote:
> It looks like this could be a problem with the allocator. I'm looking into whether we can swit... - 12:08 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- It looks like this could be a problem with the allocator. I'm looking into whether we can switch allocators as a test.
08/27/2017
- 02:07 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
- I am also getting Bluestore OSD segfaults with jemalloc enabled at runtime (jemalloc preload enabled in /etc/defaults...
08/25/2017
- 01:09 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Also, another inconsistent pg this morning....
- 12:51 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- No trim for me, unless RHEL7 is doing something dumb without my knowledge. All my OSDs are 100% spinning rust. The ...
- 10:49 AM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Brad Hubbard wrote:
> I notice trim mentioned. Are both/any of you somehow manually running trim on these devices?
...
08/24/2017
- 09:38 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- I notice trim mentioned. Are both/any of you somehow manually running trim on these devices?
- 01:27 PM Bug #21040: bluestore: multiple objects (clones?) referencing same blocks (on all replicas)
- Got another inconsistent that may or may not be related. It just cropped up overnight. "rados list-inconsistent-obj"...
Also available in: Atom