Activity
From 01/02/2018 to 01/31/2018
01/31/2018
- 08:23 AM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- I have the same problem on my cluster. Periodically I got pg inconsistent only on bluestore osd with this type of mes...
01/29/2018
- 10:43 AM Bug #21312: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
- actual_allocated_size - expected_allocated_size = 4259840 - 4194304 = 0x10000
- 04:25 AM Bug #21312: occaionsal ObjectStore/StoreTestSpecificAUSize.Many4KWritesTest/2 failure
- http://pulpito.ceph.com/yuriw-2018-01-26_18:13:44-rados-wip_yuri_master_1.26.18-distro-basic-smithi/2112995/...
- 02:27 AM Bug #22796: bluestore gets to ENOSPC with small devices
- David Turner wrote:
> I was able to resolve this issue by using the ceph-objectstore-tool to remove copies of PGs so...
01/28/2018
- 03:54 PM Bug #22796: bluestore gets to ENOSPC with small devices
- I was able to resolve this issue by using the ceph-objectstore-tool to remove copies of PGs so the osds could start. ...
01/27/2018
- 05:55 PM Bug #22102 (In Progress): BlueStore crashed on rocksdb checksum mismatch
- full logs at 5e38cf1e-532a-4aa4-8289-5b9e9c59632a
01/26/2018
- 01:27 PM Bug #22796: bluestore gets to ENOSPC with small devices
- This might be a red herring. I think Nick Fisk on the ML found the problem. Originally the output of `ceph osd df` s...
- 01:24 PM Bug #22796: bluestore gets to ENOSPC with small devices
- debug bluestore = 20 log for the same OSD as before.
ceph-post-file: 06b467b7-4a91-4263-85e0-c89268b694e3 - 01:16 PM Bug #22796: bluestore gets to ENOSPC with small devices
- Please use ceph-post-file to upload the full logs.
01/25/2018
- 02:35 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
- The arch is x68_64. Ceph was installed from eu.ceph.com deb repo. This issue isn't current for me anymore as the clus...
- 02:24 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
- Hi Mikko,
What architecture are you running on?
I tried to match your callstacks with binaries for x86_64 for "ceph... - 02:10 PM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- Martin,
For "device location [0x6d76b40000~1000]" it would be:
dd bs=4096 if=/var/lib/ceph/osd/ceph-1/block skip=... - 11:11 AM Bug #22796: bluestore gets to ENOSPC with small devices
- David Turner wrote:
> Here's a log with `debug bluestore = 5`.
- 11:10 AM Bug #22796: bluestore gets to ENOSPC with small devices
- Here's a log with `debug bluestore 5`.
- 11:00 AM Bug #22796: bluestore gets to ENOSPC with small devices
- Can you attach logs with lower debug level? E.g. debug bluestore = 5
- 10:51 AM Bug #22796 (Resolved): bluestore gets to ENOSPC with small devices
- I have a 3 node cluster with mon, mds, mgr, and osds all running on each. The steps I've recently performed on my cl...
01/23/2018
- 10:24 PM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- Hi,
how do I translate the given location, e.g. to a "dd" argument?
Meanwhile I found out that only the first m... - 12:23 PM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- Martin, your logs show places where data is located, for example: "device location [0x6d76b40000~1000]".
Is it possi... - 04:05 PM Bug #22285: _read_bdev_label unable to decode label at offset
- 10:21 AM Backport #22698: luminous: bluestore: New OSD - Caught signal - bstore_kv_sync
- @Prashant Please fix the cherry-pick conflict resolution as suggested by Igor in the PR.
- 12:16 AM Bug #22427 (Resolved): osd_fsid does not exist, fsid is generated instead
01/22/2018
- 08:17 PM Bug #22427 (Fix Under Review): osd_fsid does not exist, fsid is generated instead
- PR at https://github.com/ceph/ceph/pull/20059
- 03:51 PM Bug #22427 (Triaged): osd_fsid does not exist, fsid is generated instead
- 03:53 PM Bug #22510: osd: BlueStore.cc: BlueStore::_balance_bluefs_freespace: assert(0 == "allocate failed...
- 03:51 PM Bug #22245 (Need More Info): [segfault] ceph-bluestore-tool bluefs-log-dump
- can you still reproduce this? do you have (or can you generate) a core file? THe log doesn't tell us where it faile...
- 03:45 PM Bug #22115 (Duplicate): OSD SIGABRT on bluestore_prefer_deferred_size = 104857600: assert(_buffer...
- see #21932
- 03:43 PM Bug #22543 (Can't reproduce): OSDs can not start after shutdown, killed by OOM killer during PGs ...
- 03:40 PM Bug #22066 (Duplicate): bluestore osd asserts repeatedly with ceph-12.2.1/src/include/buffer.h: 8...
- see #21932, pending backport, should be in 12.2.3
- 03:34 PM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- Martin, can you check your dmesg/kernel log and see if there are any media errors? The crc value is for a single blo...
- 03:16 PM Backport #22264 (Resolved): luminous: bluestore: db.slow used when db is not full
- 03:02 PM Backport #22264: luminous: bluestore: db.slow used when db is not full
- luminous cherry-pick is merged.
- 06:00 AM Bug #22616: bluestore_cache_data uses too much memory
- I did some test with bluestore_default_buffered_read = false
The bluestore_cache_data now only use around a fe...
01/19/2018
- 07:12 PM Bug #22464: Bluestore: many checksum errors, always 0x6706be76 (which matches a zero block)
- Just an update:
"ceph pg repair x.yz"
changes the ceph status from HEALTH_ERR to HEALTH_OK (I have to do that e... - 03:28 PM Bug #22616: bluestore_cache_data uses too much memory
- Sage Weil wrote:
> Two things to try:
>
> - bluestore_default_buffered_read = false should make the problem go aw... - 03:03 PM Bug #22616: bluestore_cache_data uses too much memory
- Two things to try:
- bluestore_default_buffered_read = false should make the problem go away, but is more of a wor... - 03:12 PM Bug #22534: Debian's bluestore *rocksdb* does not support neither fast CRC nor compression
- 1. Characteristics of build machine MUST NOT affect builds. i.e. we should strictly override ./configure options whic...
- 03:07 PM Bug #22534 (Need More Info): Debian's bluestore *rocksdb* does not support neither fast CRC nor c...
- My guess is that the build machine or VM that debian used for the package was old and didn't have sse instructions?
- 01:04 AM Bug #22678: block checksum mismatch from rocksdb
- I thought part of the issue might be the old firmware on the 3 x LSI SAS9201-8i controller cards. So I upgraded them ...
01/18/2018
- 11:03 PM Bug #22467 (Can't reproduce): osd boot has stuck for 10min because of clear_temp_object
- 25,000 is way too many PGs for one osd. I suspect the problem is that the cache for leveldb or rocksdb is way to sma...
- 11:01 PM Bug #21556 (Can't reproduce): luminous bluestore OSDs do not recover after out of memory
- closing this, please reopen if you have more info!
- 03:26 PM Bug #22061 (Resolved): Bluestore: OSD killed due to high RAM usage
- fixed in 12.2.2
- 03:26 PM Bug #22540 (Won't Fix): bluestore crush when deleting pool
- This is teh jewel bluestore, which is experiemental and very different from the luminous version.!
- 03:24 PM Bug #22044 (Need More Info): rocksdb log replay - corruption: missing start of fragmented record
- Can you share a bit about how you reproduced this?
Our test suite is doing failure injection at the block layer th... - 07:07 AM Bug #22115 (Need More Info): OSD SIGABRT on bluestore_prefer_deferred_size = 104857600: assert(_b...
- 03:14 AM Backport #22698 (In Progress): luminous: bluestore: New OSD - Caught signal - bstore_kv_sync
- https://github.com/ceph/ceph/pull/19995
01/17/2018
- 08:16 PM Bug #22678: block checksum mismatch from rocksdb
- I'm currently backing up all the data on both CephFS and RBD so that if needed I can wipe the configuration and start...
- 08:13 PM Bug #22678: block checksum mismatch from rocksdb
- I'm able to create a crash easily by just copying files in to CephFS, but I was able to cause the crash with just RBD...
- 02:51 PM Bug #22678 (Duplicate): block checksum mismatch from rocksdb
- 02:51 PM Bug #22678: block checksum mismatch from rocksdb
- oh, see #22102.
was the workload cephfs? - 02:50 PM Bug #22678 (Need More Info): block checksum mismatch from rocksdb
- Could it be that the device has an actual media error? Can you check dmesg for errors?
01/16/2018
- 08:17 AM Backport #22698 (Resolved): luminous: bluestore: New OSD - Caught signal - bstore_kv_sync
- https://github.com/ceph/ceph/pull/19995
- 07:30 AM Bug #22102: BlueStore crashed on rocksdb checksum mismatch
- Sage Weil wrote:
> Have you seen any other instances of this? this is the first time i've heard of this particular ...
01/15/2018
- 07:21 AM Bug #22678 (Duplicate): block checksum mismatch from rocksdb
- Hi
There seems to be a crash bug in the Luminous OSD code which causes OSDs to crash.... - 02:50 AM Bug #20236: bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTest/2 failure
- /a//kchai-2018-01-11_06:11:31-rados-wip-kefu-testing-2018-01-11-1036-distro-basic-mira/2058374/teuthology.log
<pre...
01/11/2018
- 07:35 AM Bug #22616: bluestore_cache_data uses too much memory
- One more fact of my test to add.
I have 48 osd for the test and there were only a few of the osd's bluestore_cache_d... - 12:52 AM Bug #22102: BlueStore crashed on rocksdb checksum mismatch
- I seem to be getting something like this also, mostly happens when the sytem is under write load. I have created the ...
01/10/2018
- 09:38 PM Bug #22609: thrash-eio + bluestore fails with "reached maximum tries (3650) after waiting for 219...
- Is this bluestore not handling out of space conditions well?
- 04:02 PM Bug #22616: bluestore_cache_data uses too much memory
- Sage Weil wrote:
> Writes that are in flight to disk show up under bluestore_cache_data, so even if it is not *cachi... - 02:53 PM Bug #22616 (Need More Info): bluestore_cache_data uses too much memory
- Writes that are in flight to disk show up under bluestore_cache_data, so even if it is not *caching* anything you'll ...
- 03:03 AM Bug #22616: bluestore_cache_data uses too much memory
The work load of read throughput test is 6 fio server with the following parameter
[4m-seq]
description="4m-seq...- 02:24 AM Backport #22633 (In Progress): luminous: OSD crushes with FAILED assert(used_blocks.size() > coun...
- https://github.com/ceph/ceph/pull/19888
01/09/2018
- 11:24 AM Backport #22633 (Resolved): luminous: OSD crushes with FAILED assert(used_blocks.size() > count) ...
- https://github.com/ceph/ceph/pull/19888
01/08/2018
- 04:58 PM Bug #22616 (Resolved): bluestore_cache_data uses too much memory
- I was running a read throughput test and then found some of my osds were killed by oom killer and restarted.
I found... - 07:57 AM Bug #22609 (Can't reproduce): thrash-eio + bluestore fails with "reached maximum tries (3650) aft...
- http://pulpito.ceph.com/yuriw-2018-01-06_17:10:23-rados-wip-yuri-master_1.5.18-testing-basic-smithi/2036088...
01/05/2018
- 04:26 PM Bug #22543: OSDs can not start after shutdown, killed by OOM killer during PGs load
- Sage,
Unfortunately we could not wait so long and re-deployed Ceph cluster on 12/30/2017.
We have managed to sta... - 02:53 PM Bug #22543 (Need More Info): OSDs can not start after shutdown, killed by OOM killer during PGs load
- The mempool dump shows 58GB (!) of pg logs. Can you restart the osd with 'debug bluestore = 20' so we can see if it i...
- 04:01 PM Bug #22535 (Pending Backport): OSD crushes with FAILED assert(used_blocks.size() > count) during ...
- 02:56 PM Bug #22102 (Need More Info): BlueStore crashed on rocksdb checksum mismatch
- Have you seen any other instances of this? this is the first time i've heard of this particular crash. It looks lik...
- 02:54 PM Bug #21736 (Can't reproduce): Cannot create bluestore OSD
01/04/2018
- 12:30 PM Bug #22467: osd boot has stuck for 10min because of clear_temp_object
- Josh Durgin wrote:
> This is bluestore, right? It sounds like you've got too large/slow a rocksdb - you want that me...
01/03/2018
- 09:17 PM Bug #22467: osd boot has stuck for 10min because of clear_temp_object
- This is bluestore, right? It sounds like you've got too large/slow a rocksdb - you want that metadata on an ssd.
- 09:17 AM Bug #22510: osd: BlueStore.cc: BlueStore::_balance_bluefs_freespace: assert(0 == "allocate failed...
- Seems that is same as described here for bitmap allocator:
https://www.spinics.net/lists/ceph-devel/msg32462.html
...
01/02/2018
- 05:47 PM Bug #20557: segmentation fault with rocksdb|BlueStore and jemalloc
- Hi, this happens with RockDB anf filestore on luminous 12.2 on ubuntu 16.04.
Commenting /etc/default/ceph line LD_PR... - 10:23 AM Bug #22467: osd boot has stuck for 10min because of clear_temp_object
- i always got this problem , restart ceph cluster ,the OSD witch have 15000pgs ,cost almost 25min to become up state. ...
Also available in: Atom