Activity
From 07/17/2022 to 08/15/2022
08/15/2022
- 12:59 PM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
- 4) Quincy case from Telemetry: https://tracker.ceph.com/issues/56382
- 12:57 PM Bug #53002 (New): crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
- Looks like this hasn't been completely fixed yet.
We've got a bunch of new tickets from Telemetry bot which indicate...
08/11/2022
- 04:16 PM Bug #53002: crash BlueStore::Onode::put from BlueStore::TransContext::~TransContext
- according to https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/PPWIFPEI3EVBU3GQYYO6ABGF23WR5SGZ/ this ...
- 11:23 AM Bug #56424 (Resolved): bluestore_cache_other mempool entry leak
- 11:23 AM Backport #56599 (Resolved): pacific: octopus : bluestore_cache_other pool memory leak ?
- 10:22 AM Bug #52705 (Resolved): pg scrub stat mismatch with special objects that have hash 'ffffffff'
08/10/2022
- 08:07 AM Backport #49980 (Rejected): nautilus: BlueRocksEnv::GetChildren may pass trailing slashes to Blue...
- Nautilus is EOL
08/09/2022
- 08:25 AM Bug #56503: Deleting large (~850gb) objects causes OSD to crash
- Ok,
so here's what I did:
1. I moved .rgw.gc pool to different crush tree to isolate affected OSDs
2. I increased ... - 06:30 AM Bug #51682 (Resolved): bluestore repair might cause invalid write
- 06:30 AM Backport #52766 (Rejected): nautilus: bluestore repair might cause invalid write
- Nautilus is EOL
08/08/2022
- 08:32 AM Bug #47740 (Duplicate): OSD crash when increase pg_num
- 08:30 AM Bug #54650 (Duplicate): crash: BlueStore::Onode::put()
- 08:28 AM Bug #56200 (Duplicate): crash: ceph::buffer::ptr::release()
- 08:26 AM Bug #54727 (Duplicate): crash: __pthread_mutex_lock()
08/03/2022
- 03:40 PM Backport #57028 (Resolved): quincy: Bluefs might put an orpan op_update record in the log
- 03:40 PM Backport #57027 (Resolved): pacific: Bluefs might put an orpan op_update record in the log
- 03:35 PM Bug #56533 (Pending Backport): Bluefs might put an orpan op_update record in the log
- 01:25 PM Bug #57015 (New): bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file)
- /a/yuriw-2022-07-28_15:03:58-rados-wip-yuri3-testing-2022-07-27-1134-distro-default-smithi/6952123/...
08/02/2022
- 04:55 PM Backport #56669: pacific: Deferred writes might cause "rocksdb: Corruption: Bad table magic number"
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47296
merged
08/01/2022
- 05:23 PM Backport #56668: quincy: Deferred writes might cause "rocksdb: Corruption: Bad table magic number"
- Backport Bot wrote:
> https://github.com/ceph/ceph/pull/47297
merged - 08:34 AM Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- This morning, I have :
PG_NOT_DEEP_SCRUBBED: 11 pgs not deep-scrubbed in time
Never had before Pacific.
Could it...
07/29/2022
- 02:12 PM Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- I tried for a couple of hours, but it was worst than with 512k. I had a latency plateau and IOPS drops and rollback.
... - 11:33 AM Bug #56503: Deleting large (~850gb) objects causes OSD to crash
- Radoslaw Zarzynski wrote:
> Do you have logs with @debug_bluestore=20@ maybe?
>
> Mark, can this be related to th... - 11:18 AM Bug #56503: Deleting large (~850gb) objects causes OSD to crash
- Ok, I'll try to reproduce this
07/28/2022
- 04:39 PM Backport #56598: quincy: octopus : bluestore_cache_other pool memory leak ?
- https://github.com/ceph/ceph/pull/47339
- 04:27 PM Backport #56598 (In Progress): quincy: octopus : bluestore_cache_other pool memory leak ?
- 04:28 PM Backport #56599 (In Progress): pacific: octopus : bluestore_cache_other pool memory leak ?
- https://github.com/ceph/ceph/pull/47337
- 04:20 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- Correction, each test had site1 w/out BSZBD and the other with it enabled. The second round of tests were the same bu...
- 04:18 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- Two rounds of testing have occurred, one with osd debug_bluestore=20 but this one did not reproduce the perf boost se...
- 02:49 PM Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- Gilles Mocellin wrote:
> Increasing bluestore_prefer_deferred_size_hdd to 128k and even 512k helped us with slow ops... - 01:28 PM Bug #56488: BlueStore doesn't defer small writes for pre-pacific hdd osds
- Increasing bluestore_prefer_deferred_size_hdd to 128k and even 512k helped us with slow ops and reccurring high laten...
- 02:23 AM Bug #56943 (New): crash: boost::intrusive::bstree_impl<boost::intrusive::bhtraits<BlueStore::Exte...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=46670c1166822eff93a5c43e...- 02:23 AM Bug #56942 (New): crash: rocksdb::DecodeFixed32(char const*)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=fff872db27320085d5bd3c4e...- 02:23 AM Bug #56941 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d483d1bd67a50d29a49a9c2f...- 02:23 AM Bug #56940 (New): crash: virtual int BlueStore::write_meta(const string&, const string&): assert(...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b90ad279d7f5c361a824a09d...- 02:23 AM Bug #56939 (New): crash: rocksdb::DataBlockIter::SeekToFirstImpl()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=455aa69e50b040c101f55d15...- 02:23 AM Bug #56936 (New): crash: rocksdb::DataBlockIter::SeekToFirstImpl()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=8590b7fc8f5415e7225d962b...- 02:23 AM Bug #56935 (New): crash: virtual void KernelDevice::aio_submit(IOContext*): assert(r == 0)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=0a624fe2e97fa1ffd88d50dc...- 02:23 AM Bug #56933 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5f0c1131ba4800f063920642...- 02:23 AM Bug #56932 (New): crash: rocksdb::WriteBatchInternal::Put(rocksdb::WriteBatch*, unsigned int, roc...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=f6f3ea231fc56e74e4e6d6bb...- 02:23 AM Bug #56931 (New): crash: rocksdb::MemTableIterator::NextAndGetResult(rocksdb::IterateResult*)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=8926109eb0d8ab91146ec193...- 02:23 AM Bug #56930 (New): crash: base::internal::SpinLockDelay(int volatile*, int, int)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b42e6a1680ce22b480b052d6...- 02:23 AM Bug #56929 (New): crash: cfree()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b89501efbb4c8751cb1ce6c5...- 02:23 AM Bug #56928 (New): crash: void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectSto...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5315149c952c81404277d576...- 02:23 AM Bug #56927 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5602ffe6853c07359f35c069...- 02:23 AM Bug #56924 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=f38c97119a825fde03f0ce8e...- 02:23 AM Bug #56922 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=33474006b60691e46972d8f4...- 02:23 AM Bug #56921 (New): crash: rocksdb::DecodeEntry::operator()(char const*, char const*, unsigned int*...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5d3dc9333da319125cf186c8...- 02:23 AM Bug #56918 (New): crash: pthread_mutex_lock()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=9b2e91f34aa28e7988d2fc0e...- 02:23 AM Bug #56917 (New): crash: pread64()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=ec0f636281801bfa8f889580...- 02:23 AM Bug #56916 (New): crash: syscall()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b768be65fbd8841719dcb117...- 02:23 AM Bug #56915 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=73d8a758380b04ecd772b14d...- 02:23 AM Bug #56914 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d535add5e715a2e23e813913...- 02:23 AM Bug #56913 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=56806c971618016752c15b1c...- 02:23 AM Bug #56912 (New): crash: int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=6b137275d89d84d4c60afd16...- 02:23 AM Bug #56911 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=3aeea5d607ac5d4ab0d80543...- 02:22 AM Bug #56910 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d171fa279d74eacb36d7da3c...- 02:22 AM Bug #56909 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=85ed5133055b8aa9722528b9...- 02:22 AM Bug #56908 (New): crash: virtual int BlueStore::write_meta(const string&, const string&): assert(...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d2e0dd05fe7eeb5c6d2902f6...- 02:22 AM Bug #56907 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5348e00696e4a251e00fb579...- 02:22 AM Bug #56905 (New): crash: LZ4_decompress_safe()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=0ab37cb2e4890193279d234d...- 02:22 AM Bug #56902 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=3eeb235b1f1fc2558adfd76b...- 02:22 AM Bug #56899 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=e5ca6db6653c7b124e7ee407...- 02:22 AM Bug #56893 (New): crash: ceph::buffer::ptr::iterator_impl<true>::operator
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a74bc1e9c988858f79475e99...- 02:22 AM Bug #56891 (New): crash: rocksdb::DBImpl::GetIntProperty(rocksdb::ColumnFamilyHandle*, rocksdb::S...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=93e4ff70cee45151fce093c7...- 02:22 AM Bug #56885 (New): crash: pread64()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=dcd9097d27b34cfe2a6cef37...- 02:22 AM Bug #56884 (New): crash: int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a33947a3ddbb26d8cf02c253...- 02:22 AM Bug #56882 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5aeccf665a5a7739826e62db...- 02:22 AM Bug #56881 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=456b14e92628089886ace9d4...- 02:22 AM Bug #56880 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=374d78417debf1f10bc9a262...- 02:21 AM Bug #56879 (New): crash: rocksdb::IteratorWrapperBase<rocksdb::Slice>::Next()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=58891b4c629de3f8d2b35794...- 02:21 AM Bug #56877 (New): crash: rocksdb::IteratorWrapperBase<rocksdb::Slice>::NextAndGetResult(rocksdb::...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7cd4caca1d19b0ab20067b88...- 02:21 AM Bug #56876 (New): crash: pread64()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=df09937ed2deb0251d707038...- 02:21 AM Bug #56875 (New): crash: BlueStore::Onode::put()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7362aab90e47739fdd28931f...- 02:21 AM Bug #56868 (New): crash: BlueStore::log_latency(char const*, int, std::chrono::duration<unsigned ...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=54840d56a916bf7899715821...- 02:21 AM Bug #56866 (New): crash: syscall()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=749895d8e1a0bce5973293f0...- 02:21 AM Bug #56859 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b1e23039e715e2ecdb7205a9...- 02:21 AM Bug #56857 (New): crash: BlueStore::Onode::put()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=2ecac47698645f1119e47c05...- 02:21 AM Bug #56854 (New): crash: std::unique_ptr<rocksdb::Block, std::default_delete<rocksdb::Block> >::g...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=32476aa8c69d6b64f8892f80...- 02:21 AM Bug #56853 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=f3d118a9511f61db150d57f1...- 02:21 AM Bug #56852 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7cc03cdc46ada57683950f95...- 02:21 AM Bug #56851 (New): crash: int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::rea...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=825b54b05f03883919a608a9...- 02:20 AM Bug #56846 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=3785747dae8308b15b6d1ba6...- 02:20 AM Bug #56845 (New): crash: int64_t BlueFS::_read(BlueFS::FileReader*, uint64_t, size_t, ceph::buffe...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=be248808680ca921b9f51d1c...- 02:20 AM Bug #56838 (New): crash: rocksdb_cache::BinnedLRUHandleTable::FindPointer(rocksdb::Slice const&, ...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=66fd05516a21298e1ffcdfae...- 02:20 AM Bug #56834 (New): crash: ceph::PluginRegistry::get_with_load(std::basic_string<char, std::char_tr...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b386053a897d1eda5fc2198d...- 02:20 AM Bug #56829 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=f22692305dea7c0ae3001472...- 02:20 AM Bug #56828 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=5085905d74b5df92d8f9b9ee...- 02:20 AM Bug #56827 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=09acbd6c37d2afeb74cb077d...- 02:20 AM Bug #56825 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7b983065a737287fbb375b9f...- 02:20 AM Bug #56824 (New): crash: BlueStore::_txc_create(BlueStore::Collection*, BlueStore::OpSequencer*, ...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=6e74e61390a32760454390e7...- 02:20 AM Bug #56823 (New): crash: rocksdb::crc32c::Extend(unsigned int, char const*, unsigned long)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=b4832aa999ec40f147de90f6...- 02:20 AM Bug #56820 (New): crash: ceph::buffer::ptr::iterator_impl<true>::operator
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=4d59e458cfce7d94cf5b78cc...- 02:20 AM Bug #56819 (New): crash: pread64()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=4aba058b736b569ffe07a2cc...- 02:19 AM Bug #56818 (New): crash: pread64()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=2a14d56e3767b24d0345d846...- 02:19 AM Bug #56817 (New): crash: pthread_rwlock_rdlock()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=35078a6b8d06a938ebabc486...- 02:19 AM Bug #56812 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=bb50bfecaa2cde07d1f60942...- 02:19 AM Bug #56807 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=959ca98b98161f22904cba6d...- 02:19 AM Bug #56806 (New): crash: void BlueStore::_do_write_small(BlueStore::TransContext*, BlueStore::Col...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=c6aab93a57fed4eff9ca5a21...- 02:19 AM Bug #56805 (New): crash: void BlueStore::_do_write_small(BlueStore::TransContext*, BlueStore::Col...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1c077f32e70f08fce239b6d3...- 02:19 AM Bug #56803 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=1562c3fb88628d327cf7bdad...- 02:19 AM Bug #56801 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=36aad439aac04165e757a34d...- 02:19 AM Bug #56800 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=0988c6e1fa90274309e110f0...- 02:18 AM Bug #56799 (New): crash: virtual int KernelDevice::read(uint64_t, uint64_t, ceph::bufferlist*, IO...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=275b1186a0c5a700ff350229...- 02:18 AM Bug #56798 (New): crash: HybridAllocator::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uin...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=ad1067445d53dcc60397ecf3...- 02:18 AM Bug #56795 (New): crash: void BlueStore::_do_write_small(BlueStore::TransContext*, BlueStore::Col...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=8bcd735b03b19a0102739f37...- 02:17 AM Bug #55187: ceph_abort_msg(\"bluefs enospc\")
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=80c8aa5fc17d3e51500a97e14...- 02:17 AM Bug #56791 (New): crash: bool SimpleBitmap::set(uint64_t, uint64_t): assert(offset + length < m_n...
*New crash events were reported via Telemetry with newer versions (['17.1.0']) than encountered in Tracker (0.0.0)....- 02:17 AM Bug #56790 (New): crash: virtual int KernelDevice::flush(): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=9aab05f010f83cdac4badc19...- 02:17 AM Bug #56788 (New): crash: void KernelDevice::_aio_thread(): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=c1fd5f4de1d415baece7e119...- 02:17 AM Bug #56784 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=e3abc74b74df1ddc00fb39b4...- 02:16 AM Bug #56783 (New): crash: virtual void KernelDevice::aio_submit(IOContext*): assert(r == 0)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=2005126bb96e53ddaba3cd74...- 02:16 AM Bug #56782 (New): crash: bool rocksdb::InlineSkipList<rocksdb::MemTableRep::KeyComparator const&>...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=4e383a9aa9a9b5ee75e87e66...- 02:16 AM Bug #56777 (New): crash: BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lam...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=993546d443261ef33ab16d28...- 02:16 AM Bug #56775 (New): crash: void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectSto...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a854bd98ce38205d9909358c...- 02:16 AM Bug #56771 (New): crash: rocksdb::crc32c::crc32c_3way(unsigned int, char const*, unsigned long)
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d56c8a2c9634cec7ad800505...- 02:16 AM Bug #56769 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=76e8becc0b90eacf59295290...- 02:16 AM Bug #56768 (New): crash: void BlueStore::_do_write_small(BlueStore::TransContext*, BlueStore::Col...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7c8a3b629e9ce58810f204de...- 02:16 AM Bug #56767 (New): crash: int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=21ef9f7b9207c8d4a97c92c5...- 02:16 AM Bug #56766 (New): crash: int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=de4f4fe5831504c2902d7467...- 02:15 AM Bug #56765 (New): crash: int64_t BlueFS::_read_random(BlueFS::FileReader*, uint64_t, uint64_t, ch...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=61793a7ae77a9f18bb1bc4d3...- 02:15 AM Bug #56763 (New): crash: int BlueStore::_do_gc(BlueStore::TransContext*, BlueStore::CollectionRef...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=003e6c8b8938e88311f0a28c...- 02:15 AM Bug #56762 (New): crash: virtual int RocksDBStore::get(const string&, const char*, size_t, ceph::...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=821d6920aa8c1012875df41d...- 02:15 AM Bug #56760 (New): crash: virtual int KernelDevice::flush(): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a142a97ca2b72bcecf8e9bb3...- 02:14 AM Bug #56758 (New): crash: int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t): abort
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a3bf1e9df33bea667de16747...- 02:14 AM Bug #56757 (New): crash: void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectSto...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=a85cb534c8bc7cd818e51dda...- 02:14 AM Bug #56753 (New): crash: virtual int RocksDBStore::get(const string&, const char*, size_t, ceph::...
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=d666fd63a7dfdb77830d510a...- 02:14 AM Bug #56751 (New): crash: pthread_cond_wait()
http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=c1f4d2190a3e2785c3d243a5...- 12:25 AM Bug #55328: OSD crashed due to checksum error
- > After my last report, I was suffering from our testing environment changes,
Unfortunately, since our test envir...
07/27/2022
- 10:43 AM Backport #56668 (In Progress): quincy: Deferred writes might cause "rocksdb: Corruption: Bad tabl...
07/26/2022
- 06:49 PM Bug #56706 (Closed): _do_thrash:test_bluestore_reshard:Exception: ceph-bluestore-tool resharding...
- closed because this is yet merged to main
- 03:55 PM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- /a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944045/
07/25/2022
- 09:09 PM Bug #56706: _do_thrash:test_bluestore_reshard:Exception: ceph-bluestore-tool resharding failed
- Dead job:
/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943775 - 08:10 PM Bug #56706 (Closed): _do_thrash:test_bluestore_reshard:Exception: ceph-bluestore-tool resharding...
- /a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6944099...
- 07:46 PM Bug #48819: fsck error: found stray (per-pg) omap data on omap_head
- /a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943842/
07/22/2022
- 04:18 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Sébastien Bernard wrote:
> We spent full last day to recover 4 osds that wouldn't come back up.
> We saw on the mac... - 03:16 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- We spent full last day to recover 4 osds that wouldn't come back up.
We saw on the machines a lot of errors like thi... - 11:45 AM Bug #56488 (Fix Under Review): BlueStore doesn't defer small writes for pre-pacific hdd osds
07/21/2022
- 10:55 PM Backport #56669 (Resolved): pacific: Deferred writes might cause "rocksdb: Corruption: Bad table ...
- https://github.com/ceph/ceph/pull/47296
- 10:55 PM Backport #56668 (Resolved): quincy: Deferred writes might cause "rocksdb: Corruption: Bad table m...
- https://github.com/ceph/ceph/pull/47297
- 10:54 PM Bug #54547 (Pending Backport): Deferred writes might cause "rocksdb: Corruption: Bad table magic ...
- 10:48 PM Bug #54547: Deferred writes might cause "rocksdb: Corruption: Bad table magic number"
- https://github.com/ceph/ceph/pull/46890 merged
07/20/2022
- 06:31 PM Bug #56503: Deleting large (~850gb) objects causes OSD to crash
- Do you have logs with @debug_bluestore=20@ maybe?
Mark, can this be related to the RocksDB's tombstones issues? - 04:58 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- There are two test cases that would be executed to find more details on what is going on with this feature and how it...
07/19/2022
- 09:07 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- Vikhyat Umrao wrote:
> This has been already verified that by default COSbench uses random not zero.
>
> COSBench... - 09:06 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- The `bluestore_zero_block_detection` was set to false in 17.2.1. For more details please check:
https://tracker.ce... - 08:55 PM Bug #56640: RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17.2.1
- This has been already verified that by default COSbench uses random not zero.
COSBench document - page 50 - https:... - 08:53 PM Bug #56640 (New): RGW S3 workload has a huge performance boost in quincy 17.2.0 as compared to 17...
- RGW S3 *small object* workload has a huge performance boost in the quincy 17.2.0 as compared to 17.2.1 due to bluesto...
- 04:33 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Well, this doesn't look like a bluestore issue anymore. Neither I can't see any clues it's the same as the original o...
- 12:53 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Igor Fedotov wrote:
> Hi!
>
> May I have some clarifications then.
> I can see multiple backtraces for .107 like... - 10:33 AM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Hi!
May I have some clarifications then.
I can see multiple backtraces for .107 like the last one. All are happen... - 08:29 AM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Aurélien Le Clainche wrote:
> Igor Fedotov wrote:
> > Aurélien Le Clainche wrote:
> > > Igor Fedotov wrote:
> > >...
07/18/2022
- 02:24 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Igor Fedotov wrote:
> Aurélien Le Clainche wrote:
> > Igor Fedotov wrote:
> > > Aurélien Le Clainche wrote:
> > >... - 01:58 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Aurélien Le Clainche wrote:
> Igor Fedotov wrote:
> > Aurélien Le Clainche wrote:
> > > do you have a procedure or... - 01:01 PM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Igor Fedotov wrote:
> Aurélien Le Clainche wrote:
> > do you have a procedure or example to do the fsck?
>
> Not... - 09:56 AM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Aurélien Le Clainche wrote:
> do you have a procedure or example to do the fsck?
Not sure how to do that with Roo... - 09:17 AM Bug #56456: rook-ceph-v1.9.5: ceph-osd crash randomly
- Sébastien Bernard wrote:
> Regarding this problem, /var filesystem is an xfs filesystem.
> osd are setup by rook.
... - 09:10 AM Backport #56600 (Rejected): octopus: octopus : bluestore_cache_other pool memory leak ?
- 09:10 AM Backport #56599 (Resolved): pacific: octopus : bluestore_cache_other pool memory leak ?
- 09:10 AM Backport #56598 (Resolved): quincy: octopus : bluestore_cache_other pool memory leak ?
- 09:07 AM Bug #56424 (Pending Backport): bluestore_cache_other mempool entry leak
- 09:05 AM Bug #56424: bluestore_cache_other mempool entry leak
- alexandre derumier wrote:
> Hi,
>
> I think it's fixing the problem.
>
> I'll keep it running, and I'll send n...
Also available in: Atom