Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2024-03-14T13:32:10Z
Ceph
Redmine
bluestore - Backport #64928 (In Progress): reef: KeyValueDB/KVTest.RocksDB_estimate_size tests fa...
https://tracker.ceph.com/issues/64928
2024-03-14T13:32:10Z
Backport Bot
ceph-volume - Bug #64260 (Pending Backport): ceph-volume lvm migrate could assert on AttributeErr...
https://tracker.ceph.com/issues/64260
2024-01-31T03:29:03Z
Igor Fedotov
igor.fedotov@croit.io
<p>Full back trace is:</p>
<pre>
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 542, in main
self.migrate_osd()
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 448, in migrate_osd
target_lv)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 384, in migrate_to_existing
tag_tracker.remove_lvs(source_devices, target_type)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 178, in remove_lvs
remaining_devices.remove(device)
File "/usr/lib/python3.6/site-packages/ceph_volume/util/device.py", line 170, in __eq__
return self.path == other.path
AttributeError: 'NoneType' object has no attribute 'path'
</pre>
bluestore - Backport #64071 (In Progress): reef: src/common/PriorityCache.cc: FAILED ceph_assert(...
https://tracker.ceph.com/issues/64071
2024-01-18T02:05:09Z
Backport Bot
bluestore - Backport #63813 (In Progress): quincy: ObjectStore/StoreTestSpecificAUSize.SpilloverF...
https://tracker.ceph.com/issues/63813
2023-12-13T15:51:56Z
Backport Bot
bluestore - Backport #63812 (In Progress): reef: ObjectStore/StoreTestSpecificAUSize.SpilloverFix...
https://tracker.ceph.com/issues/63812
2023-12-13T15:51:49Z
Backport Bot
RADOS - Backport #63400 (In Progress): reef: pybind: ioctx.get_omap_keys asserts if start_after p...
https://tracker.ceph.com/issues/63400
2023-11-02T12:53:34Z
Backport Bot
RADOS - Backport #63399 (In Progress): quincy: pybind: ioctx.get_omap_keys asserts if start_after...
https://tracker.ceph.com/issues/63399
2023-11-02T12:53:26Z
Backport Bot
bluestore - Backport #62781 (New): reef: btree allocator doesn't pass alloctor's UTs
https://tracker.ceph.com/issues/62781
2023-09-09T03:04:40Z
Backport Bot
bluestore - Backport #62780 (New): quincy: btree allocator doesn't pass alloctor's UTs
https://tracker.ceph.com/issues/62780
2023-09-09T03:04:33Z
Backport Bot
bluestore - Bug #61949 (Pending Backport): btree allocator doesn't pass alloctor's UTs
https://tracker.ceph.com/issues/61949
2023-07-10T21:20:44Z
Igor Fedotov
igor.fedotov@croit.io
[ RUN ] Allocator/AllocTest.test_alloc_non_aligned_len/3<br />Creating alloc type btree<br />/home/if/ceph.3/src/os/bluestore/BtreeAllocator.cc: In function 'void BtreeAllocator::_remove_from_tree(uint64_t, uint64_t)' thread 7f805aee6100 time 2023-07-11T00:20:11.256261+0300<br />/home/if/ceph.3/src/os/bluestore/BtreeAllocator.cc: 172: FAILED ceph_assert(rs != range_tree.end())<br /> ceph version Development (no_version) reef (dev)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f805c8ebf97]<br /> 2: /home/if/ceph.3/build/lib/libceph-common.so.2(+0x29e1a4) [0x7f805c8ec1a4]<br /> 3: (BtreeAllocator::_remove_from_tree(unsigned long, unsigned long)+0x106) [0x5559935ee196]<br /> 4: (BtreeAllocator::_allocate(unsigned long, unsigned long, unsigned long*, unsigned long*)+0x2fe) [0x5559935f085e]<br /> 5: (BtreeAllocator::_allocate(unsigned long, unsigned long, unsigned long, long, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> ><strong>)+0xa2) [0x5559935f0d72]<br /> 6: (BtreeAllocator::allocate(unsigned long, unsigned long, unsigned long, long, std::vector<bluestore_pextent_t, mempool::pool_allocator<(mempool::pool_index_t)5, bluestore_pextent_t> ></strong>)+0xa7) [0x5559935f10a7]<br /> 7: (AllocTest_test_alloc_non_aligned_len_Test::TestBody()+0xb8) [0x55599357eb08]<br /> 8: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x4d) [0x5559935cb1bd]<br /> 9: (testing::Test::Run()+0xce) [0x5559935be8be]<br /> 10: (testing::TestInfo::Run()+0x135) [0x5559935bea15]<br /> 11: (testing::TestSuite::Run()+0xc9) [0x5559935beb09]<br /> 12: (testing::internal::UnitTestImpl::RunAllTests()+0x49c) [0x5559935bf08c]<br /> 13: (testing::UnitTest::Run()+0x82) [0x5559935bf342]<br /> 14: main()<br /> 15: /lib64/libc.so.6(+0x275b0) [0x7f805bbb95b0]<br /> 16: __libc_start_main()<br /> 17: _start()
<ul>
<li>Caught signal (Aborted) **</li>
</ul>
RADOS - Bug #58304 (Pending Backport): pybind: ioctx.get_omap_keys asserts if start_after paramet...
https://tracker.ceph.com/issues/58304
2022-12-16T16:29:11Z
Igor Fedotov
igor.fedotov@croit.io
bluestore - Bug #54555 (Fix Under Review): ceph-bluestore-tool doesn't handle 'free-fragmentation...
https://tracker.ceph.com/issues/54555
2022-03-14T18:16:43Z
Igor Fedotov
igor.fedotov@croit.io
bluestore - Bug #50992 (Pending Backport): src/common/PriorityCache.cc: FAILED ceph_assert(mem_av...
https://tracker.ceph.com/issues/50992
2021-05-27T08:29:33Z
Jiang Yu
lnsyyj@hotmail.com
<p>When I use ceph-volume to add a new OSD, the following error appears</p>
<pre>
root@yujiang-performance-dev:~/github/ceph/build# ceph-volume lvm create --bluestore --data /dev/nvme0n1
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 88631385-1de2-4168-9c4e-8b3b3e292355
Running command: /sbin/vgcreate --force --yes ceph-f32a7f97-3199-4cce-80f1-31923e35ca1f /dev/nvme0n1
stdout: Physical volume "/dev/nvme0n1" successfully created.
stdout: Volume group "ceph-f32a7f97-3199-4cce-80f1-31923e35ca1f" successfully created
Running command: /sbin/lvcreate --yes -l 476932 -n osd-block-88631385-1de2-4168-9c4e-8b3b3e292355 ceph-f32a7f97-3199-4cce-80f1-31923e35ca1f
stdout: Logical volume "osd-block-88631385-1de2-4168-9c4e-8b3b3e292355" created.
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
--> Executable selinuxenabled not in PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
Running command: /bin/chown -h ceph:ceph /dev/ceph-f32a7f97-3199-4cce-80f1-31923e35ca1f/osd-block-88631385-1de2-4168-9c4e-8b3b3e292355
Running command: /bin/chown -R ceph:ceph /dev/dm-0
Running command: /bin/ln -s /dev/ceph-f32a7f97-3199-4cce-80f1-31923e35ca1f/osd-block-88631385-1de2-4168-9c4e-8b3b3e292355 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 1
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQAPVq9gdL/3MxAALNvWGRgndR/7H0i/cc6b+g==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQAPVq9gdL/3MxAALNvWGRgndR/7H0i/cc6b+g==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 88631385-1de2-4168-9c4e-8b3b3e292355 --setuser ceph --setgroup ceph
stderr: 2021-05-27T08:19:29.280+0000 7f6f2beb3f00 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f6f0a079700 time 2021-05-27T08:19:30.129967+0000
stderr: /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0)
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x5650082ff179]
stderr: 2: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 3: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 4: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 6: clone()
stderr: *** Caught signal (Aborted) **
stderr: in thread 7f6f0a079700 thread_name:bstore_mempool
stderr: 2021-05-27T08:19:30.136+0000 7f6f0a079700 -1 /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f6f0a079700 time 2021-05-27T08:19:30.129967+0000
stderr: /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0)
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x5650082ff179]
stderr: 2: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 3: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 4: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 6: clone()
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6f2a01d980]
stderr: 2: gsignal()
stderr: 3: abort()
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x5650082ff1d4]
stderr: 5: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 6: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 7: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 9: clone()
stderr: 2021-05-27T08:19:30.140+0000 7f6f0a079700 -1 *** Caught signal (Aborted) **
stderr: in thread 7f6f0a079700 thread_name:bstore_mempool
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6f2a01d980]
stderr: 2: gsignal()
stderr: 3: abort()
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x5650082ff1d4]
stderr: 5: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 6: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 7: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 9: clone()
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: -2827> 2021-05-27T08:19:29.280+0000 7f6f2beb3f00 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: -1> 2021-05-27T08:19:30.136+0000 7f6f0a079700 -1 /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: In function 'void PriorityCache::Manager::balance()' thread 7f6f0a079700 time 2021-05-27T08:19:30.129967+0000
stderr: /tmp/release/Ubuntu/WORKDIR/ceph-16.2.4-1-g574376ae42/src/common/PriorityCache.cc: 301: FAILED ceph_assert(mem_avail >= 0)
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x5650082ff179]
stderr: 2: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 3: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 4: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 6: clone()
stderr: 0> 2021-05-27T08:19:30.140+0000 7f6f0a079700 -1 *** Caught signal (Aborted) **
stderr: in thread 7f6f0a079700 thread_name:bstore_mempool
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6f2a01d980]
stderr: 2: gsignal()
stderr: 3: abort()
stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x5650082ff1d4]
stderr: 5: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 6: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 7: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 8: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 9: clone()
stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
stderr: *** Caught signal (Segmentation fault) **
stderr: in thread 7f6f0a079700 thread_name:bstore_mempool
stderr: ceph version 16.2.4-1-g574376ae42 (574376ae42a35e132c77048bed3f4e36fc7cf68c) pacific (stable)
stderr: 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6f2a01d980]
stderr: 2: pthread_getname_np()
stderr: 3: (ceph::logging::Log::dump_recent()+0x510) [0x565008cc5920]
stderr: 4: /usr/bin/ceph-osd(+0x127bc40) [0x565008ab7c40]
stderr: 5: /lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7f6f2a01d980]
stderr: 6: gsignal()
stderr: 7: abort()
stderr: 8: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x5650082ff1d4]
stderr: 9: /usr/bin/ceph-osd(+0xac3363) [0x5650082ff363]
stderr: 10: (PriorityCache::Manager::balance()+0x444) [0x565008f43164]
stderr: 11: (BlueStore::MempoolThread::entry()+0x831) [0x565008992df1]
stderr: 12: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6f2a0126db]
stderr: 13: clone()
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
--> RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 88631385-1de2-4168-9c4e-8b3b3e292355 --setuser ceph --setgroup ceph
</pre>
bluestore - Bug #40741 (Triaged): Mass OSD failure, unable to restart
https://tracker.ceph.com/issues/40741
2019-07-11T19:26:08Z
Brett Chancellor
<p>Cluster: 14.2.1<br />OSDs: 250 spinners in default root, 63 SSDs in ssd root</p>
<p>History: 5 days ago, this cluster began losing spinning drives every couple of hours. Many of them were unable to be restarted once they went down, so they had to be rebuilt. After reaching out to the ceph users group we tried setting the bluestore_allocator and bluesfs_allocator to stupid. This allowed newly dying OSDs to be brought back online, although it didn't stop others from dying. Once the cluster finished rebalancing the performance was terrible (disks all idle and error free, individual clients getting 1-2 iops with 12k ms latency) with either allocator, We did note that performance was fine on any root other than the default root. OSDs continued to commit suicide every 30-40 minutes. In an attempt to improve performance we decided to try and move the <zone>.rgw.meta pool from spinning drives to SSD.</p>
<p>After doing this SSD's began to fail in mass. We are unable bring up the SSD OSDs with either the stupid or bitmap allocators.</p>
<p>Attached is an osd with debug_bluestore 10 set.</p>
<p>Current config<br />$ sudo ceph config dump<br />WHO MASK LEVEL OPTION VALUE RO <br />global advanced bluestore_warn_on_bluefs_spillover false <br />global advanced mon_warn_pg_not_deep_scrubbed_ratio 0.000000 <br />global advanced mon_warn_pg_not_scrubbed_ratio 0.000000 <br />global advanced osd_deep_scrub_interval 1814400.000000 <br />global advanced osd_scrub_max_interval 1814400.000000 <br />global advanced osd_scrub_min_interval 259200.000000 <br /> mon advanced mon_osd_down_out_interval 1200 <br /> mon.ceph0rdi-mon1-1-prd advanced ms_bind_msgr2 false <br /> mon.ceph0rdi-mon2-1-prd advanced ms_bind_msgr2 false <br /> mon.ceph0rdi-mon3-1-prd advanced ms_bind_msgr2 false <br /> osd advanced bluestore_bluefs_gift_ratio 0.000200 <br /> osd advanced osd_max_backfills 3 <br /> osd basic osd_memory_target 4294967296 <br /> osd advanced osd_op_thread_timeout 90 <br /> osd advanced osd_recovery_sleep_hdd 0.000000 <br /> osd advanced osd_recovery_sleep_hybrid 0.000000</p>
<p>Example stack trace:<br /> -1> 2019-07-11 18:59:24.296 7fcc1caf5d80 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7fcc1caf5d80 time 2019-07-11 18:59:24.289089<br />/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.1/rpm/el7/BUILD/ceph-14.2.1/src/os/bluestore/BlueFS.cc: 2044: abort()</p>
<pre><code>ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br /> 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xd8) [0x560dbdbc9cd0]<br /> 2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1daf) [0x560dbe2819af]<br /> 3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x560dbe281b4b]<br /> 4: (BlueRocksWritableFile::Flush()+0x3d) [0x560dbe29f84d]<br /> 5: (rocksdb::WritableFileWriter::Flush()+0x19e) [0x560dbe8cdd0e]<br /> 6: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x560dbe8cdfee]<br /> 7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt;*, std::unique_ptr&lt;rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt;, std::default_delete&lt;rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt; > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector&lt;std::unique_ptr&lt;rocksdb::IntTblPropCollectorFactory, std::default_delete&lt;rocksdb::IntTblPropCollectorFactory&gt; >, std::allocator&lt;std::unique_ptr&lt;rocksdb::IntTblPropCollectorFactory, std::default_delete&lt;rocksdb::IntTblPropCollectorFactory&gt; > > > const*, unsigned int, std::string const&, std::vector&lt;unsigned long, std::allocator&lt;unsigned long&gt; >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x2368) [0x560dbe8fb978]<br /> 8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xc66) [0x560dbe7716a6]<br /> 9: (rocksdb::DBImpl::RecoverLogFiles(std::vector&lt;unsigned long, std::allocator&lt;unsigned long&gt; > const&, unsigned long*, bool)+0x1672) [0x560dbe7735a2]<br /> 10: (rocksdb::DBImpl::Recover(std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, bool, bool, bool)+0x809) [0x560dbe774b99]<br /> 11: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; ><strong>, rocksdb::DB</strong>*, bool, bool)+0x658) [0x560dbe7759a8]<br /> 12: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; ><strong>, rocksdb::DB</strong>*)+0x24) [0x560dbe777184]<br /> 13: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector&lt;KeyValueDB::ColumnFamily, std::allocator&lt;KeyValueDB::ColumnFamily&gt; > const*)+0x1660) [0x560dbe20bde0]<br /> 14: (BlueStore::_open_db(bool, bool, bool)+0xf8e) [0x560dbe16077e]<br /> 15: (BlueStore::_open_db_and_around(bool)+0x165) [0x560dbe17dcb5]<br /> 16: (BlueStore::_mount(bool, bool)+0x6a4) [0x560dbe1ba694]<br /> 17: (OSD::init()+0x3aa) [0x560dbdd30d7a]<br /> 18: (main()+0x14fa) [0x560dbdbcd1da]<br /> 19: (__libc_start_main()+0xf5) [0x7fcc185283d5]<br /> 20: (()+0x564555) [0x560dbdcc1555]</code></pre>
<pre><code>0> 2019-07-11 18:59:24.304 7fcc1caf5d80 -1 *<strong>* Caught signal (Aborted) *</strong><br /> in thread 7fcc1caf5d80 thread_name:ceph-osd</code></pre>
<pre><code>ceph version 14.2.1 (d555a9489eb35f84f2e1ef49b77e19da9d113972) nautilus (stable)<br /> 1: (()+0xf5d0) [0x7fcc197455d0]<br /> 2: (gsignal()+0x37) [0x7fcc1853c207]<br /> 3: (abort()+0x148) [0x7fcc1853d8f8]<br /> 4: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x19c) [0x560dbdbc9d94]<br /> 5: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1daf) [0x560dbe2819af]<br /> 6: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x10b) [0x560dbe281b4b]<br /> 7: (BlueRocksWritableFile::Flush()+0x3d) [0x560dbe29f84d]<br /> 8: (rocksdb::WritableFileWriter::Flush()+0x19e) [0x560dbe8cdd0e]<br /> 9: (rocksdb::WritableFileWriter::Sync(bool)+0x2e) [0x560dbe8cdfee]<br /> 10: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt;*, std::unique_ptr&lt;rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt;, std::default_delete&lt;rocksdb::InternalIteratorBase&lt;rocksdb::Slice&gt; > >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector&lt;std::unique_ptr&lt;rocksdb::IntTblPropCollectorFactory, std::default_delete&lt;rocksdb::IntTblPropCollectorFactory&gt; >, std::allocator&lt;std::unique_ptr&lt;rocksdb::IntTblPropCollectorFactory, std::default_delete&lt;rocksdb::IntTblPropCollectorFactory&gt; > > > const*, unsigned int, std::string const&, std::vector&lt;unsigned long, std::allocator&lt;unsigned long&gt; >, unsigned long, rocksdb::SnapshotChecker*, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int, unsigned long, unsigned long, rocksdb::Env::WriteLifeTimeHint)+0x2368) [0x560dbe8fb978]<br /> 11: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0xc66) [0x560dbe7716a6]<br /> 12: (rocksdb::DBImpl::RecoverLogFiles(std::vector&lt;unsigned long, std::allocator&lt;unsigned long&gt; > const&, unsigned long*, bool)+0x1672) [0x560dbe7735a2]<br /> 13: (rocksdb::DBImpl::Recover(std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, bool, bool, bool)+0x809) [0x560dbe774b99]<br /> 14: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::string const&, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; ><strong>, rocksdb::DB</strong>*, bool, bool)+0x658) [0x560dbe7759a8]<br /> 15: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector&lt;rocksdb::ColumnFamilyDescriptor, std::allocator&lt;rocksdb::ColumnFamilyDescriptor&gt; > const&, std::vector&lt;rocksdb::ColumnFamilyHandle*, std::allocator&lt;rocksdb::ColumnFamilyHandle*&gt; ><strong>, rocksdb::DB</strong>*)+0x24) [0x560dbe777184]<br /> 16: (RocksDBStore::do_open(std::ostream&, bool, bool, std::vector&lt;KeyValueDB::ColumnFamily, std::allocator&lt;KeyValueDB::ColumnFamily&gt; > const*)+0x1660) [0x560dbe20bde0]<br /> 17: (BlueStore::_open_db(bool, bool, bool)+0xf8e) [0x560dbe16077e]<br /> 18: (BlueStore::_open_db_and_around(bool)+0x165) [0x560dbe17dcb5]<br /> 19: (BlueStore::_mount(bool, bool)+0x6a4) [0x560dbe1ba694]<br /> 20: (OSD::init()+0x3aa) [0x560dbdd30d7a]<br /> 21: (main()+0x14fa) [0x560dbdbcd1da]<br /> 22: (__libc_start_main()+0xf5) [0x7fcc185283d5]<br /> 23: (()+0x564555) [0x560dbdcc1555]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
bluestore - Bug #20236 (New): bluestore: ObjectStore/StoreTestSpecificAUSize.Many4KWritesNoCSumTe...
https://tracker.ceph.com/issues/20236
2017-06-09T15:36:29Z
Sage Weil
sage@newdream.net
<pre>
2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:seeding object 0
2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:Op 0
2017-06-09T03:37:28.314 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:37:41.290 INFO:teuthology.orchestra.run.smithi068.stderr:Op 200
2017-06-09T03:37:41.291 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:37:55.988 INFO:teuthology.orchestra.run.smithi068.stderr:Op 400
2017-06-09T03:37:55.989 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:12.470 INFO:teuthology.orchestra.run.smithi068.stderr:Op 600
2017-06-09T03:38:12.470 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:31.486 INFO:teuthology.orchestra.run.smithi068.stderr:Op 800
2017-06-09T03:38:31.486 INFO:teuthology.orchestra.run.smithi068.stderr:available_objects: 0 in_flight_objects: 1 total objects: 1 in_flight 1
2017-06-09T03:38:52.763 INFO:teuthology.orchestra.run.smithi068.stdout:/build/ceph-12.0.2-2533-g080b00b/src/test/objectstore/store_test.cc:5729: Failure
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout: Expected: res_stat.allocated
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout: Which is: 4259840
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout:To be equal to: max_object
2017-06-09T03:38:52.764 INFO:teuthology.orchestra.run.smithi068.stdout: Which is: 4194304
</pre><br />/a/yuriw-2017-06-08_19:40:51-rados-wip-yuri-testing2_2017_7_10-distro-basic-smithi/1271584