-- Logs begin at Thu 2017-07-06 16:44:09 PDT, end at Fri 2017-07-07 00:10:13 PDT. -- Jul 07 00:03:36 hostname ceph-osd[14090]: *** Caught signal (Aborted) ** Jul 07 00:03:36 hostname ceph-osd[14090]: in thread 7fa734b32700 thread_name:tp_osd_tp Jul 07 00:03:36 hostname ceph-osd[14090]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) Jul 07 00:03:36 hostname ceph-osd[14090]: 1: (()+0x95cd16) [0x55eeb36d4d16] Jul 07 00:03:36 hostname ceph-osd[14090]: 2: (()+0x11940) [0x7fa7509ca940] Jul 07 00:03:36 hostname ceph-osd[14090]: 3: (pthread_cond_wait()+0x1fd) [0x7fa7509c639d] Jul 07 00:03:36 hostname ceph-osd[14090]: 4: (Throttle::_wait(long)+0x33e) [0x55eeb370f38e] Jul 07 00:03:36 hostname ceph-osd[14090]: 5: (Throttle::get(long, long)+0x2a2) [0x55eeb3710102] Jul 07 00:03:36 hostname ceph-osd[14090]: 6: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0xde4) [0x55eeb35dc684] Jul 07 00:03:36 hostname ceph-osd[14090]: 7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x68) [0x55eeb332c2c8] Jul 07 00:03:36 hostname ceph-osd[14090]: 8: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr, ECSubWrite&, ZTracer::Trace const&, Context*)+0x960) [0x55eeb3455ca0] Jul 07 00:03:36 hostname ceph-osd[14090]: 9: (ECBackend::handle_message(boost::intrusive_ptr)+0x321) [0x55eeb346dd21] Jul 07 00:03:36 hostname ceph-osd[14090]: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x64a) [0x55eeb32d437a] Jul 07 00:03:36 hostname ceph-osd[14090]: 11: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1bc) [0x55eeb317080c] Jul 07 00:03:36 hostname ceph-osd[14090]: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr const&)+0x5a) [0x55eeb3170c3a] Jul 07 00:03:36 hostname ceph-osd[14090]: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1ae7) [0x55eeb3194037] Jul 07 00:03:36 hostname ceph-osd[14090]: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x672) [0x55eeb371e3d2] Jul 07 00:03:36 hostname ceph-osd[14090]: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55eeb3721b70] Jul 07 00:03:36 hostname ceph-osd[14090]: 16: (()+0x7297) [0x7fa7509c0297] Jul 07 00:03:36 hostname ceph-osd[14090]: 17: (clone()+0x3f) [0x7fa74fe501ef] Jul 07 00:03:36 hostname ceph-osd[14090]: 2017-07-07 00:03:36.645120 7fa734b32700 -1 *** Caught signal (Aborted) ** Jul 07 00:03:36 hostname ceph-osd[14090]: in thread 7fa734b32700 thread_name:tp_osd_tp Jul 07 00:03:36 hostname ceph-osd[14090]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) Jul 07 00:03:36 hostname ceph-osd[14090]: 1: (()+0x95cd16) [0x55eeb36d4d16] Jul 07 00:03:36 hostname ceph-osd[14090]: 2: (()+0x11940) [0x7fa7509ca940] Jul 07 00:03:36 hostname ceph-osd[14090]: 3: (pthread_cond_wait()+0x1fd) [0x7fa7509c639d] Jul 07 00:03:36 hostname ceph-osd[14090]: 4: (Throttle::_wait(long)+0x33e) [0x55eeb370f38e] Jul 07 00:03:36 hostname ceph-osd[14090]: 5: (Throttle::get(long, long)+0x2a2) [0x55eeb3710102] Jul 07 00:03:36 hostname ceph-osd[14090]: 6: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0xde4) [0x55eeb35dc684] Jul 07 00:03:36 hostname ceph-osd[14090]: 7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x68) [0x55eeb332c2c8] Jul 07 00:03:36 hostname ceph-osd[14090]: 8: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr, ECSubWrite&, ZTracer::Trace const&, Context*)+0x960) [0x55eeb3455ca0] Jul 07 00:03:36 hostname ceph-osd[14090]: 9: (ECBackend::handle_message(boost::intrusive_ptr)+0x321) [0x55eeb346dd21] Jul 07 00:03:36 hostname ceph-osd[14090]: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x64a) [0x55eeb32d437a] Jul 07 00:03:36 hostname ceph-osd[14090]: 11: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1bc) [0x55eeb317080c] Jul 07 00:03:36 hostname ceph-osd[14090]: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr const&)+0x5a) [0x55eeb3170c3a] Jul 07 00:03:36 hostname ceph-osd[14090]: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1ae7) [0x55eeb3194037] Jul 07 00:03:36 hostname ceph-osd[14090]: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x672) [0x55eeb371e3d2] Jul 07 00:03:36 hostname ceph-osd[14090]: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55eeb3721b70] Jul 07 00:03:36 hostname ceph-osd[14090]: 16: (()+0x7297) [0x7fa7509c0297] Jul 07 00:03:36 hostname ceph-osd[14090]: 17: (clone()+0x3f) [0x7fa74fe501ef] Jul 07 00:03:36 hostname ceph-osd[14090]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. Jul 07 00:03:36 hostname ceph-osd[14090]: 0> 2017-07-07 00:03:36.645120 7fa734b32700 -1 *** Caught signal (Aborted) ** Jul 07 00:03:36 hostname ceph-osd[14090]: in thread 7fa734b32700 thread_name:tp_osd_tp Jul 07 00:03:36 hostname ceph-osd[14090]: ceph version 12.1.0 (262617c9f16c55e863693258061c5b25dea5b086) luminous (dev) Jul 07 00:03:36 hostname ceph-osd[14090]: 1: (()+0x95cd16) [0x55eeb36d4d16] Jul 07 00:03:36 hostname ceph-osd[14090]: 2: (()+0x11940) [0x7fa7509ca940] Jul 07 00:03:36 hostname ceph-osd[14090]: 3: (pthread_cond_wait()+0x1fd) [0x7fa7509c639d] Jul 07 00:03:36 hostname ceph-osd[14090]: 4: (Throttle::_wait(long)+0x33e) [0x55eeb370f38e] Jul 07 00:03:36 hostname ceph-osd[14090]: 5: (Throttle::get(long, long)+0x2a2) [0x55eeb3710102] Jul 07 00:03:36 hostname ceph-osd[14090]: 6: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector >&, boost::intrusive_ptr, ThreadPool::TPHandle*)+0xde4) [0x55eeb35dc684] Jul 07 00:03:36 hostname ceph-osd[14090]: 7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector >&, boost::intrusive_ptr)+0x68) [0x55eeb332c2c8] Jul 07 00:03:36 hostname ceph-osd[14090]: 8: (ECBackend::handle_sub_write(pg_shard_t, boost::intrusive_ptr, ECSubWrite&, ZTracer::Trace const&, Context*)+0x960) [0x55eeb3455ca0] Jul 07 00:03:36 hostname ceph-osd[14090]: 9: (ECBackend::handle_message(boost::intrusive_ptr)+0x321) [0x55eeb346dd21] Jul 07 00:03:36 hostname ceph-osd[14090]: 10: (PrimaryLogPG::do_request(boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x64a) [0x55eeb32d437a] Jul 07 00:03:36 hostname ceph-osd[14090]: 11: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1bc) [0x55eeb317080c] Jul 07 00:03:36 hostname ceph-osd[14090]: 12: (PGQueueable::RunVis::operator()(boost::intrusive_ptr const&)+0x5a) [0x55eeb3170c3a] Jul 07 00:03:36 hostname ceph-osd[14090]: 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1ae7) [0x55eeb3194037] Jul 07 00:03:36 hostname ceph-osd[14090]: 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x672) [0x55eeb371e3d2] Jul 07 00:03:36 hostname ceph-osd[14090]: 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55eeb3721b70] Jul 07 00:03:36 hostname ceph-osd[14090]: 16: (()+0x7297) [0x7fa7509c0297] Jul 07 00:03:36 hostname ceph-osd[14090]: 17: (clone()+0x3f) [0x7fa74fe501ef] Jul 07 00:03:36 hostname ceph-osd[14090]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. Jul 07 00:03:48 hostname systemd[1]: ceph-osd@3.service: Main process exited, code=killed, status=6/ABRT Jul 07 00:03:48 hostname systemd[1]: ceph-osd@3.service: Unit entered failed state. Jul 07 00:03:48 hostname systemd[1]: ceph-osd@3.service: Failed with result 'signal'. Jul 07 00:04:09 hostname systemd[1]: ceph-osd@3.service: Service hold-off time over, scheduling restart. Jul 07 00:04:09 hostname systemd[1]: Stopped Ceph object storage daemon osd.3. Jul 07 00:04:09 hostname systemd[1]: Started Ceph object storage daemon osd.3. Jul 07 00:04:09 hostname ceph-osd[28543]: 2017-07-07 00:04:09.094821 7fa000d4fc80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb Jul 07 00:04:09 hostname ceph-osd[28543]: 2017-07-07 00:04:09.094950 7fa000d4fc80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb Jul 07 00:04:09 hostname ceph-osd[28543]: starting osd.3 at - osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal Jul 07 00:04:09 hostname ceph-osd[28543]: 2017-07-07 00:04:09.101776 7fa000d4fc80 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb Jul 07 00:04:39 hostname ceph-osd[28543]: 2017-07-07 00:04:39.395912 7fa000d4fc80 -1 osd.3 1668 log_to_monitors {default=true} Jul 07 00:04:40 hostname ceph-osd[28543]: 2017-07-07 00:04:40.015385 7f9fe6889700 -1 distro_detect - can't detect distro_version