Bug #7735: osd: priorityqueue debug dump crashes - Ceph - Ceph

Actions

Copy link

Bug #7735

closed

osd: priorityqueue debug dump crashes

Added by Mark Nelson about 10 years ago. Updated about 10 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Mark Nelson

Category:

Target version:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

discovered when doing backfill testing with wip-pq:

     0> 2014-03-14 21:52:47.988112 7fc425a3e700 -1 ./common/PrioritizedQueue.h: In function 'std::pair<unsigned int, T> PrioritizedQueue<T, K>::SubQueue::front() const [with T = std::pair<boost::intrusiv
e_ptr<PG>, std::tr1::shared_ptr<OpRequest> >; K = entity_inst_t]' thread 7fc425a3e700 time 2014-03-14 21:52:47.959021
./common/PrioritizedQueue.h: 130: FAILED assert(cur != q.end())

 ceph version 0.77-869-gd0f9dd6 (d0f9dd6b31c48a5850743fe0d3a40d8dc1029583)
 1: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x848) [0x61e8d8]
 2: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0x9c) [0x6608cc]
 3: (ThreadPool::worker(ThreadPool::WorkThread*)+0xaef) [0xa321af]
 4: (ThreadPool::WorkThread::entry()+0x10) [0xa330a0]
 5: (()+0x7f6e) [0x7fc43cf4bf6e]
 6: (clone()+0x6d) [0x7fc43b4f09cd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /tmp/cbt/ceph/log/osd.9.log
--- end dump of recent events ---
2014-03-14 21:52:48.108150 7fc425a3e700 -1 *** Caught signal (Aborted) **
 in thread 7fc425a3e700

 ceph version 0.77-869-gd0f9dd6 (d0f9dd6b31c48a5850743fe0d3a40d8dc1029583)
 1: ceph-osd() [0x961c2f]
 2: (()+0xfbb0) [0x7fc43cf53bb0]
 3: (gsignal()+0x37) [0x7fc43b42cf77]
 4: (abort()+0x148) [0x7fc43b4305e8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc43bd386e5]
 6: (()+0x5e856) [0x7fc43bd36856]
 7: (()+0x5e883) [0x7fc43bd36883]
 8: (()+0x5eaae) [0x7fc43bd36aae]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1f2) [0xa41382]
 10: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x848) [0x61e8d8]
 11: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0x9c) [0x6608cc]
 12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xaef) [0xa321af]
 13: (ThreadPool::WorkThread::entry()+0x10) [0xa330a0]
 14: (()+0x7f6e) [0x7fc43cf4bf6e]
 15: (clone()+0x6d) [0x7fc43b4f09cd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
     0> 2014-03-14 21:52:48.108150 7fc425a3e700 -1 *** Caught signal (Aborted) **
 in thread 7fc425a3e700

 ceph version 0.77-869-gd0f9dd6 (d0f9dd6b31c48a5850743fe0d3a40d8dc1029583)
1: ceph-osd() [0x961c2f]
 2: (()+0xfbb0) [0x7fc43cf53bb0]
 3: (gsignal()+0x37) [0x7fc43b42cf77]
 4: (abort()+0x148) [0x7fc43b4305e8]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fc43bd386e5]
 6: (()+0x5e856) [0x7fc43bd36856]
 7: (()+0x5e883) [0x7fc43bd36883]
 8: (()+0x5eaae) [0x7fc43bd36aae]
 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1f2) [0xa41382]
 10: (OSD::OpWQ::_process(boost::intrusive_ptr<PG>, ThreadPool::TPHandle&)+0x848) [0x61e8d8]
 11: (ThreadPool::WorkQueueVal<std::pair<boost::intrusive_ptr<PG>, std::tr1::shared_ptr<OpRequest> >, boost::intrusive_ptr<PG> >::_void_process(void*, ThreadPool::TPHandle&)+0x9c) [0x6608cc]
 12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xaef) [0xa321af]
 13: (ThreadPool::WorkThread::entry()+0x10) [0xa330a0]
 14: (()+0x7f6e) [0x7fc43cf4bf6e]
 15: (clone()+0x6d) [0x7fc43b4f09cd]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   0/ 0 mds
   0/ 0 mds_balancer
   0/ 0 mds_locker
   0/ 0 mds_log
   0/ 0 mds_log_expire
   0/ 0 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 0 filer
   0/ 1 striper
   0/ 0 objecter
   0/ 0 rados
   0/ 0 rbd
   0/ 0 journaler
   0/ 0 objectcacher
   0/ 0 client
   0/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   1/ 3 keyvaluestore
   0/ 0 journal
   0/ 0 ms
   0/ 0 mon
   0/ 0 monc
   0/ 0 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   0/ 0 rgw
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /tmp/cbt/ceph/log/osd.9.log
--- end dump of recent events ---