Project

General

Profile

Bug #20273

osd/OSD.h: 1957: FAILED assert(peerin g_queue.empty())

Added by Sage Weil over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Category:
Peering
Target version:
-
Start date:
06/12/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:

Description

2017-06-12T21:39:26.361 INFO:teuthology.orchestra.run.smithi136:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight'
2017-06-12T21:39:26.432 INFO:tasks.ceph.osd.1.smithi136.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.3-1508-g9960ae3/rpm/el7/BUILD/ceph-12.0.3-1508-g9960ae3/src/osd/OSD.h: In function 'virtual void 
OSD::PeeringWQ::_clear()' thread 355ce700 time 2017-06-12 21:39:26.433268
2017-06-12T21:39:26.432 INFO:tasks.ceph.osd.1.smithi136.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.0.3-1508-g9960ae3/rpm/el7/BUILD/ceph-12.0.3-1508-g9960ae3/src/osd/OSD.h: 1957: FAILED assert(peerin
g_queue.empty())
2017-06-12T21:39:26.444 INFO:tasks.ceph.osd.2.smithi136.stderr:2017-06-12 21:39:26.445498 355ce700 -1 received  signal: Hangup from  PID: 3142 task name: /usr/bin/python /bin/daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.
supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --tool=memcheck ceph-osd -f --cluster ceph -i 2  UID: 0
2017-06-12T21:39:26.454 INFO:teuthology.orchestra.run.smithi136.stderr:2017-06-12 21:39:26.456104 7f019f533700 10 monclient: _send_command 2 [{"prefix": "osd unset", "key": "nodeep-scrub"}]
2017-06-12T21:39:26.454 INFO:teuthology.orchestra.run.smithi136.stderr:2017-06-12 21:39:26.456120 7f019f533700 10 monclient: _send_mon_message to mon.a at 172.21.15.136:6789/0
2017-06-12T21:39:26.455 INFO:teuthology.orchestra.run.smithi136.stderr:2017-06-12 21:39:26.456127 7f019f533700  1 -- 172.21.15.136:0/342951207 --> 172.21.15.136:6789/0 -- mon_command({"prefix": "osd unset", "key": "nodeep-scrub"} v 0) v1 -- 0x7f0198124570 con 0
2017-06-12T21:39:26.468 INFO:tasks.ceph.osd.1.smithi136.stderr: ceph version 12.0.3-1508-g9960ae3 (9960ae3c107a7282e831b8993cf30e3cadf20844) luminous (dev)
2017-06-12T21:39:26.468 INFO:tasks.ceph.osd.1.smithi136.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0xb0c6a0]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 2: (()+0x4f11dc) [0x5f91dc]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 3: (ThreadPool::stop(bool)+0x2e5) [0xb0fe85]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 4: (OSD::shutdown()+0xa55) [0x5c2eb5]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 5: (OSD::handle_signal(int)+0x11f) [0x5c41bf]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 6: (SignalHandler::entry()+0x1d7) [0xacfe07]
2017-06-12T21:39:26.469 INFO:tasks.ceph.osd.1.smithi136.stderr: 7: (()+0x7dc5) [0xc7a4dc5]

/a/sage-2017-06-12_20:56:37-rados-wip-sage-testing-distro-basic-smithi/1280570


Related issues

Related to RADOS - Bug #20000: osd assert in shared_cache.hpp: 107: FAILED assert(weak_refs.empty()) Can't reproduce 05/22/2017

History

#1 Updated by Sage Weil over 1 year ago

  • Status changed from New to Verified

- handle_osd_map queued a write, with _write_committed as callback
- thread pools alls hut down, including peering_wq
- _write_committed queued new stuff
- tp dtor asserts out

#2 Updated by Sage Weil over 1 year ago

  • Status changed from Verified to Need Review

#3 Updated by Sage Weil over 1 year ago

  • Assignee set to Sage Weil

#4 Updated by Sage Weil over 1 year ago

  • Related to Bug #20000: osd assert in shared_cache.hpp: 107: FAILED assert(weak_refs.empty()) added

#5 Updated by Greg Farnum over 1 year ago

  • Project changed from Ceph to RADOS
  • Category set to Peering
  • Component(RADOS) OSD added

#6 Updated by Sage Weil over 1 year ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF