Project

General

Profile

Bug #10901

FAILED assert(weak_refs.empty()) (outstanding OSDMap ref at shutdown)

Added by David Zafman about 9 years ago. Updated almost 9 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
giant
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

sage-2015-02-15_07:44:23-rados-hammer-distro-basic-multi/757223

2015-02-15 12:59:29.667485 403f1c0 1 -- 10.214.137.128:0/32313 shutdown complete.
2015-02-15 12:59:29.733112 403f1c0 -1 leaked refs:
dump_weak_refs 0x41348c8 weak_refs: 11 = 0x2f23f000 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 38 = 0x2f23e600 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 101 = 0x2f5d4e00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 184 = 0x2fbb7000 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 202 = 0x2fb5cc00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 265 = 0x2f63dc00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 327 = 0x2f741200 with 4 refs
dump_weak_refs 0x41348c8 weak_refs: 330 = 0x2fb77400 with 6 refs
dump_weak_refs 0x41348c8 weak_refs: 335 = 0x3012a000 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 341 = 0x2fb77800 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 348 = 0x2f740e00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 353 = 0x2fb74c00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 359 = 0x2f216600 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 363 = 0x2f219c00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 366 = 0x30b15e00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 375 = 0x30b15400 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 377 = 0x30b15200 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 501 = 0x408a800 with 4 refs
dump_weak_refs 0x41348c8 weak_refs: 502 = 0x3144e600 with 6 refs
dump_weak_refs 0x41348c8 weak_refs: 618 = 0x2f740200 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 619 = 0x2f175800 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 658 = 0x2fbb3000 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 673 = 0x2fbb7400 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 699 = 0x2fab1e00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 739 = 0x2ffe0a00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 764 = 0x307eae00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 774 = 0x2fed7800 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 795 = 0x2ff79600 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 805 = 0x304ff200 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 808 = 0x2fbb5a00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 811 = 0x2fbb4000 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 817 = 0x30103a00 with 2 refs
dump_weak_refs 0x41348c8 weak_refs: 823 = 0x31def000 with 4 refs

2015-02-15 12:59:29.805182 403f1c0 -1 ./common/shared_cache.hpp: In function 'SharedLRU<K, V>::~SharedLRU() [with K = unsigned int; V = const OSDMap]' thread 403f1c0 time 2015-02-15 12:59:29.736570
./common/shared_cache.hpp: 104: FAILED assert(weak_refs.empty())

ceph version 0.92-1103-g7da9351 (7da93518fdf1dce693723418dedb726fb4608df8)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbc2b6b]
2: (SharedLRU&lt;unsigned int, OSDMap const&gt;::~SharedLRU()+0x332) [0x6ee222]
3: (OSDService::~OSDService()+0x16c) [0x68356c]
4: (OSD::~OSD()+0x115) [0x6c9065]
5: (OSD::~OSD()+0x9) [0x6c9bc9]
6: (main()+0x2941) [0x651711]
7: (__libc_start_main()+0xf5) [0x776aec5]
8: ceph-osd() [0x66a247]

Also in sage-2015-02-15_07:44:23-rados-hammer-distro-basic-multi/758289


Related issues

Duplicates Ceph - Bug #7995: osd shutdown: ./common/shared_cache.hpp: 93: FAILED assert(weak_refs.empty()) Can't reproduce 04/04/2014

History

#1 Updated by Yuri Weinstein about 9 years ago

  • Backport set to giant

Also in run: http://pulpito.front.sepia.ceph.com/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/
Job: ['762750']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/

Assertion: ./common/shared_cache.hpp: 101: FAILED assert(weak_refs.empty())
ceph version 0.87-159-gccb0914 (ccb0914f76da23acdd7374233cd1939ab80ef3c8)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xa9ff2f]
 2: (SharedLRU<unsigned int, OSDMap const>::~SharedLRU()+0x1ca) [0x6943fa]
 3: (OSDService::~OSDService()+0x1c8) [0x62ace8]
 4: (OSD::~OSD()+0x115) [0x640255]
 5: (OSD::~OSD()+0x9) [0x6411e9]
 6: (main()+0x1feb) [0x60d06b]
 7: (__libc_start_main()+0xed) [0x683b76d]
 8: ceph-osd() [0x611fc9]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz:2015-02-17 08:41:51.253289 4038b00 -1 *** Caught signal (Aborted) **
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: in thread 4038b00
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz:
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: ceph version 0.87-159-gccb0914 (ccb0914f76da23acdd7374233cd1939ab80ef3c8)
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 1: ceph-osd() [0x9b6625]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 2: (()+0xfcb0) [0x5290cb0]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 3: (gsignal()+0x35) [0x6850425]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 4: (abort()+0x17b) [0x6853b8b]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x60bf69d]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 6: (()+0xb5846) [0x60bd846]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 7: (()+0xb5873) [0x60bd873]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 8: (()+0xb596e) [0x60bd96e]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xaa0109]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 10: (SharedLRU<unsigned int, OSDMap const>::~SharedLRU()+0x1ca) [0x6943fa]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 11: (OSDService::~OSDService()+0x1c8) [0x62ace8]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 12: (OSD::~OSD()+0x115) [0x640255]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 13: (OSD::~OSD()+0x9) [0x6411e9]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 14: (main()+0x1feb) [0x60d06b]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 15: (__libc_start_main()+0xed) [0x683b76d]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: 16: ceph-osd() [0x611fc9]
/a/teuthology-2015-02-16_18:37:52-rados-giant-distro-basic-multi/762750/remote/plana72/log/ceph-osd.1.log.gz: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

#2 Updated by Yuri Weinstein about 9 years ago

Verified that the failed configuration

rados/verify/{1thrash/none.yaml clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rados_cls_all.yaml validater/valgrind.yaml}

passed in the giant release giant v0.87.1 RC

Job/Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-02-17_14:43:58-rados-giant-distro-basic-multi/763719/

#3 Updated by Loïc Dachary about 9 years ago

looks like a duplicate of http://tracker.ceph.com/issues/7995 ?

#4 Updated by Loïc Dachary about 9 years ago

  • Status changed from New to Duplicate

#5 Updated by Sage Weil almost 9 years ago

  • Subject changed from FAILED assert(weak_refs.empty()) to FAILED assert(weak_refs.empty()) (outstanding OSDMap ref at shutdown)
  • Status changed from Duplicate to 12
  • Source changed from other to Q/A
  • Regression set to No

This bug is still there (or a new version of it is). Saw this on current next:

/a/teuthology-2015-05-20_21:00:01-rados-next-distro-basic-multi/902503

#6 Updated by Samuel Just almost 9 years ago

bah, just saw on wip-sam-testing: ubuntu@teuthology:/a/samuelj-2015-05-28_08:23:27-rados-wip-sam-testing-distro-basic-multi/913530/remote

#7 Updated by Samuel Just almost 9 years ago

  • Status changed from 12 to Duplicate

Also available in: Atom PDF