Project

General

Profile

Bug #38822

mds: there is an assertion when calling Beacon::shutdown()

Added by huanwen ren 5 months ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
High
Assignee:
Category:
-
Target version:
Start date:
Due date:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
nautilus,mimic,luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:

Description

The construction condition is:
when you use systemctl start a non-existing ceph-mds@xx object

Error info:

2019-03-19 20:26:57.000329 7f5b6671b1c0  0 set uid:gid to 167:167 (ceph:ceph)
2019-03-19 20:26:57.000356 7f5b6671b1c0  0 ceph version 12.2.9(9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable), process ceph-mds, pid 458996
2019-03-19 20:26:57.004965 7f5b6671b1c0  0 pidfile_write: ignore empty --pid-file
2019-03-19 20:26:57.052413 7f5b6671b1c0 -1 auth: unable to find a keyring on /var/lib/ceph/mds/ceph-star/keyring: (2) No such file or directory
2019-03-19 20:26:57.052438 7f5b6671b1c0 -1 monclient: ERROR: missing keyring, cannot use cephx for authentication
2019-03-19 20:26:57.052445 7f5b6671b1c0 -1 mds.star ERROR: failed to get monmap: (2) No such file or directory
2019-03-19 20:26:57.052453 7f5b6671b1c0  1 mds.star suicide! Wanted state up:boot
2019-03-19 20:26:57.055098 7f5b6671b1c0 -1 *** Caught signal (Aborted) **
 in thread 7f5b6671b1c0 thread_name:ceph-mds

 ceph version 12.2.9(9e300932ef8a8916fb3fda78c58691a6ab0f4217) luminous (stable)
 1: (()+0x611581) [0x561a08abd581]
 2: (()+0xf6d0) [0x7f5b648116d0]
 3: (gsignal()+0x37) [0x7f5b63a36277]
 4: (abort()+0x148) [0x7f5b63a37968]
 5: (__gnu_cxx::__verbose_terminate_handler()+0x165) [0x7f5b643457d5]
 6: (()+0x5e746) [0x7f5b64343746]
 7: (()+0x5e773) [0x7f5b64343773]
 8: (()+0x5e993) [0x7f5b64343993]
 9: (std::__throw_system_error(int)+0x90) [0x7f5b64398c70]
 10: (std::thread::join()+0x18) [0x7f5b64399e28]
 11: (Beacon::shutdown()+0xa8) [0x561a087b1dd8]
 12: (MDSDaemon::suicide()+0x241) [0x561a08769f61]
 13: (MDSDaemon::init()+0x17be) [0x561a0876e86e]
 14: (main()+0xb90) [0x561a087575f0]
 15: (__libc_start_main()+0xf5) [0x7f5b63a22445]
 16: (()+0x2b8163) [0x561a08764163]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


Related issues

Copied to fs - Backport #39213: luminous: mds: there is an assertion when calling Beacon::shutdown() Resolved
Copied to fs - Backport #39214: nautilus: mds: there is an assertion when calling Beacon::shutdown() Resolved
Copied to fs - Backport #39215: mimic: mds: there is an assertion when calling Beacon::shutdown() In Progress

History

#1 Updated by huanwen ren 5 months ago

Cause:
If you run MDSDaemon::init(), an exception occurs that causes MDSDaemon::suicide()--->Beacon::shutdown()--->sender.join();
this problem occurs because sender is just Default-constructed is in Beacon.h(std::thread sender;), there is no call
to Beacon::init() to construct the sender, so the sender has no "joinable"

fix: https://github.com/ceph/ceph/pull/27063

#2 Updated by Patrick Donnelly 5 months ago

  • Status changed from New to Need Review
  • Assignee set to huanwen ren
  • Priority changed from Normal to High
  • Target version set to v15.0.0
  • Start date deleted (03/20/2019)
  • Source set to Community (dev)
  • Backport set to nautilus,mimic,luminous
  • Pull request ID set to 27063

#3 Updated by huanwen ren 5 months ago

hi Patrick @Patrick Donnelly
Can you pull me into the development of the tracker? I can't modify most of the states in this issue.

#4 Updated by Patrick Donnelly 5 months ago

huanwen ren wrote:

hi Patrick @Patrick Donnelly
Can you pull me into the development of the tracker? I can't modify most of the states in this issue.

I've added you to the ceph developers group.

#5 Updated by Patrick Donnelly 5 months ago

  • Status changed from Need Review to Pending Backport

#6 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #39213: luminous: mds: there is an assertion when calling Beacon::shutdown() added

#7 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #39214: nautilus: mds: there is an assertion when calling Beacon::shutdown() added

#8 Updated by Nathan Cutler 4 months ago

  • Copied to Backport #39215: mimic: mds: there is an assertion when calling Beacon::shutdown() added

Also available in: Atom PDF