Project

General

Profile

Actions
Copy link

Bug #44389

closed

client: fuse mount will print call trace with incorrect options

Added by Xiubo Li about 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Xiubo Li
Category:
-
Target version:
Ceph - v16.0.0
% Done:

0%

Source:
Tags:
Backport:
octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
ceph-fuse
Labels (FS):
Pull request ID:
33915
Crash signature (v1):
Crash signature (v2):

Description

2020-03-02T20:54:23.827-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-02T20:54:23.847-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-02T20:54:23.863-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 init, newargv = 0x55ec38cee780 newargc=10
fuse: invalid argument `ddd'
ceph-fuse[5867]: fuse failed to initialize
2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 fuse_parse_cmdline failed.
/data/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7f3ed958df80 time 2020-03-02T20:54:23.869371-0500
/data/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)
 ceph version Development (no_version) octopus (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f3ecfe2332a]
 2: (()+0x14775ac) [0x7f3ecfe235ac]
 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 4: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 5: (Client::~Client()+0x292) [0x55ec36c784b6]
 6: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 7: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 8: (main()+0x10db) [0x55ec36c52883]
 9: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 10: (()+0x204dd9) [0x55ec36c50dd9]
2020-03-02T20:54:23.873-0500 7f3ed958df80 -1 /data/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7f3ed958df80 time 2020-03-02T20:54:23.869371-0500
/data/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)

 ceph version Development (no_version) octopus (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f3ecfe2332a]
 2: (()+0x14775ac) [0x7f3ecfe235ac]
 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 4: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 5: (Client::~Client()+0x292) [0x55ec36c784b6]
 6: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 7: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 8: (main()+0x10db) [0x55ec36c52883]
 9: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 10: (()+0x204dd9) [0x55ec36c50dd9]

*** Caught signal (Aborted) **
 in thread 7f3ed958df80 thread_name:ceph-fuse
 ceph version Development (no_version) octopus (rc)
 1: (()+0x365c32) [0x55ec36db1c32]
 2: (()+0xf5f0) [0x7f3ecdad75f0]
 3: (gsignal()+0x37) [0x7f3ecc6b5337]
 4: (abort()+0x148) [0x7f3ecc6b6a28]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7f3ecfe234f9]
 6: (()+0x14775ac) [0x7f3ecfe235ac]
 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 8: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 9: (Client::~Client()+0x292) [0x55ec36c784b6]
 10: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 11: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 12: (main()+0x10db) [0x55ec36c52883]
 13: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 14: (()+0x204dd9) [0x55ec36c50dd9]
2020-03-02T20:54:23.877-0500 7f3ed958df80 -1 *** Caught signal (Aborted) **
 in thread 7f3ed958df80 thread_name:ceph-fuse

 ceph version Development (no_version) octopus (rc)
 1: (()+0x365c32) [0x55ec36db1c32]
 2: (()+0xf5f0) [0x7f3ecdad75f0]
 3: (gsignal()+0x37) [0x7f3ecc6b5337]
 4: (abort()+0x148) [0x7f3ecc6b6a28]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7f3ecfe234f9]
 6: (()+0x14775ac) [0x7f3ecfe235ac]
 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 8: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 9: (Client::~Client()+0x292) [0x55ec36c784b6]
 10: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 11: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 12: (main()+0x10db) [0x55ec36c52883]
 13: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 14: (()+0x204dd9) [0x55ec36c50dd9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  -141> 2020-03-02T20:54:23.827-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
   -57> 2020-03-02T20:54:23.847-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
   -48> 2020-03-02T20:54:23.863-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
    -3> 2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 init, newargv = 0x55ec38cee780 newargc=10
    -2> 2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 fuse_parse_cmdline failed.
    -1> 2020-03-02T20:54:23.873-0500 7f3ed958df80 -1 /data/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7f3ed958df80 time 2020-03-02T20:54:23.869371-0500
/data/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)

 ceph version Development (no_version) octopus (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f3ecfe2332a]
 2: (()+0x14775ac) [0x7f3ecfe235ac]
 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 4: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 5: (Client::~Client()+0x292) [0x55ec36c784b6]
 6: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 7: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 8: (main()+0x10db) [0x55ec36c52883]
 9: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 10: (()+0x204dd9) [0x55ec36c50dd9]

     0> 2020-03-02T20:54:23.877-0500 7f3ed958df80 -1 *** Caught signal (Aborted) **
 in thread 7f3ed958df80 thread_name:ceph-fuse

 ceph version Development (no_version) octopus (rc)
 1: (()+0x365c32) [0x55ec36db1c32]
 2: (()+0xf5f0) [0x7f3ecdad75f0]
 3: (gsignal()+0x37) [0x7f3ecc6b5337]
 4: (abort()+0x148) [0x7f3ecc6b6a28]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7f3ecfe234f9]
 6: (()+0x14775ac) [0x7f3ecfe235ac]
 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 8: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 9: (Client::~Client()+0x292) [0x55ec36c784b6]
 10: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 11: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 12: (main()+0x10db) [0x55ec36c52883]
 13: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 14: (()+0x204dd9) [0x55ec36c50dd9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

  -141> 2020-03-02T20:54:23.827-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
   -57> 2020-03-02T20:54:23.847-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
   -48> 2020-03-02T20:54:23.863-0500 7f3ed958df80 -1 WARNING: all dangerous and experimental features are enabled.
    -3> 2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 init, newargv = 0x55ec38cee780 newargc=10
    -2> 2020-03-02T20:54:23.868-0500 7f3ed958df80 -1 fuse_parse_cmdline failed.
    -1> 2020-03-02T20:54:23.873-0500 7f3ed958df80 -1 /data/ceph/src/common/mutex_debug.h: In function 'ceph::mutex_debug_detail::mutex_debug_impl<<anonymous> >::~mutex_debug_impl() [with bool Recursive = false]' thread 7f3ed958df80 time 2020-03-02T20:54:23.869371-0500
/data/ceph/src/common/mutex_debug.h: 111: FAILED ceph_assert(r == 0)

 ceph version Development (no_version) octopus (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1aa) [0x7f3ecfe2332a]
 2: (()+0x14775ac) [0x7f3ecfe235ac]
 3: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 4: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 5: (Client::~Client()+0x292) [0x55ec36c784b6]
 6: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 7: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 8: (main()+0x10db) [0x55ec36c52883]
 9: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 10: (()+0x204dd9) [0x55ec36c50dd9]

     0> 2020-03-02T20:54:23.877-0500 7f3ed958df80 -1 *** Caught signal (Aborted) **
 in thread 7f3ed958df80 thread_name:ceph-fuse

 ceph version Development (no_version) octopus (rc)
 1: (()+0x365c32) [0x55ec36db1c32]
 2: (()+0xf5f0) [0x7f3ecdad75f0]
 3: (gsignal()+0x37) [0x7f3ecc6b5337]
 4: (abort()+0x148) [0x7f3ecc6b6a28]
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x379) [0x7f3ecfe234f9]
 6: (()+0x14775ac) [0x7f3ecfe235ac]
 7: (ceph::mutex_debug_detail::mutex_debug_impl<false>::~mutex_debug_impl()+0x31) [0x55ec36c71253]
 8: (Finisher::~Finisher()+0xf0) [0x55ec36cfee08]
 9: (Client::~Client()+0x292) [0x55ec36c784b6]
 10: (StandaloneClient::~StandaloneClient()+0x84) [0x55ec36cf609c]
 11: (StandaloneClient::~StandaloneClient()+0x18) [0x55ec36cf60c0]
 12: (main()+0x10db) [0x55ec36c52883]
 13: (__libc_start_main()+0xf5) [0x7f3ecc6a1505]
 14: (()+0x204dd9) [0x55ec36c50dd9]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ceph-fuse [5858]: (33) Numerical argument out of domain

Related issues 2 (0 open — 2 closed)

Copied to CephFS - Backport #45495: octopus: client: fuse mount will print call trace with incorrect optionsResolvedNathan CutlerActions
Copied to CephFS - Backport #45496: nautilus: client: fuse mount will print call trace with incorrect optionsResolvedNathan CutlerActions
Actions
Copy link
 #1

Updated by Patrick Donnelly about 4 years ago

  • Description updated (diff)
  • Status changed from New to In Progress
  • Target version set to v16.0.0
  • Backport set to octopus,nautilus
Actions
Copy link
 #2

Updated by Xiubo Li about 4 years ago

  • Status changed from In Progress to Fix Under Review
Actions
Copy link
 #3

Updated by Xiubo Li about 4 years ago

The objecter_finisher is already started in Client::Client(), but in the failure path when initializing and starting the Client object, we may not get a chance to call the Client::shutdown() to stop the Finisher thread, which maybe still holding the mutex lock in it. Then when destrucing the Finisher object the pthread_mutex_destroy() will fail.

Actions
Copy link
 #4

Updated by Greg Farnum about 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link
 #5

Updated by Patrick Donnelly almost 4 years ago

  • Status changed from Resolved to Pending Backport
Actions
Copy link
 #6

Updated by Nathan Cutler almost 4 years ago

  • Copied to Backport #45495: octopus: client: fuse mount will print call trace with incorrect options added
Actions
Copy link
 #7

Updated by Nathan Cutler almost 4 years ago

  • Copied to Backport #45496: nautilus: client: fuse mount will print call trace with incorrect options added
Actions
Copy link
 #8

Updated by Nathan Cutler almost 4 years ago

  • Pull request ID set to 33915
Actions
Copy link
 #9

Updated by Nathan Cutler almost 4 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Actions
Copy link

Also available in: Atom PDF