Project

General

Profile

Bug #48946

Disable and re-enable clog_to_monitors could trigger assertion

Added by gerald yang about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus, octopus, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Steps to reproduce:

1. Disable clog_to_monitors for OSDs:
ceph tell osd.* injectargs '--clog_to_monitors=true'

2. Wait for OSDs to generate some clogs

3. Re-enable clog_to_monitors
ceph tell osd.* injectargs '--clog_to_monitors=true'

This will trigger an assertion below

-2> 2021-01-21T07:56:09.274+0000 7f9ce5a06700 10 log_client  log_queue is 900 last_log 24302 sent 18902 num 900 unsent 5400 sending 1000
-1> 2021-01-21T07:56:09.318+0000 7f9ce5a06700 -1 /home/ubuntu/ceph/src/common/LogClient.cc: In function 'ceph::ref_t<Message> LogClient::_get_mon_log_message()' thread 7f9ce5a06700 time 2021-01-21T07
:56:09.278904+0000
/home/ubuntu/ceph/src/common/LogClient.cc: 289: FAILED ceph_assert(num_unsent <= log_queue.size())
ceph version 16.0.0-9149-g0e7a7be5c0 (0e7a7be5c065bd7f792b378a59c14b503370e394) pacific (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a1) [0x55ba08a5e3bd]
2: /home/ubuntu/ceph/build/bin/ceph-osd(+0x32f265b) [0x55ba08a5e65b]
3: (LogClient::_get_mon_log_message()+0x342) [0x55ba08a1bdbe]
4: (LogClient::get_mon_log_message(bool)+0xa8) [0x55ba08a1b954]
5: (MonClient::send_log(bool)+0x57) [0x55ba08dc52b9]
6: (MonClient::tick()+0x815) [0x55ba08dcbd71]
7: /home/ubuntu/ceph/build/bin/ceph-osd(+0x366012a) [0x55ba08dcc12a]
8: /home/ubuntu/ceph/build/bin/ceph-osd(+0x366eb72) [0x55ba08ddab72]
9: (Context::complete(int)+0x2b) [0x55ba07ebc2df]
10: (SafeTimer::timer_thread()+0x390) [0x55ba08a403c2]
11: (SafeTimerThread::entry()+0x20) [0x55ba08a41c82]
12: (Thread::entry_wrapper()+0x87) [0x55ba08a33a93]
13: (Thread::_entry_func(void*)+0x1c) [0x55ba08a33a02]
14: /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f9cfa9f9609]
15: clone()

Affected releases: should be most of releases, I see this issue on Nautilus/Octopus/Master but others should have this issue too


Related issues

Copied to RADOS - Backport #49640: nautilus: Disable and re-enable clog_to_monitors could trigger assertion Resolved
Copied to RADOS - Backport #49641: octopus: Disable and re-enable clog_to_monitors could trigger assertion Resolved
Copied to RADOS - Backport #49642: pacific: Disable and re-enable clog_to_monitors could trigger assertion Resolved

History

#2 Updated by Dan Hill about 3 years ago

  • Status changed from New to In Progress
  • Assignee set to gerald yang
  • Backport set to nautilus, octopus, pacific
  • Pull request ID set to 38997

#3 Updated by Dan Hill about 3 years ago

  • Status changed from In Progress to Fix Under Review

#4 Updated by Kefu Chai about 3 years ago

  • Status changed from Fix Under Review to Pending Backport

#5 Updated by Backport Bot about 3 years ago

  • Copied to Backport #49640: nautilus: Disable and re-enable clog_to_monitors could trigger assertion added

#6 Updated by Backport Bot about 3 years ago

  • Copied to Backport #49641: octopus: Disable and re-enable clog_to_monitors could trigger assertion added

#7 Updated by Backport Bot about 3 years ago

  • Copied to Backport #49642: pacific: Disable and re-enable clog_to_monitors could trigger assertion added

#8 Updated by Loïc Dachary almost 3 years ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF