Project

General

Profile

Bug #23269

Early use of clog in OSD startup crashes OSD

Added by David Zafman almost 5 years ago. Updated almost 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This crash occurred because log_weirdness() called osd->clog->error() probably out of init() -> load_pgs() -> read_state().

2018-03-06 16:19:07.413 7f0a38ee4700 20 filestore(/home/dzafman/ceph/build/dev/osd1) sync_entry(4039): woke after 5.001077
2018-03-06 16:19:07.413 7f0a38ee4700 10 journal commit_start max_applied_seq 98, open_ops 0
2018-03-06 16:19:07.413 7f0a38ee4700 10 journal commit_start blocked, all open_ops have completed
2018-03-06 16:19:07.413 7f0a38ee4700 10 journal commit_start committing 98, still blocked
2018-03-06 16:19:07.413 7f0a38ee4700 10 journal commit_start
2018-03-06 16:19:07.413 7f0a38ee4700 15 filestore(/home/dzafman/ceph/build/dev/osd1) sync_entry(4070): committing 98
2018-03-06 16:19:07.413 7f0a38ee4700 10 journal commit_started committing 98, unblocking
2018-03-06 16:19:07.413 7f0a38ee4700 20 filestore dbobjectmap: seq is 2
2018-03-06 16:19:07.413 7f0a38ee4700 15 genericfilestorebackend(/home/dzafman/ceph/build/dev/osd1) syncfs: doing a full sync (syncfs(2) if possible)
2018-03-06 16:19:07.417 7f0a31ed6700 10 monclient: tick
2018-03-06 16:19:07.417 7f0a31ed6700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2018-03-06 16:18:37.420105)
2018-03-06 16:19:07.417 7f0a31ed6700 10 log_client  log_queue is 1 last_log 2 sent 0 num 1 unsent 2 sending 2
2018-03-06 16:19:07.417 7f0a31ed6700 -1 /home/dzafman/ceph/src/common/LogClient.cc: In function 'Message* LogClient::_get_mon_log_message()' thread 7f0a31ed6700 time 2018-03-06 16:19:07.420182
/home/dzafman/ceph/src/common/LogClient.cc: 292: FAILED assert(num_unsent <= log_queue.size())

 ceph version 13.0.1-2394-g485c784 (485c784c8f3f91e64c58f52dca852e43e99cb48e) mimic (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7f0a507f61d2]
 2: (LogClient::_get_mon_log_message()+0xdd5) [0x7f0a507b4965]
 3: (LogClient::get_mon_log_message(bool)+0x43) [0x7f0a507b4a43]
 4: (MonClient::send_log(bool)+0x1c) [0x7f0a5084859c]
 5: (MonClient::tick()+0x612) [0x7f0a50852882]
 6: (Context::complete(int)+0x9) [0x563605744579]
 7: (SafeTimer::timer_thread()+0x20f) [0x7f0a507f2a3f]
 8: (SafeTimerThread::entry()+0xd) [0x7f0a507f407d]
 9: (()+0x76ba) [0x7f0a4f3106ba]
 10: (clone()+0x6d) [0x7f0a4e59a82d]

Related issues

Related to RADOS - Feature #23242: ceph-objectstore-tool command to trim the pg log Resolved 03/06/2018

History

#1 Updated by David Zafman almost 5 years ago

  • Related to Feature #23242: ceph-objectstore-tool command to trim the pg log added

#2 Updated by David Zafman almost 5 years ago

  • Description updated (diff)

Also available in: Atom PDF