Project

General

Profile

Actions

Bug #50943

closed

mon crash due to assert failed

Added by wencong wan almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Ceph version 12.2.11
3 mons, 1 mon can't start up due to assert failed

-6> 2021-05-20 16:11:32.755959 7fffd7fd9000 10 mon.ceph-12@-1(probing).mds e54 update_logger
-5> 2021-05-20 16:11:32.758029 7fffd7fd9000 15 mon.ceph-12@-1(probing).osd e0 update_from_paxos paxos e 760619, my e 0
-4> 2021-05-20 17:04:33.640719 7fffd7fd9000 7 mon.ceph-12@-1(probing).osd e0 update_from_paxos loading latest full map e760617
-3> 2021-05-20 17:04:33.647957 7fffd7fd9000 7 mon.ceph-12@-1(probing).osd e760617 update_from_paxos loading creating_pgs last_scan_epoch 760618 with 0 pgs
-2> 2021-05-20 17:04:33.647981 7fffd7fd9000 10 mon.ceph-12@-1(probing).osd e760617 update_from_paxos pgservice is mgrstat
-1> 2021-05-20 17:04:33.648018 7fffd7fd9000 7 mon.ceph-12@-1(probing).osd e760617 update_from_paxos applying incremental 760618
0> 2021-05-20 17:04:33.661918 7fffd7fd9000 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/src/mon/OSDMonitor.cc: In function 'virtual void OSDMonitor::update_from_paxos(bool*)' thread 7fffd7fd9000 time 2021-05-20 17:04:33.658739
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.11/rpm/el7/BUILD/ceph-12.2.11/src/mon/OSDMonitor.cc: 507: FAILED assert(!inc.have_crc)

ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-ceph-12/store.db list osdmap | grep full

osdmap full_760590
osdmap full_760591
osdmap full_760592
osdmap full_760593
osdmap full_760618
osdmap full_760619
osdmap full_latest

[root@ceph001 ceph-ceph-12]# hexdump full_latest (760617)
0000000 9b29 000b 0000 0000
0000008

ceph-kvstore-tool rocksdb /var/lib/ceph/mon/ceph-ceph-12/store.db get osdmap full_760618 out full_760618

result in

(osdmap, full_760618) does not exist

but full_760619 is valid

why mon has full_760619 osdmap but full_latest record version 760617?
why assert increment 760618 does not have crc?
why key full_760618 exits in db but can't get value?

Actions #1

Updated by Neha Ojha almost 3 years ago

  • Status changed from New to Closed

Luminous is EOL, can you please redeploy the monitor and upgrade to a supported version of Ceph. Please reopen this ticket if you continue to see this issue.

Actions

Also available in: Atom PDF