Project

General

Profile

Actions
Copy link

Bug #12968

closed

mon/OSDMonitor.cc: 1864: FAILED assert(osdmap.get_num_up_osds() == 0 || osdmap.get_up_osd_features() & (1ULL<<54))

Added by Sage Weil over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Kefu Chai
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2015-09-04T14:10:23.156 INFO:tasks.ceph.mon.a.plana79.stderr:mon/OSDMonitor.cc: In function 'bool OSDMonitor::preprocess_boot(MonOpRequestRef)' thread 7f36f243a700 time 2015-09-04 14:10:23.153087
2015-09-04T14:10:23.157 INFO:tasks.ceph.mon.a.plana79.stderr:mon/OSDMonitor.cc: 1864: FAILED assert(osdmap.get_num_up_osds() == 0 || osdmap.get_up_osd_features() & (1ULL<<54))
2015-09-04T14:10:23.167 INFO:tasks.ceph.mon.a.plana79.stderr: ceph version 9.0.3-1372-g210a4b7 (210a4b7dbb0f914d45e2c194a3ec683782f37c4f)
2015-09-04T14:10:23.167 INFO:tasks.ceph.mon.a.plana79.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7f36fb91888b]
2015-09-04T14:10:23.167 INFO:tasks.ceph.mon.a.plana79.stderr: 2: (OSDMonitor::preprocess_boot(std::shared_ptr<MonOpRequest>)+0x1be6) [0x7f36fb758046]
2015-09-04T14:10:23.168 INFO:tasks.ceph.mon.a.plana79.stderr: 3: (OSDMonitor::preprocess_query(std::shared_ptr<MonOpRequest>)+0x357) [0x7f36fb7824f7]
2015-09-04T14:10:23.168 INFO:tasks.ceph.mon.a.plana79.stderr: 4: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0x833) [0x7f36fb737e03]
2015-09-04T14:10:23.168 INFO:tasks.ceph.mon.a.plana79.stderr: 5: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0x3cd) [0x7f36fb7014ed]
2015-09-04T14:10:23.168 INFO:tasks.ceph.mon.a.plana79.stderr: 6: (Monitor::dispatch(std::shared_ptr<MonOpRequest>)+0x205) [0x7f36fb702225]
2015-09-04T14:10:23.168 INFO:tasks.ceph.mon.a.plana79.stderr: 7: (Monitor::_ms_dispatch(Message*)+0x380) [0x7f36fb7033c0]
2015-09-04T14:10:23.169 INFO:tasks.ceph.mon.a.plana79.stderr: 8: (Monitor::ms_dispatch(Message*)+0x23) [0x7f36fb7212d3]
2015-09-04T14:10:23.169 INFO:tasks.ceph.mon.a.plana79.stderr: 9: (DispatchQueue::entry()+0x63a) [0x7f36fba98dfa]
2015-09-04T14:10:23.169 INFO:tasks.ceph.mon.a.plana79.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x7f36fb902e9d]
2015-09-04T14:10:23.170 INFO:tasks.ceph.mon.a.plana79.stderr: 11: (()+0x8182) [0x7f36fa353182]
2015-09-04T14:10:23.170 INFO:tasks.ceph.mon.a.plana79.stderr: 12: (clone()+0x6d) [0x7f36f88c047d]

/a/sage-2015-09-04_11:01:27-rados-wip-sage-testing---basic-multi/1043917

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #9732: ReplicatedPG::hit_set_trim osd/ReplicatedPG.cc: 11006: FAILED assert(obc)ResolvedKefu Chai10/10/2014

Actions
Actions
Copy link
 #1

Updated by Sage Weil over 8 years ago

  • Assignee set to Kefu Chai
Actions
Copy link
 #2

Updated by Kefu Chai over 8 years ago

the osd.3 is not upgraded, so the only chance is that some of existing pool is using gmt. but the pools were created when all 6 OSDs were up.

Actions
Copy link
 #3

Updated by Loïc Dachary over 8 years ago

http://pulpito.ceph.com/loic-2015-09-07_01:06:53-rados-infernalis---basic-multi/1045219/

2015-09-06T20:19:01.079 INFO:tasks.ceph.osd.3.mira097.stderr:2015-09-06 20:19:01.081461 7fa8b1454900 -1 osd.3 79 log_to_monitors {default=true}
2015-09-06T20:19:01.243 INFO:tasks.ceph.mon.b.plana44.stderr:mon/OSDMonitor.cc: In function 'bool OSDMonitor::preprocess_boot(MonOpRequestRef)' thread 7ff5dc65b700 time 2015-09-06 20:19:01.240274
2015-09-06T20:19:01.244 INFO:tasks.ceph.mon.b.plana44.stderr:mon/OSDMonitor.cc: 1864: FAILED assert(osdmap.get_num_up_osds() == 0 || osdmap.get_up_osd_features() & (1ULL<<54))
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: ceph version 9.0.3-1455-ga51c133 (a51c133bff3b6a22edd4a9a448afed51f5f4836d)
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x7ff5e5ec77fb]
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: 2: (OSDMonitor::preprocess_boot(std::shared_ptr<MonOpRequest>)+0x1be6) [0x7ff5e5d06d86]
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: 3: (OSDMonitor::preprocess_query(std::shared_ptr<MonOpRequest>)+0x357) [0x7ff5e5d31537]
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: 4: (PaxosService::dispatch(std::shared_ptr<MonOpRequest>)+0x833) [0x7ff5e5ce6b93]
2015-09-06T20:19:01.268 INFO:tasks.ceph.mon.b.plana44.stderr: 5: (Monitor::dispatch_op(std::shared_ptr<MonOpRequest>)+0x3cd) [0x7ff5e5cb029d]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 6: (Monitor::dispatch(std::shared_ptr<MonOpRequest>)+0x205) [0x7ff5e5cb0fd5]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 7: (Monitor::_ms_dispatch(Message*)+0x380) [0x7ff5e5cb2170]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 8: (Monitor::ms_dispatch(Message*)+0x23) [0x7ff5e5cd0063]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 9: (DispatchQueue::entry()+0x63a) [0x7ff5e6047d1a]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x7ff5e5eb1e0d]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 11: (()+0x8182) [0x7ff5e4902182]
2015-09-06T20:19:01.269 INFO:tasks.ceph.mon.b.plana44.stderr: 12: (clone()+0x6d) [0x7ff5e2e7038d]
2015-09-06T20:19:01.270 INFO:tasks.ceph.mon.b.plana44.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-09-06T20:19:01.270 INFO:tasks.ceph.mon.b.plana44.stderr:2015-09-06 20:19:01.267419 7ff5dc65b700 -1 mon/OSDMonitor.cc: In function 'bool OSDMonitor::preprocess_boot(MonOpRequestRef)' thread 7ff5dc65b700 time

Actions
Copy link
 #4

Updated by Kefu Chai over 8 years ago 

$ ./ceph osd lspools
0 rbd,1 metadata,2 data,4 unique_pool_0,5 unique_pool_1,6 unique_pool_2,7 test-rados-api-mira122-31528-1,8 test-rados-api-mira122-31528-2,9 test-rados-api-mira122-31528-3,10 test-rados-api-mira122-31528-4,

$ ./ceph osd pool get data all| grep use_gmt_hitset
use_gmt_hitset: 0
$ ./ceph osd pool get unique_pool_0 all| grep use_gmt_hitset
use_gmt_hitset: 1

all pools after (inclusive) unique_pool_0 are using gmt.

Actions
Copy link
 #5

Updated by Kefu Chai over 8 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #6

Updated by Kefu Chai over 8 years ago

  • Backport set to hammer

will put this fix into the backport pull request of #9732, so no need to create a backport for this.

Actions
Copy link
 #7

Updated by Loïc Dachary over 8 years ago

  • Backport deleted (hammer)

Your comment is good enough. Removing the hammer tag.

Actions
Copy link
 #8

Updated by Kefu Chai over 8 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF