Project

General

Profile

Bug #42824

mimic: rebuild_mondb.cc: FAILED assert(0) in update_osdmap()

Added by Neha Ojha 5 months ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

The assert got added recently https://github.com/ceph/ceph/commit/43662bd4266a4fcc8db62f4bc9beb0de78eef355#diff-431a930a4ccde35ae58625f1d2e167c1.

2019-11-10T01:18:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store
2019-11-10T01:18:23.225 INFO:teuthology.orchestra.run.smithi045.stderr:/build/ceph-13.2.6-923-gd203ae4/src/tools/rebuild_mondb.cc: In function 'int update_osdmap(ObjectStore&, OSDSuperblock&, MonitorDBStore&)' thread 7f91cd29fd80 time 2019-11-10 01:18:23.221665
2019-11-10T01:18:23.225 INFO:teuthology.orchestra.run.smithi045.stderr:/build/ceph-13.2.6-923-gd203ae4/src/tools/rebuild_mondb.cc: 290: FAILED assert(0)
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: ceph version 13.2.6-923-gd203ae4 (d203ae4bc82bb872129e3129796f69615aba23bf) mimic (stable)
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f91c3646f1e]
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: 2: (()+0x2fc0a7) [0x7f91c36470a7]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 3: (update_mon_db(ObjectStore&, OSDSuperblock&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27e4) [0x78f244]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 4: (main()+0x51c6) [0x6716b6]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 5: (__libc_start_main()+0xf0) [0x7f91c2312830]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 6: (_start()+0x29) [0x73d539]
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:*** Caught signal (Aborted) **
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: in thread 7f91cd29fd80 thread_name:ceph-objectstor
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: ceph version 13.2.6-923-gd203ae4 (d203ae4bc82bb872129e3129796f69615aba23bf) mimic (stable)
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: 1: (()+0x11390) [0x7f91c2df8390]
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: 2: (gsignal()+0x38) [0x7f91c2327428]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 3: (abort()+0x16a) [0x7f91c232902a]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7f91c3647026]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 5: (()+0x2fc0a7) [0x7f91c36470a7]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 6: (update_mon_db(ObjectStore&, OSDSuperblock&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27e4) [0x78f244]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 7: (main()+0x51c6) [0x6716b6]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 8: (__libc_start_main()+0xf0) [0x7f91c2312830]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 9: (_start()+0x29) [0x73d539]

/a/yuriw-2019-11-09_18:55:35-rados-wip-yuri-mimic_13.2.7_RC2-distro-basic-smithi/4490082/

Kefu, can you take a look.


Related issues

Related to Ceph - Bug #41177: ceph-objectstore-tool: update-mon-db return EINVAL with missed inc_osdmap Resolved 08/09/2019

History

#1 Updated by Brad Hubbard 5 months ago

  • Status changed from New to 4
  • Assignee changed from Kefu Chai to Brad Hubbard
(gdb) p add_inc_result
$4 = <optimized out>
(gdb) l 247
242            e <= sb.newest_map; e++) {
243         bool have_crc = false;
244         uint32_t crc = -1;
245         uint64_t features = 0;
246         // add inc maps
247         auto add_inc_result = [&] {
248           const auto oid = OSD::get_inc_osdmap_pobject_name(e);
249           bufferlist bl;
250           int nread = fs.read(ch, oid, 0, 0, bl);
251           if (nread <= 0) {
(gdb) 
252             cout << "missing " << oid << std::endl;
253             return -ENOENT;
254           }
255           t->put(prefix, e, bl);
256
257           OSDMap::Incremental inc;
258           auto p = bl.begin();
259           inc.decode(p);
260           features = inc.encode_features | CEPH_FEATURE_RESERVED;
261           if (osdmap.get_epoch() && e > 1) {
(gdb) 
262             if (osdmap.apply_incremental(inc)) {
263               cerr << "bad fsid: " 
264                    << osdmap.get_fsid() << " != " << inc.fsid << std::endl;
265               return -EINVAL;
266             }
267             have_crc = inc.have_crc;
268             if (inc.have_crc) {
269               crc = inc.full_crc;
270               bufferlist fbl;
271               osdmap.encode(fbl, features);
(gdb) 
272               if (osdmap.get_crc() != inc.full_crc) {
273                 cerr << "mismatched inc crc: " 
274                      << osdmap.get_crc() << " != " << inc.full_crc << std::endl;
275                 return -EINVAL;
276               }
277               // inc.decode() verifies `inc_crc`, so it's been taken care of.
278             }
279           }
280         }();
281         switch (add_inc_result) {
(gdb) 
282         case -ENOENT:
283           // no worries, we always have full map
284           break;
285         case -EINVAL:
286           return -EINVAL;
287         case 0:
288           break;
289         default:
290           assert(0);
291         }

The lambda didn't return an error so what would it return? Missing return statement added by https://github.com/ceph/ceph/pull/29704 include that PR and you should be apples =D

#2 Updated by Neha Ojha 5 months ago

  • Status changed from 4 to Fix Under Review
  • Pull request ID set to 31659

#3 Updated by Nathan Cutler 5 months ago

  • Status changed from Fix Under Review to Resolved

#4 Updated by Brad Hubbard 5 months ago

  • Related to Bug #41177: ceph-objectstore-tool: update-mon-db return EINVAL with missed inc_osdmap added

Also available in: Atom PDF