Bug #42824: mimic: rebuild_mondb.cc: FAILED assert(0) in update_osdmap() - RADOS - Ceph

Actions

Copy link

Bug #42824

closed

mimic: rebuild_mondb.cc: FAILED assert(0) in update_osdmap()

Added by Neha Ojha over 4 years ago. Updated over 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Brad Hubbard

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

31659

Crash signature (v1):

Crash signature (v2):

Description

The assert got added recently https://github.com/ceph/ceph/commit/43662bd4266a4fcc8db62f4bc9beb0de78eef355#diff-431a930a4ccde35ae58625f1d2e167c1.

2019-11-10T01:18:14.002 INFO:teuthology.orchestra.run.smithi045.stderr:+ sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --no-mon-config --op update-mon-db --mon-store-path /home/ubuntu/cephtest/mon-store
2019-11-10T01:18:23.225 INFO:teuthology.orchestra.run.smithi045.stderr:/build/ceph-13.2.6-923-gd203ae4/src/tools/rebuild_mondb.cc: In function 'int update_osdmap(ObjectStore&, OSDSuperblock&, MonitorDBStore&)' thread 7f91cd29fd80 time 2019-11-10 01:18:23.221665
2019-11-10T01:18:23.225 INFO:teuthology.orchestra.run.smithi045.stderr:/build/ceph-13.2.6-923-gd203ae4/src/tools/rebuild_mondb.cc: 290: FAILED assert(0)
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: ceph version 13.2.6-923-gd203ae4 (d203ae4bc82bb872129e3129796f69615aba23bf) mimic (stable)
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14e) [0x7f91c3646f1e]
2019-11-10T01:18:23.233 INFO:teuthology.orchestra.run.smithi045.stderr: 2: (()+0x2fc0a7) [0x7f91c36470a7]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 3: (update_mon_db(ObjectStore&, OSDSuperblock&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27e4) [0x78f244]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 4: (main()+0x51c6) [0x6716b6]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 5: (__libc_start_main()+0xf0) [0x7f91c2312830]
2019-11-10T01:18:23.234 INFO:teuthology.orchestra.run.smithi045.stderr: 6: (_start()+0x29) [0x73d539]
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr:*** Caught signal (Aborted) **
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: in thread 7f91cd29fd80 thread_name:ceph-objectstor
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: ceph version 13.2.6-923-gd203ae4 (d203ae4bc82bb872129e3129796f69615aba23bf) mimic (stable)
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: 1: (()+0x11390) [0x7f91c2df8390]
2019-11-10T01:18:23.235 INFO:teuthology.orchestra.run.smithi045.stderr: 2: (gsignal()+0x38) [0x7f91c2327428]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 3: (abort()+0x16a) [0x7f91c232902a]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x256) [0x7f91c3647026]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 5: (()+0x2fc0a7) [0x7f91c36470a7]
2019-11-10T01:18:23.236 INFO:teuthology.orchestra.run.smithi045.stderr: 6: (update_mon_db(ObjectStore&, OSDSuperblock&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x27e4) [0x78f244]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 7: (main()+0x51c6) [0x6716b6]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 8: (__libc_start_main()+0xf0) [0x7f91c2312830]
2019-11-10T01:18:23.237 INFO:teuthology.orchestra.run.smithi045.stderr: 9: (_start()+0x29) [0x73d539]

/a/yuriw-2019-11-09_18:55:35-rados-wip-yuri-mimic_13.2.7_RC2-distro-basic-smithi/4490082/

Kefu, can you take a look.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Brad Hubbard over 4 years ago

Status changed from New to 4
Assignee changed from Kefu Chai to Brad Hubbard

(gdb) p add_inc_result
$4 = <optimized out>
(gdb) l 247
242            e <= sb.newest_map; e++) {
243         bool have_crc = false;
244         uint32_t crc = -1;
245         uint64_t features = 0;
246         // add inc maps
247         auto add_inc_result = [&] {
248           const auto oid = OSD::get_inc_osdmap_pobject_name(e);
249           bufferlist bl;
250           int nread = fs.read(ch, oid, 0, 0, bl);
251           if (nread <= 0) {
(gdb) 
252             cout << "missing " << oid << std::endl;
253             return -ENOENT;
254           }
255           t->put(prefix, e, bl);
256
257           OSDMap::Incremental inc;
258           auto p = bl.begin();
259           inc.decode(p);
260           features = inc.encode_features | CEPH_FEATURE_RESERVED;
261           if (osdmap.get_epoch() && e > 1) {
(gdb) 
262             if (osdmap.apply_incremental(inc)) {
263               cerr << "bad fsid: " 
264                    << osdmap.get_fsid() << " != " << inc.fsid << std::endl;
265               return -EINVAL;
266             }
267             have_crc = inc.have_crc;
268             if (inc.have_crc) {
269               crc = inc.full_crc;
270               bufferlist fbl;
271               osdmap.encode(fbl, features);
(gdb) 
272               if (osdmap.get_crc() != inc.full_crc) {
273                 cerr << "mismatched inc crc: " 
274                      << osdmap.get_crc() << " != " << inc.full_crc << std::endl;
275                 return -EINVAL;
276               }
277               // inc.decode() verifies `inc_crc`, so it's been taken care of.
278             }
279           }
280         }();
281         switch (add_inc_result) {
(gdb) 
282         case -ENOENT:
283           // no worries, we always have full map
284           break;
285         case -EINVAL:
286           return -EINVAL;
287         case 0:
288           break;
289         default:
290           assert(0);
291         }

The lambda didn't return an error so what would it return? Missing return statement added by https://github.com/ceph/ceph/pull/29704 include that PR and you should be apples =D

Actions

Copy link