Bug #20974: osd/PG.cc: 3377: FAILED assert(r == 0) (update_snap_map remove fails) - RADOS - Ceph

Actions

Copy link

Bug #20974

closed

osd/PG.cc: 3377: FAILED assert(r == 0) (update_snap_map remove fails)

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:

Can't reproduce

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

 -1210> 2017-08-10 04:51:18.208183 7fbf35636700 20 snap_mapper.remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
 -1209> 2017-08-10 04:51:18.208187 7fbf35636700 20 snap_mapper._remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
 -1208> 2017-08-10 04:51:18.208195 7fbf35636700 15 bluestore(/var/lib/ceph/osd/ceph-7) omap_get_values meta oid #-1:c0371625:::snapmapper:0#
 -1207> 2017-08-10 04:51:18.208199 7fbf35636700 30 bluestore.OnodeSpace(0x7fbf65b5c7c8 in 0x7fbf65b2f6c0) lookup
 -1205> 2017-08-10 04:51:18.208202 7fbf35636700 30 bluestore.OnodeSpace(0x7fbf65b5c7c8 in 0x7fbf65b2f6c0) lookup #-1:c0371625:::snapmapper:0# hit 0x7fbf66ac0880
 -1204> 2017-08-10 04:51:18.208206 7fbf35636700 20 bluestore.onode(0x7fbf66ac0880).flush flush done
 -1197> 2017-08-10 04:51:18.208227 7fbf35636700 10 bluestore(/var/lib/ceph/osd/ceph-7) omap_get_values meta oid #-1:c0371625:::snapmapper:0# = 0
  -521> 2017-08-10 04:51:18.213084 7fbf35636700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.2-604-gfcbb717/rpm/el7/BUILD/ceph-12.1.2-604-gfcbb717/src/osd/PG.cc: In function 'void PG::update_snap_map(const std::vector<pg_log_entry_t>&, ObjectStore::Transaction&)' thread 7fbf35636700 time 2017-08-10 04:51:18.208235
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.1.2-604-gfcbb717/rpm/el7/BUILD/ceph-12.1.2-604-gfcbb717/src/osd/PG.cc: 3377: FAILED assert(r == 0)

 ceph version 12.1.2-604-gfcbb717 (fcbb7171320c178b13f039580a27b5311b80f1ff) luminous (rc)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7fbf5a923690]
 2: (PG::update_snap_map(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, ObjectStore::Transaction&)+0x278) [0x7fbf5a455b48]
 3: (PG::append_log(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, eversion_t, eversion_t, ObjectStore::Transaction&, bool)+0x4d9) [0x7fbf5a459609]
 4: (PrimaryLogPG::log_operation(std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t> const&, eversion_t const&, eversion_t const&, bool, ObjectStore::Transaction&)+0x74) [0x7fbf5a56a094]
 5: (ReplicatedBackend::do_repop(boost::intrusive_ptr<OpRequest>)+0x8b4) [0x7fbf5a677044]
 6: (ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2b4) [0x7fbf5a679f64]
 7: (PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x50) [0x7fbf5a5a3ae0]
 8: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x58e) [0x7fbf5a50fc9e]
 9: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3f9) [0x7fbf5a3a3039]
 10: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x7fbf5a612047]
 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xfce) [0x7fbf5a3ce65e]
 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x839) [0x7fbf5a9291a9]
 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x7fbf5a92b140]
 14: (()+0x7dc5) [0x7fbf5770adc5]
 15: (clone()+0x6d) [0x7fbf567fe73d]

/a/sage-2017-08-10_04:21:32-rados-wip-sage-testing-20170809a-distro-basic-smithi/1505716
rados/thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/short_pg_log.yaml backoff/normal.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml} d-require-luminous/at-mkfs.yaml msgr-failures/few.yaml msgr/random.yaml objectstore/bluestore-comp.yaml rados.yaml rocksdb.yaml thrashers/mapgap.yaml thrashosds-health.yaml workloads/snaps-few-objects.yaml}

teuthology:1505716  02:15 PM $ grep snap_mapper remote/*/log/*osd.7* | grep 2:0ad6ceaa:::smithi03729118-18:2a6
2017-08-10 04:50:29.894508 7f2f4d2ac700 20 snap_mapper.remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
2017-08-10 04:50:29.894511 7f2f4d2ac700 20 snap_mapper._remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
2017-08-10 04:50:29.894571 7f2f4d2ac700 20 snap_mapper.add_oid 2:0ad6ceaa:::smithi03729118-18:2a6 2a2
2017-08-10 04:50:29.894595 7f2f4d2ac700 20 snap_mapper.set_snaps 2:0ad6ceaa:::smithi03729118-18:2a6 2a2
2017-08-10 04:51:18.208183 7fbf35636700 20 snap_mapper.remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
2017-08-10 04:51:18.208187 7fbf35636700 20 snap_mapper._remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
 -1210> 2017-08-10 04:51:18.208183 7fbf35636700 20 snap_mapper.remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
 -1209> 2017-08-10 04:51:18.208187 7fbf35636700 20 snap_mapper._remove_oid 2:0ad6ceaa:::smithi03729118-18:2a6
teuthology:1505716  02:15 PM $ grep MAP_00000000000002A2_0000000000000002.05B63755.2a6.smithi03729118-18.. remote/*/log/*osd.7*
2017-08-10 04:50:29.894608 7f2f4d2ac700 20 snap_mapper.add_oid set MAP_00000000000002A2_0000000000000002.05B63755.2a6.smithi03729118-18..
                "MAP_00000000000002A2_0000000000000002.05B63755.2a6.smithi03729118-18..": 70
2017-08-10 04:50:29.894899 7f2f4d2ac700 30 bluestore(/var/lib/ceph/osd/ceph-7) _omap_setkeys  0x0000000000000402'.MAP_00000000000002A2_0000000000000002.05B63755.2a6.smithi03729118-18..' <- MAP_00000000000002A2_0000000000000002.05B63755.2a6.smithi03729118-18..