Project

General

Profile

Actions

Bug #35923

closed

"ceph_assert(values.size() == 2)" in PG::peek_map_epoch()

Added by Kefu Chai over 5 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

now, there are two keys to check:

/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.0.0-3041-gdadd632/rpm/el7/BUILD/ceph-14.0.0-3041-gdadd632/src/osd/PG.cc: 3673: FAILED ceph_assert(values.size() == 2)

 ceph version 14.0.0-3041-gdadd632 (dadd6326d350cd41408f29024855044a34c43c50) nautilus (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x150) [0x55f601f8fb7f]
 2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55f601f8fd04]
 3: (PG::peek_map_epoch(ObjectStore*, spg_t, unsigned int*)+0x49e) [0x55f60214589e]
 4: (OSD::load_pgs()+0x498) [0x55f6020d7768]
 5: (OSD::init()+0xe2e) [0x55f6020d93ae]
 6: (main()+0x132d) [0x55f601f931ad]
 7: (__libc_start_main()+0xf5) [0x7f95ed7023d5]
 8: (()+0x5161a5) [0x55f6020771a5]

/a/kchai-2018-09-11_07:54:44-rados-wip-kefu2-testing-2018-09-10-1716-distro-basic-smithi/3005262

looks like it's a dup of https://www.spinics.net/lists/ceph-users/msg40913.html

Actions #1

Updated by Kefu Chai over 5 years ago

#10629 has the same backtrace.

Actions #2

Updated by Sage Weil over 5 years ago

  • Project changed from Ceph to RADOS
Actions #3

Updated by Sage Weil over 5 years ago

  • Status changed from New to In Progress
  • Assignee set to Sage Weil
  • Priority changed from Normal to Urgent

This is fall-out from merge vs delete pg resurrection:

2018-09-11 11:32:12.024 7fdd90e4a700  1 osd.3 pg_epoch: 1095 pg[2.16( v 889'463 (693'338,889'463] lb MIN (bitwise) local-lis/les=921/922 n=26 ec=734/18 lis/c 921/921 les/c/f 922/922/0 1095/1095/1095) [] r=-1 lpr=1095 DELETING pi=[921,1095)/2 crt=889'463 lcod 889'461 unknown NOTIFY mbc={} ps=82] _delete_some raced w
ith merge, reinstantiating
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _osr_attach 2.16_head resurrecting zombie osr 0x55e130f2de60
2018-09-11 11:32:12.024 7fdd90e4a700 10 log is not dirty
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) queue_transactions ch 0x55e1350c4f00 2.16_head
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _txc_create osr 0x55e130f2de60 = 0x55e13b3c9200 seq 9
2018-09-11 11:32:12.024 7fdd90e4a700 15 bluestore(/var/lib/ceph/osd/ceph-3) _create_collection 2.16_head bits 5
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _create_collection 2.16_head bits 5 = 0
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _txc_add_transaction collection hint objects is a no-op,  pg_num 26 num_objects 0
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3).collection(2.16_head 0x55e1350c4f00) get_onode oid #2:68000000::::head# key 0x7f80000000000000026800000021213dfffffffffffffffeffffffffffffffff'o'
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3).collection(2.16_head 0x55e1350c4f00)  r -2 v.len 0
2018-09-11 11:32:12.024 7fdd90e4a700 15 bluestore(/var/lib/ceph/osd/ceph-3) _touch 2.16_head #2:68000000::::head#
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _assign_nid 4495
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _touch 2.16_head #2:68000000::::head# = 0
2018-09-11 11:32:12.024 7fdd90e4a700 15 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 2.16_head #2:68000000::::head#
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys  0x000000000000118f'._infover' <- _infover
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 2.16_head #2:68000000::::head# = 0
2018-09-11 11:32:12.024 7fdd90e4a700 15 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 2.16_head #2:68000000::::head#
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys  0x000000000000118f'._biginfo' <- _biginfo
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys  0x000000000000118f'._info' <- _info
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _omap_setkeys 2.16_head #2:68000000::::head# = 0
2018-09-11 11:32:12.024 7fdd90e4a700 10 bluestore(/var/lib/ceph/osd/ceph-3) _txc_calc_cost 0x55e13b3c9200 cost 6472 (1 ios * 4000 + 2472 bytes)
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _txc_write_nodes txc 0x55e13b3c9200 onodes 0x55e133a78240 shared_blobs 
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore.extentmap(0x55e133a78330) update #2:68000000::::head#
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore.extentmap(0x55e133a78330) update  inline shard 2 bytes from 0 extents
2018-09-11 11:32:12.024 7fdd90e4a700 20 bluestore(/var/lib/ceph/osd/ceph-3) _record_onode onode #2:68000000::::head# is 29 (21 bytes onode + 2 bytes spanning blobs + 6 bytes inline extents)

we set infover and biginfo, but not the pg epoch.

Actions #4

Updated by Sage Weil over 5 years ago

  • Status changed from In Progress to Fix Under Review
Actions #5

Updated by Kefu Chai over 5 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF