Bug #12528
legacy pg log is around after OSD starts
Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
see /a/ubuntu-2015-07-28_21:20:52-rados-master---basic-multi/990944/remote/plana10/log/ceph-osd.5.log.gz
2015-07-29 10:20:57.731019 4039180 20 read_log coll 0.41_head log_oid 0/41//head 2015-07-29 10:20:57.731780 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.41_head/0/41//head = 0 (size 40) 2015-07-29 10:20:57.885894 4039180 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 4039180 time 2015-07-29 10:20:57.733727 osd/PGLog.cc: 883: FAILED assert(st.st_size == 0) ceph version 9.0.2-892-g1113eb6 (1113eb6b98cb8224f8c3e407435923de7415ca3f) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xafd1ef] 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)+0x27bf) [0x736dbf] 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x48a) [0x7b320a] 4: (OSD::load_pgs()+0x680) [0x679af0] 5: (OSD::init()+0x1458) [0x68aee8] 6: (main()+0x25c7) [0x629a17] 7: (__libc_start_main()+0xed) [0x6c6776d] 8: ceph-osd() [0x62f379] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-28> 2015-07-29 10:20:57.715943 4039180 20 read_log coll 0.40_head log_oid 0/40//head -27> 2015-07-29 10:20:57.716726 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.40_head/0/40//head = 0 (size 0) -26> 2015-07-29 10:20:57.716892 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) get_omap_iterator 0.40_head/0/40//head -25> 2015-07-29 10:20:57.720003 4039180 10 read_log done -24> 2015-07-29 10:20:57.720336 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive] handle_loaded -23> 2015-07-29 10:20:57.720718 4039180 5 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] exit Initial 0.008965 0 0.000000 -22> 2015-07-29 10:20:57.721130 4039180 5 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] enter Reset -21> 2015-07-29 10:20:57.721460 4039180 20 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] set_last_peering_reset 1270 -20> 2015-07-29 10:20:57.721809 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] Clearing blocked outgoing recovery messages -19> 2015-07-29 10:20:57.722164 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] Not blocking outgoing recovery messages -18> 2015-07-29 10:20:57.722514 4039180 10 osd.5 1292 load_pgs loaded pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] log((0'0,0'0], crt=0'0) -17> 2015-07-29 10:20:57.722832 4039180 10 osd.5 1292 pgid 0.41 coll 0.41_head -16> 2015-07-29 10:20:57.723157 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head -15> 2015-07-29 10:20:57.725491 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head = 0 -14> 2015-07-29 10:20:57.725654 4039180 10 osd.5 1292 _open_lock_pg 0.41 -13> 2015-07-29 10:20:57.725743 4039180 10 osd.5 1292 _get_pool 0 -12> 2015-07-29 10:20:57.726689 4039180 5 osd.5 pg_epoch: 1286 pg[0.41(unlocked)] enter Initial -11> 2015-07-29 10:20:57.727188 4039180 20 osd.5 pg_epoch: 1286 pg[0.41(unlocked)] enter NotTrimming -10> 2015-07-29 10:20:57.727485 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head -9> 2015-07-29 10:20:57.730279 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head = 0 -8> 2015-07-29 10:20:57.731019 4039180 20 read_log coll 0.41_head log_oid 0/41//head -7> 2015-07-29 10:20:57.731780 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.41_head/0/41//head = 0 (size 40) -6> 2015-07-29 10:20:57.885894 4039180 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 4039180 time 2015-07-29 10:20:57.733727 osd/PGLog.cc: 883: FAILED assert(st.st_size == 0)
History
#1 Updated by Kefu Chai over 8 years ago
/a/ubuntu-2015-07-28_20:43:19-rados-wip-kefu-t3sting---basic-multi/990637/remote/plana93/log/ceph-osd.0.log.gz
-11> 2015-07-29 11:24:12.297108 403f940 10 osd.0 2153 pgid 0.39 coll 0.39_head -10> 2015-07-29 11:24:12.297334 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head -9> 2015-07-29 11:24:12.298596 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head = 0 -8> 2015-07-29 11:24:12.298707 403f940 10 osd.0 2153 _open_lock_pg 0.39 -7> 2015-07-29 11:24:12.298774 403f940 10 osd.0 2153 _get_pool 0 -6> 2015-07-29 11:24:12.299353 403f940 5 osd.0 pg_epoch: 2078 pg[0.39(unlocked)] enter Initial -5> 2015-07-29 11:24:12.299721 403f940 20 osd.0 pg_epoch: 2078 pg[0.39(unlocked)] enter NotTrimming -4> 2015-07-29 11:24:12.299938 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head -3> 2015-07-29 11:24:12.301414 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head = 0 -2> 2015-07-29 11:24:12.301938 403f940 20 read_log coll 0.39_head log_oid 0/39//head -1> 2015-07-29 11:24:12.302424 403f940 10 filestore(/var/lib/ceph/osd/ceph-0) stat 0.39_head/0/39//head = 0 (size 46) 0> 2015-07-29 11:24:12.325363 403f940 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 403f940 time 2015-07-29 11:24:12.303164 osd/PGLog.cc: 883: FAILED assert(st.st_size == 0) ceph version 9.0.2-881-g490bdf4 (490bdf4917d16831e0576af530339b8883f96268) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc491bb] 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)+0x174b) [0x7a65eb] 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x4b9) [0x8326c9] 4: (OSD::load_pgs()+0x603) [0x6d97b3] 5: (OSD::init()+0x73f) [0x6eb0cf] 6: (main()+0x2786) [0x66a0a6] 7: (__libc_start_main()+0xf5) [0x7991ec5] 8: ceph-osd() [0x6968e7] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
#2 Updated by Kefu Chai over 8 years ago
- Subject changed from legacy pg log is still around to legacy pg log is around after OSD starts
#3 Updated by Sage Weil almost 7 years ago
- Status changed from New to Can't reproduce