Project

General

Profile

Bug #12528

legacy pg log is around after OSD starts

Added by Kefu Chai over 8 years ago. Updated almost 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

see /a/ubuntu-2015-07-28_21:20:52-rados-master---basic-multi/990944/remote/plana10/log/ceph-osd.5.log.gz

2015-07-29 10:20:57.731019 4039180 20 read_log coll 0.41_head log_oid 0/41//head
2015-07-29 10:20:57.731780 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.41_head/0/41//head = 0 (size 40)
2015-07-29 10:20:57.885894 4039180 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 4039180 time 2015-07-29 10:20:57.733727
osd/PGLog.cc: 883: FAILED assert(st.st_size == 0)

 ceph version 9.0.2-892-g1113eb6 (1113eb6b98cb8224f8c3e407435923de7415ca3f)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xafd1ef]
 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)+0x27bf) [0x736dbf]
 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x48a) [0x7b320a]
 4: (OSD::load_pgs()+0x680) [0x679af0]
 5: (OSD::init()+0x1458) [0x68aee8]
 6: (main()+0x25c7) [0x629a17]
 7: (__libc_start_main()+0xed) [0x6c6776d]
 8: ceph-osd() [0x62f379]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
   -28> 2015-07-29 10:20:57.715943 4039180 20 read_log coll 0.40_head log_oid 0/40//head
   -27> 2015-07-29 10:20:57.716726 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.40_head/0/40//head = 0 (size 0)
   -26> 2015-07-29 10:20:57.716892 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) get_omap_iterator 0.40_head/0/40//head
   -25> 2015-07-29 10:20:57.720003 4039180 10 read_log done
   -24> 2015-07-29 10:20:57.720336 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive] handle_loaded
   -23> 2015-07-29 10:20:57.720718 4039180  5 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] exit Initial 0.008965 0 0.000000
   -22> 2015-07-29 10:20:57.721130 4039180  5 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] enter Reset
   -21> 2015-07-29 10:20:57.721460 4039180 20 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=0 pi=1044-1061/1 crt=0'0 inactive NOTIFY] set_last_peering_reset 1270
   -20> 2015-07-29 10:20:57.721809 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] Clearing blocked outgoing recovery messages
   -19> 2015-07-29 10:20:57.722164 4039180 10 osd.5 pg_epoch: 1270 pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] Not blocking outgoing recovery messages
   -18> 2015-07-29 10:20:57.722514 4039180 10 osd.5 1292 load_pgs loaded pg[0.40( empty local-les=1063 n=0 ec=1 les/c 1063/1068 1062/1062/965) [1,5] r=1 lpr=1270 pi=1044-1061/1 crt=0'0 inactive NOTIFY] log((0'0,0'0], crt=0'0)
   -17> 2015-07-29 10:20:57.722832 4039180 10 osd.5 1292 pgid 0.41 coll 0.41_head
   -16> 2015-07-29 10:20:57.723157 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head
   -15> 2015-07-29 10:20:57.725491 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head = 0
   -14> 2015-07-29 10:20:57.725654 4039180 10 osd.5 1292 _open_lock_pg 0.41
   -13> 2015-07-29 10:20:57.725743 4039180 10 osd.5 1292 _get_pool 0
   -12> 2015-07-29 10:20:57.726689 4039180  5 osd.5 pg_epoch: 1286 pg[0.41(unlocked)] enter Initial
   -11> 2015-07-29 10:20:57.727188 4039180 20 osd.5 pg_epoch: 1286 pg[0.41(unlocked)] enter NotTrimming
   -10> 2015-07-29 10:20:57.727485 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head
    -9> 2015-07-29 10:20:57.730279 4039180 15 filestore(/var/lib/ceph/osd/ceph-5) omap_get_values 0.41_head/0/41//head = 0
    -8> 2015-07-29 10:20:57.731019 4039180 20 read_log coll 0.41_head log_oid 0/41//head
    -7> 2015-07-29 10:20:57.731780 4039180 10 filestore(/var/lib/ceph/osd/ceph-5) stat 0.41_head/0/41//head = 0 (size 40)
    -6> 2015-07-29 10:20:57.885894 4039180 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 4039180 time 2015-07-29 10:20:57.733727
osd/PGLog.cc: 883: FAILED assert(st.st_size == 0)

History

#1 Updated by Kefu Chai over 8 years ago

/a/ubuntu-2015-07-28_20:43:19-rados-wip-kefu-t3sting---basic-multi/990637/remote/plana93/log/ceph-osd.0.log.gz

   -11> 2015-07-29 11:24:12.297108 403f940 10 osd.0 2153 pgid 0.39 coll 0.39_head
   -10> 2015-07-29 11:24:12.297334 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head
    -9> 2015-07-29 11:24:12.298596 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head = 0
    -8> 2015-07-29 11:24:12.298707 403f940 10 osd.0 2153 _open_lock_pg 0.39
    -7> 2015-07-29 11:24:12.298774 403f940 10 osd.0 2153 _get_pool 0
    -6> 2015-07-29 11:24:12.299353 403f940  5 osd.0 pg_epoch: 2078 pg[0.39(unlocked)] enter Initial
    -5> 2015-07-29 11:24:12.299721 403f940 20 osd.0 pg_epoch: 2078 pg[0.39(unlocked)] enter NotTrimming
    -4> 2015-07-29 11:24:12.299938 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head
    -3> 2015-07-29 11:24:12.301414 403f940 15 filestore(/var/lib/ceph/osd/ceph-0) omap_get_values 0.39_head/0/39//head = 0
    -2> 2015-07-29 11:24:12.301938 403f940 20 read_log coll 0.39_head log_oid 0/39//head
    -1> 2015-07-29 11:24:12.302424 403f940 10 filestore(/var/lib/ceph/osd/ceph-0) stat 0.39_head/0/39//head = 0 (size 46)
     0> 2015-07-29 11:24:12.325363 403f940 -1 osd/PGLog.cc: In function 'static void PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, const pg_info_t&, std::map<eversion_t, hobject_t>&, PGLog::IndexedLog&, pg_missing_t&, std::ostringstream&, std::set<std::basic_string<char> >*)' thread 403f940 time 2015-07-29 11:24:12.303164
osd/PGLog.cc: 883: FAILED assert(st.st_size == 0)

 ceph version 9.0.2-881-g490bdf4 (490bdf4917d16831e0576af530339b8883f96268)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xc491bb]
 2: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >*)+0x174b) [0x7a65eb]
 3: (PG::read_state(ObjectStore*, ceph::buffer::list&)+0x4b9) [0x8326c9]
 4: (OSD::load_pgs()+0x603) [0x6d97b3]
 5: (OSD::init()+0x73f) [0x6eb0cf]
 6: (main()+0x2786) [0x66a0a6]
 7: (__libc_start_main()+0xf5) [0x7991ec5]
 8: ceph-osd() [0x6968e7]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

#2 Updated by Kefu Chai over 8 years ago

  • Subject changed from legacy pg log is still around to legacy pg log is around after OSD starts

#3 Updated by Sage Weil almost 7 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF