Project

General

Profile

Backport #23093

luminous: last-stat-seq returns 0 because osd stats are cleared

Added by David Zafman over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
Release:
luminous

Description

I added an assert which crashes ceph-mgr because PGMap::apply_incremental() processes a osd_stat_t that is all zeros.

(gdb) bt
#0 0x00007f5e7aa8320b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1 0x00007f5e7c605307 in reraise_fatal (signum=6) at /home/dzafman/ceph/src/global/signal_handler.cc:74
#2 handle_fatal_signal (signum=6) at /home/dzafman/ceph/src/global/signal_handler.cc:138
#3 <signal handler called>
#4 0x00007f5e79a9dcc9 in GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5 0x00007f5e79aa10d8 in _GI_abort () at abort.c:89
#6 0x00007f5e7c640580 in ceph::
_ceph_assert_fail (assertion=assertion@entry=0x7f5e7c9c682b "new_stats.seq != 0", file=file@entry=0x7f5e7c9c48d8 "/home/dzafman/ceph/src/mon/PGMap.cc", line=line@entry=1163,
func=func@entry=0x7f5e7c9c9820 <PGMap::apply_incremental(CephContext*, PGMap::Incremental const&)::__PRETTY_FUNCTION
> "void PGMap::apply_incremental(CephContext*, const PGMap::Incremental&)") at /home/dzafman/ceph/src/common/assert.cc:66
#7 0x00007f5e7c4995cf in PGMap::apply_incremental (this=this@entry=0x7f5e86d6d790, cct=0x7f5e86a22380, inc=...) at /home/dzafman/ceph/src/mon/PGMap.cc:1163
#8 0x00007f5e7c4e0b94 in ClusterState::notify_osdmap (this=0x7f5e86d6d430, osd_map=...) at /home/dzafman/ceph/src/mgr/ClusterState.cc:169
#9 0x00007f5e7c51c998 in Mgr::__lambda11::operator() (_closure=_closure@entry=0x7f5e73e6c6b0, osd_map=...) at /home/dzafman/ceph/src/mgr/Mgr.cc:441
#10 0x00007f5e7c51d15b in with_osdmap<Mgr::handle_osd_map()::__lambda11> (cb=<unknown type in /src/ceph/build/bin/ceph-mgr, CU 0x17ff6eb, DIE 0x1a3c9f7>, this=<optimized out>) at /home/dzafman/ceph/src/osdc/Objecter.h:2045
#11 Mgr::handle_osd_map (this=this@entry=0x7f5e86d6d000) at /home/dzafman/ceph/src/mgr/Mgr.cc:442
#12 0x00007f5e7c51e5be in Mgr::ms_dispatch (this=this@entry=0x7f5e86d6d000, m=m@entry=0x7f5e86f393c0) at /home/dzafman/ceph/src/mgr/Mgr.cc:483
#13 0x00007f5e7c512c70 in MgrStandby::ms_dispatch (this=0x7fffe1ef4800, m=0x7f5e86f393c0) at /home/dzafman/ceph/src/mgr/MgrStandby.cc:380
#14 0x00007f5e7c8e8e2b in ms_deliver_dispatch (m=0x7f5e86f393c0, this=0x7f5e86a16e00) at /home/dzafman/ceph/src/msg/Messenger.h:668
#15 DispatchQueue::entry (this=0x7f5e86a16f58) at /home/dzafman/ceph/src/msg/DispatchQueue.cc:197
#16 0x00007f5e7c6e41fd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /home/dzafman/ceph/src/msg/DispatchQueue.h:101
#17 0x00007f5e7aa7b182 in start_thread (arg=0x7f5e73e6e700) at pthread_create.c:312
#18 0x00007f5e79b6147d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

History

#1 Updated by David Zafman over 1 year ago

  • Tracker changed from Bug to Backport

#2 Updated by David Zafman over 1 year ago

  • Status changed from Verified to In Progress

#3 Updated by David Zafman over 1 year ago

  • Status changed from In Progress to Resolved

#4 Updated by Nathan Cutler over 1 year ago

  • Target version set to v12.2.3

#5 Updated by Nathan Cutler over 1 year ago

  • Target version changed from v12.2.3 to v12.2.4

Also available in: Atom PDF