Project

General

Profile

Actions

Bug #53409

closed

seastore crashes due to no available space

Added by Xuehan Xu over 2 years ago. Updated almost 2 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

INFO 2021-11-26 19:12:19,461 [shard 0] seastore - mark open: [0,814], empty_segments 3
INFO 2021-11-26 19:12:19,463 [shard 0] seastore - mark closed: [0,325] empty_segments: 6
INFO 2021-11-26 19:12:20,079 [shard 0] osd - asok response length: 4441
INFO 2021-11-26 19:12:20,089 [shard 0] osd - asok response length: 80942
ERROR 2021-11-26 19:12:20,413 [shard 0] ms - ms_dispatch unhandled message ping magic: 0 v1
INFO 2021-11-26 19:12:20,420 [shard 0] osd - found existing watch by client.4131
INFO 2021-11-26 19:12:22,835 [shard 0] seastore - mark open: [0,1016], empty_segments 2
INFO 2021-11-26 19:12:22,835 [shard 0] seastore - mark closed: [0,1591] empty_segments: 5
INFO 2021-11-26 19:12:22,836 [shard 0] seastore - mark open: [0,1598], empty_segments 1
INFO 2021-11-26 19:12:22,836 [shard 0] seastore - mark closed: [0,1338] empty_segments: 4
INFO 2021-11-26 19:12:22,843 [shard 0] seastore - mark open: [0,1599], empty_segments 0
INFO 2021-11-26 19:12:22,843 [shard 0] seastore - mark closed: [0,814] empty_segments: 3
ERROR 2021-11-26 19:12:22,995 [shard 0] seastore - BlockSegmentManager::open: invalid segment [0,268435450]
ERROR 2021-11-26 19:12:22,995 [shard 0] none - ../src/crimson/common/errorator.h:1162 : In function 'void crimson::ct_error::assert_all::operator()(ErrorT&&) [with ErrorT = const crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::invalid_argument>))>&]', abort(s)
abort() called
Aborting on shard 0.
Backtrace:
0# gsignal in /lib64/libc.so.6
1# abort in /lib64/libc.so.6
2# ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > constx%x
) at /home/xuxuehan/src/ceph/build/../src/crimson/common/assert.cc:59
3# void crimson::ct_error::assert_all::operator()<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)22> > const&>(crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)22> > const&) at /home/xuxuehan/src/ceph/build/../src/crimson/common/errorator.h:1164
4# ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILSt4errc5EEEEEENS1_IS4_L_ZNS5_ILS6_22EEEEEENS1_IS4_L_ZNS5_ILS6_2EEEEEEEE7_futureINS_23errorated_future_markerIvEEE24_safe_then_handle_errorsINS0_IJS7_EE8futurizeIN7seastar6futureIvEEEESK_ZNS_L8composerINSG_12pass_furtherEJNS_8ct_error10assert_allEEEEDaOT_DpOT0_EUlDpOT_E_EEDaOT0_OT1 at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/invoke.h:60
5# _ZN7seastar12continuationINS_8internal22promise_base_with_typeIvEEZN7crimson9erroratorIJNS4_19unthrowable_wrapperIRKSt10error_codeL_ZNS4_2ecILSt4errc5EEEEEENS6_IS9_L_ZNSA_ILSB_22EEEEEENS6_IS9_L_ZNSA_ILSB_2EEEEEEEE7_futureINS4_23errorated_future_markerIvEEE12handle_errorIZNS4_L8composerINS5_IJSC_EE12pass_furtherEJNS4_8ct_error10assert_allEEEEDaOT_DpOT0_EUlDpOT_E_EEDaSR_EUlSR_E_ZNS_6futureIvE17then_wrapped_nrvoINSM_7_futureISI_EESZ_EENS_8futurizeISQ_E4typeEOT0_EUlOS3_RSZ_ONS_12future_stateINS1_9monostateEEEE_vE15run_and_disposeEv at /home/xuxuehan/src/ceph/build/../src/seastar/include/seastar/core/future.hh:566
6# seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/reactor.cc:2231
7# seastar::reactor::run_some_tasks() [clone .part.0] at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/reactor.cc:2640
8# seastar::reactor::do_run() at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/reactor.cc:2809
9# seastar::reactor::run() at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/reactor.cc:2697
10# seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/app-template.cc:218
11# seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /home/xuxuehan/src/ceph/build/../src/seastar/src/core/app-template.cc:125
12# main at /home/xuxuehan/src/ceph/build/../src/crimson/osd/main.cc:239
13# __libc_start_main in /lib64/libc.so.6
14# _start in /home/xuxuehan/src/ceph/build/bin/crimson-osd

Actions #1

Updated by Xuehan Xu over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Xuehan Xu over 2 years ago

  • Description updated (diff)
Actions #3

Updated by Yingxin Cheng almost 2 years ago

  • Status changed from New to Can't reproduce

GC reserved available space has been adjusted, reopen if it happens again.

Actions

Also available in: Atom PDF