Actions
Bug #58162
openseastar sharded object destructor assert when crimosd handle error
Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The backtrace :
crimson-osd: /home/ceph-2/ceph/src/seastar/include/seastar/core/sharded.hh:564: seastar::sharded<T>::~sharded() [with Service = crimson::common::PerfCountersCollection]: Assertion `_instances.empty()' failed.
Aborting on shard 0.
Backtrace:
0# gsignal in /lib64/libc.so.6
1# abort in /lib64/libc.so.6
2# 0x00007FD5D1AA7C89 in /lib64/libc.so.6
3# 0x00007FD5D1ACD3A6 in /lib64/libc.so.6
4# seastar::sharded<crimson::common::PerfCountersCollection>::~sharded() at /home/ceph-2/ceph/src/seastar/include/seastar/core/sharded.hh:565
5# 0x00007FD5D1AD71EC in /lib64/libc.so.6
6# on_exit in /lib64/libc.so.6
7# operator()<std::error_code> at /home/ceph-2/ceph/src/crimson/osd/osd.cc:194
8# operator()<seastar::future<> > at /home/ceph-2/ceph/src/crimson/common/errorator.h:765
9# void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}&&) at /home/ceph-2/ceph/src/seastar/include/seastar/util/noncopyable_function.hh:209
10# seastar::future<void>::then_wrapped_nrvo<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)> >(seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> (seastar::future<void>&&)>&, seastar::future_state<seastar::internal::monostate>&&) const at /home/ceph-2/ceph/src/seastar/include/seastar/core/future.hh:1673
11# seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /home/ceph-2/ceph/src/seastar/src/core/reactor.cc:2353
12# seastar::reactor::run_some_tasks() at /home/ceph-2/ceph/src/seastar/src/core/reactor.cc:2766
13# seastar::reactor::do_run() at /home/ceph-2/ceph/src/seastar/src/core/reactor.cc:2934
14# seastar::reactor::run() at /home/ceph-2/ceph/src/seastar/src/core/reactor.cc:2817
15# seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /home/ceph-2/ceph/src/seastar/src/core/app-template.cc:265
16# seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) at /home/ceph-2/ceph/src/seastar/src/core/app-template.cc:156
17# main at /home/ceph-2/ceph/src/crimson/osd/main.cc:219
18# __libc_start_main in /lib64/libc.so.6
19# _start in /home/ceph-2/ceph/build/bin/crimson-osd
error code is :
seastar::future<> OSD::mkfs(
FuturizedStore &store,
unsigned whoami,
uuid_d osd_uuid,
uuid_d cluster_fsid,
std::string osdspec_affinity)
{
return store.start().then([&store, osd_uuid] {
return store.mkfs(osd_uuid).handle_error(
crimson::stateful_ec::handle([] (const auto& ec) {
logger().error("error creating empty object store in {}: ({}) {}",
local_conf().get_val<std::string>("osd_data"),
ec.value(), ec.message());
std::exit(EXIT_FAILURE);
}));
}).then([&store] { ...
when mkfs handle error, std::exit will not stop sharded object but destory the sharded object directly, so the assertion happens.
Updated by Yingxin Cheng over 1 year ago
when mkfs handle error, std::exit will not stop sharded object but destory the sharded object directly, so the assertion happens.
Looks to me it is acceptable at the current stage to abort the program under the error without graceful shutdown, like many other places that call ceph_abort() under the error.
I think the more important question here is what causes the error and how to reproduce it.
Updated by Rixin Luo over 1 year ago
crimson with alien bluestore, when aio-max-nr is not enough, it will cause mkfs error. It can reproduced by reducing aio-max-nr or using more osds.
Actions