Actions
Bug #50746
openosd: terminate called after throwing an instance of 'std::out_of_range'
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
$ ../src/stop.sh
2021-05-11T12:31:36.802+0800 7f08741a31c0 -1 WARNING: all dangerous and experimental features are enabled.
2021-05-11T12:31:36.802+0800 7f08741a31c0 -1 WARNING: all dangerous and experimental features are enabled.
2021-05-11T12:31:36.894+0800 7f1056d361c0 -1 WARNING: all dangerous and experimental features are enabled.
2021-05-11T12:31:36.895+0800 7f1056d361c0 -1 WARNING: all dangerous and experimental features are enabled.
WARNING: ceph-osd ceph-mds still alive after 1 seconds
WARNING: ceph-osd ceph-mds still alive after 2 seconds
WARNING: ceph-osd ceph-mds still alive after 4 seconds
$ MDS=3 OSD=3 MON=3 MGR=1 ../src/vstart.sh -n -X -G --msgr1 --memstore
ceph-mgr dashboard not built - disabling.
rm -f core*
hostname fedora
ip 192.168.0.100
port 40728
/data/ceph/build/bin/ceph-authtool --create-keyring --gen-key --name=mon. /data/ceph/build/keyring --cap mon 'allow *'
creating /data/ceph/build/keyring
/data/ceph/build/bin/ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /data/ceph/build/keyring
/data/ceph/build/bin/monmaptool --create --clobber --addv a v1:192.168.0.100:40729 --addv b v1:192.168.0.100:40731 --addv c v1:192.168.0.100:40733 --print /tmp/ceph_monmap.105892
/data/ceph/build/bin/monmaptool: monmap file /tmp/ceph_monmap.105892
/data/ceph/build/bin/monmaptool: generated fsid 9dc503f8-44d8-4204-b669-9cec83ec3599
setting min_mon_release = octopus
epoch 0
fsid 9dc503f8-44d8-4204-b669-9cec83ec3599
last_changed 2021-05-11T12:31:47.709447+0800
created 2021-05-11T12:31:47.709447+0800
min_mon_release 15 (octopus)
election_strategy: 1
0: v1:192.168.0.100:40729/0 mon.a
1: v1:192.168.0.100:40731/0 mon.b
2: v1:192.168.0.100:40733/0 mon.c
/data/ceph/build/bin/monmaptool: writing epoch 0 to /tmp/ceph_monmap.105892 (3 monitors)
rm -rf -- /data/ceph/build/dev/mon.a
mkdir -p /data/ceph/build/dev/mon.a
/data/ceph/build/bin/ceph-mon --mkfs -c /data/ceph/build/ceph.conf -i a --monmap=/tmp/ceph_monmap.105892 --keyring=/data/ceph/build/keyring
rm -rf -- /data/ceph/build/dev/mon.b
mkdir -p /data/ceph/build/dev/mon.b
/data/ceph/build/bin/ceph-mon --mkfs -c /data/ceph/build/ceph.conf -i b --monmap=/tmp/ceph_monmap.105892 --keyring=/data/ceph/build/keyring
rm -rf -- /data/ceph/build/dev/mon.c
mkdir -p /data/ceph/build/dev/mon.c
/data/ceph/build/bin/ceph-mon --mkfs -c /data/ceph/build/ceph.conf -i c --monmap=/tmp/ceph_monmap.105892 --keyring=/data/ceph/build/keyring
rm -- /tmp/ceph_monmap.105892
/data/ceph/build/bin/ceph-mon -i a -c /data/ceph/build/ceph.conf
/data/ceph/build/bin/ceph-mon -i b -c /data/ceph/build/ceph.conf
/data/ceph/build/bin/ceph-mon -i c -c /data/ceph/build/ceph.conf
Populating config ...
[mgr]
mgr/telemetry/enable = false
mgr/telemetry/nag = false
creating /data/ceph/build/dev/mgr.x/keyring
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -i /data/ceph/build/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *'
added key for mgr.x
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf config set mgr mgr/prometheus/x/server_port 9283 --force
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf config set mgr mgr/restful/x/server_port 42728 --force
Starting mgr.x
/data/ceph/build/bin/ceph-mgr -i x -c /data/ceph/build/ceph.conf
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -h
waiting for mgr restful module to start
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -h
waiting for mgr restful module to start
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -h
waiting for mgr restful module to start
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -h
waiting for mgr restful module to start
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf -h
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf restful create-self-signed-cert
Restarting RESTful API server...
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf restful create-key admin -o /tmp/tmp.GJToTHrjqx
add osd0 39adb219-a662-47c1-a168-b288861a1d33
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf osd new 39adb219-a662-47c1-a168-b288861a1d33 -i /data/ceph/build/dev/osd0/new.json
0
/data/ceph/build/bin/ceph-osd -i 0 -c /data/ceph/build/ceph.conf --mkfs --key AQC8CJpgfnz4LhAAlincclLxoKn/cOOTCpX7zg== --osd-uuid 39adb219-a662-47c1-a168-b288861a1d33
2021-05-11T12:31:57.271+0800 7f51168cbfc0 -1 memstore(/data/ceph/build/dev/osd0) /data/ceph/build/dev/osd0
start osd.0
/data/ceph/build/bin/ceph-osd -i 0 -c /data/ceph/build/ceph.conf
add osd1 30ecdd25-6cd2-49fe-a05e-b17966fcebb1
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf osd new 30ecdd25-6cd2-49fe-a05e-b17966fcebb1 -i /data/ceph/build/dev/osd1/new.json
2021-05-11T12:31:57.485+0800 7fb541823fc0 -1 Falling back to public interface
2021-05-11T12:31:57.545+0800 7fb541823fc0 -1 osd.0 0 log_to_monitors {default=true}
2021-05-11T12:31:57.548+0800 7fb541823fc0 -1 osd.0 0 mon_cmd_maybe_osd_create fail: 'wrong fsid': (1) Operation not permitted
2021-05-11T12:31:57.549+0800 7fb541823fc0 -1 osd.0 0 init unable to update_crush_device_class: (1) Operation not permitted
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at
*** Caught signal (Aborted) **
in thread 7fb53dfc5640 thread_name:io_context_pool
ceph version 17.0.0-3870-gf7db8c12ed7 (f7db8c12ed71bbd7646c44bc7b6dfb281a09083a) quincy (dev)
1: /data/ceph/build/bin/ceph-osd(+0x3371b9a) [0x5555597f6b9a]
2: /lib64/libpthread.so.0(+0x13a00) [0x7fb541f8aa00]
3: gsignal()
4: abort()
5: /lib64/libstdc++.so.6(+0xa1a46) [0x7fb541d8fa46]
6: /lib64/libstdc++.so.6(+0xad28c) [0x7fb541d9b28c]
7: /lib64/libstdc++.so.6(+0xac2e9) [0x7fb541d9a2e9]
8: __gxx_personality_v0()
9: /lib64/libgcc_s.so.1(+0x11494) [0x7fb541b9e494]
10: _Unwind_Resume()
11: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x1ac) [0x555558cb6330]
12: (boost::asio::io_context::run()+0x48) [0x555558cb694c]
13: (ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const+0x1f) [0x555558cb82c7]
14: (void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x555558cc5784]
15: (std::__invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x555558cc441b]
16: (std::invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x555558cc1e06]
17: _ZZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_ENKUlSA_SD_E_clIS3_JEEEDaSA_SD_()
18: _ZSt13__invoke_implIvZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSB_SE_E_JS4_EESA_St14__invoke_otherOT0_DpOT1_()
19: _ZSt8__invokeIZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSB_SE_E_JS4_EENSt15__invoke_resultISA_JDpSC_EE4typeESB_SE_()
20: _ZNSt6thread8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSC_SF_E_S6_EEE9_M_invokeIJLm0ELm1EEEEvSt12_Index_tupleIJXspT_EEE()
21: _ZNSt6thread8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSC_SF_E_S6_EEEclEv()
22: _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv()
23: /lib64/libstdc++.so.6(+0xd9d54) [0x7fb541dc7d54]
24: /lib64/libpthread.so.0(+0x9299) [0x7fb541f80299]
25: clone()
1
/data/ceph/build/bin/ceph-osd -i 1 -c /data/ceph/build/ceph.conf --mkfs --key AQC9CJpg34/MGhAAL4HAAy5VoDRxI6ik5CFI3w== --osd-uuid 30ecdd25-6cd2-49fe-a05e-b17966fcebb1
2021-05-11T12:31:57.915+0800 7fe4ac428fc0 -1 memstore(/data/ceph/build/dev/osd1) /data/ceph/build/dev/osd1
start osd.1
/data/ceph/build/bin/ceph-osd -i 1 -c /data/ceph/build/ceph.conf
add osd2 cafb0db2-9a2c-418c-9e7c-99af2e7c9ab6
/data/ceph/build/bin/ceph -c /data/ceph/build/ceph.conf osd new cafb0db2-9a2c-418c-9e7c-99af2e7c9ab6 -i /data/ceph/build/dev/osd2/new.json
2021-05-11T12:31:58.068+0800 7f925ed38fc0 -1 Falling back to public interface
2021-05-11T12:31:58.119+0800 7f925ed38fc0 -1 osd.1 0 log_to_monitors {default=true}
2
/data/ceph/build/bin/ceph-osd -i 2 -c /data/ceph/build/ceph.conf --mkfs --key AQC+CJpgooIxAhAArq4w2qgcJsYYvzwxo9FF+Q== --osd-uuid cafb0db2-9a2c-418c-9e7c-99af2e7c9ab6
2021-05-11T12:31:58.500+0800 7f4de55fcfc0 -1 memstore(/data/ceph/build/dev/osd2) /data/ceph/build/dev/osd2
start osd.2
/data/ceph/build/bin/ceph-osd -i 2 -c /data/ceph/build/ceph.conf
2021-05-11T12:31:58.650+0800 7f5cb5928fc0 -1 Falling back to public interface
2021-05-11T12:31:58.703+0800 7f5cb5928fc0 -1 osd.2 0 log_to_monitors {default=true}
2021-05-11T12:31:58.708+0800 7f5cb5928fc0 -1 osd.2 0 mon_cmd_maybe_osd_create fail: 'wrong fsid': (1) Operation not permitted
2021-05-11T12:31:58.708+0800 7f5cb5928fc0 -1 osd.2 0 init unable to update_crush_device_class: (1) Operation not permitted
terminate called after throwing an instance of 'std::out_of_range'
what(): map::at
*** Caught signal (Aborted) **
in thread 7f5cb20ca640 thread_name:io_context_pool
ceph version 17.0.0-3870-gf7db8c12ed7 (f7db8c12ed71bbd7646c44bc7b6dfb281a09083a) quincy (dev)
1: /data/ceph/build/bin/ceph-osd(+0x3371b9a) [0x5629087dbb9a]
2: /lib64/libpthread.so.0(+0x13a00) [0x7f5cb608fa00]
3: gsignal()
4: abort()
5: /lib64/libstdc++.so.6(+0xa1a46) [0x7f5cb5e94a46]
6: /lib64/libstdc++.so.6(+0xad28c) [0x7f5cb5ea028c]
7: /lib64/libstdc++.so.6(+0xac2e9) [0x7f5cb5e9f2e9]
8: __gxx_personality_v0()
9: /lib64/libgcc_s.so.1(+0x11494) [0x7f5cb5ca3494]
10: _Unwind_Resume()
11: (boost::asio::detail::scheduler::run(boost::system::error_code&)+0x1ac) [0x562907c9b330]
12: (boost::asio::io_context::run()+0x48) [0x562907c9b94c]
13: (ceph::async::io_context_pool::start(short)::{lambda()#1}::operator()() const+0x1f) [0x562907c9d2c7]
14: (void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::{lambda()#1}>(std::__invoke_other, ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x562907caa784]
15: (std::__invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::__invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x562907ca941b]
16: (std::invoke_result<ceph::async::io_context_pool::start(short)::{lambda()#1}>::type std::invoke<ceph::async::io_context_pool::start(short)::{lambda()#1}>(ceph::async::io_context_pool::start(short)::{lambda()#1}&&)+0x20) [0x562907ca6e06]
17: _ZZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_ENKUlSA_SD_E_clIS3_JEEEDaSA_SD_()
18: _ZSt13__invoke_implIvZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSB_SE_E_JS4_EESA_St14__invoke_otherOT0_DpOT1_()
19: _ZSt8__invokeIZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEESt6threadSt17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSB_SE_E_JS4_EENSt15__invoke_resultISA_JDpSC_EE4typeESB_SE_()
20: _ZNSt6thread8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSC_SF_E_S6_EEE9_M_invokeIJLm0ELm1EEEEvSt12_Index_tupleIJXspT_EEE()
21: _ZNSt6thread8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSC_SF_E_S6_EEEclEv()
22: _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZ17make_named_threadIZN4ceph5async15io_context_pool5startEsEUlvE_JEES_St17basic_string_viewIcSt11char_traitsIcEEOT_DpOT0_EUlSD_SG_E_S7_EEEEE6_M_runEv()
23: /lib64/libstdc++.so.6(+0xd9d54) [0x7f5cb5eccd54]
24: /lib64/libpthread.so.0(+0x9299) [0x7f5cb6085299]
25: clone()
Updated by Neha Ojha almost 3 years ago
I ran the same command "MDS=3 OSD=3 MON=3 MGR=1 ../src/vstart.sh -n -X -G --msgr1 --memstore" and everything works fine for me.
services:
mon: 3 daemons, quorum a,b,c (age 36s)
mgr: x(active, since 28s)
mds: 1/1 daemons up, 2 standby
osd: 3 osds: 3 up (since 18s), 3 in (since 21s)
May have something to do with the environment.
Actions