Bug #56745
closedunrecognized option ssl_server_port appears to cause failure of MGR
0%
Description
This is going to take a second, sorry. This is a test environment and is open to destructive testing!
I deployed a fresh Ubuntu 20.04 system, deploy my defacto config scripts, docker and the basic Ceph Quincy 17.2.2 bootstrap to a VM, no OSDs. Then I deployed a second VM with two OSDs (one SSD and one HDD). Everything is working fine so far.
I then deployed a third install to a physical box with some drives and an SSD, monitoring services start on it and all is well. However, after deploying OSDs the the third node (second node with OSDs), withing sever seconds the manager service vanishes from the cluster.
Thought I did something wrong, wiped everything and re-deployed. Same exact issue...
I have been looking through the logs, and as best as I can tell, the MGR fails to deploy/redeploy due to these errors (private values/IPs removed):
cephadmin bash[2817]: debug -228> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/server_addr = #.#.#.# (unrecognized option) cephadmin bash[2817]: debug -227> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/server_port = 8080 (unrecognized option) cephadmin bash[2817]: debug -226> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option) cephadmin bash[2817]: debug -225> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/ssl_server_port = 8443 (unrecognized option) cephadmin bash[2817]: debug -224> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/orchestrator/orchestrator = cephadm (unrecognized option) cephadmin bash[2817]: debug -129> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/cephadm/container_init = True (unrecognized option) cephadmin bash[2817]: debug -128> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/cephadm/migration_current = 5 (unrecognized option) cephadmin bash[2817]: debug -127> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/ALERTMANAGER_API_HOST = http://###########:9093 (unrecognized option) cephadmin bash[2817]: debug -126> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/GRAFANA_API_SSL_VERIFY = false (unrecognized option) cephadmin bash[2817]: debug -125> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/GRAFANA_API_URL = https://###########:3000 (unrecognized option) cephadmin bash[2817]: debug -124> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/PROMETHEUS_API_HOST = http://###########:9095 (unrecognized option)
This causes the dashboard to attempt a starts up with a 0 for the port as best as I can tell:
from='client.? #.#.#.#:0/3815048734' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch
Everything goes down hill from there, the mgr crashes as it can't talk to its own services, finally the docker image and host services are purged...
I am not sure if it is something in my (very bare bones) config somehow, or a bug.. But... Any help would be appreciated. I have not Ceph since Luminous, so I am very out of date to say the least...
Note, I am not tagging the second and third node with "_admin", as I thought that caused it the first time.
Updated by Brian Woods over 1 year ago
Could also be related to:
https://tracker.ceph.com/issues/56726
As I am seeing this in the log, but...
1 mgr finish mon failed to return metadata for osd.6: (2) No such file or directory
Updated by Brian Woods over 1 year ago
Update, if I add an OSD to the first bootstrap node, this happens as soon as I add an OSD to the second node...
This seems like a fairly critical failure/bug...
Updated by Brian Woods over 1 year ago
I setup a watcher on the docker instance to do a snapshot as soon as the instance crashed and have been able to captured a couple of crashes:
ul 29 20:05:27 cephadmin bash[1282]: debug 2022-07-29T20:05:27.134+0000 7f20f0f81700 0 [devicehealth INFO root] creating mgr pool Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.134+0000 7f049a625700 0 mon.cephadmin@0(leader) e2 handle_command mon_command({"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32} v 0) v1 Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.134+0000 7f049a625700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.374+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e22 do_prune osdmap full prune enabled Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.374+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e22 encode_pending skipping prime_pg_temp; mapping job did not start Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.498+0000 7f0497e20700 1 mon.cephadmin@0(leader).osd e23 e23: 3 total, 3 up, 3 in Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 3314933000852226048, adjusting msgr requires Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 288514051259236352, adjusting msgr requires Jul 29 20:05:27 cephadmin bash[1283]: message repeated 2 times: [ debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 288514051259236352, adjusting msgr requires] Jul 29 20:05:27 cephadmin bash[1283]: cluster 2022-07-29T20:05:26.374617+0000 mon.cephadmin (mon.0) 378 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in Jul 29 20:05:27 cephadmin bash[1283]: audit 2022-07-29T20:05:27.136968+0000 mon.cephadmin (mon.0) 379 : audit [INF] from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 288514051259236352, adjusting msgr requires for clients Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 288514051259236352 was 288514050185503233, adjusting msgr requires for mons Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 3314933000852226048, adjusting msgr requires for osds Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f0497e20700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f0497e20700 0 log_channel(cluster) log [DBG] : osdmap e23: 3 total, 3 up, 3 in Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f049a625700 0 mon.cephadmin@0(leader) e2 handle_command mon_command({"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} v 0) v1 Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f049a625700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.266+0000 7f20e04cd700 0 [prometheus INFO cherrypy.access.139779795241728] ::ffff:192.168.10.##1 - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4" Jul 29 20:05:28 cephadmin bash[1282]: ::ffff:192.168.10.##1 - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4" Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.498+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e23 do_prune osdmap full prune enabled Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.518+0000 7f21007a0700 0 log_channel(cluster) log [DBG] : pgmap v477: 1 pgs: 1 unknown; 0 B data, 458 MiB used, 53 GiB / 53 GiB avail Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.558+0000 7f0497e20700 1 mon.cephadmin@0(leader).osd e24 e24: 3 total, 3 up, 3 in Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.582+0000 7f0497e20700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished ### Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.582+0000 7f0497e20700 0 log_channel(cluster) log [DBG] : osdmap e24: 3 total, 3 up, 3 in Jul 29 20:05:28 cephadmin bash[1282]: terminate called after throwing an instance of 'std::regex_error' Jul 29 20:05:28 cephadmin bash[1282]: what(): Invalid start of range in bracket expression. Jul 29 20:05:28 cephadmin bash[1282]: *** Caught signal (Aborted) ** Jul 29 20:05:28 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth Jul 29 20:05:28 cephadmin bash[8664]: debug 2022-07-29T20:05:28.610+0000 7fe1d1edc700 1 osd.0 pg_epoch: 23 pg[1.0( empty local-lis/les=0/0 n=0 ec=23/23 lis/c=0/0 les/c/f=0/0/0 sis=23) [1,0] r=1 lpr=23 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray Jul 29 20:05:28 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 29 20:05:28 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0] Jul 29 20:05:28 cephadmin bash[1282]: 2: gsignal() Jul 29 20:05:28 cephadmin bash[1282]: 3: abort() Jul 29 20:05:28 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b] Jul 29 20:05:28 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c] Jul 29 20:05:28 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597] Jul 29 20:05:28 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8] Jul 29 20:05:28 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea] Jul 29 20:05:28 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8] Jul 29 20:05:28 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6] Jul 29 20:05:28 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b] Jul 29 20:05:28 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a] Jul 29 20:05:28 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:28 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:28 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8] Jul 29 20:05:28 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:28 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:28 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb] Jul 29 20:05:28 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca] Jul 29 20:05:28 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486] Jul 29 20:05:28 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c] Jul 29 20:05:28 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9] Jul 29 20:05:28 cephadmin bash[1282]: 27: pysqlite_connection_init() Jul 29 20:05:28 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6] Jul 29 20:05:28 cephadmin bash[1282]: 29: PyObject_Call() Jul 29 20:05:28 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5] Jul 29 20:05:28 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1] Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.614+0000 7f20f0f81700 -1 *** Caught signal (Aborted) ** Jul 29 20:05:28 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth Jul 29 20:05:28 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 29 20:05:28 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0] Jul 29 20:05:28 cephadmin bash[1282]: 2: gsignal() Jul 29 20:05:28 cephadmin bash[1282]: 3: abort() Jul 29 20:05:28 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b] Jul 29 20:05:28 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c] Jul 29 20:05:28 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597] Jul 29 20:05:28 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8] Jul 29 20:05:28 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea] Jul 29 20:05:28 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8] Jul 29 20:05:28 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6] Jul 29 20:05:28 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b] Jul 29 20:05:28 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a] Jul 29 20:05:28 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:28 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:28 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8] Jul 29 20:05:28 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:28 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:28 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:28 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb] Jul 29 20:05:28 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca] Jul 29 20:05:28 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486] Jul 29 20:05:28 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c] Jul 29 20:05:28 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9] Jul 29 20:05:28 cephadmin bash[1282]: 27: pysqlite_connection_init() Jul 29 20:05:28 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6] Jul 29 20:05:28 cephadmin bash[1282]: 29: PyObject_Call() Jul 29 20:05:28 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5] Jul 29 20:05:28 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1] Jul 29 20:05:28 cephadmin bash[1282]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
################################################
Jul 29 20:05:29 cephadmin bash[1282]: debug -27> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 monclient: _finish_command 275 = system:0 pool '.mgr' created Jul 29 20:05:29 cephadmin bash[1282]: debug -26> 2022-07-29T20:05:27.602+0000 7f20fff9f700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -25> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 log_client handle_log_ack log(last 502) v1 Jul 29 20:05:29 cephadmin bash[1282]: debug -23> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 monclient: handle_get_version_reply finishing 120 version 23 Jul 29 20:05:29 cephadmin bash[1282]: debug -22> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}] Jul 29 20:05:29 cephadmin bash[1282]: debug -21> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 monclient: _send_command 276 [{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}] Jul 29 20:05:29 cephadmin bash[1282]: debug -20> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -19> 2022-07-29T20:05:27.934+0000 7f20fcf99700 10 monclient: tick Jul 29 20:05:29 cephadmin bash[1282]: debug -18> 2022-07-29T20:05:27.934+0000 7f20fcf99700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:57.937019+0000) Jul 29 20:05:29 cephadmin bash[1282]: debug -17> 2022-07-29T20:05:28.010+0000 7f20e4768700 10 monclient: tick Jul 29 20:05:29 cephadmin bash[1282]: debug -16> 2022-07-29T20:05:28.010+0000 7f20e4768700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.015124+0000) Jul 29 20:05:29 cephadmin bash[1282]: debug -15> 2022-07-29T20:05:28.030+0000 7f213d8b5700 10 monclient: tick Jul 29 20:05:29 cephadmin bash[1282]: debug -14> 2022-07-29T20:05:28.030+0000 7f213d8b5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.033887+0000) Jul 29 20:05:29 cephadmin bash[1282]: debug -13> 2022-07-29T20:05:28.082+0000 7f213b0b0700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -12> 2022-07-29T20:05:28.234+0000 7f20d73bb700 10 monclient: tick Jul 29 20:05:29 cephadmin bash[1282]: debug -11> 2022-07-29T20:05:28.234+0000 7f20d73bb700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.234298+0000) Jul 29 20:05:29 cephadmin bash[1282]: debug -10> 2022-07-29T20:05:28.266+0000 7f20e04cd700 0 [prometheus INFO cherrypy.access.139779795241728] ::ffff:192.168.10.### - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4" Jul 29 20:05:29 cephadmin bash[1282]: debug -9> 2022-07-29T20:05:28.518+0000 7f21007a0700 0 log_channel(cluster) log [DBG] : pgmap v477: 1 pgs: 1 unknown; 0 B data, 458 MiB used, 53 GiB / 53 GiB avail Jul 29 20:05:29 cephadmin bash[1282]: debug -8> 2022-07-29T20:05:28.518+0000 7f21007a0700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -7> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _renew_subs Jul 29 20:05:29 cephadmin bash[1282]: debug -6> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -5> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: handle_mon_command_ack 276 [{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}] Jul 29 20:05:29 cephadmin bash[1282]: debug -4> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _finish_command 276 = system:0 enabled application 'mgr' on pool '.mgr' Jul 29 20:05:29 cephadmin bash[1282]: debug -3> 2022-07-29T20:05:28.582+0000 7f20fff9f700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0 Jul 29 20:05:29 cephadmin bash[1282]: debug -2> 2022-07-29T20:05:28.610+0000 7f213f8b9700 10 monclient: handle_get_version_reply finishing 121 version 24 Jul 29 20:05:29 cephadmin bash[1282]: debug -1> 2022-07-29T20:05:28.610+0000 7f20f0f81700 5 cephsqlite: FullPathname: (client.34108) 1: /.mgr:devicehealth/main.db ### Jul 29 20:05:29 cephadmin bash[1282]: debug 0> 2022-07-29T20:05:28.614+0000 7f20f0f81700 -1 *** Caught signal (Aborted) ** Jul 29 20:05:29 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth Jul 29 20:05:29 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable) Jul 29 20:05:29 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0] Jul 29 20:05:29 cephadmin bash[1282]: 2: gsignal() Jul 29 20:05:29 cephadmin bash[1282]: 3: abort() Jul 29 20:05:29 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b] Jul 29 20:05:29 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c] Jul 29 20:05:29 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597] Jul 29 20:05:29 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8] Jul 29 20:05:29 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea] Jul 29 20:05:29 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8] Jul 29 20:05:29 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6] Jul 29 20:05:29 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b] Jul 29 20:05:29 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a] Jul 29 20:05:29 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:29 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:29 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8] Jul 29 20:05:29 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0] Jul 29 20:05:29 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:29 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:29 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:29 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12] Jul 29 20:05:29 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0] Jul 29 20:05:29 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb] Jul 29 20:05:29 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca] Jul 29 20:05:29 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486] Jul 29 20:05:29 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c] Jul 29 20:05:29 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9] Jul 29 20:05:29 cephadmin bash[1282]: 27: pysqlite_connection_init() Jul 29 20:05:29 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6] Jul 29 20:05:29 cephadmin bash[1282]: 29: PyObject_Call() Jul 29 20:05:29 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5] Jul 29 20:05:29 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1] Jul 29 20:05:29 cephadmin bash[1282]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Jul 29 20:05:29 cephadmin bash[1282]: --- logging levels --- Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 none Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 lockdep Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 context Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 crush Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_balancer Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_locker Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_log Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_log_expire Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_migrator Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 buffer Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 timer Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 filer Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 striper Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 objecter Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rados Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_mirror Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_replay Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_pwl Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 journaler Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 objectcacher Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 immutable_obj_cache Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 client Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 osd Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 optracker Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 objclass Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 filestore Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 journal Jul 29 20:05:29 cephadmin bash[1282]: 0/ 0 ms Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mon Jul 29 20:05:29 cephadmin bash[1282]: 0/10 monc Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 paxos Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 tp Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 auth Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 crypto Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 finisher Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 reserver Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 heartbeatmap Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 perfcounter Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw_sync Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw_datacache Jul 29 20:05:29 cephadmin bash[1282]: 1/10 civetweb Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 javaclient Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 asok Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 throttle Jul 29 20:05:29 cephadmin bash[1282]: 0/ 0 refs Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 compressor Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 bluestore Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 bluefs Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 bdev Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 kstore Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 rocksdb Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 leveldb Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 memdb Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 fuse Jul 29 20:05:29 cephadmin bash[1282]: 2/ 5 mgr Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mgrc Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 dpdk Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 eventtrace Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 prioritycache Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 test Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 cephfs_mirror Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 cephsqlite Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_onode Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_odata Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_omap Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_tm Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_cleaner Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_lba Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_cache Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_journal Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_device Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 alienstore Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mclock Jul 29 20:05:29 cephadmin bash[1282]: -2/-2 (syslog threshold) Jul 29 20:05:29 cephadmin bash[1282]: 99/99 (stderr threshold) Jul 29 20:05:29 cephadmin bash[1282]: --- pthread ID / name mapping for recent threads --- Jul 29 20:05:29 cephadmin bash[1282]: 7f20c3c4e700 / Jul 29 20:05:29 cephadmin bash[1282]: 7f20c4896700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20ca8e2700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cb0e3700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cb8e4700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cc0e5700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cc8e6700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cd0e7700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cd8e8700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20ce0e9700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20ce8ea700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20cf0eb700 / dashboard Jul 29 20:05:29 cephadmin bash[1282]: 7f20d3333700 / mgr-fin Jul 29 20:05:29 cephadmin bash[1282]: 7f20d6379700 / mgr-fin Jul 29 20:05:29 cephadmin bash[1282]: 7f20d73bb700 / safe_timer Jul 29 20:05:29 cephadmin bash[1282]: 7f20d83bd700 / ms_dispatch Jul 29 20:05:29 cephadmin bash[1282]: 7f20da3c1700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20dabc2700 / mgr-fin Jul 29 20:05:29 cephadmin bash[1282]: 7f20dcc06700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20ddc08700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20de409700 / mgr-fin Jul 29 20:05:29 cephadmin bash[1282]: 7f20dec0a700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20dfc8c700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20e04cd700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20e150f700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20e1f23700 / mgr-fin Jul 29 20:05:29 cephadmin bash[1282]: 7f20e2724700 / prometheus Jul 29 20:05:29 cephadmin bash[1282]: 7f20e3f27700 / prometheus
I have more of the logs, but it's about 4MBs for just a few seconds of logs and is too large for pastbin. I can find somewhere else if needed.
Updated by Brian Woods over 1 year ago
On further research, this seems to be related to: https://tracker.ceph.com/issues/55304
Not sure why this is impacting a fresh deployment.
Updated by Brian Woods over 1 year ago
Safe to close as duplicate of #55304, I don't think I can do it myself.
Updated by Konstantin Shalygin over 1 year ago
- Is duplicate of Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added
Updated by Konstantin Shalygin over 1 year ago
- Status changed from New to Duplicate