Actions
Bug #56745
closedunrecognized option ssl_server_port appears to cause failure of MGR
Added by Brian Woods almost 2 years ago. Updated almost 2 years ago.
Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
This is going to take a second, sorry. This is a test environment and is open to destructive testing!
I deployed a fresh Ubuntu 20.04 system, deploy my defacto config scripts, docker and the basic Ceph Quincy 17.2.2 bootstrap to a VM, no OSDs. Then I deployed a second VM with two OSDs (one SSD and one HDD). Everything is working fine so far.
I then deployed a third install to a physical box with some drives and an SSD, monitoring services start on it and all is well. However, after deploying OSDs the the third node (second node with OSDs), withing sever seconds the manager service vanishes from the cluster.
Thought I did something wrong, wiped everything and re-deployed. Same exact issue...
I have been looking through the logs, and as best as I can tell, the MGR fails to deploy/redeploy due to these errors (private values/IPs removed):
cephadmin bash[2817]: debug -228> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/server_addr = #.#.#.# (unrecognized option) cephadmin bash[2817]: debug -227> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/server_port = 8080 (unrecognized option) cephadmin bash[2817]: debug -226> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/ssl = false (unrecognized option) cephadmin bash[2817]: debug -225> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/dashboard/ssl_server_port = 8443 (unrecognized option) cephadmin bash[2817]: debug -224> 2022-07-27T23:49:30.438+0000 7fb0c162b700 10 set_mon_vals mgr/orchestrator/orchestrator = cephadm (unrecognized option) cephadmin bash[2817]: debug -129> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/cephadm/container_init = True (unrecognized option) cephadmin bash[2817]: debug -128> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/cephadm/migration_current = 5 (unrecognized option) cephadmin bash[2817]: debug -127> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/ALERTMANAGER_API_HOST = http://###########:9093 (unrecognized option) cephadmin bash[2817]: debug -126> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/GRAFANA_API_SSL_VERIFY = false (unrecognized option) cephadmin bash[2817]: debug -125> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/GRAFANA_API_URL = https://###########:3000 (unrecognized option) cephadmin bash[2817]: debug -124> 2022-07-27T23:49:30.450+0000 7fb09a6b2700 10 set_mon_vals mgr/dashboard/PROMETHEUS_API_HOST = http://###########:9095 (unrecognized option)
This causes the dashboard to attempt a starts up with a 0 for the port as best as I can tell:
from='client.? #.#.#.#:0/3815048734' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch
Everything goes down hill from there, the mgr crashes as it can't talk to its own services, finally the docker image and host services are purged...
I am not sure if it is something in my (very bare bones) config somehow, or a bug.. But... Any help would be appreciated. I have not Ceph since Luminous, so I am very out of date to say the least...
Note, I am not tagging the second and third node with "_admin", as I thought that caused it the first time.
Updated by Brian Woods almost 2 years ago
Could also be related to:
https://tracker.ceph.com/issues/56726
As I am seeing this in the log, but...
1 mgr finish mon failed to return metadata for osd.6: (2) No such file or directory
Updated by Brian Woods almost 2 years ago
Update, if I add an OSD to the first bootstrap node, this happens as soon as I add an OSD to the second node...
This seems like a fairly critical failure/bug...
Updated by Brian Woods almost 2 years ago
I setup a watcher on the docker instance to do a snapshot as soon as the instance crashed and have been able to captured a couple of crashes:
ul 29 20:05:27 cephadmin bash[1282]: debug 2022-07-29T20:05:27.134+0000 7f20f0f81700 0 [devicehealth INFO root] creating mgr pool
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.134+0000 7f049a625700 0 mon.cephadmin@0(leader) e2 handle_command mon_command({"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32} v 0) v1
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.134+0000 7f049a625700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.374+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e22 do_prune osdmap full prune enabled
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.374+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e22 encode_pending skipping prime_pg_temp; mapping job did not start
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.498+0000 7f0497e20700 1 mon.cephadmin@0(leader).osd e23 e23: 3 total, 3 up, 3 in
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 3314933000852226048, adjusting msgr requires
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 288514051259236352, adjusting msgr requires
Jul 29 20:05:27 cephadmin bash[1283]: message repeated 2 times: [ debug 2022-07-29T20:05:27.526+0000 7f0497e20700 0 mon.cephadmin@0(leader).osd e23 crush map has features 288514051259236352, adjusting msgr requires]
Jul 29 20:05:27 cephadmin bash[1283]: cluster 2022-07-29T20:05:26.374617+0000 mon.cephadmin (mon.0) 378 : cluster [DBG] osdmap e22: 3 total, 3 up, 3 in
Jul 29 20:05:27 cephadmin bash[1283]: audit 2022-07-29T20:05:27.136968+0000 mon.cephadmin (mon.0) 379 : audit [INF] from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch
Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 288514051259236352, adjusting msgr requires for clients
Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 288514051259236352 was 288514050185503233, adjusting msgr requires for mons
Jul 29 20:05:27 cephadmin bash[8664]: debug 2022-07-29T20:05:27.562+0000 7fe1ddef4700 0 osd.0 23 crush map has features 3314933000852226048, adjusting msgr requires for osds
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f0497e20700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f0497e20700 0 log_channel(cluster) log [DBG] : osdmap e23: 3 total, 3 up, 3 in
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f049a625700 0 mon.cephadmin@0(leader) e2 handle_command mon_command({"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} v 0) v1
Jul 29 20:05:27 cephadmin bash[1283]: debug 2022-07-29T20:05:27.602+0000 7f049a625700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch
Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.266+0000 7f20e04cd700 0 [prometheus INFO cherrypy.access.139779795241728] ::ffff:192.168.10.##1 - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4"
Jul 29 20:05:28 cephadmin bash[1282]: ::ffff:192.168.10.##1 - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4"
Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.498+0000 7f049ce2a700 1 mon.cephadmin@0(leader).osd e23 do_prune osdmap full prune enabled
Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.518+0000 7f21007a0700 0 log_channel(cluster) log [DBG] : pgmap v477: 1 pgs: 1 unknown; 0 B data, 458 MiB used, 53 GiB / 53 GiB avail
Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.558+0000 7f0497e20700 1 mon.cephadmin@0(leader).osd e24 e24: 3 total, 3 up, 3 in
Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.582+0000 7f0497e20700 0 log_channel(audit) log [INF] : from='mgr.34106 192.168.10.##1:0/4277796309' entity='mgr.cephadmin.wyrydc' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished
###
Jul 29 20:05:28 cephadmin bash[1283]: debug 2022-07-29T20:05:28.582+0000 7f0497e20700 0 log_channel(cluster) log [DBG] : osdmap e24: 3 total, 3 up, 3 in
Jul 29 20:05:28 cephadmin bash[1282]: terminate called after throwing an instance of 'std::regex_error'
Jul 29 20:05:28 cephadmin bash[1282]: what(): Invalid start of range in bracket expression.
Jul 29 20:05:28 cephadmin bash[1282]: *** Caught signal (Aborted) **
Jul 29 20:05:28 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth
Jul 29 20:05:28 cephadmin bash[8664]: debug 2022-07-29T20:05:28.610+0000 7fe1d1edc700 1 osd.0 pg_epoch: 23 pg[1.0( empty local-lis/les=0/0 n=0 ec=23/23 lis/c=0/0 les/c/f=0/0/0 sis=23) [1,0] r=1 lpr=23 crt=0'0 mlcod 0'0 unknown mbc={}] state<Start>: transitioning to Stray
Jul 29 20:05:28 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)
Jul 29 20:05:28 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0]
Jul 29 20:05:28 cephadmin bash[1282]: 2: gsignal()
Jul 29 20:05:28 cephadmin bash[1282]: 3: abort()
Jul 29 20:05:28 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b]
Jul 29 20:05:28 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c]
Jul 29 20:05:28 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597]
Jul 29 20:05:28 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8]
Jul 29 20:05:28 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea]
Jul 29 20:05:28 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8]
Jul 29 20:05:28 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6]
Jul 29 20:05:28 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b]
Jul 29 20:05:28 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a]
Jul 29 20:05:28 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:28 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:28 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8]
Jul 29 20:05:28 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:28 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:28 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb]
Jul 29 20:05:28 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca]
Jul 29 20:05:28 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486]
Jul 29 20:05:28 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c]
Jul 29 20:05:28 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9]
Jul 29 20:05:28 cephadmin bash[1282]: 27: pysqlite_connection_init()
Jul 29 20:05:28 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6]
Jul 29 20:05:28 cephadmin bash[1282]: 29: PyObject_Call()
Jul 29 20:05:28 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5]
Jul 29 20:05:28 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1]
Jul 29 20:05:28 cephadmin bash[1282]: debug 2022-07-29T20:05:28.614+0000 7f20f0f81700 -1 *** Caught signal (Aborted) **
Jul 29 20:05:28 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth
Jul 29 20:05:28 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)
Jul 29 20:05:28 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0]
Jul 29 20:05:28 cephadmin bash[1282]: 2: gsignal()
Jul 29 20:05:28 cephadmin bash[1282]: 3: abort()
Jul 29 20:05:28 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b]
Jul 29 20:05:28 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c]
Jul 29 20:05:28 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597]
Jul 29 20:05:28 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8]
Jul 29 20:05:28 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea]
Jul 29 20:05:28 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8]
Jul 29 20:05:28 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6]
Jul 29 20:05:28 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b]
Jul 29 20:05:28 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a]
Jul 29 20:05:28 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:28 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:28 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8]
Jul 29 20:05:28 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:28 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:28 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:28 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb]
Jul 29 20:05:28 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca]
Jul 29 20:05:28 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486]
Jul 29 20:05:28 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c]
Jul 29 20:05:28 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9]
Jul 29 20:05:28 cephadmin bash[1282]: 27: pysqlite_connection_init()
Jul 29 20:05:28 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6]
Jul 29 20:05:28 cephadmin bash[1282]: 29: PyObject_Call()
Jul 29 20:05:28 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5]
Jul 29 20:05:28 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1]
Jul 29 20:05:28 cephadmin bash[1282]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
################################################
Jul 29 20:05:29 cephadmin bash[1282]: debug -27> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 monclient: _finish_command 275 = system:0 pool '.mgr' created
Jul 29 20:05:29 cephadmin bash[1282]: debug -26> 2022-07-29T20:05:27.602+0000 7f20fff9f700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -25> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 log_client handle_log_ack log(last 502) v1
Jul 29 20:05:29 cephadmin bash[1282]: debug -23> 2022-07-29T20:05:27.602+0000 7f213f8b9700 10 monclient: handle_get_version_reply finishing 120 version 23
Jul 29 20:05:29 cephadmin bash[1282]: debug -22> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]
Jul 29 20:05:29 cephadmin bash[1282]: debug -21> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 monclient: _send_command 276 [{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]
Jul 29 20:05:29 cephadmin bash[1282]: debug -20> 2022-07-29T20:05:27.602+0000 7f20f0f81700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -19> 2022-07-29T20:05:27.934+0000 7f20fcf99700 10 monclient: tick
Jul 29 20:05:29 cephadmin bash[1282]: debug -18> 2022-07-29T20:05:27.934+0000 7f20fcf99700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:57.937019+0000)
Jul 29 20:05:29 cephadmin bash[1282]: debug -17> 2022-07-29T20:05:28.010+0000 7f20e4768700 10 monclient: tick
Jul 29 20:05:29 cephadmin bash[1282]: debug -16> 2022-07-29T20:05:28.010+0000 7f20e4768700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.015124+0000)
Jul 29 20:05:29 cephadmin bash[1282]: debug -15> 2022-07-29T20:05:28.030+0000 7f213d8b5700 10 monclient: tick
Jul 29 20:05:29 cephadmin bash[1282]: debug -14> 2022-07-29T20:05:28.030+0000 7f213d8b5700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.033887+0000)
Jul 29 20:05:29 cephadmin bash[1282]: debug -13> 2022-07-29T20:05:28.082+0000 7f213b0b0700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -12> 2022-07-29T20:05:28.234+0000 7f20d73bb700 10 monclient: tick
Jul 29 20:05:29 cephadmin bash[1282]: debug -11> 2022-07-29T20:05:28.234+0000 7f20d73bb700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2022-07-29T20:04:58.234298+0000)
Jul 29 20:05:29 cephadmin bash[1282]: debug -10> 2022-07-29T20:05:28.266+0000 7f20e04cd700 0 [prometheus INFO cherrypy.access.139779795241728] ::ffff:192.168.10.### - - [29/Jul/2022:20:05:28] "GET /metrics HTTP/1.1" 200 100244 "" "Prometheus/2.33.4"
Jul 29 20:05:29 cephadmin bash[1282]: debug -9> 2022-07-29T20:05:28.518+0000 7f21007a0700 0 log_channel(cluster) log [DBG] : pgmap v477: 1 pgs: 1 unknown; 0 B data, 458 MiB used, 53 GiB / 53 GiB avail
Jul 29 20:05:29 cephadmin bash[1282]: debug -8> 2022-07-29T20:05:28.518+0000 7f21007a0700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -7> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _renew_subs
Jul 29 20:05:29 cephadmin bash[1282]: debug -6> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -5> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: handle_mon_command_ack 276 [{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]
Jul 29 20:05:29 cephadmin bash[1282]: debug -4> 2022-07-29T20:05:28.582+0000 7f213f8b9700 10 monclient: _finish_command 276 = system:0 enabled application 'mgr' on pool '.mgr'
Jul 29 20:05:29 cephadmin bash[1282]: debug -3> 2022-07-29T20:05:28.582+0000 7f20fff9f700 10 monclient: _send_mon_message to mon.cephadmin at v2:192.168.10.###:3300/0
Jul 29 20:05:29 cephadmin bash[1282]: debug -2> 2022-07-29T20:05:28.610+0000 7f213f8b9700 10 monclient: handle_get_version_reply finishing 121 version 24
Jul 29 20:05:29 cephadmin bash[1282]: debug -1> 2022-07-29T20:05:28.610+0000 7f20f0f81700 5 cephsqlite: FullPathname: (client.34108) 1: /.mgr:devicehealth/main.db
###
Jul 29 20:05:29 cephadmin bash[1282]: debug 0> 2022-07-29T20:05:28.614+0000 7f20f0f81700 -1 *** Caught signal (Aborted) **
Jul 29 20:05:29 cephadmin bash[1282]: in thread 7f20f0f81700 thread_name:devicehealth
Jul 29 20:05:29 cephadmin bash[1282]: ceph version 17.2.2 (b6e46b8939c67a6cc754abb4d0ece3c8918eccc3) quincy (stable)
Jul 29 20:05:29 cephadmin bash[1282]: 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f2147ec6ce0]
Jul 29 20:05:29 cephadmin bash[1282]: 2: gsignal()
Jul 29 20:05:29 cephadmin bash[1282]: 3: abort()
Jul 29 20:05:29 cephadmin bash[1282]: 4: /lib64/libstdc++.so.6(+0x9009b) [0x7f21472c509b]
Jul 29 20:05:29 cephadmin bash[1282]: 5: /lib64/libstdc++.so.6(+0x9653c) [0x7f21472cb53c]
Jul 29 20:05:29 cephadmin bash[1282]: 6: /lib64/libstdc++.so.6(+0x96597) [0x7f21472cb597]
Jul 29 20:05:29 cephadmin bash[1282]: 7: /lib64/libstdc++.so.6(+0x967f8) [0x7f21472cb7f8]
Jul 29 20:05:29 cephadmin bash[1282]: 8: (std::__throw_regex_error(std::regex_constants::error_type, char const*)+0x4a) [0x561d7cf6beea]
Jul 29 20:05:29 cephadmin bash[1282]: 9: (bool std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_expression_term<false, false>(std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_BracketState&, std::__detail::_BracketMatcher<std::__cxx11::regex_traits<char>, false, false>&)+0x578) [0x561d7cf770a8]
Jul 29 20:05:29 cephadmin bash[1282]: 10: (void std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_insert_bracket_matcher<false, false>(bool)+0x146) [0x561d7cf786b6]
Jul 29 20:05:29 cephadmin bash[1282]: 11: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_bracket_expression()+0x6b) [0x561d7cf7c63b]
Jul 29 20:05:29 cephadmin bash[1282]: 12: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x6a) [0x561d7cf7c71a]
Jul 29 20:05:29 cephadmin bash[1282]: 13: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:29 cephadmin bash[1282]: 14: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:29 cephadmin bash[1282]: 15: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_atom()+0x338) [0x561d7cf7c9e8]
Jul 29 20:05:29 cephadmin bash[1282]: 16: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0xd0) [0x561d7cf7cca0]
Jul 29 20:05:29 cephadmin bash[1282]: 17: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:29 cephadmin bash[1282]: 18: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:29 cephadmin bash[1282]: 19: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:29 cephadmin bash[1282]: 20: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_alternative()+0x42) [0x561d7cf7cc12]
Jul 29 20:05:29 cephadmin bash[1282]: 21: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_M_disjunction()+0x30) [0x561d7cf7cdf0]
Jul 29 20:05:29 cephadmin bash[1282]: 22: (std::__detail::_Compiler<std::__cxx11::regex_traits<char> >::_Compiler(char const*, char const*, std::locale const&, std::regex_constants::syntax_option_type)+0x36b) [0x561d7cf7d3cb]
Jul 29 20:05:29 cephadmin bash[1282]: 23: /lib64/libcephsqlite.so(+0x1b7ca) [0x7f214a6b07ca]
Jul 29 20:05:29 cephadmin bash[1282]: 24: /lib64/libcephsqlite.so(+0x24486) [0x7f214a6b9486]
Jul 29 20:05:29 cephadmin bash[1282]: 25: /lib64/libsqlite3.so.0(+0x75f1c) [0x7f214a3f6f1c]
Jul 29 20:05:29 cephadmin bash[1282]: 26: /lib64/libsqlite3.so.0(+0xdd4c9) [0x7f214a45e4c9]
Jul 29 20:05:29 cephadmin bash[1282]: 27: pysqlite_connection_init()
Jul 29 20:05:29 cephadmin bash[1282]: 28: /lib64/libpython3.6m.so.1.0(+0x13afc6) [0x7f2149f78fc6]
Jul 29 20:05:29 cephadmin bash[1282]: 29: PyObject_Call()
Jul 29 20:05:29 cephadmin bash[1282]: 30: /lib64/python3.6/lib-dynload/_sqlite3.cpython-36m-x86_64-linux-gnu.so(+0xa1f5) [0x7f2138dfc1f5]
Jul 29 20:05:29 cephadmin bash[1282]: 31: /lib64/libpython3.6m.so.1.0(+0x19d5f1) [0x7f2149fdb5f1]
Jul 29 20:05:29 cephadmin bash[1282]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 29 20:05:29 cephadmin bash[1282]: --- logging levels ---
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 none
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 lockdep
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 context
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 crush
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_balancer
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_locker
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_log
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_log_expire
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mds_migrator
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 buffer
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 timer
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 filer
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 striper
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 1 objecter
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rados
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_mirror
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_replay
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 rbd_pwl
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 journaler
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 objectcacher
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 immutable_obj_cache
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 client
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 osd
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 optracker
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 objclass
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 filestore
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 journal
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 0 ms
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mon
Jul 29 20:05:29 cephadmin bash[1282]: 0/10 monc
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 paxos
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 tp
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 auth
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 crypto
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 finisher
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 reserver
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 heartbeatmap
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 perfcounter
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw_sync
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 rgw_datacache
Jul 29 20:05:29 cephadmin bash[1282]: 1/10 civetweb
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 javaclient
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 asok
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 1 throttle
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 0 refs
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 compressor
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 bluestore
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 bluefs
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 3 bdev
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 kstore
Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 rocksdb
Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 leveldb
Jul 29 20:05:29 cephadmin bash[1282]: 4/ 5 memdb
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 fuse
Jul 29 20:05:29 cephadmin bash[1282]: 2/ 5 mgr
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mgrc
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 dpdk
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 eventtrace
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 prioritycache
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 test
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 cephfs_mirror
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 cephsqlite
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_onode
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_odata
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_omap
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_tm
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_cleaner
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_lba
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_cache
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_journal
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 seastore_device
Jul 29 20:05:29 cephadmin bash[1282]: 0/ 5 alienstore
Jul 29 20:05:29 cephadmin bash[1282]: 1/ 5 mclock
Jul 29 20:05:29 cephadmin bash[1282]: -2/-2 (syslog threshold)
Jul 29 20:05:29 cephadmin bash[1282]: 99/99 (stderr threshold)
Jul 29 20:05:29 cephadmin bash[1282]: --- pthread ID / name mapping for recent threads ---
Jul 29 20:05:29 cephadmin bash[1282]: 7f20c3c4e700 /
Jul 29 20:05:29 cephadmin bash[1282]: 7f20c4896700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20ca8e2700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cb0e3700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cb8e4700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cc0e5700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cc8e6700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cd0e7700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cd8e8700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20ce0e9700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20ce8ea700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20cf0eb700 / dashboard
Jul 29 20:05:29 cephadmin bash[1282]: 7f20d3333700 / mgr-fin
Jul 29 20:05:29 cephadmin bash[1282]: 7f20d6379700 / mgr-fin
Jul 29 20:05:29 cephadmin bash[1282]: 7f20d73bb700 / safe_timer
Jul 29 20:05:29 cephadmin bash[1282]: 7f20d83bd700 / ms_dispatch
Jul 29 20:05:29 cephadmin bash[1282]: 7f20da3c1700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20dabc2700 / mgr-fin
Jul 29 20:05:29 cephadmin bash[1282]: 7f20dcc06700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20ddc08700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20de409700 / mgr-fin
Jul 29 20:05:29 cephadmin bash[1282]: 7f20dec0a700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20dfc8c700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20e04cd700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20e150f700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20e1f23700 / mgr-fin
Jul 29 20:05:29 cephadmin bash[1282]: 7f20e2724700 / prometheus
Jul 29 20:05:29 cephadmin bash[1282]: 7f20e3f27700 / prometheus
I have more of the logs, but it's about 4MBs for just a few seconds of logs and is too large for pastbin. I can find somewhere else if needed.
Updated by Brian Woods almost 2 years ago
On further research, this seems to be related to: https://tracker.ceph.com/issues/55304
Not sure why this is impacting a fresh deployment.
Updated by Brian Woods almost 2 years ago
Safe to close as duplicate of #55304, I don't think I can do it myself.
Updated by Konstantin Shalygin almost 2 years ago
- Is duplicate of Bug #55304: libcephsqlite: crash when compiled with gcc12 cause of regex treating '-' as a range operator added
Updated by Konstantin Shalygin almost 2 years ago
- Status changed from New to Duplicate
Actions