Project

General

Profile

Download (66.6 KB)

Bug #50659 » osd-segfault-when-crush-location-hook-configured.log

systemd OSD logs of segfault - Andrew Davidoff, 05/05/2021 02:20 PM

 
May 04 15:40:05 02.ceph-kubernetes.dev.lax1.REDACTED.net systemd[1]: Started Ceph osd.2 for 87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef.
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/ceph --cluster ceph --name client.osd-lockbox.3c48b9f9-ee36-416f-a653-8d018cbf8875 --keyring /var/lib/ceph/osd/ceph-2/lockbox.keyring config-key get dm-crypt/osd/3c48b9f9-ee36-416f-a653-8d018cbf8875/luks
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-add3a9f2-0da2-4675-ab9c-fe5aff30b6f3/osd-block-3c48b9f9-ee36-416f-a653-8d018cbf8875 gTCQdK-HJgL-zLfg-EkTI-1BHu-QdPu-afsGdI
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: stderr: Device gTCQdK-HJgL-zLfg-EkTI-1BHu-QdPu-afsGdI already exists.
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/mapper/gTCQdK-HJgL-zLfg-EkTI-1BHu-QdPu-afsGdI --path /var/lib/ceph/osd/ceph-2 --no-mon-config
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/ln -snf /dev/mapper/gTCQdK-HJgL-zLfg-EkTI-1BHu-QdPu-afsGdI /var/lib/ceph/osd/ceph-2/block
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32525]: --> ceph-volume lvm activate successful for osd ID: 2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: *** Caught signal (Segmentation fault) **
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: in thread 7f4e61e54080 thread_name:ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 set uid:gid to 167:167 (ceph:ceph)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process ceph-osd, pid 6
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4e5fbbbb20]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: /lib64/libc.so.6(+0x9a3da) [0x7f4e5e8863da]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 3: (SubProcess::add_cmd_arg(char const*)+0x4c) [0x56504e693b2c]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4: (SubProcess::add_cmd_args(char const*, ...)+0x75) [0x56504e693cc5]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 5: (ceph::crush::CrushLocation::update_from_hook()+0x2d4) [0x56504e883304]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 6: (ceph::crush::CrushLocation::init_on_startup()+0x3f5) [0x56504e884455]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 7: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0xcd1) [0x56504e5305b1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 8: main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 9: __libc_start_main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 10: _start()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug 2021-05-04T15:40:07.914+0000 7f4e61e54080 -1 *** Caught signal (Segmentation fault) **
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: in thread 7f4e61e54080 thread_name:ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4e5fbbbb20]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: /lib64/libc.so.6(+0x9a3da) [0x7f4e5e8863da]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 3: (SubProcess::add_cmd_arg(char const*)+0x4c) [0x56504e693b2c]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4: (SubProcess::add_cmd_args(char const*, ...)+0x75) [0x56504e693cc5]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 5: (ceph::crush::CrushLocation::update_from_hook()+0x2d4) [0x56504e883304]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 6: (ceph::crush::CrushLocation::init_on_startup()+0x3f5) [0x56504e884455]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 7: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0xcd1) [0x56504e5305b1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 8: main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 9: __libc_start_main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 10: _start()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- begin dump of recent events ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -104> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command assert hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -103> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command abort hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -102> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command leak_some_memory hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -101> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perfcounters_dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -100> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command 1 hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -99> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -98> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perfcounters_schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -97> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf histogram dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -96> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command 2 hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -95> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -94> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf histogram schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -93> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf reset hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -92> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config show hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -91> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config help hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -90> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config set hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -89> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config unset hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -88> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config get hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -87> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config diff hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -86> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config diff get hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -85> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command injectargs hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -84> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log flush hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -83> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -82> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log reopen hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -81> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command dump_mempools hook 0x565058206328
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -80> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: get_monmap_and_config
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -79> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: build_initial_monmap
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -78> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: monmap:
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: epoch 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: fsid 87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: last_changed 2021-05-04T15:40:07.910610+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: created 2021-05-04T15:40:07.910610+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: min_mon_release 0 (unknown)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: election_strategy: 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0: [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0] mon.noname-a
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0] mon.noname-b
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0] mon.noname-c
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -77> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -76> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -75> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -74> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -73> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -72> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -71> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -70> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -69> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -68> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -67> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -66> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -65> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -64> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -63> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -62> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -61> 2021-05-04T15:40:07.906+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -60> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: init
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -59> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -58> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -57> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -56> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -55> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -54> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -53> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -52> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -51> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -50> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -49> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -48> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -47> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -46> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -45> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -44> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -43> 2021-05-04T15:40:07.910+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -42> 2021-05-04T15:40:07.910+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -41> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: _reopen_session rank -1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -40> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: _add_conns ranks=[0,1,2]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -39> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-a con 0x565058214c00 addr [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -38> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-b con 0x565058215000 addr [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -37> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-c con 0x565058215400 addr [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -36> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -35> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -34> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -33> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): _renew_subs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -32> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): authenticate will time out at 2021-05-04T15:45:07.912128+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -31> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): get_auth_request con 0x565058215000 auth_method 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -30> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -29> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): _init_auth method 2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -28> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): _init_auth creating new auth
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -27> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more payload 9
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -26> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more payload_len 9
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -25> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -24> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_done global_id 1316491 payload 274
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -23> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient: _finish_hunting 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -22> 2021-05-04T15:40:07.910+0000 7f4e5abca700 1 monclient: found mon.noname-b
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -21> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient: _send_mon_message to mon.noname-b at v2:10.0.114.16:3300/0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -20> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: handle_monmap mon_map magic: 0 v1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -19> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: got monmap 13 from mon.noname-b (according to old e13)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -18> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: dump:
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: epoch 13
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: fsid 87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: last_changed 2021-04-29T15:09:01.891190+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: created 2020-12-14T18:42:30.246595+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: min_mon_release 16 (pacific)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: election_strategy: 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0: [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0] mon.03
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0] mon.02
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0] mon.01
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -17> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _finish_auth 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -16> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-05-04T15:39:37.913786+0000)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -15> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _send_mon_message to mon.02 at v2:10.0.114.16:3300/0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -14> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 monclient: authenticate success, global_id 1316491
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -13> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: handle_config config(7 keys) v1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -12> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: get_monmap_and_config success
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -11> 2021-05-04T15:40:07.910+0000 7f4e61e54080 4 set_mon_vals no callback set
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -10> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals cluster_network = 10.0.0.0/8
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -9> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals container_image = docker.artifactory.prod.adnxs.net/ceph/ceph@sha256:e5a456db2779401b485345a4eadc595b611b48faf7304bf6d246fd4707e896c7
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -8> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals crush_location_hook = /var/log/ceph/crush_location
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -7> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_memory_target = 4294967296
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -6> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_pool_default_crush_rule = 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -5> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_pool_default_pg_autoscale_mode = warn
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -4> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals public_network = 10.0.0.0/8
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -3> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: shutdown
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -2> 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 set uid:gid to 167:167 (ceph:ceph)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -1> 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process ceph-osd, pid 6
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug 0> 2021-05-04T15:40:07.914+0000 7f4e61e54080 -1 *** Caught signal (Segmentation fault) **
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: in thread 7f4e61e54080 thread_name:ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4e5fbbbb20]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: /lib64/libc.so.6(+0x9a3da) [0x7f4e5e8863da]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 3: (SubProcess::add_cmd_arg(char const*)+0x4c) [0x56504e693b2c]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4: (SubProcess::add_cmd_args(char const*, ...)+0x75) [0x56504e693cc5]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 5: (ceph::crush::CrushLocation::update_from_hook()+0x2d4) [0x56504e883304]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 6: (ceph::crush::CrushLocation::init_on_startup()+0x3f5) [0x56504e884455]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 7: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0xcd1) [0x56504e5305b1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 8: main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 9: __libc_start_main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 10: _start()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- logging levels ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 lockdep
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 context
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 crush
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_balancer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_locker
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_log
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_log_expire
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_migrator
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 buffer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 timer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 filer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 striper
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 objecter
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rados
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_mirror
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_replay
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_pwl
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 journaler
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 objectcacher
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 immutable_obj_cache
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 client
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 optracker
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 objclass
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 filestore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 journal
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 0 ms
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mon
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/10 monc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 paxos
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 tp
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 auth
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 crypto
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 finisher
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 reserver
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 heartbeatmap
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 perfcounter
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 rgw
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 rgw_sync
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/10 civetweb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 javaclient
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 asok
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 throttle
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 0 refs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 compressor
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 bluestore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 bluefs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 bdev
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 kstore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 rocksdb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 leveldb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 memdb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 fuse
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mgr
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mgrc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 dpdk
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 eventtrace
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 prioritycache
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 test
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 cephfs_mirror
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 cephsqlite
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: -2/-2 (syslog threshold)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 99/99 (stderr threshold)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- pthread ID / name mapping for recent threads ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974498096896 /
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974506489600 / msgr-worker-2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974626590848 / ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: max_recent 10000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: max_new 10000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: log_file
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- end dump of recent events ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- begin dump of recent events ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -104> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command assert hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -103> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command abort hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -102> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command leak_some_memory hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -101> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perfcounters_dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -100> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command 1 hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -99> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -98> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perfcounters_schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -97> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf histogram dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -96> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command 2 hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -95> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -94> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf histogram schema hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -93> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command perf reset hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -92> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config show hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -91> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config help hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -90> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config set hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -89> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config unset hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -88> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config get hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -87> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config diff hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -86> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command config diff get hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -85> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command injectargs hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -84> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log flush hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -83> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log dump hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -82> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command log reopen hook 0x565058166540
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -81> 2021-05-04T15:40:07.898+0000 7f4e61e54080 5 asok(0x565058218000) register_command dump_mempools hook 0x565058206328
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -80> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: get_monmap_and_config
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -79> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: build_initial_monmap
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -78> 2021-05-04T15:40:07.906+0000 7f4e61e54080 10 monclient: monmap:
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: epoch 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: fsid 87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: last_changed 2021-05-04T15:40:07.910610+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: created 2021-05-04T15:40:07.910610+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: min_mon_release 0 (unknown)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: election_strategy: 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0: [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0] mon.noname-a
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0] mon.noname-b
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0] mon.noname-c
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -77> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -76> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -75> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -74> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding auth protocol: none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -73> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -72> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -71> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -70> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -69> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -68> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -67> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -66> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -65> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -64> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -63> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -62> 2021-05-04T15:40:07.906+0000 7f4e61e54080 5 AuthRegistry(0x565058dea140) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -61> 2021-05-04T15:40:07.906+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -60> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: init
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -59> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -58> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -57> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: cephx
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -56> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding auth protocol: none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -55> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -54> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -53> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -52> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -51> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -50> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -49> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -48> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -47> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -46> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -45> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: crc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -44> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 AuthRegistry(0x7ffdc3d36430) adding con mode: secure
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -43> 2021-05-04T15:40:07.910+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -42> 2021-05-04T15:40:07.910+0000 7f4e61e54080 2 auth: KeyRing::load: loaded key file /var/lib/ceph/osd/ceph-2/keyring
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -41> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: _reopen_session rank -1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -40> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: _add_conns ranks=[0,1,2]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -39> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-a con 0x565058214c00 addr [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -38> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-b con 0x565058215000 addr [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -37> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): picked mon.noname-c con 0x565058215400 addr [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -36> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -35> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -34> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): start opening mon connection
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -33> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): _renew_subs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -32> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient(hunting): authenticate will time out at 2021-05-04T15:45:07.912128+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -31> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): get_auth_request con 0x565058215000 auth_method 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -30> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -29> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): _init_auth method 2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -28> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): _init_auth creating new auth
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -27> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more payload 9
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -26> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more payload_len 9
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -25> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -24> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient(hunting): handle_auth_done global_id 1316491 payload 274
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -23> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient: _finish_hunting 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -22> 2021-05-04T15:40:07.910+0000 7f4e5abca700 1 monclient: found mon.noname-b
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -21> 2021-05-04T15:40:07.910+0000 7f4e5abca700 10 monclient: _send_mon_message to mon.noname-b at v2:10.0.114.16:3300/0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -20> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: handle_monmap mon_map magic: 0 v1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -19> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: got monmap 13 from mon.noname-b (according to old e13)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -18> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: dump:
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: epoch 13
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: fsid 87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: last_changed 2021-04-29T15:09:01.891190+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: created 2020-12-14T18:42:30.246595+0000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: min_mon_release 16 (pacific)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: election_strategy: 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0: [v2:10.0.114.17:3300/0,v1:10.0.114.17:6789/0] mon.03
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: [v2:10.0.114.16:3300/0,v1:10.0.114.16:6789/0] mon.02
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: [v2:10.0.114.9:3300/0,v1:10.0.114.9:6789/0] mon.01
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -17> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _finish_auth 0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -16> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2021-05-04T15:39:37.913786+0000)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -15> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: _send_mon_message to mon.02 at v2:10.0.114.16:3300/0
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -14> 2021-05-04T15:40:07.910+0000 7f4e61e54080 5 monclient: authenticate success, global_id 1316491
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -13> 2021-05-04T15:40:07.910+0000 7f4e5a3c9700 10 monclient: handle_config config(7 keys) v1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -12> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: get_monmap_and_config success
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -11> 2021-05-04T15:40:07.910+0000 7f4e61e54080 4 set_mon_vals no callback set
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -10> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals cluster_network = 10.0.0.0/8
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -9> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals container_image = docker.artifactory.prod.adnxs.net/ceph/ceph@sha256:e5a456db2779401b485345a4eadc595b611b48faf7304bf6d246fd4707e896c7
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -8> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals crush_location_hook = /var/log/ceph/crush_location
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -7> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_memory_target = 4294967296
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -6> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_pool_default_crush_rule = 1
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -5> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals osd_pool_default_pg_autoscale_mode = warn
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -4> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 set_mon_vals public_network = 10.0.0.0/8
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -3> 2021-05-04T15:40:07.910+0000 7f4e61e54080 10 monclient: shutdown
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -2> 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 set uid:gid to 167:167 (ceph:ceph)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug -1> 2021-05-04T15:40:07.910+0000 7f4e61e54080 0 ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable), process ceph-osd, pid 6
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: debug 0> 2021-05-04T15:40:07.914+0000 7f4e61e54080 -1 *** Caught signal (Segmentation fault) **
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: in thread 7f4e61e54080 thread_name:ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: ceph version 16.2.1 (afb9061ab4117f798c858c741efa6390e48ccf10) pacific (stable)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1: /lib64/libpthread.so.0(+0x12b20) [0x7f4e5fbbbb20]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 2: /lib64/libc.so.6(+0x9a3da) [0x7f4e5e8863da]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 3: (SubProcess::add_cmd_arg(char const*)+0x4c) [0x56504e693b2c]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4: (SubProcess::add_cmd_args(char const*, ...)+0x75) [0x56504e693cc5]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 5: (ceph::crush::CrushLocation::update_from_hook()+0x2d4) [0x56504e883304]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 6: (ceph::crush::CrushLocation::init_on_startup()+0x3f5) [0x56504e884455]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 7: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char const*> >&, unsigned int, code_environment_t, int, bool)+0xcd1) [0x56504e5305b1]
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 8: main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 9: __libc_start_main()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 10: _start()
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- logging levels ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 none
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 lockdep
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 context
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 crush
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_balancer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_locker
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_log
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_log_expire
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mds_migrator
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 buffer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 timer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 filer
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 striper
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 1 objecter
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rados
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_mirror
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_replay
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 rbd_pwl
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 journaler
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 objectcacher
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 immutable_obj_cache
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 client
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 optracker
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 objclass
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 filestore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 journal
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 0 ms
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mon
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/10 monc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 paxos
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 tp
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 auth
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 crypto
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 finisher
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 reserver
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 heartbeatmap
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 perfcounter
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 rgw
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 rgw_sync
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/10 civetweb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 javaclient
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 asok
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 1 throttle
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 0 refs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 compressor
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 bluestore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 bluefs
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 3 bdev
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 kstore
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 rocksdb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 leveldb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 4/ 5 memdb
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 fuse
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mgr
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 mgrc
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 dpdk
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 eventtrace
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 1/ 5 prioritycache
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 test
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 cephfs_mirror
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 0/ 5 cephsqlite
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: -2/-2 (syslog threshold)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 99/99 (stderr threshold)
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- pthread ID / name mapping for recent threads ---
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974498096896 /
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974506489600 / msgr-worker-2
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: 139974626590848 / ceph-osd
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: max_recent 10000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: max_new 10000
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: log_file /var/lib/ceph/crash/2021-05-04T15:40:07.918919Z_fc38cc93-ef43-4bf3-aed9-0c587d0de619/log
May 04 15:40:07 02.ceph-kubernetes.dev.lax1.REDACTED.net bash[32744]: --- end dump of recent events ---
May 04 15:40:08 02.ceph-kubernetes.dev.lax1.REDACTED.net systemd[1]: ceph-87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef@osd.2.service: Main process exited, code=exited, status=139/n/a
May 04 15:40:09 02.ceph-kubernetes.dev.lax1.REDACTED.net systemd[1]: ceph-87309d4d-e1fc-4be6-9f59-1f77ce2cc5ef@osd.2.service: Failed with result 'exit-code'.

(1-1/2)