2019-12-22 12:20:17.347 7fce967872c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-12-22 12:20:17.347 7fce967872c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7040 2019-12-22 12:20:17.347 7fce967872c0 0 pidfile_write: ignore empty --pid-file 2019-12-22 12:20:17.351 7fce848d7700 1 mds.ceph1 Updating MDS map to version 331 from mon.0 2019-12-22 12:20:21.811 7fce848d7700 1 mds.ceph1 Updating MDS map to version 332 from mon.0 2019-12-22 12:20:21.811 7fce848d7700 1 mds.ceph1 Map has assigned me to become a standby 2019-12-22 12:20:21.815 7fce848d7700 1 mds.ceph1 Updating MDS map to version 333 from mon.0 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 handle_mds_map i am now mds.0.333 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 handle_mds_map state change up:boot --> up:replay 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 replay_start 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 recovery set is 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 waiting for osdmap 6044 (which blacklists prior instance) 2019-12-22 12:20:21.819 7fce7d8c9700 0 mds.0.cache creating system inode with ino:0x100 2019-12-22 12:20:21.819 7fce7d8c9700 0 mds.0.cache creating system inode with ino:0x1 2019-12-22 12:20:22.055 7fce7c8c7700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) 2019-12-22 12:20:22.055 7fce7c8c7700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} 2019-12-22 12:20:22.055 7fce7c8c7700 0 mds.0.journal journal ambig_subtrees: 2019-12-22 12:20:22.059 7fce7c8c7700 -1 *** Caught signal (Segmentation fault) ** in thread 7fce7c8c7700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7fce8d26d890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -272> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command assert hook 0x562d33cf4500 -271> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command abort hook 0x562d33cf4500 -270> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perfcounters_dump hook 0x562d33cf4500 -269> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command 1 hook 0x562d33cf4500 -268> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perf dump hook 0x562d33cf4500 -267> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perfcounters_schema hook 0x562d33cf4500 -266> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perf histogram dump hook 0x562d33cf4500 -265> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command 2 hook 0x562d33cf4500 -264> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perf schema hook 0x562d33cf4500 -263> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perf histogram schema hook 0x562d33cf4500 -262> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command perf reset hook 0x562d33cf4500 -261> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config show hook 0x562d33cf4500 -260> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config help hook 0x562d33cf4500 -259> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config set hook 0x562d33cf4500 -258> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config unset hook 0x562d33cf4500 -257> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config get hook 0x562d33cf4500 -256> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config diff hook 0x562d33cf4500 -255> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command config diff get hook 0x562d33cf4500 -254> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command log flush hook 0x562d33cf4500 -253> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command log dump hook 0x562d33cf4500 -252> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command log reopen hook 0x562d33cf4500 -251> 2019-12-22 12:20:17.331 7fce967872c0 5 asok(0x562d33d82000) register_command dump_mempools hook 0x562d3498c068 -250> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient: get_monmap_and_config -249> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient: build_initial_monmap -248> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:17.343610 created 2019-12-22 12:20:17.343610 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -247> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding auth protocol: cephx -246> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding auth protocol: cephx -245> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding auth protocol: cephx -244> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -243> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -242> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -241> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -240> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -239> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -238> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -237> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -236> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -235> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -234> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: crc -233> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x562d33dc6140) adding con mode: secure -232> 2019-12-22 12:20:17.339 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -231> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient: init -230> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding auth protocol: cephx -229> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding auth protocol: cephx -228> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding auth protocol: cephx -227> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -226> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -225> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -224> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -223> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -222> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -221> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -220> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -219> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -218> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -217> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: crc -216> 2019-12-22 12:20:17.339 7fce967872c0 5 AuthRegistry(0x7ffd444b6bc8) adding con mode: secure -215> 2019-12-22 12:20:17.339 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -214> 2019-12-22 12:20:17.339 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -213> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient: _reopen_session rank -1 -212> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x562d33e1d680 addr v1:10.10.0.20:6789/0 -211> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): picked mon.noname-c-legacy con 0x562d33e1cd80 addr v1:10.10.0.30:6789/0 -210> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): picked mon.noname-c con 0x562d33e1db00 addr v2:10.10.0.30:3300/0 -209> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): start opening mon connection -208> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): _renew_subs -207> 2019-12-22 12:20:17.339 7fce967872c0 10 monclient(hunting): authenticate will time out at 2019-12-22 12:25:17.344924 -206> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): get_auth_request con 0x562d33e1db00 auth_method 0 -205> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -204> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): _init_auth method 2 -203> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more payload 9 -202> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -201> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -200> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient(hunting): handle_auth_done global_id 554753 payload 386 -199> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient: _finish_hunting 0 -198> 2019-12-22 12:20:17.343 7fce88ac7700 1 monclient: found mon.noname-c -197> 2019-12-22 12:20:17.343 7fce88ac7700 10 monclient: _send_mon_message to mon.noname-c at v2:10.10.0.30:3300/0 -196> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: handle_monmap mon_map magic: 0 v1 -195> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) -194> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -193> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: _finish_auth 0 -192> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:47.347388) -191> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: _send_mon_message to mon.ceph3 at v2:10.10.0.30:3300/0 -190> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: handle_config config(2 keys) v1 -189> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: handle_monmap mon_map magic: 0 v1 -188> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: got monmap 1 from mon.ceph3 (according to old e1) -187> 2019-12-22 12:20:17.343 7fce872c4700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -186> 2019-12-22 12:20:17.343 7fce967872c0 5 monclient: authenticate success, global_id 554753 -185> 2019-12-22 12:20:17.343 7fce967872c0 10 monclient: get_monmap_and_config success -184> 2019-12-22 12:20:17.343 7fce967872c0 10 monclient: shutdown -183> 2019-12-22 12:20:17.343 7fce85ac1700 4 set_mon_vals no callback set -182> 2019-12-22 12:20:17.343 7fce85ac1700 10 set_mon_vals mon_target_pg_per_osd = 50 -181> 2019-12-22 12:20:17.347 7fce967872c0 0 set uid:gid to 64045:64045 (ceph:ceph) -180> 2019-12-22 12:20:17.347 7fce967872c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7040 -179> 2019-12-22 12:20:17.347 7fce967872c0 0 pidfile_write: ignore empty --pid-file -178> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) init /var/run/ceph/ceph-mds.ceph1.asok -177> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) bind_and_listen /var/run/ceph/ceph-mds.ceph1.asok -176> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) register_command 0 hook 0x562d33cf2ac8 -175> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) register_command version hook 0x562d33cf2ac8 -174> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) register_command git_version hook 0x562d33cf2ac8 -173> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) register_command help hook 0x562d33cf4380 -172> 2019-12-22 12:20:17.347 7fce967872c0 5 asok(0x562d33d82000) register_command get_command_descriptions hook 0x562d33cf4460 -171> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding auth protocol: cephx -170> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding auth protocol: cephx -169> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding auth protocol: cephx -168> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -167> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -166> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -165> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -164> 2019-12-22 12:20:17.347 7fce862c2700 5 asok(0x562d33d82000) entry start -163> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -162> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -161> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -160> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -159> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -158> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -157> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: crc -156> 2019-12-22 12:20:17.347 7fce967872c0 5 AuthRegistry(0x562d33dc6a40) adding con mode: secure -155> 2019-12-22 12:20:17.347 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -154> 2019-12-22 12:20:17.347 7fce967872c0 10 monclient: build_initial_monmap -153> 2019-12-22 12:20:17.347 7fce967872c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:17.352920 created 2019-12-22 12:20:17.352920 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -152> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient: init -151> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding auth protocol: cephx -150> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding auth protocol: cephx -149> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding auth protocol: cephx -148> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -147> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -146> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -145> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -144> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -143> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -142> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -141> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -140> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -139> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -138> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: crc -137> 2019-12-22 12:20:17.351 7fce967872c0 5 AuthRegistry(0x7ffd444b8958) adding con mode: secure -136> 2019-12-22 12:20:17.351 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -135> 2019-12-22 12:20:17.351 7fce967872c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -134> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient: _reopen_session rank -1 -133> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): picked mon.noname-c con 0x562d33e1d680 addr v2:10.10.0.30:3300/0 -132> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): picked mon.noname-a-legacy con 0x562d33e1cd80 addr v1:10.10.0.10:6789/0 -131> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): picked mon.noname-a con 0x562d33e1df80 addr v2:10.10.0.10:3300/0 -130> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): start opening mon connection -129> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): start opening mon connection -128> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient(hunting): _renew_subs -127> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): get_auth_request con 0x562d33e1df80 auth_method 0 -126> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -125> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): _init_auth method 2 -124> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more payload 9 -123> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -122> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -121> 2019-12-22 12:20:17.351 7fce882c6700 10 monclient(hunting): get_auth_request con 0x562d33e1d680 auth_method 0 -120> 2019-12-22 12:20:17.351 7fce882c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -119> 2019-12-22 12:20:17.351 7fce882c6700 10 monclient(hunting): _init_auth method 2 -118> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient(hunting): handle_auth_done global_id 554631 payload 1102 -117> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient: _finish_hunting 0 -116> 2019-12-22 12:20:17.351 7fce88ac7700 1 monclient: found mon.noname-a -115> 2019-12-22 12:20:17.351 7fce88ac7700 10 monclient: _send_mon_message to mon.noname-a at v2:10.10.0.10:3300/0 -114> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 -113> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: handle_monmap mon_map magic: 0 v1 -112> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) -111> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -110> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: _finish_auth 0 -109> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:47.354648) -108> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -107> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: handle_config config(2 keys) v1 -106> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: handle_monmap mon_map magic: 0 v1 -105> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: got monmap 1 from mon.ceph1 (according to old e1) -104> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -103> 2019-12-22 12:20:17.351 7fce830d4700 4 set_mon_vals no callback set -102> 2019-12-22 12:20:17.351 7fce967872c0 5 monclient: authenticate success, global_id 554631 -101> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: _finish_auth 0 -100> 2019-12-22 12:20:17.351 7fce848d7700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-12-22 12:19:47.355045) -99> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient: wait_auth_rotating done -98> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient: _renew_subs -97> 2019-12-22 12:20:17.351 7fce967872c0 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -96> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command status hook 0x562d33cf47c0 -95> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump_ops_in_flight hook 0x562d33cf47c0 -94> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command ops hook 0x562d33cf47c0 -93> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump_blocked_ops hook 0x562d33cf47c0 -92> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump_historic_ops hook 0x562d33cf47c0 -91> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump_historic_ops_by_duration hook 0x562d33cf47c0 -90> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command scrub_path hook 0x562d33cf47c0 -89> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command tag path hook 0x562d33cf47c0 -88> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command flush_path hook 0x562d33cf47c0 -87> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command export dir hook 0x562d33cf47c0 -86> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump cache hook 0x562d33cf47c0 -85> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command cache status hook 0x562d33cf47c0 -84> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump tree hook 0x562d33cf47c0 -83> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump loads hook 0x562d33cf47c0 -82> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump snaps hook 0x562d33cf47c0 -81> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command session evict hook 0x562d33cf47c0 -80> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command osdmap barrier hook 0x562d33cf47c0 -79> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command session ls hook 0x562d33cf47c0 -78> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command flush journal hook 0x562d33cf47c0 -77> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command force_readonly hook 0x562d33cf47c0 -76> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command get subtrees hook 0x562d33cf47c0 -75> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dirfrag split hook 0x562d33cf47c0 -74> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dirfrag merge hook 0x562d33cf47c0 -73> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dirfrag ls hook 0x562d33cf47c0 -72> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command openfiles ls hook 0x562d33cf47c0 -71> 2019-12-22 12:20:17.351 7fce967872c0 5 asok(0x562d33d82000) register_command dump inode hook 0x562d33cf47c0 -70> 2019-12-22 12:20:17.351 7fce818d1700 5 mds.beacon.ceph1 Sending beacon up:boot seq 1 -69> 2019-12-22 12:20:17.351 7fce818d1700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -68> 2019-12-22 12:20:17.351 7fce848d7700 1 mds.ceph1 Updating MDS map to version 331 from mon.0 -67> 2019-12-22 12:20:17.355 7fce848d7700 4 mgrc handle_mgr_map Got map version 130 -66> 2019-12-22 12:20:17.355 7fce848d7700 4 mgrc handle_mgr_map Active mgr is now [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -65> 2019-12-22 12:20:17.355 7fce848d7700 4 mgrc reconnect Starting new session with [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -64> 2019-12-22 12:20:17.355 7fce882c6700 10 monclient: get_auth_request con 0x562d33e1f600 auth_method 0 -63> 2019-12-22 12:20:17.355 7fce848d7700 4 mgrc handle_mgr_configure stats_period=5 -62> 2019-12-22 12:20:17.355 7fce848d7700 4 mgrc handle_mgr_configure updated stats threshold: 5 -61> 2019-12-22 12:20:21.351 7fce818d1700 5 mds.beacon.ceph1 Sending beacon up:boot seq 2 -60> 2019-12-22 12:20:21.371 7fce818d1700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -59> 2019-12-22 12:20:21.811 7fce848d7700 1 mds.ceph1 Updating MDS map to version 332 from mon.0 -58> 2019-12-22 12:20:21.811 7fce848d7700 5 mds.beacon.ceph1 set_want_state: up:boot -> up:standby -57> 2019-12-22 12:20:21.811 7fce848d7700 1 mds.ceph1 Map has assigned me to become a standby -56> 2019-12-22 12:20:21.811 7fce88ac7700 5 mds.beacon.ceph1 received beacon reply up:boot seq 2 rtt 0.459994 -55> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.ceph1 Updating MDS map to version 333 from mon.0 -54> 2019-12-22 12:20:21.815 7fce848d7700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -53> 2019-12-22 12:20:21.815 7fce848d7700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -52> 2019-12-22 12:20:21.815 7fce848d7700 5 asok(0x562d33d82000) register_command objecter_requests hook 0x562d33cf4760 -51> 2019-12-22 12:20:21.815 7fce848d7700 10 monclient: _renew_subs -50> 2019-12-22 12:20:21.815 7fce848d7700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -49> 2019-12-22 12:20:21.815 7fce848d7700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -48> 2019-12-22 12:20:21.815 7fce848d7700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -47> 2019-12-22 12:20:21.815 7fce848d7700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -46> 2019-12-22 12:20:21.815 7fce848d7700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries -45> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 handle_mds_map i am now mds.0.333 -44> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 handle_mds_map state change up:boot --> up:replay -43> 2019-12-22 12:20:21.815 7fce848d7700 5 mds.beacon.ceph1 set_want_state: up:standby -> up:replay -42> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 replay_start -41> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 recovery set is -40> 2019-12-22 12:20:21.815 7fce848d7700 1 mds.0.333 waiting for osdmap 6044 (which blacklists prior instance) -39> 2019-12-22 12:20:21.819 7fce848d7700 4 mds.0.333 handle_osd_map epoch 6044, 0 new blacklist entries -38> 2019-12-22 12:20:21.819 7fce848d7700 10 monclient: _renew_subs -37> 2019-12-22 12:20:21.819 7fce848d7700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -36> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: opening inotable -35> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: opening sessionmap -34> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: opening mds log -33> 2019-12-22 12:20:21.819 7fce7e0ca700 5 mds.0.log open discovering log bounds -32> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: opening purge queue (async) -31> 2019-12-22 12:20:21.819 7fce7e0ca700 4 mds.0.purge_queue open: opening -30> 2019-12-22 12:20:21.819 7fce7e0ca700 1 mds.0.journaler.pq(ro) recover start -29> 2019-12-22 12:20:21.819 7fce7e0ca700 1 mds.0.journaler.pq(ro) read_head -28> 2019-12-22 12:20:21.819 7fce7d8c9700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -27> 2019-12-22 12:20:21.819 7fce882c6700 10 monclient: get_auth_request con 0x562d34add200 auth_method 0 -26> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: loading open file table (async) -25> 2019-12-22 12:20:21.819 7fce7e0ca700 2 mds.0.333 Booting: 0: opening snap table -24> 2019-12-22 12:20:21.819 7fce87ac5700 10 monclient: get_auth_request con 0x562d34add680 auth_method 0 -23> 2019-12-22 12:20:21.819 7fce87ac5700 10 monclient: get_auth_request con 0x562d34adcd80 auth_method 0 -22> 2019-12-22 12:20:21.819 7fce88ac7700 10 monclient: get_auth_request con 0x562d34addb00 auth_method 0 -21> 2019-12-22 12:20:21.819 7fce7f0cc700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4320033, write 4320033, stream_format 1). probing for end of log (from 4320033)... -20> 2019-12-22 12:20:21.819 7fce7f0cc700 1 mds.0.journaler.pq(ro) probing for end of the log -19> 2019-12-22 12:20:21.819 7fce7d8c9700 1 mds.0.journaler.mdlog(ro) recover start -18> 2019-12-22 12:20:21.819 7fce7d8c9700 1 mds.0.journaler.mdlog(ro) read_head -17> 2019-12-22 12:20:21.819 7fce7f0cc700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4320033 (header had 4320033). recovered. -16> 2019-12-22 12:20:21.819 7fce7f0cc700 4 mds.0.purge_queue operator(): open complete -15> 2019-12-22 12:20:21.819 7fce7f0cc700 1 mds.0.journaler.pq(ro) set_writeable -14> 2019-12-22 12:20:21.819 7fce7d8c9700 4 mds.0.log Waiting for journal 0x200 to recover... -13> 2019-12-22 12:20:21.819 7fce7e0ca700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 805306368, expire 808964827, write 812539373, stream_format 1). probing for end of log (from 812539373)... -12> 2019-12-22 12:20:21.819 7fce7e0ca700 1 mds.0.journaler.mdlog(ro) probing for end of the log -11> 2019-12-22 12:20:21.819 7fce7e0ca700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 812539516 (header had 812539373). recovered. -10> 2019-12-22 12:20:21.819 7fce7d8c9700 4 mds.0.log Journal 0x200 recovered. -9> 2019-12-22 12:20:21.819 7fce7d8c9700 4 mds.0.log Recovered journal 0x200 in format 1 -8> 2019-12-22 12:20:21.819 7fce7d8c9700 2 mds.0.333 Booting: 1: loading/discovering base inodes -7> 2019-12-22 12:20:21.819 7fce7d8c9700 0 mds.0.cache creating system inode with ino:0x100 -6> 2019-12-22 12:20:21.819 7fce7d8c9700 0 mds.0.cache creating system inode with ino:0x1 -5> 2019-12-22 12:20:21.823 7fce7e0ca700 2 mds.0.333 Booting: 2: replaying mds log -4> 2019-12-22 12:20:21.823 7fce7e0ca700 2 mds.0.333 Booting: 2: waiting for purge queue recovered -3> 2019-12-22 12:20:22.055 7fce7c8c7700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) -2> 2019-12-22 12:20:22.055 7fce7c8c7700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -1> 2019-12-22 12:20:22.055 7fce7c8c7700 0 mds.0.journal journal ambig_subtrees: 0> 2019-12-22 12:20:22.059 7fce7c8c7700 -1 *** Caught signal (Segmentation fault) ** in thread 7fce7c8c7700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7fce8d26d890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.ceph1.log --- end dump of recent events --- 2019-12-22 12:20:22.607 7f550c06c2c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-12-22 12:20:22.607 7f550c06c2c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7073 2019-12-22 12:20:22.611 7f550c06c2c0 0 pidfile_write: ignore empty --pid-file 2019-12-22 12:20:22.615 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 333 from mon.1 2019-12-22 12:20:22.819 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 334 from mon.1 2019-12-22 12:20:22.819 7f54fa1bc700 1 mds.ceph1 Map has assigned me to become a standby 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 335 from mon.1 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 handle_mds_map i am now mds.0.335 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 handle_mds_map state change up:boot --> up:replay 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 replay_start 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 recovery set is 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 waiting for osdmap 6045 (which blacklists prior instance) 2019-12-22 12:20:22.867 7f54f39af700 0 mds.0.cache creating system inode with ino:0x100 2019-12-22 12:20:22.867 7f54f39af700 0 mds.0.cache creating system inode with ino:0x1 2019-12-22 12:20:22.903 7f54f21ac700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) 2019-12-22 12:20:22.903 7f54f21ac700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} 2019-12-22 12:20:22.903 7f54f21ac700 0 mds.0.journal journal ambig_subtrees: 2019-12-22 12:20:22.907 7f54f21ac700 -1 *** Caught signal (Segmentation fault) ** in thread 7f54f21ac700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7f5502b52890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -272> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command assert hook 0x55d06f8ac500 -271> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command abort hook 0x55d06f8ac500 -270> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perfcounters_dump hook 0x55d06f8ac500 -269> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command 1 hook 0x55d06f8ac500 -268> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perf dump hook 0x55d06f8ac500 -267> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perfcounters_schema hook 0x55d06f8ac500 -266> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perf histogram dump hook 0x55d06f8ac500 -265> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command 2 hook 0x55d06f8ac500 -264> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perf schema hook 0x55d06f8ac500 -263> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perf histogram schema hook 0x55d06f8ac500 -262> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command perf reset hook 0x55d06f8ac500 -261> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config show hook 0x55d06f8ac500 -260> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config help hook 0x55d06f8ac500 -259> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config set hook 0x55d06f8ac500 -258> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config unset hook 0x55d06f8ac500 -257> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config get hook 0x55d06f8ac500 -256> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config diff hook 0x55d06f8ac500 -255> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command config diff get hook 0x55d06f8ac500 -254> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command log flush hook 0x55d06f8ac500 -253> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command log dump hook 0x55d06f8ac500 -252> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command log reopen hook 0x55d06f8ac500 -251> 2019-12-22 12:20:22.595 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump_mempools hook 0x55d070544068 -250> 2019-12-22 12:20:22.603 7f550c06c2c0 10 monclient: get_monmap_and_config -249> 2019-12-22 12:20:22.603 7f550c06c2c0 10 monclient: build_initial_monmap -248> 2019-12-22 12:20:22.603 7f550c06c2c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:22.607767 created 2019-12-22 12:20:22.607767 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -247> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding auth protocol: cephx -246> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding auth protocol: cephx -245> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding auth protocol: cephx -244> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -243> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -242> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -241> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -240> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -239> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -238> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -237> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -236> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -235> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -234> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: crc -233> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x55d06f97e140) adding con mode: secure -232> 2019-12-22 12:20:22.603 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -231> 2019-12-22 12:20:22.603 7f550c06c2c0 10 monclient: init -230> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding auth protocol: cephx -229> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding auth protocol: cephx -228> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding auth protocol: cephx -227> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -226> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -225> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -224> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -223> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -222> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -221> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -220> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -219> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -218> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -217> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: crc -216> 2019-12-22 12:20:22.603 7f550c06c2c0 5 AuthRegistry(0x7ffe15922db8) adding con mode: secure -215> 2019-12-22 12:20:22.603 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -214> 2019-12-22 12:20:22.603 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -213> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient: _reopen_session rank -1 -212> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x55d06f9d5680 addr v1:10.10.0.20:6789/0 -211> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): picked mon.noname-a con 0x55d06f9d4d80 addr v2:10.10.0.10:3300/0 -210> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): picked mon.noname-b con 0x55d06f9d5b00 addr v2:10.10.0.20:3300/0 -209> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): start opening mon connection -208> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): start opening mon connection -207> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): _renew_subs -206> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient(hunting): authenticate will time out at 2019-12-22 12:25:22.609116 -205> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): get_auth_request con 0x55d06f9d4d80 auth_method 0 -204> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -203> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): _init_auth method 2 -202> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): handle_auth_reply_more payload 9 -201> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -200> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -199> 2019-12-22 12:20:22.607 7f54fe3ac700 10 monclient(hunting): get_auth_request con 0x55d06f9d5b00 auth_method 0 -198> 2019-12-22 12:20:22.607 7f54fe3ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -197> 2019-12-22 12:20:22.607 7f54fe3ac700 10 monclient(hunting): _init_auth method 2 -196> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient(hunting): handle_auth_done global_id 554637 payload 386 -195> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient: _finish_hunting 0 -194> 2019-12-22 12:20:22.607 7f54fd3aa700 1 monclient: found mon.noname-a -193> 2019-12-22 12:20:22.607 7f54fd3aa700 10 monclient: _send_mon_message to mon.noname-a at v2:10.10.0.10:3300/0 -192> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: handle_monmap mon_map magic: 0 v1 -191> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) -190> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -189> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: _finish_auth 0 -188> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:52.610435) -187> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -186> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: handle_config config(2 keys) v1 -185> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: handle_monmap mon_map magic: 0 v1 -184> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: got monmap 1 from mon.ceph1 (according to old e1) -183> 2019-12-22 12:20:22.607 7f54fb3a6700 4 set_mon_vals no callback set -182> 2019-12-22 12:20:22.607 7f54fcba9700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -181> 2019-12-22 12:20:22.607 7f54fb3a6700 10 set_mon_vals mon_target_pg_per_osd = 50 -180> 2019-12-22 12:20:22.607 7f550c06c2c0 5 monclient: authenticate success, global_id 554637 -179> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient: get_monmap_and_config success -178> 2019-12-22 12:20:22.607 7f550c06c2c0 10 monclient: shutdown -177> 2019-12-22 12:20:22.607 7f550c06c2c0 0 set uid:gid to 64045:64045 (ceph:ceph) -176> 2019-12-22 12:20:22.607 7f550c06c2c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7073 -175> 2019-12-22 12:20:22.611 7f550c06c2c0 0 pidfile_write: ignore empty --pid-file -174> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) init /var/run/ceph/ceph-mds.ceph1.asok -173> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) bind_and_listen /var/run/ceph/ceph-mds.ceph1.asok -172> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) register_command 0 hook 0x55d06f8aaac8 -171> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) register_command version hook 0x55d06f8aaac8 -170> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) register_command git_version hook 0x55d06f8aaac8 -169> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) register_command help hook 0x55d06f8ac480 -168> 2019-12-22 12:20:22.611 7f550c06c2c0 5 asok(0x55d06f93a000) register_command get_command_descriptions hook 0x55d06f8ac490 -167> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding auth protocol: cephx -166> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding auth protocol: cephx -165> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding auth protocol: cephx -164> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -163> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -162> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -161> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -160> 2019-12-22 12:20:22.611 7f54fbba7700 5 asok(0x55d06f93a000) entry start -159> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -158> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -157> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -156> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -155> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -154> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -153> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: crc -152> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x55d06f97ea40) adding con mode: secure -151> 2019-12-22 12:20:22.611 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -150> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient: build_initial_monmap -149> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:22.615334 created 2019-12-22 12:20:22.615334 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -148> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient: init -147> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding auth protocol: cephx -146> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding auth protocol: cephx -145> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding auth protocol: cephx -144> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -143> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -142> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -141> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -140> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -139> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -138> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -137> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -136> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -135> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -134> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: crc -133> 2019-12-22 12:20:22.611 7f550c06c2c0 5 AuthRegistry(0x7ffe15924b48) adding con mode: secure -132> 2019-12-22 12:20:22.611 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -131> 2019-12-22 12:20:22.611 7f550c06c2c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -130> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient: _reopen_session rank -1 -129> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient(hunting): picked mon.noname-c-legacy con 0x55d06f9d5680 addr v1:10.10.0.30:6789/0 -128> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x55d06f9d4d80 addr v1:10.10.0.20:6789/0 -127> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient(hunting): picked mon.noname-b con 0x55d06f9d5f80 addr v2:10.10.0.20:3300/0 -126> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient(hunting): start opening mon connection -125> 2019-12-22 12:20:22.611 7f550c06c2c0 10 monclient(hunting): _renew_subs -124> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): get_auth_request con 0x55d06f9d5f80 auth_method 0 -123> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -122> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): _init_auth method 2 -121> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): handle_auth_reply_more payload 9 -120> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -119> 2019-12-22 12:20:22.611 7f54fe3ac700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -118> 2019-12-22 12:20:22.611 7f54fa1bc700 10 monclient(hunting): _init_auth method 2 -117> 2019-12-22 12:20:22.611 7f54fa1bc700 10 monclient(hunting): my global_id is 554762 -116> 2019-12-22 12:20:22.615 7f54fe3ac700 10 monclient(hunting): handle_auth_done global_id 554743 payload 1102 -115> 2019-12-22 12:20:22.615 7f54fe3ac700 10 monclient: _finish_hunting 0 -114> 2019-12-22 12:20:22.615 7f54fe3ac700 1 monclient: found mon.noname-b -113> 2019-12-22 12:20:22.615 7f54fe3ac700 10 monclient: _send_mon_message to mon.noname-b at v2:10.10.0.20:3300/0 -112> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: handle_monmap mon_map magic: 0 v1 -111> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) -110> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -109> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: _finish_auth 0 -108> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:52.617728) -107> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: _send_mon_message to mon.ceph2 at v2:10.10.0.20:3300/0 -106> 2019-12-22 12:20:22.615 7f550c06c2c0 5 monclient: authenticate success, global_id 554743 -105> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: handle_config config(2 keys) v1 -104> 2019-12-22 12:20:22.615 7f550c06c2c0 10 monclient: wait_auth_rotating waiting (until 2019-12-22 12:20:52.617799) -103> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: handle_monmap mon_map magic: 0 v1 -102> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: got monmap 1 from mon.ceph2 (according to old e1) -101> 2019-12-22 12:20:22.615 7f54f89b9700 4 set_mon_vals no callback set -100> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -99> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: _finish_auth 0 -98> 2019-12-22 12:20:22.615 7f54fa1bc700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-12-22 12:19:52.618067) -97> 2019-12-22 12:20:22.615 7f550c06c2c0 10 monclient: wait_auth_rotating done -96> 2019-12-22 12:20:22.615 7f550c06c2c0 10 monclient: _renew_subs -95> 2019-12-22 12:20:22.615 7f550c06c2c0 10 monclient: _send_mon_message to mon.ceph2 at v2:10.10.0.20:3300/0 -94> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command status hook 0x55d06f8ac7c0 -93> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump_ops_in_flight hook 0x55d06f8ac7c0 -92> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command ops hook 0x55d06f8ac7c0 -91> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump_blocked_ops hook 0x55d06f8ac7c0 -90> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump_historic_ops hook 0x55d06f8ac7c0 -89> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump_historic_ops_by_duration hook 0x55d06f8ac7c0 -88> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command scrub_path hook 0x55d06f8ac7c0 -87> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command tag path hook 0x55d06f8ac7c0 -86> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command flush_path hook 0x55d06f8ac7c0 -85> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command export dir hook 0x55d06f8ac7c0 -84> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump cache hook 0x55d06f8ac7c0 -83> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command cache status hook 0x55d06f8ac7c0 -82> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump tree hook 0x55d06f8ac7c0 -81> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump loads hook 0x55d06f8ac7c0 -80> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump snaps hook 0x55d06f8ac7c0 -79> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command session evict hook 0x55d06f8ac7c0 -78> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command osdmap barrier hook 0x55d06f8ac7c0 -77> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command session ls hook 0x55d06f8ac7c0 -76> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command flush journal hook 0x55d06f8ac7c0 -75> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command force_readonly hook 0x55d06f8ac7c0 -74> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command get subtrees hook 0x55d06f8ac7c0 -73> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dirfrag split hook 0x55d06f8ac7c0 -72> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dirfrag merge hook 0x55d06f8ac7c0 -71> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dirfrag ls hook 0x55d06f8ac7c0 -70> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command openfiles ls hook 0x55d06f8ac7c0 -69> 2019-12-22 12:20:22.615 7f550c06c2c0 5 asok(0x55d06f93a000) register_command dump inode hook 0x55d06f8ac7c0 -68> 2019-12-22 12:20:22.615 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 333 from mon.1 -67> 2019-12-22 12:20:22.615 7f54f71b6700 5 mds.beacon.ceph1 Sending beacon up:boot seq 1 -66> 2019-12-22 12:20:22.615 7f54fa1bc700 4 mgrc handle_mgr_map Got map version 130 -65> 2019-12-22 12:20:22.615 7f54fa1bc700 4 mgrc handle_mgr_map Active mgr is now [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -64> 2019-12-22 12:20:22.615 7f54fa1bc700 4 mgrc reconnect Starting new session with [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -63> 2019-12-22 12:20:22.619 7f54fdbab700 10 monclient: get_auth_request con 0x55d06f9d7600 auth_method 0 -62> 2019-12-22 12:20:22.619 7f54fa1bc700 4 mgrc handle_mgr_configure stats_period=5 -61> 2019-12-22 12:20:22.619 7f54fa1bc700 4 mgrc handle_mgr_configure updated stats threshold: 5 -60> 2019-12-22 12:20:22.635 7f54f71b6700 10 monclient: _send_mon_message to mon.ceph2 at v2:10.10.0.20:3300/0 -59> 2019-12-22 12:20:22.819 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 334 from mon.1 -58> 2019-12-22 12:20:22.819 7f54fa1bc700 5 mds.beacon.ceph1 set_want_state: up:boot -> up:standby -57> 2019-12-22 12:20:22.819 7f54fa1bc700 1 mds.ceph1 Map has assigned me to become a standby -56> 2019-12-22 12:20:22.823 7f54fe3ac700 5 mds.beacon.ceph1 received beacon reply up:boot seq 1 rtt 0.207997 -55> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.ceph1 Updating MDS map to version 335 from mon.1 -54> 2019-12-22 12:20:22.827 7f54fa1bc700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -53> 2019-12-22 12:20:22.827 7f54fa1bc700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -52> 2019-12-22 12:20:22.827 7f54fa1bc700 5 asok(0x55d06f93a000) register_command objecter_requests hook 0x55d06f8ac320 -51> 2019-12-22 12:20:22.827 7f54fa1bc700 10 monclient: _renew_subs -50> 2019-12-22 12:20:22.827 7f54fa1bc700 10 monclient: _send_mon_message to mon.ceph2 at v2:10.10.0.20:3300/0 -49> 2019-12-22 12:20:22.827 7f54fa1bc700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -48> 2019-12-22 12:20:22.827 7f54fa1bc700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -47> 2019-12-22 12:20:22.827 7f54fa1bc700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -46> 2019-12-22 12:20:22.827 7f54fa1bc700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries -45> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 handle_mds_map i am now mds.0.335 -44> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 handle_mds_map state change up:boot --> up:replay -43> 2019-12-22 12:20:22.827 7f54fa1bc700 5 mds.beacon.ceph1 set_want_state: up:standby -> up:replay -42> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 replay_start -41> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 recovery set is -40> 2019-12-22 12:20:22.827 7f54fa1bc700 1 mds.0.335 waiting for osdmap 6045 (which blacklists prior instance) -39> 2019-12-22 12:20:22.831 7f54fa1bc700 4 mds.0.335 handle_osd_map epoch 6045, 0 new blacklist entries -38> 2019-12-22 12:20:22.831 7f54fa1bc700 10 monclient: _renew_subs -37> 2019-12-22 12:20:22.831 7f54fa1bc700 10 monclient: _send_mon_message to mon.ceph2 at v2:10.10.0.20:3300/0 -36> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: opening inotable -35> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: opening sessionmap -34> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: opening mds log -33> 2019-12-22 12:20:22.831 7f54f39af700 5 mds.0.log open discovering log bounds -32> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: opening purge queue (async) -31> 2019-12-22 12:20:22.831 7f54f39af700 4 mds.0.purge_queue open: opening -30> 2019-12-22 12:20:22.831 7f54f31ae700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -29> 2019-12-22 12:20:22.831 7f54f39af700 1 mds.0.journaler.pq(ro) recover start -28> 2019-12-22 12:20:22.831 7f54f39af700 1 mds.0.journaler.pq(ro) read_head -27> 2019-12-22 12:20:22.831 7f54fd3aa700 10 monclient: get_auth_request con 0x55d070698d80 auth_method 0 -26> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: loading open file table (async) -25> 2019-12-22 12:20:22.831 7f54fdbab700 10 monclient: get_auth_request con 0x55d070699200 auth_method 0 -24> 2019-12-22 12:20:22.831 7f54f39af700 2 mds.0.335 Booting: 0: opening snap table -23> 2019-12-22 12:20:22.831 7f54fd3aa700 10 monclient: get_auth_request con 0x55d070699680 auth_method 0 -22> 2019-12-22 12:20:22.831 7f54fe3ac700 10 monclient: get_auth_request con 0x55d070699b00 auth_method 0 -21> 2019-12-22 12:20:22.831 7f54f31ae700 1 mds.0.journaler.mdlog(ro) recover start -20> 2019-12-22 12:20:22.831 7f54f31ae700 1 mds.0.journaler.mdlog(ro) read_head -19> 2019-12-22 12:20:22.831 7f54f31ae700 4 mds.0.log Waiting for journal 0x200 to recover... -18> 2019-12-22 12:20:22.831 7f54f39af700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 805306368, expire 808964827, write 812539373, stream_format 1). probing for end of log (from 812539373)... -17> 2019-12-22 12:20:22.831 7f54f39af700 1 mds.0.journaler.mdlog(ro) probing for end of the log -16> 2019-12-22 12:20:22.855 7f54f49b1700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4320033, write 4320033, stream_format 1). probing for end of log (from 4320033)... -15> 2019-12-22 12:20:22.855 7f54f49b1700 1 mds.0.journaler.pq(ro) probing for end of the log -14> 2019-12-22 12:20:22.855 7f54f49b1700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4320033 (header had 4320033). recovered. -13> 2019-12-22 12:20:22.855 7f54f49b1700 4 mds.0.purge_queue operator(): open complete -12> 2019-12-22 12:20:22.855 7f54f49b1700 1 mds.0.journaler.pq(ro) set_writeable -11> 2019-12-22 12:20:22.867 7f54f39af700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 812539516 (header had 812539373). recovered. -10> 2019-12-22 12:20:22.867 7f54f31ae700 4 mds.0.log Journal 0x200 recovered. -9> 2019-12-22 12:20:22.867 7f54f31ae700 4 mds.0.log Recovered journal 0x200 in format 1 -8> 2019-12-22 12:20:22.867 7f54f39af700 2 mds.0.335 Booting: 1: loading/discovering base inodes -7> 2019-12-22 12:20:22.867 7f54f39af700 0 mds.0.cache creating system inode with ino:0x100 -6> 2019-12-22 12:20:22.867 7f54f39af700 0 mds.0.cache creating system inode with ino:0x1 -5> 2019-12-22 12:20:22.867 7f54f39af700 2 mds.0.335 Booting: 2: replaying mds log -4> 2019-12-22 12:20:22.867 7f54f39af700 2 mds.0.335 Booting: 2: waiting for purge queue recovered -3> 2019-12-22 12:20:22.903 7f54f21ac700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) -2> 2019-12-22 12:20:22.903 7f54f21ac700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -1> 2019-12-22 12:20:22.903 7f54f21ac700 0 mds.0.journal journal ambig_subtrees: 0> 2019-12-22 12:20:22.907 7f54f21ac700 -1 *** Caught signal (Segmentation fault) ** in thread 7f54f21ac700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7f5502b52890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.ceph1.log --- end dump of recent events --- 2019-12-22 12:20:23.391 7f48efc612c0 0 set uid:gid to 64045:64045 (ceph:ceph) 2019-12-22 12:20:23.391 7f48efc612c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7106 2019-12-22 12:20:23.391 7f48efc612c0 0 pidfile_write: ignore empty --pid-file 2019-12-22 12:20:23.395 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 335 from mon.0 2019-12-22 12:20:27.803 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 336 from mon.0 2019-12-22 12:20:27.803 7f48dddb1700 1 mds.ceph1 Map has assigned me to become a standby 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 337 from mon.0 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 handle_mds_map i am now mds.0.337 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 handle_mds_map state change up:boot --> up:replay 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 replay_start 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 recovery set is 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 waiting for osdmap 6046 (which blacklists prior instance) 2019-12-22 12:20:27.819 7f48d6da3700 0 mds.0.cache creating system inode with ino:0x100 2019-12-22 12:20:27.819 7f48d6da3700 0 mds.0.cache creating system inode with ino:0x1 2019-12-22 12:20:27.895 7f48d5da1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) 2019-12-22 12:20:27.895 7f48d5da1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} 2019-12-22 12:20:27.895 7f48d5da1700 0 mds.0.journal journal ambig_subtrees: 2019-12-22 12:20:27.899 7f48d5da1700 -1 *** Caught signal (Segmentation fault) ** in thread 7f48d5da1700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7f48e6747890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -270> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command assert hook 0x56188bbc8500 -269> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command abort hook 0x56188bbc8500 -268> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perfcounters_dump hook 0x56188bbc8500 -267> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command 1 hook 0x56188bbc8500 -266> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perf dump hook 0x56188bbc8500 -265> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perfcounters_schema hook 0x56188bbc8500 -264> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perf histogram dump hook 0x56188bbc8500 -263> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command 2 hook 0x56188bbc8500 -262> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perf schema hook 0x56188bbc8500 -261> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perf histogram schema hook 0x56188bbc8500 -260> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command perf reset hook 0x56188bbc8500 -259> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config show hook 0x56188bbc8500 -258> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config help hook 0x56188bbc8500 -257> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config set hook 0x56188bbc8500 -256> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config unset hook 0x56188bbc8500 -255> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config get hook 0x56188bbc8500 -254> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config diff hook 0x56188bbc8500 -253> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command config diff get hook 0x56188bbc8500 -252> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command log flush hook 0x56188bbc8500 -251> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command log dump hook 0x56188bbc8500 -250> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command log reopen hook 0x56188bbc8500 -249> 2019-12-22 12:20:23.379 7f48efc612c0 5 asok(0x56188bc56000) register_command dump_mempools hook 0x56188c860068 -248> 2019-12-22 12:20:23.383 7f48efc612c0 10 monclient: get_monmap_and_config -247> 2019-12-22 12:20:23.383 7f48efc612c0 10 monclient: build_initial_monmap -246> 2019-12-22 12:20:23.383 7f48efc612c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:23.388937 created 2019-12-22 12:20:23.388937 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -245> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding auth protocol: cephx -244> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding auth protocol: cephx -243> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding auth protocol: cephx -242> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -241> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -240> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -239> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -238> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -237> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -236> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -235> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -234> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -233> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -232> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: crc -231> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x56188bc9a140) adding con mode: secure -230> 2019-12-22 12:20:23.387 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -229> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient: init -228> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding auth protocol: cephx -227> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding auth protocol: cephx -226> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding auth protocol: cephx -225> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -224> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -223> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -222> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -221> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -220> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -219> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -218> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -217> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -216> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -215> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: crc -214> 2019-12-22 12:20:23.387 7f48efc612c0 5 AuthRegistry(0x7ffc3226a458) adding con mode: secure -213> 2019-12-22 12:20:23.387 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -212> 2019-12-22 12:20:23.387 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -211> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient: _reopen_session rank -1 -210> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x56188bcf1680 addr v1:10.10.0.20:6789/0 -209> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): picked mon.noname-a-legacy con 0x56188bcf0d80 addr v1:10.10.0.10:6789/0 -208> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): picked mon.noname-a con 0x56188bcf1b00 addr v2:10.10.0.10:3300/0 -207> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): start opening mon connection -206> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): _renew_subs -205> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient(hunting): authenticate will time out at 2019-12-22 12:25:23.390276 -204> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): get_auth_request con 0x56188bcf1b00 auth_method 0 -203> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -202> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): _init_auth method 2 -201> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more payload 9 -200> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -199> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -198> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient(hunting): handle_auth_done global_id 554640 payload 386 -197> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient: _finish_hunting 0 -196> 2019-12-22 12:20:23.387 7f48e1fa1700 1 monclient: found mon.noname-a -195> 2019-12-22 12:20:23.387 7f48e1fa1700 10 monclient: _send_mon_message to mon.noname-a at v2:10.10.0.10:3300/0 -194> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) v1 -193> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: handle_monmap mon_map magic: 0 v1 -192> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) -191> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -190> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: _finish_auth 0 -189> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:53.391286) -188> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -187> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: handle_config config(2 keys) v1 -186> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: handle_monmap mon_map magic: 0 v1 -185> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: got monmap 1 from mon.ceph1 (according to old e1) -184> 2019-12-22 12:20:23.387 7f48e079e700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -183> 2019-12-22 12:20:23.387 7f48efc612c0 5 monclient: authenticate success, global_id 554640 -182> 2019-12-22 12:20:23.387 7f48def9b700 4 set_mon_vals no callback set -181> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient: get_monmap_and_config success -180> 2019-12-22 12:20:23.387 7f48efc612c0 10 monclient: shutdown -179> 2019-12-22 12:20:23.387 7f48def9b700 10 set_mon_vals mon_target_pg_per_osd = 50 -178> 2019-12-22 12:20:23.391 7f48efc612c0 0 set uid:gid to 64045:64045 (ceph:ceph) -177> 2019-12-22 12:20:23.391 7f48efc612c0 0 ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable), process ceph-mds, pid 7106 -176> 2019-12-22 12:20:23.391 7f48efc612c0 0 pidfile_write: ignore empty --pid-file -175> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) init /var/run/ceph/ceph-mds.ceph1.asok -174> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) bind_and_listen /var/run/ceph/ceph-mds.ceph1.asok -173> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) register_command 0 hook 0x56188bbc6ac8 -172> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) register_command version hook 0x56188bbc6ac8 -171> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) register_command git_version hook 0x56188bbc6ac8 -170> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) register_command help hook 0x56188bbc8380 -169> 2019-12-22 12:20:23.391 7f48efc612c0 5 asok(0x56188bc56000) register_command get_command_descriptions hook 0x56188bbc8460 -168> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding auth protocol: cephx -167> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding auth protocol: cephx -166> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding auth protocol: cephx -165> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -164> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -163> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -162> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -161> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -160> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -159> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -158> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -157> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -156> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -155> 2019-12-22 12:20:23.391 7f48df79c700 5 asok(0x56188bc56000) entry start -154> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: crc -153> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x56188bc9aa40) adding con mode: secure -152> 2019-12-22 12:20:23.391 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -151> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient: build_initial_monmap -150> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient: monmap: epoch 0 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-12-22 12:20:23.396016 created 2019-12-22 12:20:23.396016 min_mon_release 0 (unknown) 0: v1:10.10.0.10:6789/0 mon.noname-a-legacy 1: v1:10.10.0.20:6789/0 mon.noname-b-legacy 2: v1:10.10.0.30:6789/0 mon.noname-c-legacy 3: v2:10.10.0.10:3300/0 mon.noname-a 4: v2:10.10.0.20:3300/0 mon.noname-b 5: v2:10.10.0.30:3300/0 mon.noname-c -149> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient: init -148> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding auth protocol: cephx -147> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding auth protocol: cephx -146> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding auth protocol: cephx -145> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -144> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -143> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -142> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -141> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -140> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -139> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -138> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -137> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -136> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -135> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: crc -134> 2019-12-22 12:20:23.391 7f48efc612c0 5 AuthRegistry(0x7ffc3226c1e8) adding con mode: secure -133> 2019-12-22 12:20:23.391 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -132> 2019-12-22 12:20:23.391 7f48efc612c0 2 auth: KeyRing::load: loaded key file /var/lib/ceph/mds/ceph-ceph1/keyring -131> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient: _reopen_session rank -1 -130> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient(hunting): picked mon.noname-a-legacy con 0x56188bcf1680 addr v1:10.10.0.10:6789/0 -129> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient(hunting): picked mon.noname-b-legacy con 0x56188bcf0d80 addr v1:10.10.0.20:6789/0 -128> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient(hunting): picked mon.noname-a con 0x56188bcf1f80 addr v2:10.10.0.10:3300/0 -127> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient(hunting): start opening mon connection -126> 2019-12-22 12:20:23.391 7f48efc612c0 10 monclient(hunting): _renew_subs -125> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): get_auth_request con 0x56188bcf1f80 auth_method 0 -124> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [1,2] -123> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): _init_auth method 2 -122> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more payload 9 -121> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 -120> 2019-12-22 12:20:23.391 7f48e1fa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes -119> 2019-12-22 12:20:23.391 7f48dddb1700 10 monclient(hunting): _init_auth method 2 -118> 2019-12-22 12:20:23.391 7f48dddb1700 10 monclient(hunting): my global_id is 554649 -117> 2019-12-22 12:20:23.395 7f48e1fa1700 10 monclient(hunting): handle_auth_done global_id 554646 payload 1102 -116> 2019-12-22 12:20:23.395 7f48e1fa1700 10 monclient: _finish_hunting 0 -115> 2019-12-22 12:20:23.395 7f48e1fa1700 1 monclient: found mon.noname-a -114> 2019-12-22 12:20:23.395 7f48e1fa1700 10 monclient: _send_mon_message to mon.noname-a at v2:10.10.0.10:3300/0 -113> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: handle_monmap mon_map magic: 0 v1 -112> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) -111> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -110> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: _finish_auth 0 -109> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: _check_auth_rotating renewing rotating keys (they expired before 2019-12-22 12:19:53.397436) -108> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -107> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: handle_config config(2 keys) v1 -106> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: handle_monmap mon_map magic: 0 v1 -105> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: got monmap 1 from mon.ceph1 (according to old e1) -104> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: dump: epoch 1 fsid 3ac4d7e5-98ca-4b75-bb88-02386caa5793 last_changed 2019-10-20 21:47:34.775628 created 2019-10-20 21:47:34.775628 min_mon_release 14 (nautilus) 0: [v2:10.10.0.10:3300/0,v1:10.10.0.10:6789/0] mon.ceph1 1: [v2:10.10.0.20:3300/0,v1:10.10.0.20:6789/0] mon.ceph2 2: [v2:10.10.0.30:3300/0,v1:10.10.0.30:6789/0] mon.ceph3 -103> 2019-12-22 12:20:23.395 7f48dc5ae700 4 set_mon_vals no callback set -102> 2019-12-22 12:20:23.395 7f48efc612c0 5 monclient: authenticate success, global_id 554646 -101> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: _finish_auth 0 -100> 2019-12-22 12:20:23.395 7f48dddb1700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-12-22 12:19:53.397749) -99> 2019-12-22 12:20:23.395 7f48efc612c0 10 monclient: wait_auth_rotating done -98> 2019-12-22 12:20:23.395 7f48efc612c0 10 monclient: _renew_subs -97> 2019-12-22 12:20:23.395 7f48efc612c0 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -96> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command status hook 0x56188bbc87d0 -95> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump_ops_in_flight hook 0x56188bbc87d0 -94> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command ops hook 0x56188bbc87d0 -93> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump_blocked_ops hook 0x56188bbc87d0 -92> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump_historic_ops hook 0x56188bbc87d0 -91> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump_historic_ops_by_duration hook 0x56188bbc87d0 -90> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command scrub_path hook 0x56188bbc87d0 -89> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command tag path hook 0x56188bbc87d0 -88> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command flush_path hook 0x56188bbc87d0 -87> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command export dir hook 0x56188bbc87d0 -86> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump cache hook 0x56188bbc87d0 -85> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command cache status hook 0x56188bbc87d0 -84> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump tree hook 0x56188bbc87d0 -83> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump loads hook 0x56188bbc87d0 -82> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump snaps hook 0x56188bbc87d0 -81> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command session evict hook 0x56188bbc87d0 -80> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command osdmap barrier hook 0x56188bbc87d0 -79> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command session ls hook 0x56188bbc87d0 -78> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command flush journal hook 0x56188bbc87d0 -77> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command force_readonly hook 0x56188bbc87d0 -76> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command get subtrees hook 0x56188bbc87d0 -75> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dirfrag split hook 0x56188bbc87d0 -74> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dirfrag merge hook 0x56188bbc87d0 -73> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dirfrag ls hook 0x56188bbc87d0 -72> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command openfiles ls hook 0x56188bbc87d0 -71> 2019-12-22 12:20:23.395 7f48efc612c0 5 asok(0x56188bc56000) register_command dump inode hook 0x56188bbc87d0 -70> 2019-12-22 12:20:23.395 7f48dadab700 5 mds.beacon.ceph1 Sending beacon up:boot seq 1 -69> 2019-12-22 12:20:23.395 7f48dadab700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -68> 2019-12-22 12:20:23.395 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 335 from mon.0 -67> 2019-12-22 12:20:23.395 7f48dddb1700 4 mgrc handle_mgr_map Got map version 130 -66> 2019-12-22 12:20:23.395 7f48dddb1700 4 mgrc handle_mgr_map Active mgr is now [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -65> 2019-12-22 12:20:23.395 7f48dddb1700 4 mgrc reconnect Starting new session with [v2:10.10.0.20:6800/1434,v1:10.10.0.20:6801/1434] -64> 2019-12-22 12:20:23.395 7f48e17a0700 10 monclient: get_auth_request con 0x56188bcf3600 auth_method 0 -63> 2019-12-22 12:20:23.399 7f48dddb1700 4 mgrc handle_mgr_configure stats_period=5 -62> 2019-12-22 12:20:23.399 7f48dddb1700 4 mgrc handle_mgr_configure updated stats threshold: 5 -61> 2019-12-22 12:20:27.395 7f48dadab700 5 mds.beacon.ceph1 Sending beacon up:boot seq 2 -60> 2019-12-22 12:20:27.395 7f48dadab700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -59> 2019-12-22 12:20:27.803 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 336 from mon.0 -58> 2019-12-22 12:20:27.803 7f48dddb1700 5 mds.beacon.ceph1 set_want_state: up:boot -> up:standby -57> 2019-12-22 12:20:27.803 7f48dddb1700 1 mds.ceph1 Map has assigned me to become a standby -56> 2019-12-22 12:20:27.803 7f48e1fa1700 5 mds.beacon.ceph1 received beacon reply up:boot seq 2 rtt 0.407994 -55> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.ceph1 Updating MDS map to version 337 from mon.0 -54> 2019-12-22 12:20:27.811 7f48dddb1700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -53> 2019-12-22 12:20:27.811 7f48dddb1700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -52> 2019-12-22 12:20:27.811 7f48dddb1700 5 asok(0x56188bc56000) register_command objecter_requests hook 0x56188bbc8980 -51> 2019-12-22 12:20:27.811 7f48dddb1700 10 monclient: _renew_subs -50> 2019-12-22 12:20:27.811 7f48dddb1700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -49> 2019-12-22 12:20:27.811 7f48dddb1700 10 log_channel(cluster) update_config to_monitors: true to_syslog: false syslog_facility: daemon prio: info to_graylog: false graylog_host: 127.0.0.1 graylog_port: 12201) -48> 2019-12-22 12:20:27.811 7f48dddb1700 4 mds.0.purge_queue operator(): data pool 11 not found in OSDMap -47> 2019-12-22 12:20:27.811 7f48dddb1700 4 mds.0.purge_queue operator(): data pool 12 not found in OSDMap -46> 2019-12-22 12:20:27.811 7f48dddb1700 4 mds.0.0 handle_osd_map epoch 0, 0 new blacklist entries -45> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 handle_mds_map i am now mds.0.337 -44> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 handle_mds_map state change up:boot --> up:replay -43> 2019-12-22 12:20:27.811 7f48dddb1700 5 mds.beacon.ceph1 set_want_state: up:standby -> up:replay -42> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 replay_start -41> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 recovery set is -40> 2019-12-22 12:20:27.811 7f48dddb1700 1 mds.0.337 waiting for osdmap 6046 (which blacklists prior instance) -39> 2019-12-22 12:20:27.815 7f48dddb1700 4 mds.0.337 handle_osd_map epoch 6046, 0 new blacklist entries -38> 2019-12-22 12:20:27.815 7f48dddb1700 10 monclient: _renew_subs -37> 2019-12-22 12:20:27.815 7f48dddb1700 10 monclient: _send_mon_message to mon.ceph1 at v2:10.10.0.10:3300/0 -36> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: opening inotable -35> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: opening sessionmap -34> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: opening mds log -33> 2019-12-22 12:20:27.815 7f48d75a4700 5 mds.0.log open discovering log bounds -32> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: opening purge queue (async) -31> 2019-12-22 12:20:27.815 7f48d75a4700 4 mds.0.purge_queue open: opening -30> 2019-12-22 12:20:27.815 7f48d75a4700 1 mds.0.journaler.pq(ro) recover start -29> 2019-12-22 12:20:27.815 7f48d75a4700 1 mds.0.journaler.pq(ro) read_head -28> 2019-12-22 12:20:27.815 7f48d6da3700 4 mds.0.journalpointer Reading journal pointer '400.00000000' -27> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: loading open file table (async) -26> 2019-12-22 12:20:27.815 7f48e17a0700 10 monclient: get_auth_request con 0x56188c9ab200 auth_method 0 -25> 2019-12-22 12:20:27.815 7f48d75a4700 2 mds.0.337 Booting: 0: opening snap table -24> 2019-12-22 12:20:27.815 7f48e0f9f700 10 monclient: get_auth_request con 0x56188c9aad80 auth_method 0 -23> 2019-12-22 12:20:27.815 7f48e0f9f700 10 monclient: get_auth_request con 0x56188c9ab680 auth_method 0 -22> 2019-12-22 12:20:27.815 7f48e1fa1700 10 monclient: get_auth_request con 0x56188c9abb00 auth_method 0 -21> 2019-12-22 12:20:27.815 7f48d85a6700 1 mds.0.journaler.pq(ro) _finish_read_head loghead(trim 4194304, expire 4320033, write 4320033, stream_format 1). probing for end of log (from 4320033)... -20> 2019-12-22 12:20:27.815 7f48d85a6700 1 mds.0.journaler.pq(ro) probing for end of the log -19> 2019-12-22 12:20:27.815 7f48d6da3700 1 mds.0.journaler.mdlog(ro) recover start -18> 2019-12-22 12:20:27.815 7f48d6da3700 1 mds.0.journaler.mdlog(ro) read_head -17> 2019-12-22 12:20:27.815 7f48d6da3700 4 mds.0.log Waiting for journal 0x200 to recover... -16> 2019-12-22 12:20:27.815 7f48d85a6700 1 mds.0.journaler.pq(ro) _finish_probe_end write_pos = 4320033 (header had 4320033). recovered. -15> 2019-12-22 12:20:27.815 7f48d85a6700 4 mds.0.purge_queue operator(): open complete -14> 2019-12-22 12:20:27.815 7f48d85a6700 1 mds.0.journaler.pq(ro) set_writeable -13> 2019-12-22 12:20:27.815 7f48d75a4700 1 mds.0.journaler.mdlog(ro) _finish_read_head loghead(trim 805306368, expire 808964827, write 812539373, stream_format 1). probing for end of log (from 812539373)... -12> 2019-12-22 12:20:27.815 7f48d75a4700 1 mds.0.journaler.mdlog(ro) probing for end of the log -11> 2019-12-22 12:20:27.819 7f48d75a4700 1 mds.0.journaler.mdlog(ro) _finish_probe_end write_pos = 812539516 (header had 812539373). recovered. -10> 2019-12-22 12:20:27.819 7f48d6da3700 4 mds.0.log Journal 0x200 recovered. -9> 2019-12-22 12:20:27.819 7f48d6da3700 4 mds.0.log Recovered journal 0x200 in format 1 -8> 2019-12-22 12:20:27.819 7f48d6da3700 2 mds.0.337 Booting: 1: loading/discovering base inodes -7> 2019-12-22 12:20:27.819 7f48d6da3700 0 mds.0.cache creating system inode with ino:0x100 -6> 2019-12-22 12:20:27.819 7f48d6da3700 0 mds.0.cache creating system inode with ino:0x1 -5> 2019-12-22 12:20:27.819 7f48d75a4700 2 mds.0.337 Booting: 2: replaying mds log -4> 2019-12-22 12:20:27.819 7f48d75a4700 2 mds.0.337 Booting: 2: waiting for purge queue recovered -3> 2019-12-22 12:20:27.895 7f48d5da1700 -1 log_channel(cluster) log [ERR] : replayed ESubtreeMap at 809504954 subtree root 0x1 is not mine in cache (it's -2,-2) -2> 2019-12-22 12:20:27.895 7f48d5da1700 0 mds.0.journal journal subtrees: {0x1=[],0x100=[]} -1> 2019-12-22 12:20:27.895 7f48d5da1700 0 mds.0.journal journal ambig_subtrees: 0> 2019-12-22 12:20:27.899 7f48d5da1700 -1 *** Caught signal (Segmentation fault) ** in thread 7f48d5da1700 thread_name:md_log_replay ceph version 14.2.5 (ad5bd132e1492173c85fda2cc863152730b16a92) nautilus (stable) 1: (()+0x12890) [0x7f48e6747890] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 0 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace 1/ 5 prioritycache -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-mds.ceph1.log --- end dump of recent events ---