Bug #51815
Updated by Neha Ojha over 2 years ago
<pre>
2021-07-22T03:33:49.333 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x
2021-07-22T03:33:49.335 INFO:tasks.ceph.mgr.x:Started
...
2021-07-22T03:35:27.068 INFO:tasks.ceph.mon.a.smithi050.stderr:2021-07-22T03:35:27.067+0000 7fd05e2be700 -1 log_channel(cluster) log [ERR] : Health check failed: no active mgr (MGR_DOWN)
...
2021-07-22T03:35:38.826 DEBUG:teuthology.orchestra.run.smithi050:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json
2021-07-22T03:37:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 124
</pre>
<pre>
2021-07-22T03:33:49.395+0000 7efe70d3d380 0 ceph version 16.2.5-107-g25279b58 (25279b580b12bb6c4029912fd2e89174d631ce62) pacific (stable), process ceph-mgr, pid 23364
...
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 <== mon.2 v1:172.21.15.50:6791/0 2 ==== auth_reply(proto 2 -13 (13) Permission denied) v1 ==== 24+0+0 (unknown 4080774440 0 0) 0x55e9b0762ea0 con 0x55e9aee70800
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: 0x55e9aed87ee0 handle_response ret = -13
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 >> v1:172.21.15.50:6791/0 conn(0x55e9aee70800 legacy=0x55e9af9e0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 <== mon.0 v1:172.21.15.50:6789/0 2 ==== auth_reply(proto 2 -13 (13) Permission denied) v1 ==== 24+0+0 (unknown 4080774440 0 0) 0x55e9b076dd40 con 0x55e9aee70c00
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: 0x55e9aed87e10 handle_response ret = -13
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 >> v1:172.21.15.50:6789/0 conn(0x55e9aee70c00 legacy=0x55e9af9e1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 <== mon.1 v1:172.21.15.50:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623661766 0 0) 0x55e9aed816c0 con 0x55e9b32ee000
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 monclient(hunting): _init_auth method 2
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 monclient(hunting): _init_auth already have auth, reseting
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: reset
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: 0x55e9aed871e0 handle_response ret = 0
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: got initial server challenge 215dfa99bc67a265
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: validate_tickets: want=55 need=0 have=55
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx: validate_tickets want 55 have 23 need 32
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: want=55 need=32 have=23
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: build_request
2021-07-22T03:33:53.063+0000 7efe6bcf1700 20 cephx client: old ticket len=96
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: get auth session key: client_challenge 90515daa5a75b25c
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 --> v1:172.21.15.50:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x55e9b32f8780 con 0x55e9b32ee000
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 <== mon.1 v1:172.21.15.50:6790/0 2 ==== auth_reply(proto 2 -13 (13) Permission denied) v1 ==== 24+0+0 (unknown 4080774440 0 0) 0x55e9b0762d00 con 0x55e9b32ee000
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 cephx client: 0x55e9aed871e0 handle_response ret = -13
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 -- 172.21.15.50:0/23364 >> v1:172.21.15.50:6790/0 conn(0x55e9b32ee000 legacy=0x55e9b32f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 monclient: _finish_hunting -13
2021-07-22T03:33:53.063+0000 7efe6bcf1700 1 monclient: no mon sessions established
2021-07-22T03:33:53.063+0000 7efe6bcf1700 10 monclient: _finish_auth -13
</pre>
rados/singleton/{all/rebuild-mondb mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-comp-lz4 rados supported-random-distro$/{ubuntu_latest}}
/a/yuriw-2021-07-21_19:04:13-rados-wip-yuri4-testing-2021-07-21-0859-pacific-distro-basic-smithi/6285312