Bug #56487: Error EPERM: problem getting command descriptions from mon, when execute "ceph -s". - RADOS - Ceph

Actions

Copy link

Bug #56487

open

Error EPERM: problem getting command descriptions from mon, when execute "ceph -s".

Added by liqun zhang almost 2 years ago. Updated almost 2 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v15.2.13

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

version 15.2.13
disable cephx, and excute "ceph -s" every 1 second,
A great chance to reproduce this error. log as below:

Mon Jul 4 15:31:19 CST 2022
2022-07-04T15:31:20.219+0800 7f8595551700 10 monclient: get_monmap_and_config
2022-07-04T15:31:20.219+0800 7f8595551700 10 monclient: build_initial_monmap
2022-07-04T15:31:20.219+0800 7f8595551700 10 monclient: monmap:
epoch 0
fsid 7a9e6d1a-fb5c-11ec-bf4c-0e03b1ba142b
last_changed 2022-07-04T15:31:20.220530+0800
created 2022-07-04T15:31:20.220530+0800
min_mon_release 0 (unknown)
0: v1:192.168.38.16:6789/0 mon.a
1: v1:192.168.38.17:6789/0 mon.b
2: v1:192.168.38.18:6789/0 mon.c

2022-07-04T15:31:20.221+0800 7f8595551700 10 monclient: init
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient: _reopen_session rank -1
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient: _add_conns ranks=[1,2,0]
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient(hunting): picked mon.b con 0x7f859006c640 addr v1:192.168.38.17:6789/0
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient(hunting): picked mon.c con 0x7f859006f3b0 addr v1:192.168.38.18:6789/0
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient(hunting): picked mon.a con 0x7f859007cbd0 addr v1:192.168.38.16:6789/0
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient(hunting): _renew_subs
2022-07-04T15:31:20.222+0800 7f8595551700 10 monclient(hunting): authenticate will time out at 2022-07-04T15:36:20.223898+0800
2022-07-04T15:31:20.352+0800 7f858dffb700 10 monclient(hunting): _init_auth method 1
2022-07-04T15:31:20.352+0800 7f858dffb700 10 monclient(hunting): _init_auth creating new auth
2022-07-04T15:31:20.352+0800 7f858dffb700 10 monclient(hunting): my global_id is 152691
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: _finish_hunting 0
2022-07-04T15:31:20.353+0800 7f858dffb700 1 monclient: found mon.a
2022-07-04T15:31:20.353+0800 7f858dffb700 20 monclient: _un_backoff reopen_interval_multipler now 1
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: _send_mon_message to mon.a at v1:192.168.38.16:6789/0
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: _finish_auth 0
2022-07-04T15:31:20.353+0800 7f858dffb700 20 monclient: _check_auth_rotating not needed by client.admin
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: handle_monmap mon_map magic: 0 v1
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: got monmap 3 from mon.a (according to old e3)
2022-07-04T15:31:20.353+0800 7f858dffb700 10 monclient: dump:
epoch 3
fsid 7a9e6d1a-fb5c-11ec-bf4c-0e03b1ba142b
last_changed 2022-07-04T13:45:31.648208+0800
created 2022-07-04T13:45:16.451723+0800
min_mon_release 15 (octopus)
0: v1:192.168.38.16:6789/0 mon.a
1: v1:192.168.38.17:6789/0 mon.b
2: v1:192.168.38.18:6789/0 mon.c

2022-07-04T15:31:20.354+0800 7f8595551700 5 monclient: authenticate success, global_id 152691
2022-07-04T15:31:20.354+0800 7f8595551700 20 monclient: get_monmap_and_config waiting for monmap|config
2022-07-04T15:31:20.359+0800 7f858dffb700 10 monclient: handle_config config(0 keys) v1
2022-07-04T15:31:20.359+0800 7f858dffb700 10 monclient: handle_monmap mon_map magic: 0 v1
2022-07-04T15:31:20.359+0800 7f858dffb700 10 monclient: got monmap 3 from mon.a (according to old e3)
2022-07-04T15:31:20.359+0800 7f858dffb700 10 monclient: dump:
epoch 3
fsid 7a9e6d1a-fb5c-11ec-bf4c-0e03b1ba142b
last_changed 2022-07-04T13:45:31.648208+0800
created 2022-07-04T13:45:16.451723+0800
min_mon_release 15 (octopus)
0: v1:192.168.38.16:6789/0 mon.a
1: v1:192.168.38.17:6789/0 mon.b
2: v1:192.168.38.18:6789/0 mon.c

2022-07-04T15:31:20.359+0800 7f8595551700 10 monclient: get_monmap_and_config success
2022-07-04T15:31:20.359+0800 7f8595551700 10 monclient: shutdown
2022-07-04T15:31:20.359+0800 7f8595551700 20 monclient: shutdown discarding 0 pending message(s)
2022-07-04T15:31:20.359+0800 7f857ffff700 4 set_mon_vals no callback set
2022-07-04T15:31:20.369+0800 7f8595551700 10 monclient: build_initial_monmap
2022-07-04T15:31:20.369+0800 7f8595551700 10 monclient: monmap:
epoch 0
fsid 00000000-0000-0000-0000-000000000000
last_changed 0.000000
created 0.000000
min_mon_release 0 (unknown)
0: v1:192.168.38.16:6789/0 mon.noname-a
1: v1:192.168.38.17:6789/0 mon.noname-b
2: v1:192.168.38.18:6789/0 mon.noname-c

2022-07-04T15:31:20.369+0800 7f8595551700 1 librados: starting msgr at
2022-07-04T15:31:20.370+0800 7f8595551700 1 librados: starting objecter
2022-07-04T15:31:20.371+0800 7f8595551700 1 librados: setting wanted keys
2022-07-04T15:31:20.371+0800 7f8595551700 1 librados: calling monclient init
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient: init
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient: _reopen_session rank -1
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient: _add_conns ranks=[1,2,0]
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient(hunting): picked mon.noname-b con 0x7f859006c640 addr v1:192.168.38.17:6789/0
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient(hunting): picked mon.noname-c con 0x7f859006f3b0 addr v1:192.168.38.18:6789/0
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient(hunting): picked mon.noname-a con 0x7f859007cbd0 addr v1:192.168.38.16:6789/0
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient(hunting): _renew_subs
2022-07-04T15:31:20.371+0800 7f8595551700 10 monclient(hunting): authenticate will time out at 2022-07-04T15:36:20.372880+0800
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient(hunting): _init_auth method 1
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient(hunting): _init_auth creating new auth
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient(hunting): my global_id is 152697
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient: _finish_hunting 0
2022-07-04T15:31:20.452+0800 7f857ffff700 1 monclient: found mon.noname-a
2022-07-04T15:31:20.452+0800 7f857ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient: _send_mon_message to mon.noname-a at v1:192.168.38.16:6789/0
2022-07-04T15:31:20.452+0800 7f857ffff700 10 monclient: _finish_auth 0
2022-07-04T15:31:20.452+0800 7f857ffff700 20 monclient: _check_auth_rotating not needed by client.admin
2022-07-04T15:31:20.455+0800 7f8595551700 5 monclient: authenticate success, global_id 152697
2022-07-04T15:31:20.455+0800 7f8595551700 10 monclient: _renew_subs
2022-07-04T15:31:20.455+0800 7f8595551700 10 monclient: _send_mon_message to mon.noname-a at v1:192.168.38.16:6789/0
2022-07-04T15:31:20.455+0800 7f8595551700 10 monclient: _renew_subs
2022-07-04T15:31:20.455+0800 7f8595551700 10 monclient: _send_mon_message to mon.noname-a at v1:192.168.38.16:6789/0
2022-07-04T15:31:20.455+0800 7f8595551700 1 librados: init done
2022-07-04T15:31:20.457+0800 7f8595551700 10 monclient: start_mon_command cmd=[{"prefix": "get_command_descriptions"}]
2022-07-04T15:31:20.457+0800 7f8595551700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}]
2022-07-04T15:31:20.457+0800 7f8595551700 10 monclient: _send_mon_message to mon.noname-a at v1:192.168.38.16:6789/0
2022-07-04T15:31:20.460+0800 7f857ffff700 10 monclient: handle_monmap mon_map magic: 0 v1
2022-07-04T15:31:20.463+0800 7f857ffff700 10 monclient: got monmap 3 from mon.noname-a (according to old e3)
2022-07-04T15:31:20.463+0800 7f857ffff700 10 monclient: dump:
epoch 3
fsid 7a9e6d1a-fb5c-11ec-bf4c-0e03b1ba142b
last_changed 2022-07-04T13:45:31.648208+0800
created 2022-07-04T13:45:16.451723+0800
min_mon_release 15 (octopus)
0: v1:192.168.38.16:6789/0 mon.a
1: v1:192.168.38.17:6789/0 mon.b
2: v1:192.168.38.18:6789/0 mon.c

2022-07-04T15:31:20.488+0800 7f857ffff700 10 monclient: handle_config config(0 keys) v1
2022-07-04T15:31:20.488+0800 7f857ffff700 10 monclient: handle_monmap mon_map magic: 0 v1
2022-07-04T15:31:20.488+0800 7f857ffff700 10 monclient: got monmap 3 from mon.a (according to old e3)
2022-07-04T15:31:20.488+0800 7f857ffff700 10 monclient: dump:
epoch 3
fsid 7a9e6d1a-fb5c-11ec-bf4c-0e03b1ba142b
last_changed 2022-07-04T13:45:31.648208+0800
created 2022-07-04T13:45:16.451723+0800
min_mon_release 15 (octopus)
0: v1:192.168.38.16:6789/0 mon.a
1: v1:192.168.38.17:6789/0 mon.b
2: v1:192.168.38.18:6789/0 mon.c

2022-07-04T15:31:20.489+0800 7f857e7fc700 4 set_mon_vals no callback set
2022-07-04T15:31:20.523+0800 7f857ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}]
2022-07-04T15:31:20.523+0800 7f857ffff700 10 monclient: _finish_command 1 = -1 wrong fsid
Error EPERM: problem getting command descriptions from mon.

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by liqun zhang almost 2 years ago

in the case, cephx is disabled. test script as below:

#/usr/bin/bash
while true
do
echo `date` >> /tmp/o.log
ret=`ceph s >> /tmp/o.log 2>&1 `
sleep 1
echo '' >> /tmp/o.log
echo '---------------------------' >> /tmp/o.log
done

Actions

Copy link