Project

General

Profile

Bug #50020

qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)"

Added by Patrick Donnelly 26 days ago. Updated 15 days ago.

Status:
Pending Backport
Priority:
Low
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-03-25T07:13:14.867 DEBUG:teuthology.orchestra.run.smithi191:mirror status for fs: cephfs> ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@38
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:no valid command found; 10 closest matches:
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:0
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:1
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:2
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:abort
2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:assert
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config diff
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config diff get <var>
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config get <var>
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config help [<var>]
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config set <var> <val>...
2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:admin_socket: invalid command
2021-03-25T07:13:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 22
2021-03-25T07:13:14.981 WARNING:tasks.cephfs.test_mirroring:mirror daemon command with label "mirror status for fs: cephfs" failed: Command failed (mirror status for fs: cephfs) on smithi191 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@38'
2021-03-25T07:13:17.669 INFO:tasks.ceph.mgr.x.smithi191.stderr:2021-03-25T07:13:17.666+0000 7fad4dafa700 -1 mgr load Failed to construct class in 'mirroring'
2021-03-25T07:13:17.776 INFO:tasks.ceph.mgr.x.smithi191.stderr:2021-03-25T07:13:17.666+0000 7fad4dafa700 -1 mgr load Traceback (most recent call last):
2021-03-25T07:13:17.776 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/dir_map/load.py", line 34, in load_dir_map
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:    ioctx.operate_read_op(read_op, MIRROR_OBJECT_NAME)
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "rados.pyx", line 3682, in rados.Ioctx.operate_read_op
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:rados.ObjectNotFound: [errno 2] RADOS object not found (Failed to operate read op for oid cephfs_mirror)
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:During handling of the above exception, another exception occurred:
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:Traceback (most recent call last):
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/module.py", line 12, in __init__
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:    self.fs_snapshot_mirror = FSSnapshotMirror(self)
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 287, in __init__
2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:    self.refresh_pool_policy()
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 502, in refresh_pool_policy
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:    self.refresh_pool_policy_locked()
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 498, in refresh_pool_policy_locked
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:    self.init_pool_policy(filesystem)
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 476, in init_pool_policy
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:    dir_mapping = load_dir_map(ioctx)
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:  File "/usr/share/ceph/mgr/mirroring/fs/dir_map/load.py", line 44, in load_dir_map
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:    raise Exception(-e.errno)
2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:Exception: -2

From: /ceph/teuthology-archive/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/5995923/teuthology.log


Related issues

Copied to CephFS - Backport #50185: pacific: qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)" In Progress

History

#1 Updated by Venky Shankar 25 days ago

The index object (cephfs_mirror) is missing in the rados pool. This is created when mirroring is enabled (via mgr/mirroring).

I think to simulate the init failure, the test enabled mirroring via mon interface so that the mirroring daemon does not find the index obejct. We probably just need to disable mgr/mirroring for this test (before executing the mon command).

I'll push a PR.

#2 Updated by Venky Shankar 23 days ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 40465

#3 Updated by Venky Shankar 23 days ago

The real failure was the "failed" state not reflecting in mirror status. From the daemon logs, the mirror daemon restarted mirroring for the filesystem and the retry interval was low to not observe the failed state.

#4 Updated by Venky Shankar 23 days ago

  • Priority changed from High to Low

test case fix: prio -> low

#5 Updated by Patrick Donnelly 15 days ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot 15 days ago

  • Copied to Backport #50185: pacific: qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)" added

Also available in: Atom PDF