Actions
Bug #50020
closedqa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)"
% Done:
0%
Source:
Q/A
Tags:
Backport:
pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-03-25T07:13:14.867 DEBUG:teuthology.orchestra.run.smithi191:mirror status for fs: cephfs> ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@38 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:no valid command found; 10 closest matches: 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:0 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:1 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:2 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:abort 2021-03-25T07:13:14.979 INFO:teuthology.orchestra.run.smithi191.stderr:assert 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config diff 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config diff get <var> 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config get <var> 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config help [<var>] 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:config set <var> <val>... 2021-03-25T07:13:14.980 INFO:teuthology.orchestra.run.smithi191.stderr:admin_socket: invalid command 2021-03-25T07:13:14.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-25T07:13:14.981 WARNING:tasks.cephfs.test_mirroring:mirror daemon command with label "mirror status for fs: cephfs" failed: Command failed (mirror status for fs: cephfs) on smithi191 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@38' 2021-03-25T07:13:17.669 INFO:tasks.ceph.mgr.x.smithi191.stderr:2021-03-25T07:13:17.666+0000 7fad4dafa700 -1 mgr load Failed to construct class in 'mirroring' 2021-03-25T07:13:17.776 INFO:tasks.ceph.mgr.x.smithi191.stderr:2021-03-25T07:13:17.666+0000 7fad4dafa700 -1 mgr load Traceback (most recent call last): 2021-03-25T07:13:17.776 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/dir_map/load.py", line 34, in load_dir_map 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: ioctx.operate_read_op(read_op, MIRROR_OBJECT_NAME) 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "rados.pyx", line 3682, in rados.Ioctx.operate_read_op 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:rados.ObjectNotFound: [errno 2] RADOS object not found (Failed to operate read op for oid cephfs_mirror) 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:During handling of the above exception, another exception occurred: 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr:Traceback (most recent call last): 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/module.py", line 12, in __init__ 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: self.fs_snapshot_mirror = FSSnapshotMirror(self) 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 287, in __init__ 2021-03-25T07:13:17.777 INFO:tasks.ceph.mgr.x.smithi191.stderr: self.refresh_pool_policy() 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 502, in refresh_pool_policy 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: self.refresh_pool_policy_locked() 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 498, in refresh_pool_policy_locked 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: self.init_pool_policy(filesystem) 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/snapshot_mirror.py", line 476, in init_pool_policy 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: dir_mapping = load_dir_map(ioctx) 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: File "/usr/share/ceph/mgr/mirroring/fs/dir_map/load.py", line 44, in load_dir_map 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr: raise Exception(-e.errno) 2021-03-25T07:13:17.778 INFO:tasks.ceph.mgr.x.smithi191.stderr:Exception: -2
From: /ceph/teuthology-archive/pdonnell-2021-03-24_23:26:35-fs-wip-pdonnell-testing-20210324.190252-distro-basic-smithi/5995923/teuthology.log
Updated by Venky Shankar about 3 years ago
The index object (cephfs_mirror) is missing in the rados pool. This is created when mirroring is enabled (via mgr/mirroring).
I think to simulate the init failure, the test enabled mirroring via mon interface so that the mirroring daemon does not find the index obejct. We probably just need to disable mgr/mirroring for this test (before executing the mon command).
I'll push a PR.
Updated by Venky Shankar about 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 40465
Updated by Venky Shankar about 3 years ago
The real failure was the "failed" state not reflecting in mirror status. From the daemon logs, the mirror daemon restarted mirroring for the filesystem and the retry interval was low to not observe the failed state.
Updated by Venky Shankar about 3 years ago
- Priority changed from High to Low
test case fix: prio -> low
Updated by Patrick Donnelly about 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot about 3 years ago
- Copied to Backport #50185: pacific: qa: "RADOS object not found (Failed to operate read op for oid cephfs_mirror)" added
Updated by Loïc Dachary almost 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions