Actions
Bug #63089
openqa: tasks/mirror times out
Status:
New
Priority:
Urgent
Assignee:
Category:
Administration/Usability
Target version:
% Done:
0%
Source:
Tags:
Backport:
reef,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
cephfs-mirror
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
/a/vshankar-2023-09-28_07:23:59-fs-wip-vshankar-testing-20230926.081818-testing-default-smithi/7405363
2023-09-28T11:15:33.524 DEBUG:teuthology.orchestra.run.smithi105:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs mirror enable cephfs 2023-09-28T11:15:33.549 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.549+0000 7f1d69c56040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-28T11:15:33.604 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.605+0000 7f1d69c56040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-28T11:15:33.657 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.657+0000 7f1d69c56040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-28T11:15:33.721 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.721+0000 7f1d69c56040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-28T11:15:33.794 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.794+0000 7f1d69c56040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-28T11:15:33.935 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:33.935+0000 7f1d69c56040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-28T11:15:34.002 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.002+0000 7f1d69c56040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-28T11:15:34.056 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.056+0000 7f1d69c56040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-28T11:15:34.118 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.118+0000 7f1d69c56040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-28T11:15:34.172 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.172+0000 7f1d69c56040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-28T11:15:34.534 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-09-28T11:15:34.534 DEBUG:teuthology.orchestra.run.smithi105:mirror status for fs: cephfs> ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@56 2023-09-28T11:15:34.572 INFO:tasks.ceph.mgr.x.smithi105.stderr:2023-09-28T11:15:34.572+0000 7f1d69c56040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:no valid command found; 1 closest matches: 2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:fs mirror status cephfs@54 2023-09-28T11:15:34.726 INFO:teuthology.orchestra.run.smithi105.stderr:admin_socket: invalid command 2023-09-28T11:15:34.729 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-09-28T11:15:34.730 WARNING:tasks.cephfs.test_mirroring:mirror daemon command with label "mirror status for fs: cephfs" failed: Command failed (mirror status for fs: cephfs) on smithi105 with status 22: 'ceph --admin-daemon /var/run/ceph/cephfs-mirror.asok fs mirror status cephfs@56'
Updated by Venky Shankar 7 months ago
Another instance, this time from reef branch: vshankar-2023-09-27_10:23:33-fs-wip-vshankar-testing-reef-20230927.021134-testing-default-smithi/7402858
From logs:
2023-09-27T13:52:11.070+0000 d0c7640 20 cephfs::mirror::Mirror schedule_mirror_update_task: scheduling fs mirror update (0x7083620) after 2 seconds
2023-09-27T13:52:11.071+0000 c8c6640 20 cephfs::mirror::FSMirror ~FSMirror
2023-09-27T13:52:11.071+0000 c8c6640 10 cephfs::mirror::Mirror enable_mirroring: starting FSMirror: filesystem={fscid=52, fs_name=cephfs}
2023-09-27T13:52:11.071+0000 c8c6640 10 cephfs::mirror::ServiceDaemon: 0x8fdf7e0 add_or_update_fs_attribute: fscid=52
2023-09-27T13:52:11.071+0000 c8c6640 10 cephfs::mirror::ServiceDaemon: 0x8fdf7e0 schedule_update_status
2023-09-27T13:52:11.071+0000 c8c6640 20 cephfs::mirror::FSMirror init
2023-09-27T13:52:11.071+0000 c8c6640 20 cephfs::mirror::Utils connect: connecting to cluster=ceph, client=client.mirror, mon_host=
2023-09-27T13:52:11.465+0000 c8c6640 10 cephfs::mirror::Utils connect: using mon addr=172.21.15.17
2023-09-27T13:52:12.071+0000 110cf640 20 cephfs::mirror::ServiceDaemon: 0x8fdf7e0 update_status: 1 filesystem(s)
2023-09-27T13:52:13.070+0000 d0c7640 20 cephfs::mirror::Mirror update_fs_mirrors
2023-09-27T13:52:22.110+0000 c8c6640 10 cephfs::mirror::Utils connect: connected to cluster=ceph using client=client.mirror
2023-09-27T13:52:22.169+0000 c8c6640 20 cephfs::mirror::Utils mount: filesystem={fscid=52, fs_name=cephfs}
2023-09-27T13:52:22.609+0000 c8c6640 10 cephfs::mirror::Utils mount: mounted filesystem={fscid=52, fs_name=cephfs}
2023-09-27T13:52:22.609+0000 c8c6640 10 cephfs::mirror::FSMirror init: rados addrs=172.21.15.17:0/3359552797
2023-09-27T13:52:22.609+0000 c8c6640 20 cephfs::mirror::FSMirror init_instance_watcher
2023-09-27T13:52:22.609+0000 c8c6640 20 cephfs::mirror::InstanceWatcher init
2023-09-27T13:52:22.609+0000 c8c6640 20 cephfs::mirror::InstanceWatcher create_instance
The daemon never returned from creating an instance object. Another observation is that the failures are with valgrind/
Actions