Ceph » CephFS

Andreas Teuchert wrote:

When disabling mirroring on a FS with "ceph fs snapshot mirror disable <source-fs>" the "ceph.mirror.info" xattr is not removed from the target FS.

This causes an attempt to use the former target FS as a source FS ("ceph fs snapshot mirror enable <target-fs>") to fail with "Error EINVAL: file system is an active peer for file system: <source-fs>".

This was tested with 16.2.7.

(Apologies for late reply, Andreas)

We do remove the xattr when mirroring disabled - https://github.com/ceph/ceph/blob/main/src/pybind/mgr/mirroring/fs/snapshot_mirror.py#L438

Do you have debug mgr logs?

Hi Venky,

I tried it again, now with 17.2.1, and I could reproduce the issue. The mgr debug log is below.

As far as I understand the code, the purge_mirror_info() function that removes the xattr is not called when mirroring is disabled (which is handled by the disable_mirror() function).

However, when removing the peer with ceph fs snapshot mirror peer_remove <fs> <peer-uuid>, peer_remove() is called, which calls purge_mirror_info() and indeed removes the xattr.

So, maybe disabling the mirroring should be disallowed while there are still peers configured or disable_mirror() should also cleanly remove the peers (the peers are gone anyway after disabling and enabling mirroring).

Aug  8 16:34:20 HOSTNAME ceph-mgr[12991]: log_channel(audit) log [DBG] : from='client.204861 -' entity='client.admin' cmd=[{"prefix": "fs snapshot mirror disable", "fs_name": "fs-a", "target": ["mon-mgr", ""]}]: dispatch
Aug  8 16:34:20 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.snapshot_mirror] disabling mirror for filesystem fs-a
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring INFO mgr_util] scanning for idle connections..
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mgr_util] fs_name (fs-a) connections ([])
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring INFO mgr_util] cleaning up connections: []
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.notify
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.handle_notify r=0 acks=[(204819, 94678628866688, b'{"addr":"10.1.2.139:0/4162393046"}')] timeouts=[]
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.handle_notify: 204819: {'addr': '10.1.2.139:0/4162393046'}
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG root] mon_command: 'fs mirror disable' -> 0 in 0.969s
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] got notify type fs_map
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] refreshing policy for []
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.snapshot_mirror] shutdown pool policy for fs-a
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] FSPolicy.shutdown
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] canceling update timer task
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] update timer task canceled
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] stopping instance watcher
Aug  8 16:34:21 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.notify] InstanceWatcher.wait_and_stop
Aug  8 16:34:22 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.notify
Aug  8 16:34:22 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.handle_notify r=0 acks=[(204819, 94678628866688, b'{"addr":"10.1.2.139:0/4162393046"}')] timeouts=[]
Aug  8 16:34:22 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.handle_notify: 204819: {'addr': '10.1.2.139:0/4162393046'}
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.notify
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.notify] InstanceWatcher.handle_notify r=0 acks=[] timeouts=[]
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG root] update_instances: added={}, removed={'204819': '10.1.2.139:0/4162393046'}
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.snapshot_mirror] blocklisting instance_id: 204819 addr: 10.1.2.139:0/4162393046
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] async request_id: c7a4daae-d500-4ed9-80c7-90260bdf140d
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.utils] start_async_op: 1
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.dir_map.update] updating instances
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] pending purges: 1
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.notify] waiting done
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] stopping instance watcher
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.utils] wait_for_ops: 1
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] handle_remove: r=-2
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] pending purges: 0
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] pending updates: 0+1
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] applying 0 updates, 1 deletes
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.dir_map.update] handle_update: r=0
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.dir_map.update] finish: r=0
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.utils] calling <bound method FSPolicy.handle_update_instances of <mirroring.fs.snapshot_mirror.FSPolicy object at 0x7fdea47169b0>> params [[], ['204819'], 'c7a4daae-d500-4ed9-80c7-90260bdf140d', 0]
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring INFO mirroring.fs.snapshot_mirror] handle_update_instances: [] ['204819'] c7a4daae-d500-4ed9-80c7-90260bdf140d 0
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] handle_update_instances: policy shutting down
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.utils] finish_async_op: 0
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.utils] done
Aug  8 16:34:23 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mirroring.fs.snapshot_mirror] FSPolicy.shutdown done
Aug  8 16:34:51 HOSTNAME ceph-mgr[12991]: [mirroring INFO mgr_util] scanning for idle connections..
Aug  8 16:34:51 HOSTNAME ceph-mgr[12991]: [mirroring DEBUG mgr_util] fs_name (fs-a) connections ([])
Aug  8 16:34:51 HOSTNAME ceph-mgr[12991]: [mirroring INFO mgr_util] cleaning up connections: []