Project

General

Profile

Actions
Copy link

Bug #58196

closed

CephFS mirroring fails when using with Rook

Added by Deepika Upadhyay over 1 year ago. Updated over 1 year ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Reproducer:
Configure FS mirroring in rook ceph
https://rook.io/docs/rook/v1.10/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-mirroring/

I used 1 hr as the snapshot schedule interval.

[root@centos-8gb-hel1-1 ~]# kubectl logs -nrook-ceph rook-ceph-fs-mirror-dc5b66b74-f8bb4
Defaulted container "fs-mirror" out of: fs-mirror, log-collector, chown-container-data-dir (init)
debug 2022-12-05T15:12:29.819+0000 7f8c9cabd100  0 set uid:gid to 167:167 (ceph:ceph)
debug 2022-12-05T15:12:29.819+0000 7f8c9cabd100  0 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable), process cephfs-mirror, pid 1634
debug 2022-12-05T15:12:29.819+0000 7f8c9cabd100  0 pidfile_write: ignore empty --pid-file
debug 2022-12-05T15:12:29.832+0000 7f8c9cabd100  1 mgrc service_daemon_register cephfs-mirror.2179192 metadata {arch=x86_64,ceph_release=quincy,ceph_version=ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable),ceph_version_short=17.2.5,container_hostname=centos-8gb-hel1-1,container_image=quay.io/ceph/ceph:v17.2.5,cpu=Intel Xeon Processor (Skylake, IBRS),distro=centos,distro_description=CentOS Stream 8,distro_version=8,hostname=centos-8gb-hel1-1,id=fs-mirror,instance_id=2179192,kernel_description=#1 SMP Mon Jul 18 17:42:52 UTC 2022,kernel_version=4.18.0-408.el8.x86_64,mem_swap_kb=0,mem_total_kb=7755460,os=Linux,pod_name=rook-ceph-fs-mirror-dc5b66b74-f8bb4,pod_namespace=rook-ceph}
debug 2022-12-05T15:12:33.956+0000 7f8c97649700  0 cephfs::mirror::PeerReplayer(4c35ed9f-d8e0-49b7-8525-d58cf02b61ee) init: remote monitor host=[v2:65.109.139.111:3300,v1:65.109.139.111:6789],[v2:95.217.219.154:3300,v1:95.217.219.154:6789],[v2:65.21.2.149:3300,v1:65.21.2.149:6789]
debug 2022-12-05T15:12:35.102+0000 7f8c7a60f700 -1 cephfs::mirror::PeerReplayer(4c35ed9f-d8e0-49b7-8525-d58cf02b61ee) build_snap_map: failed to open local snap directory=/tmp/registry/.snap: (2) No such file or directory
debug 2022-12-05T15:12:35.102+0000 7f8c7a60f700 -1 cephfs::mirror::PeerReplayer(4c35ed9f-d8e0-49b7-8525-d58cf02b61ee) do_sync_snaps: failed to build local snap map
debug 2022-12-05T15:12:35.102+0000 7f8c7a60f700 -1 cephfs::mirror::PeerReplayer(4c35ed9f-d8e0-49b7-8525-d58cf02b61ee) sync_snaps: failed to sync snapshots for dir_root=/tmp/registry
debug 2022-12-05T15:12:36.076+0000 7f8c97649700  0 cephfs::mirror::PeerReplayer(f63ff9e0-3ecb-4406-a001-1bcf81e53bee) init: remote monitor host=[v2:65.109.139.111:3300,v1:65.109.139.111:6789],[v2:95.217.219.154:3300,v1:95.217.219.154:6789],[v2:65.21.2.149:3300,v1:65.21.2.149:6789]
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/tools/cephfs_mirror/FSMirror.cc: In function 'void cephfs::mirror::FSMirror::add_peer(const Peer&)' thread 7f8c97649700 time 2022-12-05T15:12:36.218409+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/tools/cephfs_mirror/FSMirror.cc: 396: FAILED ceph_assert(m_peer_replayers.size() == 1)
 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8ca006743f]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7f8ca0067605]
 3: (cephfs::mirror::FSMirror::add_peer(Peer const&)+0x527) [0x5634e525edb7]
 4: (Context::complete(int)+0xd) [0x5634e5258bad]
 5: (cephfs::mirror::Mirror::update_fs_mirrors()+0x6b8) [0x5634e5257118]
 6: (Context::complete(int)+0xd) [0x5634e5258bad]
 7: (CommonSafeTimer<std::mutex>::timer_thread()+0x12f) [0x7f8ca0166c6f]
 8: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f8ca0167d61]
 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c9f2921ca]
 10: clone()
*** Caught signal (Aborted) **
 in thread 7f8c97649700 thread_name:safe_timer
 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c9f29ccf0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8ca0067499]
 5: /usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7f8ca0067605]
 6: (cephfs::mirror::FSMirror::add_peer(Peer const&)+0x527) [0x5634e525edb7]
 7: (Context::complete(int)+0xd) [0x5634e5258bad]
 8: (cephfs::mirror::Mirror::update_fs_mirrors()+0x6b8) [0x5634e5257118]
 9: (Context::complete(int)+0xd) [0x5634e5258bad]
 10: (CommonSafeTimer<std::mutex>::timer_thread()+0x12f) [0x7f8ca0166c6f]
 11: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f8ca0167d61]
 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c9f2921ca]
 13: clone()
debug 2022-12-05T15:12:36.219+0000 7f8c97649700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/tools/cephfs_mirror/FSMirror.cc: In function 'void cephfs::mirror::FSMirror::add_peer(const Peer&)' thread 7f8c97649700 time 2022-12-05T15:12:36.218409+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.5/rpm/el8/BUILD/ceph-17.2.5/src/tools/cephfs_mirror/FSMirror.cc: 396: FAILED ceph_assert(m_peer_replayers.size() == 1)

 ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8ca006743f]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7f8ca0067605]
 3: (cephfs::mirror::FSMirror::add_peer(Peer const&)+0x527) [0x5634e525edb7]
 4: (Context::complete(int)+0xd) [0x5634e5258bad]
 5: (cephfs::mirror::Mirror::update_fs_mirrors()+0x6b8) [0x5634e5257118]
 6: (Context::complete(int)+0xd) [0x5634e5258bad]
 7: (CommonSafeTimer<std::mutex>::timer_thread()+0x12f) [0x7f8ca0166c6f]
 8: (CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f8ca0167d61]
 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c9f2921ca]
 10: clone()
Actions
Copy link
 #1

Updated by Venky Shankar over 1 year ago

Hi Deepika,

It seems that multiple file system peers have been added to the primary cluster mirror sync? cephfs-mirror supports onyl a single peer as of now.

Actions
Copy link
 #2

Updated by Venky Shankar over 1 year ago

  • Status changed from New to Rejected

Hi Deepika,

If you wanted to add multiple sync peers, then please create a feature tracker.

Closing this tracker since it uses a unsupported config.

Cheers,
Venky

Actions
Copy link

Also available in: Atom PDF