Bug #50696
Updated by Ramana Raja about 3 years ago
See, https://pulpito.ceph.com/yuriw-2021-05-04_15:32:03-multimds-wip-yuri3-testing-2021-04-29-1036-nautilus-distro-basic-smithi/6094471/
Description: multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
<pre>
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36801f537d3dceb7c135151b37ba843b7c595bbe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-05-05T03:31:23.774 DEBUG:teuthology.run_tasks:Unwinding manager kclient
2021-05-05T03:31:23.787 INFO:tasks.kclient:Unmounting kernel clients...
2021-05-05T03:31:23.790 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2021-05-05T03:31:23.791 INFO:teuthology.orchestra.run:Running command with timeout 900
2021-05-05T03:31:23.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo umount /home/ubuntu/cephtest/mnt.0
</pre>
See this ceph_assert earlier in the log,
<pre>
2021-05-05T03:26:19.414 INFO:tasks.mds_thrash.fs.[None]:no change
2021-05-05T03:26:19.781 INFO:tasks.ceph.mds.d.smithi102.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.20-192-g36801f53/rpm/el7/BUILD/ceph-14.2.20-192-g36801f53/src/msg/async/ProtocolV1.cc: In function 'Ct<ProtocolV1>* ProtocolV1::handle_message_footer(char*, int)' thread 7fb8eff59700 time 2021-05-05 03:26:19.786792
2021-05-05T03:26:19.781 INFO:tasks.ceph.mds.d.smithi102.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.20-192-g36801f53/rpm/el7/BUILD/ceph-14.2.20-192-g36801f53/src/msg/async/ProtocolV1.cc: 967: FAILED ceph_assert(0 == "old msgs despite reconnect_seq feature")
2021-05-05T03:26:19.782 INFO:tasks.ceph.mds.d.smithi102.stderr: ceph version 14.2.20-192-g36801f537d3 (36801f537d3dceb7c135151b37ba843b7c595bbe) nautilus (stable)
2021-05-05T03:26:19.782 INFO:tasks.ceph.mds.d.smithi102.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7fb8f69e9467]
2021-05-05T03:26:19.783 INFO:tasks.ceph.mds.d.smithi102.stderr: 2: (()+0x25d62f) [0x7fb8f69e962f]
2021-05-05T03:26:19.783 INFO:tasks.ceph.mds.d.smithi102.stderr: 3: (ProtocolV1::handle_message_footer(char*, int)+0xf3d) [0x7fb8f6cd985d]
2021-05-05T03:26:19.783 INFO:tasks.ceph.mds.d.smithi102.stderr: 4: (()+0x54605d) [0x7fb8f6cd205d]
2021-05-05T03:26:19.783 INFO:tasks.ceph.mds.d.smithi102.stderr: 5: (AsyncConnection::process()+0x186) [0x7fb8f6cc3c66]
2021-05-05T03:26:19.784 INFO:tasks.ceph.mds.d.smithi102.stderr: 6: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7fb8f6d132d5]
2021-05-05T03:26:19.784 INFO:tasks.ceph.mds.d.smithi102.stderr: 7: (()+0x58dc37) [0x7fb8f6d19c37]
2021-05-05T03:26:19.784 INFO:tasks.ceph.mds.d.smithi102.stderr: 8: (()+0x82d53f) [0x7fb8f6fb953f]
2021-05-05T03:26:19.784 INFO:tasks.ceph.mds.d.smithi102.stderr: 9: (()+0x7ea5) [0x7fb8f489eea5]
2021-05-05T03:26:19.784 INFO:tasks.ceph.mds.d.smithi102.stderr: 10: (clone()+0x6d) [0x7fb8f354b9fd]
2021-05-05T03:26:19.785 INFO:tasks.ceph.mds.d.smithi102.stderr:*** Caught signal (Aborted) **
2021-05-05T03:26:19.785 INFO:tasks.ceph.mds.d.smithi102.stderr: in thread 7fb8eff59700 thread_name:msgr-worker-1
</pre>