Bug #50696
Updated by Ramana Raja about 3 years ago
See, https://pulpito.ceph.com/yuriw-2021-05-04_15:32:03-multimds-wip-yuri3-testing-2021-04-29-1036-nautilus-distro-basic-smithi/6094471/
Description: multimds/thrash/{0-supported-random-distro$/{centos_latest} begin ceph-thrash/default clusters/3-mds-2-standby conf/{client mds mon osd} mount/kclient/{mount overrides/{distro/rhel/{k-distro rhel_latest} ms-die-on-skipped}} msgr-failures/osd-mds-delay objectstore-ec/bluestore-comp overrides/{fuse-default-perm-no thrash/{frag_enable session_timeout whitelist_health whitelist_wrongly_marked_down} thrash_debug} tasks/cfuse_workunit_suites_fsstress}
<pre>
teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsstress.sh) on smithi160 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=36801f537d3dceb7c135151b37ba843b7c595bbe TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh'
2021-05-05T03:31:23.774 DEBUG:teuthology.run_tasks:Unwinding manager kclient
2021-05-05T03:31:23.787 INFO:tasks.kclient:Unmounting kernel clients...
2021-05-05T03:31:23.790 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2021-05-05T03:31:23.791 INFO:teuthology.orchestra.run:Running command with timeout 900
2021-05-05T03:31:23.793 DEBUG:teuthology.orchestra.run.smithi160:> sudo umount /home/ubuntu/cephtest/mnt.0
</pre>
See this ceph_assert earlier in the log, /src/msg/async/ProtocolV1.cc: 967: FAILED ceph_assert(0 == "old msgs despite reconnect_seq feature")
<pre>
2021-05-05T03:27:39.806 INFO:tasks.ceph.mds.c.smithi094.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.20-192-g36801f53/rpm/el7/BUILD/ceph-14.2.20-192-g36801f53/src/msg/async/ProtocolV1.cc: 967: FAILED ceph_assert(0 == "old msgs despite reconnect_seq feature")
2021-05-05T03:27:39.807 INFO:tasks.ceph.mds.c.smithi094.stderr:
2021-05-05T03:27:39.807 INFO:tasks.ceph.mds.c.smithi094.stderr: ceph version 14.2.20-192-g36801f537d3 (36801f537d3dceb7c135151b37ba843b7c595bbe) nautilus (stable)
2021-05-05T03:27:39.807 INFO:tasks.ceph.mds.c.smithi094.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f8896fd6467]
2021-05-05T03:27:39.807 INFO:tasks.ceph.mds.c.smithi094.stderr: 2: (()+0x25d62f) [0x7f8896fd662f]
2021-05-05T03:27:39.808 INFO:tasks.ceph.mds.c.smithi094.stderr: 3: (ProtocolV1::handle_message_footer(char*, int)+0xf3d) [0x7f88972c685d]
2021-05-05T03:27:39.808 INFO:tasks.ceph.mds.c.smithi094.stderr: 4: (()+0x54605d) [0x7f88972bf05d]
2021-05-05T03:27:39.808 INFO:tasks.ceph.mds.c.smithi094.stderr: 5: (AsyncConnection::process()+0x186) [0x7f88972b0c66]
2021-05-05T03:27:39.808 INFO:tasks.ceph.mds.c.smithi094.stderr: 6: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7f88973002d5]
2021-05-05T03:27:39.809 INFO:tasks.ceph.mds.c.smithi094.stderr: 7: (()+0x58dc37) [0x7f8897306c37]
2021-05-05T03:27:39.809 INFO:tasks.ceph.mds.c.smithi094.stderr: 8: (()+0x82d53f) [0x7f88975a653f]
2021-05-05T03:27:39.809 INFO:tasks.ceph.mds.c.smithi094.stderr: 9: (()+0x7ea5) [0x7f8894e8bea5]
2021-05-05T03:27:39.809 INFO:tasks.ceph.mds.c.smithi094.stderr: 10: (clone()+0x6d) [0x7f8893b389fd]
2021-05-05T03:27:39.809 INFO:tasks.ceph.mds.c.smithi094.stderr:
2021-05-05T03:27:39.810 INFO:tasks.ceph.mds.c.smithi094.stderr: 0> 2021-05-05 03:27:39.787 7f8890546700 -1 *** Caught signal (Aborted) **
2021-05-05T03:27:39.810 INFO:tasks.ceph.mds.c.smithi094.stderr: in thread 7f8890546700 thread_name:msgr-worker-1
2021-05-05T03:27:39.810 INFO:tasks.ceph.mds.c.smithi094.stderr:
2021-05-05T03:27:39.810 INFO:tasks.ceph.mds.c.smithi094.stderr: ceph version 14.2.20-192-g36801f537d3 (36801f537d3dceb7c135151b37ba843b7c595bbe) nautilus (stable)
2021-05-05T03:27:39.811 INFO:tasks.ceph.mds.c.smithi094.stderr: 1: (()+0xf630) [0x7f8894e93630]
2021-05-05T03:27:39.811 INFO:tasks.ceph.mds.c.smithi094.stderr: 2: (gsignal()+0x37) [0x7f8893a703d7]
2021-05-05T03:27:39.811 INFO:tasks.ceph.mds.c.smithi094.stderr: 3: (abort()+0x148) [0x7f8893a71ac8]
2021-05-05T03:27:39.811 INFO:tasks.ceph.mds.c.smithi094.stderr: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f8896fd64b6]
2021-05-05T03:27:39.811 INFO:tasks.ceph.mds.c.smithi094.stderr: 5: (()+0x25d62f) [0x7f8896fd662f]
2021-05-05T03:27:39.812 INFO:tasks.ceph.mds.c.smithi094.stderr: 6: (ProtocolV1::handle_message_footer(char*, int)+0xf3d) [0x7f88972c685d]
2021-05-05T03:27:39.812 INFO:tasks.ceph.mds.c.smithi094.stderr: 7: (()+0x54605d) [0x7f88972bf05d]
2021-05-05T03:27:39.812 INFO:tasks.ceph.mds.c.smithi094.stderr: 8: (AsyncConnection::process()+0x186) [0x7f88972b0c66]
2021-05-05T03:27:39.812 INFO:tasks.ceph.mds.c.smithi094.stderr: 9: (EventCenter::process_events(unsigned int, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> >*)+0xa15) [0x7f88973002d5]
2021-05-05T03:27:39.813 INFO:tasks.ceph.mds.c.smithi094.stderr: 10: (()+0x58dc37) [0x7f8897306c37]
2021-05-05T03:27:39.813 INFO:tasks.ceph.mds.c.smithi094.stderr: 11: (()+0x82d53f) [0x7f88975a653f]
2021-05-05T03:27:39.813 INFO:tasks.ceph.mds.c.smithi094.stderr: 12: (()+0x7ea5) [0x7f8894e8bea5]
2021-05-05T03:27:39.813 INFO:tasks.ceph.mds.c.smithi094.stderr: 13: (clone()+0x6d) [0x7f8893b389fd]
2021-05-05T03:27:39.813 INFO:tasks.ceph.mds.c.smithi094.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2021-05-05T03:27:39.814 INFO:tasks.ceph.mds.c.smithi094.stderr:
2021-05-05T03:27:40.100 INFO:tasks.ceph.mds.c.smithi094.stderr:daemon-helper: command crashed with signal 6
</pre>