Bug #52437: mds: InoTable::replay_release_ids abort via test_inotable_sync - CephFS - Ceph

Actions

Copy link

Bug #52437

open

mds: InoTable::replay_release_ids abort via test_inotable_sync

Added by Patrick Donnelly over 2 years ago. Updated almost 2 years ago.

Status:

New

Priority:

High

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

pacific,octopus

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS

Labels (FS):

qa, qa-failure

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

    -4> 2021-08-27T17:07:21.961+0000 7f1ab57b7700 10 mds.0.log _replay 4226387~56 / 4226463 0.000000: Eurged 500 inos, inotable v7
    -3> 2021-08-27T17:07:21.961+0000 7f1ab57b7700 10 mds.0.journal EPurged.replay inotable 3 < 7
    -2> 2021-08-27T17:07:21.961+0000 7f1ab57b7700 10 mds.0.inotable: replay_release_ids [0x10000000003~0x1f4]
    -1> 2021-08-27T17:07:21.962+0000 7f1ab57b7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7314-g571ce12c/rpm/el8/BUILD/ceph-17.0.0-7314-g571ce12c/src/include/interval_set.h: In function 'void interval_set<T, C>::insert(T, T, T*, T*) [with T = inodeno_t; C = std::map]' thread 7f1ab57b7700 time 2021-08-27T17:07:21.963074+0000
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7314-g571ce12c/rpm/el8/BUILD/ceph-17.0.0-7314-g571ce12c/src/include/interval_set.h: 477: ceph_abort_msg("abort() called")

 ceph version 17.0.0-7314-g571ce12c (571ce12cf5ad339da7d8efb36315a33417d719bb) quincy (dev)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x7f1ac57e6459]
 2: (InoTable::replay_release_ids(interval_set<inodeno_t, std::map>&)+0x871) [0x55dec323e791]
 3: (EPurged::replay(MDSRank*)+0x490) [0x55dec33195e0]
 4: (MDLog::_replay_thread()+0xcd1) [0x55dec32a4551]
 5: (MDLog::ReplayThread::entry()+0x11) [0x55dec2f95b11]
 6: /lib64/libpthread.so.0(+0x814a) [0x7f1ac47be14a]
 7: clone()

From: /ceph/teuthology-archive/pdonnell-2021-08-27_16:46:16-fs-wip-pdonnell-testing-20210827.024746-distro-basic-smithi/6362977/remote/smithi162/log/ceph-mds.d.log.gz

symptom:

2021-08-27T17:07:31.066 INFO:tasks.cephfs_test_runner:======================================================================
2021-08-27T17:07:31.066 INFO:tasks.cephfs_test_runner:ERROR: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub)
2021-08-27T17:07:31.066 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-08-27T17:07:31.067 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-08-27T17:07:31.067 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_571ce12cf5ad339da7d8efb36315a33417d719bb/qa/tasks/cephfs/test_forward_scrub.py", line 256, in test_inotable_sync
2021-08-27T17:07:31.067 INFO:tasks.cephfs_test_runner:    self.fs.wait_for_daemons()
2021-08-27T17:07:31.067 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_571ce12cf5ad339da7d8efb36315a33417d719bb/qa/tasks/cephfs/filesystem.py", line 1057, in wait_for_daemons
2021-08-27T17:07:31.068 INFO:tasks.cephfs_test_runner:    if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check):
2021-08-27T17:07:31.068 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_571ce12cf5ad339da7d8efb36315a33417d719bb/qa/tasks/cephfs/filesystem.py", line 909, in are_daemons_healthy
2021-08-27T17:07:31.068 INFO:tasks.cephfs_test_runner:    mds.check_status()
2021-08-27T17:07:31.069 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/daemon/state.py", line 43, in check_status
2021-08-27T17:07:31.069 INFO:tasks.cephfs_test_runner:    return self.proc.poll()
2021-08-27T17:07:31.069 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/run.py", line 211, in poll
2021-08-27T17:07:31.069 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-08-27T17:07:31.070 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_d0ccb5e7543966c9868cca0e1d0b1e1f5b5df280/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-08-27T17:07:31.070 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-08-27T17:07:31.070 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi162 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d'