Ceph » CephFS

Bug discovered on QA run for PR - https://github.com/ceph/ceph/pull/45556

Teuthology run: https://pulpito.ceph.com/vshankar-2022-04-26_06:23:29-fs:workload-wip-45556-20220418-102656-testing-default-smithi/. These 4 jobs failed with same error: 6806467, 6806495, 6806509, 6806523.

The failure can be reproduced easily. Mount a subvolume, get pjdfstests -

$ wget http://download.ceph.com/qa/pjd-fstest-20090130-RC-aclfixes.tgz
$ tar zxvf pjd*.tgz

And then run -

$ # /mnt/cephfs is the cephfs mountpoint
$ cd /mnt/cephfs && sudo cp -r ~/pjd-fstest-20090130-RC ./ && cd pjd-fstest-20090130-RC/ && sudo make clean && sudo make && cd ../ && sudo mkdir tmp && cd tmp

And then run the failing testsuites -

$ sudo prove -r -v --exec 'bash -x' ../pjd*/tests/link/02.t
$ sudo prove -r -v --exec 'bash -x' ../pjd*/tests/link/03.t

    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:../pjd-fstest-20090130-RC/tests/link/02.t ......
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:1..10
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:ok 1
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:ok 2
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:ok 3
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:not ok 4
    2022-04-26T06:59:11.148 INFO:tasks.workunit.client.0.smithi047.stdout:ok 5
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:not ok 6
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:ok 7
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:ok 8
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:ok 9
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:ok 10
    2022-04-26T06:59:11.149 INFO:tasks.workunit.client.0.smithi047.stdout:Failed 2/10 subtests

02.t test #4: expect 0 link ${n0} ${name255}
02.t test #6: expect 0 unlink ${name255}

    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:../pjd-fstest-20090130-RC/tests/link/03.t ......
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:1..16
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 1
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 2
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 3
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 4
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 5
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 6
    2022-04-26T06:59:11.346 INFO:tasks.workunit.client.0.smithi047.stdout:ok 7
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:not ok 8
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:not ok 9
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 10
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 11
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 12
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 13
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 14
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 15
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:ok 16
    2022-04-26T06:59:11.347 INFO:tasks.workunit.client.0.smithi047.stdout:Failed 2/16 subtests

03.t test #8: expect 0 link ${n0} ${path1023}
03.t test #9: expect 0 unlink ${path1023}

Final lines of output of pjd.sh -

    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:ok 40
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:ok 41
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:ok 42
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:ok
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:Test Summary Report
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:-------------------
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:../pjd-fstest-20090130-RC/tests/link/02.t    (Wstat: 0 Tests: 10 Failed: 2)
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:  Failed tests:  4, 6
    2022-04-26T06:59:51.015 INFO:tasks.workunit.client.0.smithi047.stdout:../pjd-fstest-20090130-RC/tests/link/03.t    (Wstat: 0 Tests: 16 Failed: 2)
    2022-04-26T06:59:51.016 INFO:tasks.workunit.client.0.smithi047.stdout:  Failed tests:  8-9
    2022-04-26T06:59:51.016 INFO:tasks.workunit.client.0.smithi047.stdout:Files=191, Tests=2286, 69 wallclock secs ( 1.30 usr  0.61 sys +  8.17 cusr  6.84 csys = 16.92 CPU)
    2022-04-26T06:59:51.016 INFO:tasks.workunit.client.0.smithi047.stdout:Result: FAIL
    2022-04-26T06:59:51.016 INFO:tasks.workunit:Stopping ['suites/pjd.sh'] on client.0...
    2022-04-26T06:59:51.016 DEBUG:teuthology.orchestra.run.smithi047:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0

Error -

    teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/pjd.sh) on smithi047 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1ccbc711b8876e630c0358e1d8d923daa34dca1e TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/pjd.sh'

Traceback from the job 6806467 -

    Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/run_tasks.py", line 103, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/run_tasks.py", line 82, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/workunit.py", line 148, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/workunit.py", line 298, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/parallel.py", line 84, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/parallel.py", line 98, in __next__
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/parallel.py", line 30, in resurrect_traceback
    raise exc.exc_info[1]
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/parallel.py", line 23, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_1ccbc711b8876e630c0358e1d8d923daa34dca1e/qa/tasks/workunit.py", line 427, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/orchestra/remote.py", line 509, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_788cfdd8098ad222aa448289edcfa4436091c32c/teuthology/orchestra/run.py", line 183, in _raise_for_status