Project

General

Profile

Bug #58220

Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:

Added by Venky Shankar about 1 year ago. Updated 5 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Tags:
backport_processed
Backport:
reef,quincy,pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS, ceph-fuse
Labels (FS):
quotas
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/vshankar-2022-12-08_04:33:46-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7107722

Description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}

Seems like a failure in quota enforcement:

2022-12-08T05:39:44.399 INFO:tasks.workunit.client.3.smithi138.stderr:+ set -x
2022-12-08T05:39:44.399 INFO:tasks.workunit.client.3.smithi138.stderr:+ mv files limit/
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ expect_false mv files limit/
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ set -x
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ mv files limit/
2022-12-08T05:39:44.412 INFO:tasks.workunit.client.1.smithi138.stderr:+ return 1
2022-12-08T05:39:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-12-08T05:39:44.415 INFO:tasks.workunit:Stopping ['fs/quota'] on client.1...
2022-12-08T05:39:44.415 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1
2022-12-08T05:39:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-12-08T05:39:44.417 INFO:tasks.workunit.client.3.smithi138.stderr:+ return 1
2022-12-08T05:39:44.418 INFO:tasks.workunit:Stopping ['fs/quota'] on client.3...
2022-12-08T05:39:44.418 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.3 /home/ubuntu/cephtest/clone.client.3

Related issues

Copied to CephFS - Backport #59014: quincy: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: Rejected
Copied to CephFS - Backport #59015: pacific: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: Rejected
Copied to CephFS - Backport #59410: reef: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: In Progress

History

#1 Updated by Venky Shankar about 1 year ago

  • Status changed from New to Duplicate

#2 Updated by Venky Shankar about 1 year ago

  • Status changed from Duplicate to New

Sorry - didn't mean to mark this tracker as a duplicate.

#3 Updated by Venky Shankar about 1 year ago

  • Assignee set to Rishabh Dave

#4 Updated by Venky Shankar about 1 year ago

  • Status changed from New to Triaged

#5 Updated by Venky Shankar about 1 year ago

  • Severity changed from 3 - minor to 2 - major

#6 Updated by Rishabh Dave about 1 year ago

  • Status changed from Triaged to In Progress

#7 Updated by Rishabh Dave about 1 year ago

  • Status changed from In Progress to Fix Under Review

#8 Updated by Rishabh Dave about 1 year ago

  • Pull request ID set to 50127

#9 Updated by Rishabh Dave 12 months ago

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

    2023-03-04T09:56:26.878 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_disk_quota_exceeeded_error ... FAIL
    2023-03-04T09:56:26.891 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_empty_snapshot_info ... ERROR
    2023-03-04T09:56:26.896 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_snapshot_info ... ERROR
    2023-03-04T09:56:26.899 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_post_mount ... ERROR
    2023-03-04T09:56:26.902 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout ... ERROR
    2023-03-04T09:56:26.971 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_lt0 ... ERROR
    2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:

Traceback for test_disk_quota_exceeeded_error -

    2023-03-04T09:56:26.983 INFO:tasks.workunit.client.0.smithi114.stderr:======================================================================
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:FAIL: test_cephfs.test_disk_quota_exceeeded_error
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:    self.test(*self.arg)
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 859, in test_disk_quota_exceeeded_error
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:    assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz1234567890qwertyuioddd", -1)

Every test that failed after test_disk_quota_exceeeded_error has same traceback -

2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:ERROR: test_cephfs.test_empty_snapshot_info
2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 267, in setUp
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:    try_run(self.test, names)
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:    return func()
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 29, in setup_test
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:    cephfs.rmdir(b"/" + dent.d_name)
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:  File "cephfs.pyx", line 1276, in cephfs.LibCephFS.rmdir
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:cephfs.ObjectNotEmpty: error in rmdir /dir-1: Directory not empty [Errno 39]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -

    @with_setup(setup_test)
    def test_disk_quota_exceeeded_error():
        cephfs.mkdir("/dir-1", 0o755)
        cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
        fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
        assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)
        cephfs.close(fd)
        cephfs.unlink(b"/dir-1/file-1")

Specifically following code from above test is quota related code where failure occurs -

        cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
        fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
        assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

In short the cause for this bug is same as that of this tracker ticket.

#10 Updated by Venky Shankar 12 months ago

Rishabh Dave wrote:

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

[...]

Traceback for test_disk_quota_exceeeded_error -
[...]

Every test that failed after test_disk_quota_exceeeded_error has same traceback -
[...]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -
[...]

Specifically following code from above test is quota related code where failure occurs -
[...]

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

Rishabh, this is not related to the quota.sh workunit failure, isn't it?

#11 Updated by Venky Shankar 12 months ago

  • Status changed from Fix Under Review to Pending Backport

#12 Updated by Rishabh Dave 12 months ago

Venky Shankar wrote:

Rishabh Dave wrote:

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

[...]

Traceback for test_disk_quota_exceeeded_error -
[...]

Every test that failed after test_disk_quota_exceeeded_error has same traceback -
[...]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -
[...]

Specifically following code from above test is quota related code where failure occurs -
[...]

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

Rishabh, this is not related to the quota.sh workunit failure, isn't it?

It has the same root cause. So this test failure too should be fixed with the patch for this ticket.

#13 Updated by Backport Bot 12 months ago

  • Copied to Backport #59014: quincy: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: added

#14 Updated by Backport Bot 12 months ago

  • Copied to Backport #59015: pacific: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: added

#15 Updated by Backport Bot 12 months ago

  • Tags set to backport_processed

#16 Updated by Venky Shankar 11 months ago

  • Tags deleted (backport_processed)
  • Backport changed from pacific,quincy to reef,quincy,pacific

#17 Updated by Backport Bot 11 months ago

  • Copied to Backport #59410: reef: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: added

#18 Updated by Backport Bot 11 months ago

  • Tags set to backport_processed

Also available in: Atom PDF