Bug #58220: Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1: - CephFS - Ceph

Actions

Copy link

Bug #58220

open

Command failed (workunit test fs/quota/quota.sh) on smithi081 with status 1:

Added by Venky Shankar over 1 year ago. Updated 7 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Rishabh Dave

Category:

Correctness/Safety

Target version:

Ceph - v18.0.0

% Done:

Source:

Tags:

backport_processed

Backport:

reef,quincy,pacific

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

MDS, ceph-fuse

Labels (FS):

quotas

Pull request ID:

50127

Crash signature (v1):

Crash signature (v2):

Description

/a/vshankar-2022-12-08_04:33:46-fs-wip-vshankar-testing-20221130.043104-testing-default-smithi/7107722

Description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/workunit/quota}

Seems like a failure in quota enforcement:

2022-12-08T05:39:44.399 INFO:tasks.workunit.client.3.smithi138.stderr:+ set -x
2022-12-08T05:39:44.399 INFO:tasks.workunit.client.3.smithi138.stderr:+ mv files limit/
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ expect_false mv files limit/
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ set -x
2022-12-08T05:39:44.400 INFO:tasks.workunit.client.1.smithi138.stderr:+ mv files limit/
2022-12-08T05:39:44.412 INFO:tasks.workunit.client.1.smithi138.stderr:+ return 1
2022-12-08T05:39:44.414 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-12-08T05:39:44.415 INFO:tasks.workunit:Stopping ['fs/quota'] on client.1...
2022-12-08T05:39:44.415 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.1 /home/ubuntu/cephtest/clone.client.1
2022-12-08T05:39:44.416 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-12-08T05:39:44.417 INFO:tasks.workunit.client.3.smithi138.stderr:+ return 1
2022-12-08T05:39:44.418 INFO:tasks.workunit:Stopping ['fs/quota'] on client.3...
2022-12-08T05:39:44.418 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.3 /home/ubuntu/cephtest/clone.client.3

Related issues 3 (1 open — 2 closed)

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Status changed from New to Duplicate

Duplicate of https://tracker.ceph.com/issues/56695

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Status changed from Duplicate to New

Sorry - didn't mean to mark this tracker as a duplicate.

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Assignee set to Rishabh Dave

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Status changed from New to Triaged

Actions

Copy link

Updated by Venky Shankar about 1 year ago

Severity changed from 3 - minor to 2 - major

Latest instance - https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/7138138/

Rishabh, please do an RCA.

Actions

Copy link

Updated by Rishabh Dave about 1 year ago

Status changed from Triaged to In Progress

Actions

Copy link

Updated by Rishabh Dave about 1 year ago

Status changed from In Progress to Fix Under Review

Actions

Copy link

Updated by Rishabh Dave about 1 year ago

Pull request ID set to 50127

Actions

Copy link

Updated by Rishabh Dave about 1 year ago

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

    2023-03-04T09:56:26.878 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_disk_quota_exceeeded_error ... FAIL
    2023-03-04T09:56:26.891 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_empty_snapshot_info ... ERROR
    2023-03-04T09:56:26.896 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_snapshot_info ... ERROR
    2023-03-04T09:56:26.899 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_post_mount ... ERROR
    2023-03-04T09:56:26.902 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout ... ERROR
    2023-03-04T09:56:26.971 INFO:tasks.workunit.client.0.smithi114.stderr:test_cephfs.test_set_mount_timeout_lt0 ... ERROR
    2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:

Traceback for test_disk_quota_exceeeded_error -

    2023-03-04T09:56:26.983 INFO:tasks.workunit.client.0.smithi114.stderr:======================================================================
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:FAIL: test_cephfs.test_disk_quota_exceeeded_error
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
    2023-03-04T09:56:26.984 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:    self.test(*self.arg)
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 859, in test_disk_quota_exceeeded_error
    2023-03-04T09:56:26.985 INFO:tasks.workunit.client.0.smithi114.stderr:    assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz1234567890qwertyuioddd", -1)

Every test that failed after test_disk_quota_exceeeded_error has same traceback -

2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:ERROR: test_cephfs.test_empty_snapshot_info
2023-03-04T09:56:26.972 INFO:tasks.workunit.client.0.smithi114.stderr:----------------------------------------------------------------------
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:Traceback (most recent call last):
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/case.py", line 267, in setUp
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:    try_run(self.test, names)
2023-03-04T09:56:26.973 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/usr/lib/python3/dist-packages/nose/util.py", line 471, in try_run
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:    return func()
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:  File "/home/ubuntu/cephtest/clone.client.0/src/test/pybind/test_cephfs.py", line 29, in setup_test
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:    cephfs.rmdir(b"/" + dent.d_name)
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:  File "cephfs.pyx", line 1276, in cephfs.LibCephFS.rmdir
2023-03-04T09:56:26.974 INFO:tasks.workunit.client.0.smithi114.stderr:cephfs.ObjectNotEmpty: error in rmdir /dir-1: Directory not empty [Errno 39]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -

    @with_setup(setup_test)
    def test_disk_quota_exceeeded_error():
        cephfs.mkdir("/dir-1", 0o755)
        cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
        fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
        assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)
        cephfs.close(fd)
        cephfs.unlink(b"/dir-1/file-1")

Specifically following code from above test is quota related code where failure occurs -

        cephfs.setxattr("/dir-1", "ceph.quota.max_bytes", b"5", 0)
        fd = cephfs.open(b'/dir-1/file-1', 'w', 0o755)
        assert_raises(libcephfs.DiskQuotaExceeded, cephfs.write, fd, b"abcdeghiklmnopqrstuvwxyz", 0)

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

In short the cause for this bug is same as that of this tracker ticket.

Actions

Copy link

#10

Updated by Venky Shankar about 1 year ago

Rishabh Dave wrote:

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

[...]

Traceback for test_disk_quota_exceeeded_error -
[...]

Every test that failed after test_disk_quota_exceeeded_error has same traceback -
[...]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -
[...]

Specifically following code from above test is quota related code where failure occurs -
[...]

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

Rishabh, this is not related to the quota.sh workunit failure, isn't it?

Actions

Copy link

#11

Updated by Venky Shankar about 1 year ago

Status changed from Fix Under Review to Pending Backport

Actions

Copy link

#12

Updated by Rishabh Dave about 1 year ago

Venky Shankar wrote:

Rishabh Dave wrote:

test_disk_quota_exceeeded_error from src/test/pybind/test_cephfs.py fails on this teuthology run - http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7193845

[...]

Traceback for test_disk_quota_exceeeded_error -
[...]

Every test that failed after test_disk_quota_exceeeded_error has same traceback -
[...]

setUp() hits error on running rmdir /dir-1. Normally this doesn't occur which means this dir-1 is normally empty. This implies unclean cleanup/teardown in previous test which is test_disk_quota_exceeeded_error.

Code of test_disk_quota_exceeeded_error -
[...]

Specifically following code from above test is quota related code where failure occurs -
[...]

And the last line of this test method (which is cephfs.unlink(b"/dir-1/file-1") is not executed. Thus, dir-1 is not empty and setUp() fails on attempting to run rmdir /dir-1.

Rishabh, this is not related to the quota.sh workunit failure, isn't it?