Bug #56644: qa: test_rapid_creation fails with "No space left on device" - CephFS - Ceph

Actions

Copy link

Bug #56644

open

qa: test_rapid_creation fails with "No space left on device"

Added by Rishabh Dave almost 2 years ago. Updated 7 months ago.

Status:

Triaged

Priority:

Normal

Assignee:

Rishabh Dave

Category:

Testing

Target version:

% Done:

Source:

Tags:

Backport:

quincy,pacific

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

qa-suite

Labels (FS):

qa, qa-failure

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6921226
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6931962
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-19_18:30:20-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/

Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)

2022-07-15T07:53:25.144 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits daemon-helper kill python3 -c '
2022-07-15T07:53:25.145 DEBUG:teuthology.orchestra.run.smithi159:> import os
2022-07-15T07:53:25.146 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> n = 400
2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> path = "/home/ubuntu/cephtest/mnt.0/splitdir/file" 
2022-07-15T07:53:25.148 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.150 DEBUG:teuthology.orchestra.run.smithi159:> dpath = os.path.dirname(path)
2022-07-15T07:53:25.151 DEBUG:teuthology.orchestra.run.smithi159:> fnameprefix = os.path.basename(path)
2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(dpath, exist_ok=True)
2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:> try:
2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:>     dirfd = os.open(dpath, os.O_DIRECTORY)
2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:>     for i in range(n):
2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:>         fpath = os.path.join(dpath, f"{fnameprefix}_{i}")
2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:>         with open(fpath, '"'"'w'"'"') as f:
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>             f.write(f"{i}")
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>             if False:
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>                 f.flush()
2022-07-15T07:53:25.161 DEBUG:teuthology.orchestra.run.smithi159:>                 os.fsync(f.fileno())
2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:>         if False:
2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:>             os.unlink(fpath)
2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:>         if False:
2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:>             os.fsync(dirfd)
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:>     if False:
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:>         os.fsync(dirfd)
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:> finally:
2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:>     os.close(dirfd)
2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:> '
2022-07-15T07:53:25.478 INFO:teuthology.orchestra.run.smithi159.stderr:Traceback (most recent call last):
2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:  File "<string>", line 16, in <module>
2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/splitdir/file_376'
2022-07-15T07:53:25.667 INFO:teuthology.orchestra.run.smithi159.stderr:daemon-helper: command failed with exit status 1
2022-07-15T07:53:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1

2022-07-15T07:53:36.666 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR
2022-07-15T07:53:36.667 INFO:tasks.cephfs_test_runner:
2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:======================================================================
2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:    self.mount_a.create_n_files("splitdir/file", split_size * 4)
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 1173, in create_n_files
2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner:    self.run_python(pyscript)
2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 706, in run_python
2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner:    p.wait()
2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 161, in wait
2022-07-15T07:53:36.673 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-07-15T07:53:36.675 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n    dirfd = os.open(dpath, os.O_DIRECTORY)\n\n    for i in range(n):\n        fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n        with open(fpath, \'"\'"\'w\'"\'"\') as f:\n            f.write(f"{i}")\n            if False:\n                f.flush()\n                os.fsync(f.fileno())\n        if False:\n            os.unlink(fpath)\n        if False:\n            os.fsync(dirfd)\n    if False:\n        os.fsync(dirfd)\nfinally:\n    os.close(dirfd)\n\''
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:Ran 3 tests in 218.476s

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Rishabh Dave almost 2 years ago

Category set to Testing
Target version set to v18.0.0
Backport set to quincy,pacific

From https://tracker.ceph.com/issues/56507 -¶

https://pulpito.ceph.com/yuriw-2022-07-06_13:57:53-fs-wip-yuri4-testing-2022-07-05-0719-pacific-distro-default-smithi/6917405/

2022-07-06T14:31:21.553 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR
2022-07-06T14:31:21.555 INFO:tasks.cephfs_test_runner:
2022-07-06T14:31:21.556 INFO:tasks.cephfs_test_runner:======================================================================
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:    self.mount_a.create_n_files("splitdir/file", split_size * 4)
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 1074, in create_n_files
2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner:    self.run_python(pyscript)
2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 658, in run_python
2022-07-06T14:31:21.562 INFO:tasks.cephfs_test_runner:    p.wait()
2022-07-06T14:31:21.563 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath
= os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n    dirfd = os.open(dpath, os.O_DIRECTORY)\n\n    for i in range(n):\n        fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n        with open(fpath, \'"\'"\'w\'"\'"\') as f:\n            f.write(f"{i}")\n            if False:\n                f.flush()\n                os.fsync(f.fileno())\n        if False:\n            os.unlink(fpath)\n        if False:\n            os.fsync(dirfd)\n    if False:\n        os.fsync(dirfd)\nfinally:\n    os.close(dirfd)\n\''

Failure while creating files. Could be a permission/sudo thing (no mds crashes or errors).

Actions

Copy link