Project

General

Profile

Bug #56644

qa: test_rapid_creation fails with "No space left on device"

Added by Rishabh Dave 4 months ago. Updated 3 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6921226
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6931962
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-19_18:30:20-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/

Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-15T07:53:25.144 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits daemon-helper kill python3 -c '
2022-07-15T07:53:25.145 DEBUG:teuthology.orchestra.run.smithi159:> import os
2022-07-15T07:53:25.146 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> n = 400
2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> path = "/home/ubuntu/cephtest/mnt.0/splitdir/file" 
2022-07-15T07:53:25.148 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.150 DEBUG:teuthology.orchestra.run.smithi159:> dpath = os.path.dirname(path)
2022-07-15T07:53:25.151 DEBUG:teuthology.orchestra.run.smithi159:> fnameprefix = os.path.basename(path)
2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(dpath, exist_ok=True)
2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:> try:
2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:>     dirfd = os.open(dpath, os.O_DIRECTORY)
2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:>
2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:>     for i in range(n):
2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:>         fpath = os.path.join(dpath, f"{fnameprefix}_{i}")
2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:>         with open(fpath, '"'"'w'"'"') as f:
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>             f.write(f"{i}")
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>             if False:
2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:>                 f.flush()
2022-07-15T07:53:25.161 DEBUG:teuthology.orchestra.run.smithi159:>                 os.fsync(f.fileno())
2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:>         if False:
2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:>             os.unlink(fpath)
2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:>         if False:
2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:>             os.fsync(dirfd)
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:>     if False:
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:>         os.fsync(dirfd)
2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:> finally:
2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:>     os.close(dirfd)
2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:> '
2022-07-15T07:53:25.478 INFO:teuthology.orchestra.run.smithi159.stderr:Traceback (most recent call last):
2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:  File "<string>", line 16, in <module>
2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/splitdir/file_376'
2022-07-15T07:53:25.667 INFO:teuthology.orchestra.run.smithi159.stderr:daemon-helper: command failed with exit status 1
2022-07-15T07:53:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-07-15T07:53:36.666 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR
2022-07-15T07:53:36.667 INFO:tasks.cephfs_test_runner:
2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:======================================================================
2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:    self.mount_a.create_n_files("splitdir/file", split_size * 4)
2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 1173, in create_n_files
2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner:    self.run_python(pyscript)
2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 706, in run_python
2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner:    p.wait()
2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 161, in wait
2022-07-15T07:53:36.673 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-07-15T07:53:36.675 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n    dirfd = os.open(dpath, os.O_DIRECTORY)\n\n    for i in range(n):\n        fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n        with open(fpath, \'"\'"\'w\'"\'"\') as f:\n            f.write(f"{i}")\n            if False:\n                f.flush()\n                os.fsync(f.fileno())\n        if False:\n            os.unlink(fpath)\n        if False:\n            os.fsync(dirfd)\n    if False:\n        os.fsync(dirfd)\nfinally:\n    os.close(dirfd)\n\''
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:Ran 3 tests in 218.476s

Related issues

Related to CephFS - Bug #56507: pacific: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) Duplicate

History

#1 Updated by Rishabh Dave 4 months ago

  • Category set to Testing
  • Target version set to v18.0.0
  • Backport set to quincy,pacific

From https://tracker.ceph.com/issues/56507 -

https://pulpito.ceph.com/yuriw-2022-07-06_13:57:53-fs-wip-yuri4-testing-2022-07-05-0719-pacific-distro-default-smithi/6917405/

2022-07-06T14:31:21.553 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR
2022-07-06T14:31:21.555 INFO:tasks.cephfs_test_runner:
2022-07-06T14:31:21.556 INFO:tasks.cephfs_test_runner:======================================================================
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:    self.mount_a.create_n_files("splitdir/file", split_size * 4)
2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 1074, in create_n_files
2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner:    self.run_python(pyscript)
2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 658, in run_python
2022-07-06T14:31:21.562 INFO:tasks.cephfs_test_runner:    p.wait()
2022-07-06T14:31:21.563 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait
2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status
2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath
= os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n    dirfd = os.open(dpath, os.O_DIRECTORY)\n\n    for i in range(n):\n        fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n        with open(fpath, \'"\'"\'w\'"\'"\') as f:\n            f.write(f"{i}")\n            if False:\n                f.flush()\n                os.fsync(f.fileno())\n        if False:\n            os.unlink(fpath)\n        if False:\n            os.fsync(dirfd)\n    if False:\n        os.fsync(dirfd)\nfinally:\n    os.close(dirfd)\n\''

Failure while creating files. Could be a permission/sudo thing (no mds crashes or errors).

#2 Updated by Rishabh Dave 4 months ago

  • Related to Bug #56507: pacific: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) added

#3 Updated by Venky Shankar 4 months ago

Rishabh,

Do we know why the space issue started to show up recently?

#7 Updated by Venky Shankar 3 months ago

  • Status changed from New to Triaged
  • Assignee set to Rishabh Dave

Also available in: Atom PDF