Actions
Bug #56644
openqa: test_rapid_creation fails with "No space left on device"
% Done:
0%
Source:
Tags:
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6921226
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6931962
http://pulpito.front.sepia.ceph.com/rishabh-2022-07-19_18:30:20-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/
Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation)
2022-07-15T07:53:25.144 DEBUG:teuthology.orchestra.run.smithi159:> adjust-ulimits daemon-helper kill python3 -c ' 2022-07-15T07:53:25.145 DEBUG:teuthology.orchestra.run.smithi159:> import os 2022-07-15T07:53:25.146 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> n = 400 2022-07-15T07:53:25.147 DEBUG:teuthology.orchestra.run.smithi159:> path = "/home/ubuntu/cephtest/mnt.0/splitdir/file" 2022-07-15T07:53:25.148 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-15T07:53:25.150 DEBUG:teuthology.orchestra.run.smithi159:> dpath = os.path.dirname(path) 2022-07-15T07:53:25.151 DEBUG:teuthology.orchestra.run.smithi159:> fnameprefix = os.path.basename(path) 2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:> os.makedirs(dpath, exist_ok=True) 2022-07-15T07:53:25.152 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:> try: 2022-07-15T07:53:25.155 DEBUG:teuthology.orchestra.run.smithi159:> dirfd = os.open(dpath, os.O_DIRECTORY) 2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:> 2022-07-15T07:53:25.158 DEBUG:teuthology.orchestra.run.smithi159:> for i in range(n): 2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:> fpath = os.path.join(dpath, f"{fnameprefix}_{i}") 2022-07-15T07:53:25.159 DEBUG:teuthology.orchestra.run.smithi159:> with open(fpath, '"'"'w'"'"') as f: 2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:> f.write(f"{i}") 2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:> if False: 2022-07-15T07:53:25.160 DEBUG:teuthology.orchestra.run.smithi159:> f.flush() 2022-07-15T07:53:25.161 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(f.fileno()) 2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:> if False: 2022-07-15T07:53:25.162 DEBUG:teuthology.orchestra.run.smithi159:> os.unlink(fpath) 2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:> if False: 2022-07-15T07:53:25.163 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(dirfd) 2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:> if False: 2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:> os.fsync(dirfd) 2022-07-15T07:53:25.164 DEBUG:teuthology.orchestra.run.smithi159:> finally: 2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:> os.close(dirfd) 2022-07-15T07:53:25.165 DEBUG:teuthology.orchestra.run.smithi159:> ' 2022-07-15T07:53:25.478 INFO:teuthology.orchestra.run.smithi159.stderr:Traceback (most recent call last): 2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr: File "<string>", line 16, in <module> 2022-07-15T07:53:25.479 INFO:teuthology.orchestra.run.smithi159.stderr:OSError: [Errno 28] No space left on device: '/home/ubuntu/cephtest/mnt.0/splitdir/file_376' 2022-07-15T07:53:25.667 INFO:teuthology.orchestra.run.smithi159.stderr:daemon-helper: command failed with exit status 1 2022-07-15T07:53:25.690 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-07-15T07:53:36.666 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR 2022-07-15T07:53:36.667 INFO:tasks.cephfs_test_runner: 2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-15T07:53:36.668 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) 2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-15T07:53:36.669 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation 2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner: self.mount_a.create_n_files("splitdir/file", split_size * 4) 2022-07-15T07:53:36.670 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 1173, in create_n_files 2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner: self.run_python(pyscript) 2022-07-15T07:53:36.671 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 706, in run_python 2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner: p.wait() 2022-07-15T07:53:36.672 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 161, in wait 2022-07-15T07:53:36.673 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2022-07-15T07:53:36.675 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 183, in _raise_for_status 2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label 2022-07-15T07:53:36.676 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi159 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n dirfd = os.open(dpath, os.O_DIRECTORY)\n\n for i in range(n):\n fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n with open(fpath, \'"\'"\'w\'"\'"\') as f:\n f.write(f"{i}")\n if False:\n f.flush()\n os.fsync(f.fileno())\n if False:\n os.unlink(fpath)\n if False:\n os.fsync(dirfd)\n if False:\n os.fsync(dirfd)\nfinally:\n os.close(dirfd)\n\'' 2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner: 2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-15T07:53:36.677 INFO:tasks.cephfs_test_runner:Ran 3 tests in 218.476s
Updated by Rishabh Dave over 1 year ago
- Category set to Testing
- Target version set to v18.0.0
- Backport set to quincy,pacific
From https://tracker.ceph.com/issues/56507 -¶
2022-07-06T14:31:21.553 INFO:tasks.cephfs_test_runner:test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) ... ERROR 2022-07-06T14:31:21.555 INFO:tasks.cephfs_test_runner: 2022-07-06T14:31:21.556 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:ERROR: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) 2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-06T14:31:21.559 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/test_fragment.py", line 119, in test_rapid_creation 2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner: self.mount_a.create_n_files("splitdir/file", split_size * 4) 2022-07-06T14:31:21.560 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 1074, in create_n_files 2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner: self.run_python(pyscript) 2022-07-06T14:31:21.561 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_2103eaf02dd3c9da5290a33a5b7ec7f42138d76a/qa/tasks/cephfs/mount.py", line 658, in run_python 2022-07-06T14:31:21.562 INFO:tasks.cephfs_test_runner: p.wait() 2022-07-06T14:31:21.563 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 161, in wait 2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2022-07-06T14:31:21.564 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_f58709e3fda190343157063e5edb0d7a36a41afc/teuthology/orchestra/run.py", line 183, in _raise_for_status 2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label 2022-07-06T14:31:21.565 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi123 with status 1: 'adjust-ulimits daemon-helper kill python3 -c \'\nimport os\n\nn = 400\npath = "/home/ubuntu/cephtest/mnt.0/splitdir/file"\n\ndpath = os.path.dirname(path)\nfnameprefix = os.path.basename(path)\nos.makedirs(dpath, exist_ok=True)\n\ntry:\n dirfd = os.open(dpath, os.O_DIRECTORY)\n\n for i in range(n):\n fpath = os.path.join(dpath, f"{fnameprefix}_{i}")\n with open(fpath, \'"\'"\'w\'"\'"\') as f:\n f.write(f"{i}")\n if False:\n f.flush()\n os.fsync(f.fileno())\n if False:\n os.unlink(fpath)\n if False:\n os.fsync(dirfd)\n if False:\n os.fsync(dirfd)\nfinally:\n os.close(dirfd)\n\''
Failure while creating files. Could be a permission/sudo thing (no mds crashes or errors).
Updated by Rishabh Dave over 1 year ago
- Related to Bug #56507: pacific: Test failure: test_rapid_creation (tasks.cephfs.test_fragment.TestFragmentation) added
Updated by Venky Shankar over 1 year ago
Rishabh,
Do we know why the space issue started to show up recently?
Updated by Kotresh Hiremath Ravishankar over 1 year ago
Updated by Venky Shankar over 1 year ago
- Status changed from New to Triaged
- Assignee set to Rishabh Dave
Updated by Venky Shankar over 1 year ago
Another instance - https://pulpito.ceph.com/yuriw-2022-12-07_15:45:30-fs-wip-yuri4-testing-2022-12-05-0921-pacific-distro-default-smithi/7106317/
Rishabh, did we RCA this?
Actions