Bug #56632
Updated by Rishabh Dave almost 2 years ago
100% reproducible so far. http://pulpito.front.sepia.ceph.com/rishabh-2022-07-08_23:53:34-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6921138 http://pulpito.front.sepia.ceph.com/rishabh-2022-07-15_06:42:04-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6931951 http://pulpito.front.sepia.ceph.com/rishabh-2022-07-19_13:40:01-fs-wip-rishabh-testing-2022Jul08-1820-testing-default-smithi/6938375 Failure message on job: <pre> Test @Test failure: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) </pre> (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones)@ <pre> 2022-07-19T14:53:23.439 DEBUG:teuthology.orchestra.run.smithi063:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000001034444/dfc97c72-1fb6-4715-af16-cd100be4dfb3/subvolume_file.42 bs=1M conv=fdatasync count=1 seek=0) 2022-07-19T14:53:23.521 INFO:teuthology.orchestra.run.smithi063.stderr:dd: error writing 'volumes/_nogroup/subvolume_0000000001034444/dfc97c72-1fb6-4715-af16-cd100be4dfb3/subvolume_file.42': Disk quota exceeded 2022-07-19T14:53:23.522 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2022-07-19T14:53:23.522 INFO:teuthology.orchestra.run.smithi063.stderr:0+0 records out 2022-07-19T14:53:23.523 INFO:teuthology.orchestra.run.smithi063.stderr:917504 bytes (918 kB, 896 KiB) copied, 0.0132426 s, 69.3 MB/s 2022-07-19T14:53:23.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 </pre> <pre> 2022-07-09T09:06:56.709 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-09T09:06:56.709 INFO:tasks.cephfs_test_runner:ERROR: test_subvolume_snapshot_clone_quota_exceeded (tasks.cephfs.test_volumes.TestSubvolumeSnapshotClones) 2022-07-09T09:06:56.709 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-09T09:06:56.710 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-09T09:06:56.710 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/test_volumes.py", line 6111, in test_subvolume_snapshot_clone_quota_exceeded 2022-07-09T09:06:56.710 INFO:tasks.cephfs_test_runner: self._do_subvolume_io(subvolume, number_of_files=50) 2022-07-09T09:06:56.711 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/test_volumes.py", line 288, in _do_subvolume_io 2022-07-09T09:06:56.711 INFO:tasks.cephfs_test_runner: self.mount_a.write_n_mb(os.path.join(io_path, filename), file_size) 2022-07-09T09:06:56.711 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 1056, in write_n_mb 2022-07-09T09:06:56.712 INFO:tasks.cephfs_test_runner: ], wait=wait) 2022-07-09T09:06:56.712 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_10d6351a921d0691675d827b5bf030ef8a89b733/qa/tasks/cephfs/mount.py", line 724, in run_shell 2022-07-09T09:06:56.712 INFO:tasks.cephfs_test_runner: omit_sudo=omit_sudo, **kwargs) 2022-07-09T09:06:56.713 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/remote.py", line 510, in run 2022-07-09T09:06:56.713 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2022-07-09T09:06:56.713 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 455, in run 2022-07-09T09:06:56.714 INFO:tasks.cephfs_test_runner: r.wait() 2022-07-09T09:06:56.714 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 161, in wait 2022-07-09T09:06:56.714 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2022-07-09T09:06:56.715 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_10062088f503b43eff3624326bda825b23438f9b/teuthology/orchestra/run.py", line 183, in _raise_for_status 2022-07-09T09:06:56.715 INFO:tasks.cephfs_test_runner: node=self.hostname, label=self.label 2022-07-09T09:06:56.715 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 1: '(cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/subvolume_0000000000196238/8a431604-8247-41ec-a9be-c889c65f46d1/subvolume_file.43 bs=1M conv=fdatasync count=1 seek=0)' 2022-07-09T09:06:56.715 INFO:tasks.cephfs_test_runner: 2022-07-09T09:06:56.716 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-09T09:06:56.716 INFO:tasks.cephfs_test_runner:Ran 26 tests in 2158.897s </pre>