Bug #58645
openUnclear error when creating new subvolume when subvolumegroup has ceph.dir.subvolume attribute set to 1
0%
Description
When an empty name is given while creating a subvolume, this will turn the subvolumegroup into a subvolume instead of giving a proper error or warning.
Example below uses a subvolumegroup argument for clarity (note that if omitted, the default "_nogroup" subbvolumegroup will be used).
ceph fs subvolumegroup create ceph-filesystem examplegroup ceph fs subvolume create ceph-filesystem example1 --group_name examplegroup ceph fs subvolume create ceph-filesystem "" --group_name examplegroup # note the empty name ceph fs subvolume create ceph-filesystem example2 --group_name examplegroup
This last command shows the following error:
Error EINVAL: invalid value specified for ceph.dir.subvolume
The error is correct, but not really helpful:
- the "invalid value specified" seems to be meant for the subvolumegroup, not the subvolume we try to create, which is confusing and misleading
- it is unclear which value is invalid, in this case a 1 needs to be a 0, but it requires thorough investigation to come to this conclusion
And to make it even more confusing; this extended attribute can't be read with getfattr, although it is used by Ceph:
# getfattr -n ceph.dir.subvolume /mnt/cephfs/volumes/examplegroup /mnt/cephfs/volumes/examplegroup: ceph.dir.subvolume: No such attribute
Eventhough this extended attribute can't be read, it can be set with setfattr, to resolve the issue you need to change the "ceph.dir.subvolume" attribute on the filesystem to "0"
mount -t ceph 10.10.10.100:6789,10.10.10.101:6789,10.10.10.102:6789:/ /mnt/cephfs/ -o name=admin,secret=admin_secret setfattr -n ceph.dir.subvolume -v 0 /mnt/cephfs/volumes/examplegroup
Tested on versions:
- quincy (17) shows error only as shown in example
- octopus (15) and pacific (16) show multiple stacktraces, result (unusable subbvolumegroup) is the same.
Stacktrace octopus|pacific:
Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 271, in discover self.fs.stat(self.base_path) File "cephfs.pyx", line 1338, in cephfs.LibCephFS.stat cephfs.ObjectNotFound: error in stat: /volumes/examplegroup/example2: No such file or directory [Errno 2] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 164, in create_subvolume with open_subvol(self.mgr, fs_handle, self.volspec, group, subvolname, SubvolumeOpType.CREATE) as subvolume: File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 72, in open_subvol subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname) File "/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py", line 95, in get_subvolume_object subvolume.discover() File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 283, in discover raise VolumeException(-errno.ENOENT, "subvolume '{0}' does not exist".format(self.subvolname)) volumes.fs.exception.VolumeException: -2 (subvolume 'example2' does not exist) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 101, in mark_subvolume self.fs.setxattr(self.base_path, 'ceph.dir.subvolume', b'1', 0) File "cephfs.pyx", line 1264, in cephfs.LibCephFS.setxattr cephfs.InvalidValue: error in setxattr: Invalid argument [Errno 22] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 171, in create self.mark_subvolume() File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 103, in mark_subvolume raise VolumeException(-errno.EINVAL, "invalid value specified for ceph.dir.subvolume") volumes.fs.exception.VolumeException: -22 (invalid value specified for ceph.dir.subvolume) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 99, in dump self.fs.rename(path, self.unique_trash_path) File "cephfs.pyx", line 1545, in cephfs.LibCephFS.rename cephfs.Error: error in rename /volumes/examplegroup/example2 to /volumes/_deleting/4dd2069d-7cf9-44fe-b1c9-1c3c8e80ae28: Invalid cross-device link [Errno 18] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1212, in _handle_command return self.handle_command(inbuf, cmd) File "/usr/share/ceph/mgr/volumes/module.py", line 466, in handle_command return handler(inbuf, cmd) File "/usr/share/ceph/mgr/volumes/module.py", line 34, in wrap return f(self, inbuf, cmd) File "/usr/share/ceph/mgr/volumes/module.py", line 520, in _cmd_fs_subvolume_create namespace_isolated=cmd.get('namespace_isolated', False)) File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 176, in create_subvolume self._create_subvolume(fs_handle, volname, group, subvolname, **kwargs) File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 142, in _create_subvolume self.mgr, fs_handle, self.volspec, group, subvolname, size, isolate_nspace, pool, oct_mode, uid, gid) File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 26, in create_subvol subvolume.create(size, isolate_nspace, pool, mode, uid, gid) File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 193, in create self._remove_on_failure(subvol_path, retained) File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 151, in _remove_on_failure self.remove() File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 346, in remove self.trash_base_dir() File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 301, in trash_base_dir self._trash_dir(self.base_path) File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 289, in _trash_dir trashcan.dump(path) File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 101, in dump raise VolumeException(-e.args[0], e.args[1]) TypeError: bad operand type for unary -: 'str'
Updated by Venky Shankar about 1 year ago
- Status changed from New to Triaged
- Assignee set to Milind Changire
- Target version set to v18.0.0
- Backport set to pacific,quincy
- Component(FS) deleted (
cephfs.pyx)
Updated by Venky Shankar about 1 year ago
- Status changed from Triaged to Fix Under Review
- Pull request ID set to 50028
Updated by Patrick Donnelly 7 months ago
- Target version changed from v18.0.0 to v19.0.0
- Backport changed from pacific,quincy to reef,quincy
Updated by Milind Changire 6 months ago
- Pull request ID changed from 50028 to 53989