Ceph » CephFS

In response to: https://tracker.ceph.com/issues/36593#note-14

Yes, there is not an easy solution here. I guess we have to transmit the truncate as soon as it occurs.

After discussing this with Jeff on the mailing-list¹ we agreed that the best thing to do is to simply revert to returning -EXDEV when doing cross quotarealms renames. Any other solution would have a big performance impact. This effectively means reverting commit dffdcd71458e ("ceph: allow rename operation under different quota realms").

[1] https://www.spinics.net/lists/ceph-devel/msg49850.html

Luis Henriques wrote:

After discussing this with Jeff on the mailing-list¹ we agreed that the best thing to do is to simply revert to returning -EXDEV when doing cross quotarealms renames. Any other solution would have a big performance impact. This effectively means reverting commit dffdcd71458e ("ceph: allow rename operation under different quota realms").

[1] https://www.spinics.net/lists/ceph-devel/msg49850.html

Also need to revert the userspace change too. That was done for issue #39715.

To avoid forgetting why this is not a good idea, let's also add some good comments about why we don't allow cross-quota renames.

Patrick Donnelly wrote:

Luis Henriques wrote:

After discussing this with Jeff on the mailing-list¹ we agreed that the best thing to do is to simply revert to returning -EXDEV when doing cross quotarealms renames. Any other solution would have a big performance impact. This effectively means reverting commit dffdcd71458e ("ceph: allow rename operation under different quota realms").

[1] https://www.spinics.net/lists/ceph-devel/msg49850.html

Also need to revert the userspace change too. That was done for issue #39715.

I can submit the revert for the fuse client, but I wonder if that's really necessary (other than for consistency between the two clients, of course). AFAICS the fuse client does not have the same problem because the truncate operations are sync. Or did you saw a similar failure?

To avoid forgetting why this is not a good idea, let's also add some good comments about why we don't allow cross-quota renames.

The kernel client revert is currently queued on the testing branch, and I tried to describe there the problem with the commit.

Luis Henriques wrote:

Patrick Donnelly wrote:

Luis Henriques wrote:

After discussing this with Jeff on the mailing-list¹ we agreed that the best thing to do is to simply revert to returning -EXDEV when doing cross quotarealms renames. Any other solution would have a big performance impact. This effectively means reverting commit dffdcd71458e ("ceph: allow rename operation under different quota realms").

[1] https://www.spinics.net/lists/ceph-devel/msg49850.html

Also need to revert the userspace change too. That was done for issue #39715.

I can submit the revert for the fuse client, but I wonder if that's really necessary (other than for consistency between the two clients, of course). AFAICS the fuse client does not have the same problem because the truncate operations are sync. Or did you saw a similar failure?

The issue is not confined to a single client's consistent view of the quotas. We could have another client truncating files during the rename.

To avoid forgetting why this is not a good idea, let's also add some good comments about why we don't allow cross-quota renames.

The kernel client revert is currently queued on the testing branch, and I tried to describe there the problem with the commit.

Thanks!

Patrick Donnelly wrote:

Hey Luis, I think this is still broken; the revert didn't work: https://pulpito.ceph.com/teuthology-2021-01-03_03:15:02-fs-master-distro-basic-smithi/5751916/

Can you take a look?

Hmm... It looks like a distro kernel (kernel-4.18.0-240.1.1.el8_3.x86_64) is being used. Maybe the fix hasn't been backported to it? Or am I not looking at it right?

Dear all, I went through the discussion about the failed test and would like to ask you to consider an alternative solution to this problem:

Please introduce a mount option "fast_move" (or similar) to re-enable the code path with a proper move instead of falling back to cp+rm, and to document the implications of this fast_move option in such a way that operators can either prevent it or be prepared for accidents.

To be honest, the "corner case" that was discovered is certainly of academic interest and I understand the desire to have a file system pass all sorts of tests. However, there is also the practical side that some operators face a situation where users sometimes need to move large amounts of data (I talk hundreds of TB) across quota realms and the "fix" to remove quotas temporarily for this purpose does not really work.

"Moving" a large amount of data with "cp+rm" is really harmful. Several hundred TB usually require an amount of time that contains a service window and users will end up with interrupted operations even though a move is expected to be an atomic operation.

I would like to add to this that on our system (I would actually assume on any properly configured system) an ordinary user cannot even execute a truncate+mv in the way it is executed in the fs test suite. Root of course can, but root always has the power to screw up and also fix a system again. I don't think this is enough reason to disallow fast moves for everybody.

In my specific situation, I need to deploy a server where users can see a higher level view of our file systems (a server where /home and another directory do not show up as separate file systems, but as subdirectories of the same file system) and users can execute moves between different quota realms within this file system. On this occasion I would need an option like "fast_move" for the mount on this machine.

In general, whenever a decision between "clean theory" and "real life of operators" implies such a conflicting outcome, I would prefer if an implementation option similar to an additional mount flag is chosen, with a safe default and performance improved expert settings with known and documented risks involved.

At this very moment I'm exceedingly happy that I still have a server with a client that still does the fast moves. However, I need to update and really really need the fast move functionality. We are migrating a huge amount of data and this "cp+rm" default is a serious bottleneck, if not a real danger.

Thank you very much for your considerations.