Project

General

Profile

Documentation #24642

Document visibility semantics to other clients

Added by Niklas Hambuechen 6 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
06/24/2018
Due date:
% Done:

0%

Tags:
Backport:
mimic,luminous
Reviewed:
Affected Versions:
Labels (FS):
offline
Pull request ID:

Description

I believe to just have run into a situation where one CephFS (fuse) mount created a file, and another CephFS mount still did not see that file as existent 3 minutes later.

So far I could not see anything in the CephFS docs that explains whether or not there are any guarantees on visibility semantics for other clients, or upper bounds or operations after which created files are supposed to be visible to other mount clients.

The only thing possibly relevant I have found so far is from http://docs.ceph.com/docs/mimic/cephfs/client-config-ref/ has

client oc max dirty age`
Description: Set the maximum age in seconds of dirty data in the object cache before writeback.
Default: 5.0 (seconds)

It would be great if this topic could be documented explicitly, as it is rather important to know when building distributed systems where order of events is important (e.g. where other nodes need to know at what point they can expect a file to be on the FS after another client has written it).

History

#1 Updated by Niklas Hambuechen 6 months ago

User `SeanR` on freenode `#ceph` reports:

nh2, I found (depening on mount options) if I don't call fsync() then other clients may not see the data until the dirty pages are flushed - I even had a case where another client would see the files metadata from the mds, but on trying to read the contents it would get 0 bytes (as the contents was not yet flushed), worse the read would populate the fscache on the reading client.

#2 Updated by Patrick Donnelly 6 months ago

Niklas Hambuechen wrote:

I believe to just have run into a situation where one CephFS (fuse) mount created a file, and another CephFS mount still did not see that file as existent 3 minutes later.

It sounds like you may have found a bug (possibly relating to a recent readdir bug). Can you post more information about your cluster version/setup and what client versions you're using?

#3 Updated by Zheng Yan 6 months ago

you found a bug. this can be http://tracker.ceph.com/issues/23894

#4 Updated by Niklas Hambuechen 6 months ago

Thanks everyone for the quick answers.

Patrick Donnelly wrote:

It sounds like you may have found a bug (possibly relating to a recent readdir bug). Can you post more information about your cluster version/setup and what client versions you're using?

ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)

and using the fuse client from that same version.

As far as I can tell, the fix for http://tracker.ceph.com/issues/23894 is in 13.2.0, so I can go upgrade and try out if that makes it never appear again (though it already took me a while to trigger it, so it seems relatively rare at least under my current workload, so I may want to allow for some time before I report back).

I also noticed that http://tracker.ceph.com/issues/23894 still has a `Backport: Luminous` tag -- do you plan to backport the fix to that?

#5 Updated by Patrick Donnelly 6 months ago

  • Status changed from New to Closed

Niklas Hambuechen wrote:

Thanks everyone for the quick answers.

Patrick Donnelly wrote:

It sounds like you may have found a bug (possibly relating to a recent readdir bug). Can you post more information about your cluster version/setup and what client versions you're using?

ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable)

and using the fuse client from that same version.

As far as I can tell, the fix for http://tracker.ceph.com/issues/23894 is in 13.2.0, so I can go upgrade and try out if that makes it never appear again (though it already took me a while to trigger it, so it seems relatively rare at least under my current workload, so I may want to allow for some time before I report back).

I also noticed that http://tracker.ceph.com/issues/23894 still has a `Backport: Luminous` tag -- do you plan to backport the fix to that?

It has already been backported to the Luminous branch: http://tracker.ceph.com/issues/24049

12.2.6 should be out soon so you can also wait for that or upgrade to Mimic.

#6 Updated by Niklas Hambuechen 6 months ago

I have upgraded to Mimic now and will check out if I see three minute delays again.

I would like to ask you to reopen the issue though, because I originally filed this as a documentation issue:

To find out and document what ceph's guarantees, non-guarantees and expected timeouts are.

(The fact that a three-minute delay is unexpected is only one bit of that information; I certainly appreciate a related issue being fixed, but that issue is not this issue.)

#7 Updated by Patrick Donnelly 6 months ago

  • Status changed from Closed to New
  • Target version set to v14.0.0
  • Backport set to mimic,luminous
  • Affected Versions deleted (v12.2.5)
  • Labels (FS) offline added

Niklas Hambuechen wrote:

I have upgraded to Mimic now and will check out if I see three minute delays again.

I would like to ask you to reopen the issue though, because I originally filed this as a documentation issue:

To find out and document what ceph's guarantees, non-guarantees and expected timeouts are.

(The fact that a three-minute delay is unexpected is only one bit of that information; I certainly appreciate a related issue being fixed, but that issue is not this issue.)

Thanks, we'll try to address this documentation fix.

Also available in: Atom PDF