Ceph » CephFS

Venky Shankar wrote:

which prompted a variety of code changes to workaround the problem. This all carries a sizeable baggage of fixes/workarounds which I think can be done away with if the metadata is stored directly in RADOS.

Why didn't we go with directly storing into RADOS back when we implemented subvolumes? Is there any catch here?

Dhairya Parmar wrote:

Venky Shankar wrote:

which prompted a variety of code changes to workaround the problem. This all carries a sizeable baggage of fixes/workarounds which I think can be done away with if the metadata is stored directly in RADOS.

Why didn't we go with directly storing into RADOS back when we implemented subvolumes? Is there any catch here?

Honestly, I don't recall exactly why, but one of the factors that effected the decision was surely timelines :)

Dhairya Parmar wrote:

Venky Shankar wrote:

which prompted a variety of code changes to workaround the problem. This all carries a sizeable baggage of fixes/workarounds which I think can be done away with if the metadata is stored directly in RADOS.

Why didn't we go with directly storing into RADOS back when we implemented subvolumes? Is there any catch here?

The reason is because legacy subvolume interface was handled by a python library (used by Openstack Manila). That library only had libcephfs and acted as a regular client.

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

Patrick Donnelly wrote:

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

I've been pretty aggressively NAKing this due to the situations we've seen with Rook users losing data. If we lose the object containing the sqlite database, we can no longer map any subvolumes for Ceph-CSI, and there's no recovery option at all (unless we maintain a backup elsewhere, in which case...why bother with the database?).

Venky, is there a design for storing the metadata in omap? It's not clear to me how adding "yet another version" will clean up the issues we've hit, most of which actually stem from trying to correctly identify which version of the metadata we're working with...

Greg Farnum wrote:

Patrick Donnelly wrote:

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

I've been pretty aggressively NAKing this due to the situations we've seen with Rook users losing data. If we lose the object containing the sqlite database, we can no longer map any subvolumes for Ceph-CSI, and there's no recovery option at all (unless we maintain a backup elsewhere, in which case...why bother with the database?).

Two items:

- Usability: sqlite is suprior to manipulating the omap. It's too easy to hit scaling issues with the omap. Programmability is as unfriendly and tricky. Recovering a corrupt (probably partially written) .meta file is not fun and complicates the volumes plugin with recovery steps. There will be similar issues with omap when the metadata spans multiple objects (which it must for clones). Sqlite allows us to neatly avoid this problem.

- Recoverability: There are two approaches to this: (a) distributing the metadata as-is currently done so that a partial loss doesn't mean complete loss. So if one subvolume is corrupt, not all are impacted. This is what we currently have and we're talking about changing it. (b) centralizing the metadata for ease of inspection/modification. Centralizing the store doesn't necessarily mean we stop duplicating the data in the subvolume tree structure. It's just not authoritative. We could recover the database in the event of a catastrophe (probably with out-of-the-mgr tools). Of course, we can also use sqlite3 existing backup solutions to put periodic backups in CephFS itself to rollback to.

Venky, is there a design for storing the metadata in omap? It's not clear to me how adding "yet another version" will clean up the issues we've hit, most of which actually stem from trying to correctly identify which version of the metadata we're working with...

The directory layout of the subvolumes has always complicated how we can store metadata. "Legacy" subvolumes from openstack manila (via the infamous ceph_volume_plugin.py library) would have no metadata and simple directory structures (/volumes/<group>/<root of subvolume>/). This made doing snapshots harder: if you snapshot the subvolume then you can no longer "delete" it until its snapshots are gone (creating an incompatiblity with CSI). This and other issues prompted the v2 rework.

It doesn't sound to me like Venky is proposing a new hierarchy. Just moving the metadata to solve a specific problem of reliability. SQLite can help solve that.

Greg Farnum wrote:

Patrick Donnelly wrote:

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

I've been pretty aggressively NAKing this due to the situations we've seen with Rook users losing data. If we lose the object containing the sqlite database, we can no longer map any subvolumes for Ceph-CSI, and there's no recovery option at all (unless we maintain a backup elsewhere, in which case...why bother with the database?).

Venky, is there a design for storing the metadata in omap? It's not clear to me how adding "yet another version" will clean up the issues we've hit, most of which actually stem from trying to correctly identify which version of the metadata we're working with...

The feature is more to do with being able to do away with a bunch of stuff to workaround cases such as a half-baked .meta due to ENOSPC and another issue which involved a failed metadata update when removing subvolumes since it involved changing state (and persisting the state in .meta).

Patrick Donnelly wrote:

Greg Farnum wrote:

Patrick Donnelly wrote:

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

I've been pretty aggressively NAKing this due to the situations we've seen with Rook users losing data. If we lose the object containing the sqlite database, we can no longer map any subvolumes for Ceph-CSI, and there's no recovery option at all (unless we maintain a backup elsewhere, in which case...why bother with the database?).

Two items:

- Usability: sqlite is suprior to manipulating the omap. It's too easy to hit scaling issues with the omap.

To avoid scalability issues with omap, having one object per subvolume instead of stashing lots of omap entries in one object would be preferred.

Programmability is as unfriendly and tricky.

Perhaps my previous experience of working with RADOS OMAP APIs are contributing my bias towards using omap. I'm not sure if the RADOS OMAP APIs are too unfriendly - so I'm more curious about what trickery is in there? (not the implementation, of course)

Recovering a corrupt (probably partially written) .meta file is not fun and complicates the volumes plugin with recovery steps. There will be similar issues with omap when the metadata spans multiple objects (which it must for clones). Sqlite allows us to neatly avoid this problem.

Fair enough, however, in the current implementation, one of the places where a bit more attention is given during metadata updation is when a subvolume clone finishes and the plugin needs to update metadata on the source subvolume (removing some meta) followed by updating metadata on the clone (changing state) - the order in which we update is important (update clone state and then the source subvolume) and I do agree that libcephsqlite can be useful in this case.

The partial metadata that I mention in the tracker description is the half-baked metadata file for a subvolume. This is worked around by writing to a temp file and then renaming it over the real (existing) one.

- Recoverability: There are two approaches to this: (a) distributing the metadata as-is currently done so that a partial loss doesn't mean complete loss. So if one subvolume is corrupt, not all are impacted. This is what we currently have and we're talking about changing it. (b) centralizing the metadata for ease of inspection/modification. Centralizing the store doesn't necessarily mean we stop duplicating the data in the subvolume tree structure. It's just not authoritative. We could recover the database in the event of a catastrophe (probably with out-of-the-mgr tools). Of course, we can also use sqlite3 existing backup solutions to put periodic backups in CephFS itself to rollback to.

Venky, is there a design for storing the metadata in omap? It's not clear to me how adding "yet another version" will clean up the issues we've hit, most of which actually stem from trying to correctly identify which version of the metadata we're working with...

The directory layout of the subvolumes has always complicated how we can store metadata. "Legacy" subvolumes from openstack manila (via the infamous ceph_volume_plugin.py library) would have no metadata and simple directory structures (/volumes/<group>/<root of subvolume>/). This made doing snapshots harder: if you snapshot the subvolume then you can no longer "delete" it until its snapshots are gone (creating an incompatiblity with CSI). This and other issues prompted the v2 rework.

It doesn't sound to me like Venky is proposing a new hierarchy. Just moving the metadata to solve a specific problem of reliability. SQLite can help solve that.

That's correct. Another thing that I this would bring us is avoiding hand-editing the metadata which has been a source of abuse lately.

Venky Shankar wrote:

Patrick Donnelly wrote:

Greg Farnum wrote:

Patrick Donnelly wrote:

If we are going to move the metadata out of CephFS, I think it should go in cephsqlite. There is already a database available to the volumes plugin for this potential purpose.

I've been pretty aggressively NAKing this due to the situations we've seen with Rook users losing data. If we lose the object containing the sqlite database, we can no longer map any subvolumes for Ceph-CSI, and there's no recovery option at all (unless we maintain a backup elsewhere, in which case...why bother with the database?).

Two items:

- Usability: sqlite is suprior to manipulating the omap. It's too easy to hit scaling issues with the omap.

To avoid scalability issues with omap, having one object per subvolume instead of stashing lots of omap entries in one object would be preferred.

Right, but then comes the consistency issues.

Programmability is as unfriendly and tricky.

Perhaps my previous experience of working with RADOS OMAP APIs are contributing my bias towards using omap. I'm not sure if the RADOS OMAP APIs are too unfriendly - so I'm more curious about what trickery is in there? (not the implementation, of course)

omap may be straightforward when you have a static key-value format to persist but the moment it becomes hierarchical or tabular (and it will!), then you are wading into painful territory. That is one of my main concerns. It's also expensive to answer questions concerning multiple subvolumes as you must do omap queries spanning all subvolumes. e.g.: list subvolumes by quota; list all cloned subvolumes; etc.