Ceph

Any news on this please? I can barely use my cluster since upgrade to Giant -- OSDs are crashing during backfill all over the place -- everything is very slow and I'm already tired of restarting OSDs...
Please help.

Are you saying these osds are using the KV backend?

Because the bug you say this duplicates is about KV osds.

Please let me know if you need more logs.
Due to this error all OSDs (except few) in my cluster are down or crashing within minute(s) from start even before peering is completed...
Now without access to data I have little left to do than to gather logs...
This is happening when only filestore-based OSD are up, as well as when filestore and keyvaluestore OSD are up.
Type of OSD is certainly doesn't matter...

Type of osd probably does matter if the KV osds are distributing faulty information.

Samuel Just wrote:

Type of osd probably does matter if the KV osds are distributing faulty information.

Earlier I stopped all KV OSDs and still had this issue...

Hmm, thanks.

I'm not sure who is working or dive into it. I need more free time to diagnose it. EC+KeyValueStore is a good match.

And Is there any other complains or reports about this assertion without KeyValueStore involved in?

Haomai Wang wrote:

EC+KeyValueStore is a good match.

Ironically that was the worst thing I ever tried in Ceph... Last week was miserable -- I did nothing but tried to recover my cluster but no matter what I do I can't get enough OSDs up to bring up EC pools... What happens is eventually KV OSD poisons newly created filestore OSDs during backfill even if all KV OSD are out... The only way I can bring poisoned OSD up is to delete folder with PG causing crash... Anyway it seems that I've lost (m) OSDs and my EC pools are doomed... It feels very bad to lose data on Ceph for the third time even after using it for over 10 months. It feels so fragile... I'm beginning to consider giving up on it... :(

And Is there any other complains or reports about this assertion without KeyValueStore involved in?

See above except for the rant... It seems that problem originates from KV OSD when some bad EC replica migrates from it. Then the problem persists even when KV OSD is "out" and "down"...

It's a pity to see it.

Thanks to Dmitry Smirnov, could you give a detail summary about what you done and suggestion for EC+KV. And what's the problem you guess.

I will dive into it in the near future.

Thanks and sorry for lack of details. I was curious about KV-based OSDs for a long time and after upgrading to 0.87 I've decided to try introducing new KV OSD. Initially it was added with primary-affinity=0 and it was working like this for a week with no problems whatsoever. Some in-out tests were run as well as (deep-)scrubs etc. Then I've taken testing a bit further and tried primary-affinity=1 for some time and again found that new OSD is stable and responding properly. Then few more KV OSDs were added and I've let 'em to fill up to 1 TiB each. Problem hit accidentally and with catastrophic scale probably after reboot of host with filestore-based OSDs: OSD crashed as described following by crash of another OSD etc. very fast so cascade of crashes seemed almost simultaneous. OSD crash happens early after boot when peering is over so no replicas is sent. Then like a bushfire it spreads to next OSD where another replica of EC pool is located and so on. I use the following erasure profile on two pools:

k=2
m=3
plugin=jerasure
ruleset-failure-domain=host
technique=reed_sol_van

As for suggestions I'd recommend more testing but I also recognise what looks like a fundamental flaw in Ceph development. OSDs stability issues are not new and I experienced 'em first hand ever since 0.72. Bugs happen that's fine but plugging untested code paths with asserts is wrong and ugly. OSDs must be stable and when something wrong happens, log the problem loud enough, stop operating on problematic PG and possibly mark it "down" while keep OSD up and running rather than committing suicide. If in doubt make this behaviour configurable. What happened to me is not (just) because of the bug but more due to systematic error in OSD development -- this is not how mission-critical software should work.

Do you need any additional "debug_keyvaluestore=20/20" logs? It's been another week... Is there any progress? Any hope to revive my cluster?

Yep, if free you can paste crash logs with debug_keyvaluestore=20/20

It's been another week -- is there any chance to get this fixed please?

Hi Dmitry,

A related bug is fixed. But I'm not fully sure whether fix this problem. So could you give a crash keyvaluestore log with debug_keyvaluestore=20/20.

Haomai Wang wrote:

A related bug is fixed. But I'm not fully sure whether fix this problem.

I'm not sure which bug you have in mind. Fix for #10119 did not help my problem at all. :(

So could you give a crash keyvaluestore log with debug_keyvaluestore=20/20.

Unfortunately after weeks of downtime no KV OSDs are starting any more:

read meta/23c2fcde/osd_superblock/0//-1 0~0
lookup_strip_header failed to get strip_header  cid meta oid 23c2fcde/osd_superblock/0//-1
read meta/23c2fcde/osd_superblock/0//-1 0~0 header isn't exist: r = -2
OSD::init() : unable to read osd superblock
 ** ERROR: osd init failed: (22) Invalid argument

I reported KV OSD init problem as #10225.

Can this bug get a little attention please? It has profound effect on my crippled cluster and I'm talking about filestore OSDs, not KV ones...

Dmitry Smirnov wrote:

Can this bug get a little attention please? It has profound effect on my crippled cluster and I'm talking about filestore OSDs, not KV ones...

This looks like fallout from a KV OSD elsewhere in the cluster. This assert happens when a replica/shard doesn't have the object it's supposed to. You might be able to recover to shutting down the KV OSDs.

No, I was not able to recover by stopping all KV OSDs. It did not help. I don't know why you guys do not understand that the problem is keep happening even without KV OSDs. Here is what I did:

• Applied https://github.com/ceph/ceph/pull/2952 from #9943 on top of 0.89 (seemingly fixes ECBackend::handle_sub_read)

• Disabled backfilling completely (`osd max backfills = 0` in `/etc/ceph/ceph.conf`) to avoid other asserts like `OnRecoveryReadComplete::finish` / `osd/ECBackend.cc: 198: FAILED assert(res.r == 0)`.

• Run the following loop to keep OSDs up: `while sleep 66; do /etc/init.d/ceph start osd ; done`.

• Mark especially unstable OSDs as "out" (`ceph osd out NN`).

• Copied all data from degraded EC pool. (Another EC pool did not survive due to unfound object despite redundancy that supposed to be enough to tolerate failure of 3 hosts).
  Not deleting files from degraded EC pools helped to keep OSDs alive. It was possible to use replicated pool(s) as this problem affects only EC pools. Copying data took about a week.

• Dropped all EC pools.

• Decommissioned Ceph in favour of different storage solution (LizardFS, unless better alternative will be found in a week).

It was possible to read data from degraded EC pools but backfill was causing OSD crash in seconds. I've just finished recovering data so I can't tell how much corruption is there. I expect some. So far some files seems to be recovered successfully while at least one ~350 GiB file failed integrity check...

The OSDs share all kinds of information with each other and that information is generally trusted. That means that a misbehaving OSD that shares bad metadata with its peers can break the rest of the cluster. The fact that all KV OSDs are now stopped and/or removed from the cluster doesn't change that when they were present the may have 'poisoned' some PGs with incorrect metadata.

I'm really really sorry that it wasn't clear that keyvaluestore-dev was an experimental and untested backend that should not be trusted with important data. I had thought the '-dev' suffix would be sufficient to scare people off but I was mistaken. I'm afraid this has unduly soured your ceph experience. Unfortunately we have enough to do that we can't sink a lot of developer time into repairing corruptions caused by unsupported code. I'm really sorry. :(

If you have a few minutes, I would appreciate it if you reviewed this thread about how to more aggressively flag experimental features so that others do not repeat your experience:

http://marc.info/?t=141780100300006&r=1&w=2

In particular, it would be helpful to know if the third option (requiring a config option like 'enable unrecoverable data corrupting features = keyvaluestore' in order to use an untested feature) sounds like a good one to you.

Thanks, Dmitry!

Dear Sage,
Thanks for your kind words but please do not draw a wrong conclusion from this incident:
the problem was not that I did not realise that KV backend is experimental. I took full responsibility for trying it and I knowingly considered risk of evaluating KV OSDs.

The real problem is that such things as OSDs "poisoning" is possible. Earlier I've already experienced similar problems like cascade of "domino effect" OSD failures spreading like a bush fire. I tried KV backend only because I was super unhappy with fragile behaviour of filestore OSDs and it was naive to expect KV to be any better.

This was not a bad experience comparing to other incidents and I would give up on Ceph anyway -- this was just a very last chance I've gave it before walking away... Merely a problem in experimental KV backend wouldn't be a reason to give up. Unfortunately I see a bigger picture where Ceph is full of design flaws making it unfit for its purpose in my eyes... That is my conclusion. Sorry.

Unfortunately, we encountered this bug recently. I found a workaround and hope anyone who came to this page desperately could try it.
We using rgw+EC to store big files, the cluster is about 3PB. We encountered this bug when adding new nodes to the cluster, one pg is down, three OSDs kept coredumped after backfilling from the old osd sets to new osd sets, because the handle_sub_read caused a assertion.
The method is quite simple, we stop backfilling by "ceph osd set nobackfill", after that, no backfill would happen, and so, no osd coredump, no data loss.
Then we found the reason for the coredump, we found a suspicious object that could not found in local file system. we decided to backup the object and then delete the object, then we started backfilling, the backfilling procedure runs happily.