Ceph

Copy the below email thread and open an issue to track the enhancement.

Date: Wed, 29 Oct 2014 08:11:01 -0700
From: sage@newdream.net
To: yguang11@outlook.com
CC: ceph-devel@vger.kernel.org
Subject: Re: OSD crashed due to filestore EIO

On Wed, 29 Oct 2014, GuangYang wrote:
> Recently we observed an OSD crash due to file corruption in filesystem,
> which leads to an assertion failure at FileStore::read as EIO is not
> tolerated. As file corruption is normal in large deployment, I am
> thinking if that behavior is too aggressive, especially for EC pool.
>
> After searching, I found this flag might help : filestore_fail_eio,
> which can make the OSD survive an EIO failure, it is true by default
> though. I haven't tested it yet.

 That will reove the immediate assert. Currently, for an object being read
 by a client, it will just pass EIO back to the client, though, which is
 clearly not what we want.

> Does it make sense to adjust the behavior a little bit, if the filestore
> read fail due to file corruption, return back the failure and at the
> same time mark the PG as inconsistent, due the redundancy (replication
> or EC), the request can still be served, and at the same time, we can
> get alert saying there is inconsistency and manually trigger a PG
> repair?

 That would be ideal, yeah. I think that initially it makes sense to doing
 *just that read* via a replica but letting the admin trigger the repair.
 This most closely mirrors what scrub currently does on EIO (mark
 inconsistent but let admin repair). Later, when we support automatic
 repair, that option can affect both scrub and client-triggered EIOs?

 We just need to be careful that any EIO on *metadata* still triggers a
 failure as we need to be especially careful about handling that. IIRC
 there is a flag passed to read indicating whether EIO is okay; we should
 probably use that so that EIO-ok vs EIO-notok cases are still clearly
 annotated.