Bug #12811
closedENXIO from kernel client (cephfs)
0%
Description
this is on the lab cluster. It has happened twice in 2 days.
In each case, the requests are sent to the second acting OSD instead of the first.
Teuthology kernel is v4.1-rc1-1-gb9ddddf.
The first time, we mounted the same thing on flab (3.16) and the same files were accessible.
The second time, flab was still mounted and it also couldn't access the same file. This suggests it may be triggered by the stream of osdmap incrementals?
Updated by Sage Weil over 8 years ago
during this time the apama osds were flapping and we were rebalancing lots of data... either of those may be the trigger?
Updated by Zheng Yan over 8 years ago
I compared osdmaps of good and bad mounts, found that the 'exist' flag is missing in bad mount's osdmap.
root@mira049:~# ceph osd map data 1000b1dd68e.00000000 osdmap e600761 pool 'data' (0) object '1000b1dd68e.00000000' -> pg 0.5c0c9f2c (0.f2c) -> up ([4,24,127], p4) acting ([4,24,127], p4)
root@teuthology:/tmp# diff osdmap.good osdmap.bad -u --- osdmap.good 2015-08-28 01:48:23.219372417 -0700 +++ osdmap.bad 2015-08-28 01:49:00.679826791 -0700 @@ -1,4 +1,4 @@ -epoch 600790 +epoch 600800 flags pool 0 pg_num 4096 (4095) read_tier -1 write_tier -1 pool 1 pg_num 64 (63) read_tier -1 write_tier -1 @@ -8,7 +8,7 @@ osd1 10.214.133.104:6804 100% (exists, up) 100% osd2 10.214.134.140:6807 100% (exists, up) 100% osd3 10.214.134.140:6813 100% (exists, up) 100% -osd4 10.214.133.104:6808 90% (exists, up) 100% +osd4 10.214.133.104:6808 90% (up) 100% osd5 10.214.133.104:6812 100% (exists, up) 100% osd6 10.214.133.104:6801 100% (exists, up) 100% osd7 10.214.133.114:6804 100% (exists, up) 100%
Updated by Zheng Yan over 8 years ago
- Status changed from New to Fix Under Review
https://github.com/ceph/ceph-client/commit/4a18ede97ba5029587b362446360cfd42379f9d0
diff --git a/net/ceph/osdmap.c b/net/ceph/osdmap.c index 4a31258..7d8f581 100644 --- a/net/ceph/osdmap.c +++ b/net/ceph/osdmap.c @@ -1300,7 +1300,7 @@ struct ceph_osdmap *osdmap_apply_incremental(void **p, void *end, ceph_decode_addr(&addr); pr_info("osd%d up\n", osd); BUG_ON(osd >= map->max_osd); - map->osd_state[osd] |= CEPH_OSD_UP; + map->osd_state[osd] |= CEPH_OSD_UP | CEPH_OSD_EXISTS; map->osd_addr[osd] = addr; }
Updated by Sage Weil over 8 years ago
- Status changed from Fix Under Review to Resolved
fix deployed to teuthology, so far so good! Reviewed-by: