Ceph » RADOS

We have a 30 host, 1080 OSD cluster with a mix of replicated and EC 8+3 pools, running Jewel on SL7.

ceph version 10.2.3 (ecc23778eb545d8dd55e2e4735b53cc93f92e65b)

The problem was first seen when creating a large pool (16384 PGs) where a number of them would stay in peering states. Restarting the last primary would get the PG to peer.

PGs in other pools would also go into peering states.

It was also seen when creating a smaller pool and expanding the number of PGs (when increasing both pg_num and pgp_num). The problem could be mitigated by increasing pg_num (or pgp_num) in very small steps - anything above 128 PGs would result in some of them getting stuck.

After gradually increasing the PGs in the pool and getting everything active+clean, we started a rados bench to run overnight.

In the morning, all the OSDs on a host were down (36 in total) and a number of PGs in aforementioned pool in peering states.

Sample ceph health detail output:

pg 19.4a4 is down+remapped+peering, acting [86,812,2147483647,209,622,306,420,1029,394,266,204]
pg 19.380 is peering, acting [514,301,97,347,366,206,438,738,947,431,982]

Throughout the whole process the stuck PGs would peer after restarting their primary OSD, it was observed that the acting sets contained CRUSH_ITEM_NONE (2147483647) mappings rather than actual OSDs.

Sample ceph pg query past_interval output for pg 19.380:

                {
                    "first": 2035,
                    "last": 2036,
                    "maybe_went_rw": 1,
                    "up": [
                        514,
                        301,
                        97,
                        347,
                        366,
                        206,
                        438,
                        738,
                        947,
                        431,
                        982
                    ],
                    "acting": [
                        2147483647,
                        301,
                        97,
                        347,
                        366,
                        206,
                        438,
                        738,
                        947,
                        431,
                        982
                    ],
                    "primary": 301,
                    "up_primary": 514
                }

Some PGs in EC 8+3 pools would report up to 8 of their acting set as CRUSH_ITEM_NONE, often the primary being one of them.