Project

General

Profile

Support #18508 » echo-pg-peering-ceph-health-detail-2017-01-12 10.42.40.txt

ceph health detail output with a node down - George Vasilakakos, 01/12/2017 11:28 AM

 
[root@ceph-mon1 ~]# ceph health detail
HEALTH_ERR 13 pgs are stuck inactive for more than 300 seconds; 10 pgs down; 13 pgs peering; 13 pgs stuck inactive; 75 requests are blocked > 32 sec; 13 osds have slow requests
pg 19.52 is stuck inactive for 35290.947511, current state down+remapped+peering, last acting [2147483647,650,273,193,610,799,593,49,808,493,675]
pg 19.4a4 is stuck inactive for 35291.655185, current state down+remapped+peering, last acting [86,812,2147483647,209,622,306,420,1029,394,266,204]
pg 19.344c is stuck inactive for 35218.192411, current state remapped+peering, last acting [332,294,2147483647,503,384,610,336,8,1042,429,527]
pg 19.3390 is stuck inactive for 35304.106635, current state down+remapped+peering, last acting [2147483647,487,908,711,80,420,111,1078,192,601,49]
pg 19.340a is stuck inactive for 35309.821889, current state down+remapped+peering, last acting [2147483647,434,1038,921,1026,999,611,80,13,54,966]
pg 19.2fbc is stuck inactive for 35218.210703, current state peering, last acting [839,1008,34,505,227,527,460,104,315,682,829]
pg 19.380 is stuck inactive since forever, current state peering, last acting [514,301,97,347,366,206,438,738,947,431,982]
pg 19.1fda is stuck inactive for 35309.403963, current state down+remapped+peering, last acting [2147483647,424,879,972,2147483647,134,442,157,58,83,2147483647]
pg 19.2b61 is stuck inactive for 35301.815507, current state down+remapped+peering, last acting [2147483647,908,46,471,380,805,483,801,1024,221,967]
pg 19.214d is stuck inactive for 35309.967389, current state down+remapped+peering, last acting [280,2147483647,726,441,213,723,340,2147483647,2147483647,57,246]
pg 19.3d11 is stuck inactive for 35358.826608, current state down+remapped+peering, last acting [2147483647,823,25,2147483647,877,2147483647,599,345,2147483647,2147483647,857]
pg 19.3b77 is stuck inactive for 35296.249419, current state down+remapped+peering, last acting [2147483647,71,396,46,718,774,296,619,218,101,604]
pg 19.2007 is stuck inactive for 35450.591308, current state down+remapped+peering, last acting [2147483647,968,713,2147483647,2147483647,221,632,179,386,311,906]
pg 19.3d11 is down+remapped+peering, acting [2147483647,823,25,2147483647,877,2147483647,599,345,2147483647,2147483647,857]
pg 19.3b77 is down+remapped+peering, acting [2147483647,71,396,46,718,774,296,619,218,101,604]
pg 19.344c is remapped+peering, acting [332,294,2147483647,503,384,610,336,8,1042,429,527]
pg 19.340a is down+remapped+peering, acting [2147483647,434,1038,921,1026,999,611,80,13,54,966]
pg 19.3390 is down+remapped+peering, acting [2147483647,487,908,711,80,420,111,1078,192,601,49]
pg 19.2fbc is peering, acting [839,1008,34,505,227,527,460,104,315,682,829]
pg 19.2b61 is down+remapped+peering, acting [2147483647,908,46,471,380,805,483,801,1024,221,967]
pg 19.214d is down+remapped+peering, acting [280,2147483647,726,441,213,723,340,2147483647,2147483647,57,246]
pg 19.2007 is down+remapped+peering, acting [2147483647,968,713,2147483647,2147483647,221,632,179,386,311,906]
pg 19.1fda is down+remapped+peering, acting [2147483647,424,879,972,2147483647,134,442,157,58,83,2147483647]
pg 19.4a4 is down+remapped+peering, acting [86,812,2147483647,209,622,306,420,1029,394,266,204]
pg 19.380 is peering, acting [514,301,97,347,366,206,438,738,947,431,982]
pg 19.52 is down+remapped+peering, acting [2147483647,650,273,193,610,799,593,49,808,493,675]
9 ops are blocked > 67108.9 sec on osd.968
7 ops are blocked > 67108.9 sec on osd.908
10 ops are blocked > 67108.9 sec on osd.839
4 ops are blocked > 67108.9 sec on osd.823
8 ops are blocked > 67108.9 sec on osd.332
5 ops are blocked > 67108.9 sec on osd.280
6 ops are blocked > 67108.9 sec on osd.86
9 ops are blocked > 67108.9 sec on osd.71
4 ops are blocked > 67108.9 sec on osd.424
5 ops are blocked > 67108.9 sec on osd.434
3 ops are blocked > 67108.9 sec on osd.487
4 ops are blocked > 67108.9 sec on osd.514
1 ops are blocked > 67108.9 sec on osd.650
13 osds have slow requests
(2-2/3)