Project

General

Profile

Bug #22847 » ceph_health_detail.log

ceph health details - Frank Li, 02/01/2018 04:28 AM

 
HEALTH_WARN nodown flag(s) set; 32 osds down; 4 hosts (32 osds) down; 2 racks (32 osds) down; 1 row (32 osds) down; Reduced data availability: 10240 pgs inactive, 5 pgs down, 3431 pgs peering, 3643 pgs stale; Degraded data redundancy: 215798/552339 objects degraded (39.070%), 10240 pgs unclean, 4666 pgs degraded, 4666 pgs undersized
OSDMAP_FLAGS nodown flag(s) set
OSD_DOWN 32 osds down
osd.0 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.1 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.2 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.3 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.4 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.5 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.6 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.7 (root=openstack,row=1,rack=602,host=dl1-cephosd101) is down
osd.8 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.9 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.10 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.11 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.12 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.13 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.14 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.15 (root=openstack,row=1,rack=603,host=dl1-cephosd201) is down
osd.16 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.17 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.18 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.19 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.20 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.21 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.22 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.23 (root=openstack,row=1,rack=602,host=dl1-cephosd102) is down
osd.24 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.25 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.26 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.27 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.28 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.29 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.30 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
osd.31 (root=openstack,row=1,rack=603,host=dl1-cephosd202) is down
OSD_HOST_DOWN 4 hosts (32 osds) down
host dl1-cephosd101 (root=openstack,row=1,rack=602) (8 osds) is down
host dl1-cephosd102 (root=openstack,row=1,rack=602) (8 osds) is down
host dl1-cephosd201 (root=openstack,row=1,rack=603) (8 osds) is down
host dl1-cephosd202 (root=openstack,row=1,rack=603) (8 osds) is down
OSD_RACK_DOWN 2 racks (32 osds) down
rack 602 (root=openstack,row=1) (16 osds) is down
rack 603 (root=openstack,row=1) (16 osds) is down
OSD_ROW_DOWN 1 row (32 osds) down
row 1 (root=openstack) (32 osds) is down
PG_AVAILABILITY Reduced data availability: 10240 pgs inactive, 5 pgs down, 3431 pgs peering, 3643 pgs stale
pg 1.1fcd is stuck inactive for 6415.071555, current state peering, last acting [10,28,55]
pg 1.1fce is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fcf is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fd0 is stuck peering for 25786.183809, current state peering, last acting [50,59]
pg 1.1fd1 is stuck stale for 11565.240195, current state stale+undersized+degraded+peered, last acting [6]
pg 1.1fd2 is stuck peering for 25772.727420, current state peering, last acting [1,20,63]
pg 1.1fd3 is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fd4 is stuck stale for 11565.240197, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fd5 is stuck stale for 11565.240197, current state stale+undersized+degraded+peered, last acting [7]
pg 1.1fd6 is stuck stale for 11565.240198, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fd7 is stuck stale for 11565.240198, current state stale+undersized+degraded+peered, last acting [2]
pg 1.1fd8 is stuck stale for 11565.240199, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fd9 is stuck stale for 11565.240200, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1fda is stuck stale for 11565.240200, current state stale+undersized+degraded+peered, last acting [31]
pg 1.1fdb is stuck stale for 11565.240201, current state stale+undersized+degraded+peered, last acting [16]
pg 1.1fdc is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fdd is stuck peering for 25789.507905, current state peering, last acting [40,30]
pg 1.1fde is stuck peering for 25783.103856, current state peering, last acting [51,63]
pg 1.1fdf is stuck inactive for 25771.709997, current state undersized+degraded+peered, last acting [63]
pg 1.1fe0 is stuck stale for 11565.240202, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1fe1 is stuck peering for 26523.274018, current state peering, last acting [40,37,11]
pg 1.1fe2 is stuck stale for 7148.035823, current state stale+peering, last acting [13,63]
pg 1.1fe3 is stuck peering for 25783.098959, current state peering, last acting [16,25]
pg 1.1fe4 is stuck stale for 11565.240209, current state stale+undersized+degraded+peered, last acting [33]
pg 1.1fe5 is stuck peering for 25786.195140, current state peering, last acting [47,6]
pg 1.1fe6 is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fe7 is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fe8 is stuck stale for 11565.240210, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fe9 is stuck stale for 11565.240211, current state stale+undersized+degraded+peered, last acting [1]
pg 1.1fea is stuck peering for 25786.199662, current state peering, last acting [52,7]
pg 1.1feb is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fec is stuck stale for 11565.240211, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fed is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fee is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1fef is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1ff0 is stuck stale for 11565.240227, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1ff1 is stuck stale for 11565.240227, current state stale+undersized+degraded+peered, last acting [0]
pg 1.1ff2 is stuck peering for 25770.632924, current state peering, last acting [9,3]
pg 1.1ff3 is stuck inactive for 25782.074274, current state undersized+degraded+peered, last acting [57]
pg 1.1ff4 is stuck peering for 25783.096624, current state peering, last acting [25,4]
pg 1.1ff5 is stuck peering for 25776.467718, current state peering, last acting [52,31]
pg 1.1ff6 is stuck stale for 11565.240237, current state stale+undersized+degraded+peered, last acting [25]
pg 1.1ff7 is stuck peering for 25778.778252, current state peering, last acting [6,17]
pg 1.1ff8 is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1ff9 is stuck stale for 11565.240257, current state stale+undersized+degraded+peered, last acting [6]
pg 1.1ffa is stuck stale for 7148.035831, current state stale+peering, last acting [13,4]
pg 1.1ffb is stuck peering for 25786.171740, current state peering, last acting [29,51]
pg 1.1ffc is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1ffd is stuck inactive for 30221.951178, current state unknown, last acting []
pg 1.1ffe is stuck stale for 11565.240263, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fff is stuck inactive for 30221.951178, current state unknown, last acting []
PG_DEGRADED Degraded data redundancy: 215798/552339 objects degraded (39.070%), 10240 pgs unclean, 4666 pgs degraded, 4666 pgs undersized
pg 1.1fcd is stuck unclean for 26213.638775, current state peering, last acting [10,28,55]
pg 1.1fce is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fcf is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fd0 is stuck unclean for 25788.313231, current state peering, last acting [50,59]
pg 1.1fd1 is stuck undersized for 15556.585572, current state stale+undersized+degraded+peered, last acting [6]
pg 1.1fd2 is stuck unclean for 25772.727420, current state peering, last acting [1,20,63]
pg 1.1fd3 is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fd4 is stuck undersized for 15526.837130, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fd5 is stuck undersized for 15556.288593, current state stale+undersized+degraded+peered, last acting [7]
pg 1.1fd6 is stuck undersized for 15546.524192, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fd7 is stuck undersized for 15556.592113, current state stale+undersized+degraded+peered, last acting [2]
pg 1.1fd8 is stuck undersized for 15546.516976, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fd9 is stuck undersized for 15556.827903, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1fda is stuck undersized for 15540.223467, current state stale+undersized+degraded+peered, last acting [31]
pg 1.1fdb is stuck undersized for 15546.313751, current state stale+undersized+degraded+peered, last acting [16]
pg 1.1fdc is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fdd is stuck unclean for 26203.148219, current state peering, last acting [40,30]
pg 1.1fde is stuck unclean for 25785.208621, current state peering, last acting [51,63]
pg 1.1fdf is stuck undersized for 15546.647448, current state undersized+degraded+peered, last acting [63]
pg 1.1fe0 is stuck undersized for 15556.836267, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1fe1 is stuck unclean for 26523.274018, current state peering, last acting [40,37,11]
pg 1.1fe2 is stuck unclean for 25786.220114, current state stale+peering, last acting [13,63]
pg 1.1fe3 is stuck unclean for 30097.504822, current state peering, last acting [16,25]
pg 1.1fe4 is stuck undersized for 15546.482837, current state stale+undersized+degraded+peered, last acting [33]
pg 1.1fe5 is stuck unclean for 30220.857111, current state peering, last acting [47,6]
pg 1.1fe6 is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fe7 is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fe8 is stuck undersized for 15526.792522, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fe9 is stuck undersized for 15556.007507, current state stale+undersized+degraded+peered, last acting [1]
pg 1.1fea is stuck unclean for 30221.917471, current state peering, last acting [52,7]
pg 1.1feb is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fec is stuck undersized for 15526.686958, current state stale+undersized+degraded+peered, last acting [29]
pg 1.1fed is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fee is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1fef is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1ff0 is stuck undersized for 15556.819248, current state stale+undersized+degraded+peered, last acting [3]
pg 1.1ff1 is stuck undersized for 15556.151991, current state stale+undersized+degraded+peered, last acting [0]
pg 1.1ff2 is stuck unclean for 25772.739719, current state peering, last acting [9,3]
pg 1.1ff3 is stuck undersized for 15546.926259, current state undersized+degraded+peered, last acting [57]
pg 1.1ff4 is stuck unclean for 88019.620739, current state peering, last acting [25,4]
pg 1.1ff5 is stuck unclean for 25778.781465, current state peering, last acting [52,31]
pg 1.1ff6 is stuck undersized for 15540.377043, current state stale+undersized+degraded+peered, last acting [25]
pg 1.1ff7 is stuck unclean for 25779.890723, current state peering, last acting [6,17]
pg 1.1ff8 is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1ff9 is stuck undersized for 15556.575201, current state stale+undersized+degraded+peered, last acting [6]
pg 1.1ffa is stuck unclean for 30199.625515, current state stale+peering, last acting [13,4]
pg 1.1ffb is stuck unclean for 44171.550888, current state peering, last acting [29,51]
pg 1.1ffc is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1ffd is stuck unclean for 30221.951178, current state unknown, last acting []
pg 1.1ffe is stuck undersized for 15546.458395, current state stale+undersized+degraded+peered, last acting [9]
pg 1.1fff is stuck unclean for 30221.951178, current state unknown, last acting []
(3-3/5)