|
2021-11-02T16:25:25.258590+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8123 : cluster [DBG] osdmap e1809331: 125 total, 125 up, 125 in
|
|
2021-11-02T16:25:25.375159+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8124 : cluster [INF] osd.13 marked itself down
|
|
2021-11-02T16:25:25.821439+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11053 : cluster [DBG] pgmap v11596: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 4127 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 90 MiB/s rd, 88 MiB/s wr, 109 op/s
|
|
2021-11-02T16:25:26.269541+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8125 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)
|
|
2021-11-02T16:25:26.297861+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8126 : cluster [DBG] osdmap e1809332: 125 total, 124 up, 125 in
|
|
2021-11-02T16:25:27.334284+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8127 : cluster [DBG] osdmap e1809333: 125 total, 124 up, 125 in
|
|
2021-11-02T16:25:27.829339+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11054 : cluster [DBG] pgmap v11599: 4129 pgs: 3 activating+undersized, 4 active+undersized+degraded+wait, 40 peering, 18 stale+active+clean, 2 active+clean+scrubbing+deep+repair, 4062 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 92 MiB/s rd, 102 MiB/s wr, 120 op/s; 13356/23541294 objects degraded (0.057%)
|
|
2021-11-02T16:25:28.319139+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8128 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)
|
|
2021-11-02T16:25:28.319212+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8129 : cluster [WRN] Health check failed: Degraded data redundancy: 13356/23541294 objects degraded (0.057%), 4 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:25:29.844204+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11055 : cluster [DBG] pgmap v11600: 4129 pgs: 3 activating+undersized, 56 active+undersized+degraded+wait, 40 peering, 2 active+clean+scrubbing+deep+repair, 4028 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 77 MiB/s rd, 88 MiB/s wr, 105 op/s; 112710/23541336 objects degraded (0.479%)
|
|
2021-11-02T16:25:31.524887+0100 osd.88 (osd.88) 21 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:31.850132+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11056 : cluster [DBG] pgmap v11601: 4129 pgs: 3 activating+undersized, 60 active+undersized+degraded+wait, 37 peering, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 46 MiB/s rd, 55 MiB/s wr, 67 op/s; 120210/23541345 objects degraded (0.511%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:31.887781+0100 osd.19 (osd.19) 8 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:32.056540+0100 osd.31 (osd.31) 57 : cluster [WRN] slow request osd_op(client.31809201.0:44028932 1.ec 1:37017179:::b1-168958f9274046bcb17b4186579219eb-junk-0:head [writefull 0~988424 in=988424b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.578139+0100 currently delayed
|
|
2021-11-02T16:25:32.241100+0100 osd.41 (osd.41) 15 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:32.563188+0100 osd.88 (osd.88) 22 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:32.652474+0100 osd.10 (osd.10) 14 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:32.910977+0100 osd.19 (osd.19) 9 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:33.061115+0100 osd.31 (osd.31) 58 : cluster [WRN] slow request osd_op(client.31809201.0:44028932 1.ec 1:37017179:::b1-168958f9274046bcb17b4186579219eb-junk-0:head [writefull 0~988424 in=988424b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.578139+0100 currently delayed
|
|
2021-11-02T16:25:33.211765+0100 osd.41 (osd.41) 16 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:33.534959+0100 osd.88 (osd.88) 23 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:33.632014+0100 osd.10 (osd.10) 15 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:33.870684+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11057 : cluster [DBG] pgmap v11602: 4129 pgs: 100 active+undersized+degraded+wait, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 55 MiB/s rd, 61 MiB/s wr, 76 op/s; 189476/23541318 objects degraded (0.805%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:33.976265+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8131 : cluster [WRN] Health check update: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)
|
|
2021-11-02T16:25:33.976347+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8132 : cluster [WRN] Health check update: Degraded data redundancy: 120210/23541345 objects degraded (0.511%), 60 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:25:33.948970+0100 osd.19 (osd.19) 10 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:34.083576+0100 osd.31 (osd.31) 59 : cluster [WRN] slow request osd_op(client.31809201.0:44028932 1.ec 1:37017179:::b1-168958f9274046bcb17b4186579219eb-junk-0:head [writefull 0~988424 in=988424b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.578139+0100 currently delayed
|
|
2021-11-02T16:25:34.247324+0100 osd.41 (osd.41) 17 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:34.443046+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8133 : cluster [WRN] Health check failed: 2 slow ops, oldest one blocked for 7 sec, daemons [osd.41,osd.88] have slow ops. (SLOW_OPS)
|
|
2021-11-02T16:25:34.443122+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8134 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering)
|
|
2021-11-02T16:25:34.498851+0100 osd.88 (osd.88) 24 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:34.667924+0100 osd.10 (osd.10) 16 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:34.940113+0100 osd.19 (osd.19) 11 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:35.272624+0100 osd.41 (osd.41) 18 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:35.548327+0100 osd.88 (osd.88) 25 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:35.640785+0100 osd.10 (osd.10) 17 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:35.884853+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11058 : cluster [DBG] pgmap v11603: 4129 pgs: 2 active+undersized+degraded, 98 active+undersized+degraded+wait, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 56 MiB/s rd, 62 MiB/s wr, 79 op/s; 189476/23541315 objects degraded (0.805%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:35.817620+0100 osd.52 (osd.52) 13 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:35.966000+0100 osd.19 (osd.19) 12 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:36.236359+0100 osd.41 (osd.41) 19 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:36.533912+0100 osd.88 (osd.88) 26 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:36.680212+0100 osd.10 (osd.10) 18 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:36.972823+0100 osd.19 (osd.19) 13 : cluster [WRN] slow request osd_op(client.31809201.0:44028899 1.b07 1:e0d115c5:::b1-f7b10ac684df4befb5f4e8bff24cb30d-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809331) initiated 2021-11-02T16:25:26.289986+0100 currently delayed
|
|
2021-11-02T16:25:36.848796+0100 osd.52 (osd.52) 14 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:37.223732+0100 osd.41 (osd.41) 20 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:37.488208+0100 osd.88 (osd.88) 27 : cluster [WRN] slow request osd_op(client.31809201.0:44028915 1.e9c 1:397e931e:::b1-b17c70876486454280638d44492cfad7-index:head [writefull 0~31 in=31b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.438551+0100 currently delayed
|
|
2021-11-02T16:25:37.492609+0100 osd.97 (osd.97) 10 : cluster [WRN] slow request osd_op(client.31809201.0:44029112 1.d6d 1:b6baf7cc:::b1-8eed54f8ea8c4789bff769eea2a8b9f9-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809333) initiated 2021-11-02T16:25:31.598941+0100 currently delayed
|
|
2021-11-02T16:25:37.728100+0100 osd.85 (osd.85) 24 : cluster [WRN] slow request osd_op(client.31809201.0:44029211 1.eb6 1:6d75abce:::b1-f30a4427675a455db6e4903eaef6a52e-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.377172+0100 currently delayed
|
|
2021-11-02T16:25:37.901414+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11059 : cluster [DBG] pgmap v11604: 4129 pgs: 17 active+undersized+degraded, 83 active+undersized+degraded+wait, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 55 MiB/s rd, 64 MiB/s wr, 77 op/s; 189477/23541378 objects degraded (0.805%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:38.002677+0100 osd.109 (osd.109) 14 : cluster [WRN] slow request osd_op(client.31809201.0:44029241 1.2d0 1:0b4511cb:::b1-0bde1d5c64a94cd69200686cd68b58eb-index:head [writefull 0~41 in=41b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.642512+0100 currently delayed
|
|
2021-11-02T16:25:37.831354+0100 osd.52 (osd.52) 15 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:38.234358+0100 osd.41 (osd.41) 21 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:38.351379+0100 osd.89 (osd.89) 31 : cluster [WRN] slow request osd_op(client.31809201.0:44029259 1.298 1:1942a064:::b1-045e8d3047344ee4bc0855316185043d-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.746114+0100 currently delayed
|
|
2021-11-02T16:25:38.718641+0100 osd.10 (osd.10) 19 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:38.982024+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8137 : cluster [WRN] Health check update: Degraded data redundancy: 189477/23541378 objects degraded (0.805%), 100 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:25:39.029241+0100 osd.109 (osd.109) 15 : cluster [WRN] slow request osd_op(client.31809201.0:44029241 1.2d0 1:0b4511cb:::b1-0bde1d5c64a94cd69200686cd68b58eb-index:head [writefull 0~41 in=41b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.642512+0100 currently delayed
|
|
2021-11-02T16:25:38.859124+0100 osd.52 (osd.52) 16 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:39.070371+0100 osd.11 (osd.11) 9 : cluster [WRN] slow request osd_op(client.31809201.0:44029365 1.8da 1:5b1d17b5:::b1-e71cfef1a53e44b1ab1111fae554c992-junk-12582912:head [writefull 0~38677 in=38677b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:33.611917+0100 currently delayed
|
|
2021-11-02T16:25:39.258945+0100 osd.41 (osd.41) 22 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:39.364083+0100 osd.89 (osd.89) 32 : cluster [WRN] slow request osd_op(client.31809201.0:44029259 1.298 1:1942a064:::b1-045e8d3047344ee4bc0855316185043d-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.746114+0100 currently delayed
|
|
2021-11-02T16:25:39.428342+0100 osd.65 (osd.65) 30 : cluster [WRN] slow request osd_op(client.31809201.0:44029353 1.790 1:09eb7eb3:::b1-fdd7ee4c9dd24f2584523b88c2f6c475-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:33.491024+0100 currently delayed
|
|
2021-11-02T16:25:39.700068+0100 osd.10 (osd.10) 20 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:39.916879+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11060 : cluster [DBG] pgmap v11605: 4129 pgs: 44 active+undersized+degraded, 56 active+undersized+degraded+wait, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 66 MiB/s rd, 76 MiB/s wr, 88 op/s; 189478/23541459 objects degraded (0.805%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:40.079387+0100 osd.109 (osd.109) 16 : cluster [WRN] slow request osd_op(client.31809201.0:44029241 1.2d0 1:0b4511cb:::b1-0bde1d5c64a94cd69200686cd68b58eb-index:head [writefull 0~41 in=41b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.642512+0100 currently delayed
|
|
2021-11-02T16:25:40.356501+0100 osd.89 (osd.89) 33 : cluster [WRN] slow request osd_op(client.31809201.0:44029259 1.298 1:1942a064:::b1-045e8d3047344ee4bc0855316185043d-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.746114+0100 currently delayed
|
|
2021-11-02T16:25:39.816468+0100 osd.52 (osd.52) 17 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:40.274589+0100 osd.41 (osd.41) 23 : cluster [WRN] slow request osd_op(client.31809201.0:44028920 1.a9a 1:595bc342:::b1-a026adbe158241c58dafbb2081f380b9-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.499089+0100 currently delayed
|
|
2021-11-02T16:25:40.462239+0100 osd.65 (osd.65) 31 : cluster [WRN] slow request osd_op(client.31809201.0:44029353 1.790 1:09eb7eb3:::b1-fdd7ee4c9dd24f2584523b88c2f6c475-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:33.491024+0100 currently delayed
|
|
2021-11-02T16:25:40.589138+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8138 : cluster [WRN] Health check update: 5 slow ops, oldest one blocked for 10 sec, daemons [osd.10,osd.19,osd.41,osd.52,osd.85] have slow ops. (SLOW_OPS)
|
|
2021-11-02T16:25:40.689426+0100 osd.10 (osd.10) 21 : cluster [WRN] slow request osd_op(client.31809201.0:44028956 1.cfa 1:5f3552d3:::b1-acec7710c43b4e478608dbf9a9d1172b-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809332) initiated 2021-11-02T16:25:26.768574+0100 currently delayed
|
|
2021-11-02T16:25:41.122613+0100 osd.109 (osd.109) 17 : cluster [WRN] slow request osd_op(client.31809201.0:44029241 1.2d0 1:0b4511cb:::b1-0bde1d5c64a94cd69200686cd68b58eb-index:head [writefull 0~41 in=41b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:32.642512+0100 currently delayed
|
|
2021-11-02T16:25:40.834794+0100 osd.52 (osd.52) 18 : cluster [WRN] slow request osd_op(client.31809201.0:44029008 1.380 1:01cba887:::b1-c1b7f2807987483cbe5a397c60d1cec1-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809333) initiated 2021-11-02T16:25:30.764748+0100 currently delayed
|
|
2021-11-02T16:25:41.113298+0100 osd.11 (osd.11) 10 : cluster [WRN] slow request osd_op(client.31809201.0:44029365 1.8da 1:5b1d17b5:::b1-e71cfef1a53e44b1ab1111fae554c992-junk-12582912:head [writefull 0~38677 in=38677b] snapc 0=[] ondisk+write+known_if_redirected e1809333) initiated 2021-11-02T16:25:33.611917+0100 currently delayed
|
|
2021-11-02T16:25:41.924068+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11061 : cluster [DBG] pgmap v11606: 4129 pgs: 47 active+undersized+degraded, 53 active+undersized+degraded+wait, 2 active+clean+scrubbing+deep+repair, 4027 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 59 MiB/s rd, 71 MiB/s wr, 80 op/s; 189477/23541477 objects degraded (0.805%); 0 B/s, 0 objects/s recovering
|
|
2021-11-02T16:25:43.943494+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11062 : cluster [DBG] pgmap v11607: 4129 pgs: 88 active+undersized+degraded, 12 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 87 MiB/s rd, 99 MiB/s wr, 117 op/s; 189482/23541504 objects degraded (0.805%)
|
|
2021-11-02T16:25:43.987963+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8139 : cluster [WRN] Health check update: Degraded data redundancy: 189477/23541477 objects degraded (0.805%), 100 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:25:44.929706+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8140 : cluster [INF] Health check cleared: SLOW_OPS (was: 4 slow ops, oldest one blocked for 10 sec, daemons [osd.10,osd.41,osd.52,osd.85] have slow ops.)
|
|
2021-11-02T16:25:45.957470+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11063 : cluster [DBG] pgmap v11608: 4129 pgs: 99 active+undersized+degraded, 1 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 88 MiB/s rd, 104 MiB/s wr, 119 op/s; 189479/23541567 objects degraded (0.805%)
|
|
2021-11-02T16:25:47.974982+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11064 : cluster [DBG] pgmap v11609: 4129 pgs: 100 active+undersized+degraded, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 106 MiB/s rd, 123 MiB/s wr, 137 op/s; 189481/23541642 objects degraded (0.805%)
|
|
2021-11-02T16:25:48.994062+0100 mon.csdeveubs-u02c01mon01 (mon.0) 8141 : cluster [WRN] Health check update: Degraded data redundancy: 189481/23541642 objects degraded (0.805%), 100 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:25:49.989421+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11065 : cluster [DBG] pgmap v11610: 4129 pgs: 100 active+undersized+degraded, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 106 MiB/s rd, 119 MiB/s wr, 135 op/s; 189482/23541696 objects degraded (0.805%)
|
|
2021-11-02T16:25:51.997223+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 11066 : cluster [DBG] pgmap v11611: 4129 pgs: 100 active+undersized+degraded, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 90 MiB/s rd, 101 MiB/s wr, 117 op/s; 189481/23541702 objects degraded (0.805%)
|