Ceph » RADOS

2021-11-02T16:20:16.641899+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7663 : cluster [DBG] osdmap e1809249: 125 total, 125 up, 125 in

2021-11-02T16:20:18.101818+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10855 : cluster [DBG] pgmap v11361: 4129 pgs: 3 active+clean+scrubbing+deep+repair, 4126 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 87 MiB/s rd, 90 MiB/s wr, 122 op/s

2021-11-02T16:20:18.723100+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7664 : cluster [DBG] osdmap e1809250: 125 total, 125 up, 125 in

2021-11-02T16:20:20.105696+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10856 : cluster [DBG] pgmap v11363: 4129 pgs: 3 active+clean+scrubbing+deep+repair, 4126 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 82 MiB/s rd, 86 MiB/s wr, 117 op/s

2021-11-02T16:20:20.774450+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7665 : cluster [DBG] osdmap e1809251: 125 total, 125 up, 125 in

2021-11-02T16:20:22.094637+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7666 : cluster [DBG] osd.7 reported immediately failed by osd.6

2021-11-02T16:20:22.094722+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7667 : cluster [INF] osd.7 failed (root=default,host=csdeveubs-u02c01b01) (connection refused reported by osd.6)

2021-11-02T16:20:22.206502+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7668 : cluster [DBG] osd.7 reported immediately failed by osd.75

<removed 250 more lines>

2021-11-02T16:20:22.618396+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7920 : cluster [DBG] osd.7 reported immediately failed by osd.119

2021-11-02T16:20:22.618442+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7921 : cluster [DBG] osd.7 reported immediately failed by osd.5

2021-11-02T16:20:22.618510+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7922 : cluster [DBG] osd.7 reported immediately failed by osd.38

2021-11-02T16:20:22.693787+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7923 : cluster [DBG] osd.7 reported immediately failed by osd.6

2021-11-02T16:20:22.127137+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10857 : cluster [DBG] pgmap v11365: 4129 pgs: 3 active+clean+scrubbing+deep+repair, 4126 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 141 MiB/s rd, 144 MiB/s wr, 193 op/s

2021-11-02T16:20:22.792914+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7924 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN)

2021-11-02T16:20:22.816977+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7925 : cluster [DBG] osdmap e1809252: 125 total, 124 up, 125 in

2021-11-02T16:20:23.841017+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7926 : cluster [DBG] osdmap e1809253: 125 total, 124 up, 125 in

2021-11-02T16:20:24.131189+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10858 : cluster [DBG] pgmap v11368: 4129 pgs: 10 peering, 26 stale+active+clean, 3 active+clean+scrubbing+deep+repair, 4090 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 96 MiB/s rd, 91 MiB/s wr, 122 op/s

2021-11-02T16:20:24.858162+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7927 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY)

2021-11-02T16:20:24.902272+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7928 : cluster [DBG] osdmap e1809254: 125 total, 124 up, 125 in

2021-11-02T16:20:26.136279+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10859 : cluster [DBG] pgmap v11370: 4129 pgs: 10 peering, 26 stale+active+clean, 3 active+clean+scrubbing+deep+repair, 4090 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 98 MiB/s rd, 93 MiB/s wr, 124 op/s

2021-11-02T16:20:27.765459+0100 osd.6 (osd.6) 20 : cluster [WRN] slow request osd_op(client.31809201.0:43989711 1.845 1:a21b9fa9:::b1-f3230e2382284f73b1da76132a1ecc6c-junk-12582912:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.036727+0100 currently delayed

2021-11-02T16:20:28.394093+0100 osd.35 (osd.35) 8 : cluster [WRN] slow request osd_op(client.31809201.0:43989817 1.d91 1:89babef2:::b1-8c9221a2976c44da98e3f0177f9fc6bf-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.974483+0100 currently delayed

2021-11-02T16:20:28.508340+0100 osd.113 (osd.113) 7 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:28.579141+0100 osd.53 (osd.53) 36 : cluster [WRN] slow request osd_op(client.31809201.0:43989780 1.74d 1:b2e8aff8:::b1-20632f423bb84eff8e13c1d6040ae0de-junk-0:head [writefull 0~4194304 in=4194304b] snapc 0=[] ondisk+write+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.645539+0100 currently delayed

2021-11-02T16:20:28.610917+0100 osd.99 (osd.99) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43989722 1.42b 1:d429e2d2:::b1-a61fa72578154154a838a3154990a30f-junk-16777216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863079+0100 currently delayed

2021-11-02T16:20:28.158466+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10860 : cluster [DBG] pgmap v11371: 4129 pgs: 95 active+undersized+degraded+wait, 5 peering, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 89 MiB/s rd, 78 MiB/s wr, 113 op/s; 180736/23540508 objects degraded (0.768%)

2021-11-02T16:20:28.787049+0100 osd.6 (osd.6) 21 : cluster [WRN] slow request osd_op(client.31809201.0:43989711 1.845 1:a21b9fa9:::b1-f3230e2382284f73b1da76132a1ecc6c-junk-12582912:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.036727+0100 currently delayed

2021-11-02T16:20:28.814707+0100 osd.37 (osd.37) 19 : cluster [WRN] slow request osd_op(client.31809201.0:43989918 1.3aa 1:55cc9832:::b1-3e489b0e647e4230ab8d1b82f7d5fa47-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:23.759194+0100 currently delayed

2021-11-02T16:20:28.908904+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7930 : cluster [WRN] Health check failed: Degraded data redundancy: 180736/23540508 objects degraded (0.768%), 95 pgs degraded (PG_DEGRADED)

2021-11-02T16:20:29.383445+0100 osd.35 (osd.35) 9 : cluster [WRN] slow request osd_op(client.31809201.0:43989817 1.d91 1:89babef2:::b1-8c9221a2976c44da98e3f0177f9fc6bf-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.974483+0100 currently delayed

2021-11-02T16:20:29.472158+0100 osd.113 (osd.113) 8 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:29.554255+0100 osd.53 (osd.53) 37 : cluster [WRN] slow request osd_op(client.31809201.0:43989780 1.74d 1:b2e8aff8:::b1-20632f423bb84eff8e13c1d6040ae0de-junk-0:head [writefull 0~4194304 in=4194304b] snapc 0=[] ondisk+write+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.645539+0100 currently delayed

2021-11-02T16:20:29.586170+0100 osd.99 (osd.99) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43989722 1.42b 1:d429e2d2:::b1-a61fa72578154154a838a3154990a30f-junk-16777216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863079+0100 currently delayed

2021-11-02T16:20:29.356893+0100 osd.5 (osd.5) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43989819 1.c6f 1:f63a5a5c:::b1-980575befee740f295b7e13e2a5c827d-junk-8388608:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.980535+0100 currently delayed

2021-11-02T16:20:29.401448+0100 osd.116 (osd.116) 9 : cluster [WRN] slow request osd_op(client.31809201.0:43989984 1.fdf 1:fbf23cc1:::b1-d72b11f03bb6466fa7d0b7009098862c-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.314934+0100 currently delayed

2021-11-02T16:20:29.763546+0100 osd.6 (osd.6) 22 : cluster [WRN] slow request osd_op(client.31809201.0:43989711 1.845 1:a21b9fa9:::b1-f3230e2382284f73b1da76132a1ecc6c-junk-12582912:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.036727+0100 currently delayed

2021-11-02T16:20:29.763586+0100 osd.6 (osd.6) 23 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:29.847268+0100 osd.37 (osd.37) 20 : cluster [WRN] slow request osd_op(client.31809201.0:43989918 1.3aa 1:55cc9832:::b1-3e489b0e647e4230ab8d1b82f7d5fa47-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:23.759194+0100 currently delayed

2021-11-02T16:20:30.347902+0100 osd.38 (osd.38) 6 : cluster [WRN] slow request osd_op(client.31809201.0:43990070 1.cf4 1:2f37acf6:::b1-1b54f6c5f78b493ab4ee9510ff2923a9-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.048964+0100 currently delayed

2021-11-02T16:20:30.420475+0100 osd.35 (osd.35) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43989817 1.d91 1:89babef2:::b1-8c9221a2976c44da98e3f0177f9fc6bf-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.974483+0100 currently delayed

2021-11-02T16:20:30.593290+0100 osd.99 (osd.99) 12 : cluster [WRN] slow request osd_op(client.31809201.0:43989722 1.42b 1:d429e2d2:::b1-a61fa72578154154a838a3154990a30f-junk-16777216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863079+0100 currently delayed

2021-11-02T16:20:30.164461+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10861 : cluster [DBG] pgmap v11372: 4129 pgs: 100 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 66 MiB/s rd, 54 MiB/s wr, 83 op/s; 190418/23540484 objects degraded (0.809%)

2021-11-02T16:20:30.332549+0100 osd.5 (osd.5) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43989819 1.c6f 1:f63a5a5c:::b1-980575befee740f295b7e13e2a5c827d-junk-8388608:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.980535+0100 currently delayed

2021-11-02T16:20:30.447692+0100 osd.116 (osd.116) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43989984 1.fdf 1:fbf23cc1:::b1-d72b11f03bb6466fa7d0b7009098862c-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.314934+0100 currently delayed

2021-11-02T16:20:30.723548+0100 osd.6 (osd.6) 24 : cluster [WRN] slow request osd_op(client.31809201.0:43989711 1.845 1:a21b9fa9:::b1-f3230e2382284f73b1da76132a1ecc6c-junk-12582912:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.036727+0100 currently delayed

2021-11-02T16:20:30.723562+0100 osd.6 (osd.6) 25 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:30.855261+0100 osd.37 (osd.37) 21 : cluster [WRN] slow request osd_op(client.31809201.0:43989918 1.3aa 1:55cc9832:::b1-3e489b0e647e4230ab8d1b82f7d5fa47-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:23.759194+0100 currently delayed

2021-11-02T16:20:30.947660+0100 osd.48 (osd.48) 58 : cluster [WRN] slow request osd_op(client.31809201.0:43990123 1.798 1:19e3fe5b:::b1-0a066a0f3b7d41efb53e2a055bca63b8-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.453072+0100 currently delayed

2021-11-02T16:20:30.947791+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7931 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering)

2021-11-02T16:20:31.393486+0100 osd.35 (osd.35) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43989817 1.d91 1:89babef2:::b1-8c9221a2976c44da98e3f0177f9fc6bf-index:head [writefull 0~3 in=3b] snapc 0=[] ondisk+write+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.974483+0100 currently delayed

2021-11-02T16:20:31.484658+0100 osd.22 (osd.22) 28 : cluster [WRN] slow request osd_op(client.31809201.0:43990160 1.592 1:49a03058:::b1-4821dc145dbc411b961a4dac9cd312c2-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.744625+0100 currently delayed

2021-11-02T16:20:31.515807+0100 osd.113 (osd.113) 9 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:31.547653+0100 osd.99 (osd.99) 13 : cluster [WRN] slow request osd_op(client.31809201.0:43989722 1.42b 1:d429e2d2:::b1-a61fa72578154154a838a3154990a30f-junk-16777216:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863079+0100 currently delayed

2021-11-02T16:20:31.319280+0100 osd.5 (osd.5) 12 : cluster [WRN] slow request osd_op(client.31809201.0:43989819 1.c6f 1:f63a5a5c:::b1-980575befee740f295b7e13e2a5c827d-junk-8388608:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.980535+0100 currently delayed

2021-11-02T16:20:31.472912+0100 osd.116 (osd.116) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43989984 1.fdf 1:fbf23cc1:::b1-d72b11f03bb6466fa7d0b7009098862c-index:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.314934+0100 currently delayed

2021-11-02T16:20:31.707664+0100 osd.6 (osd.6) 26 : cluster [WRN] slow request osd_op(client.31809201.0:43989711 1.845 1:a21b9fa9:::b1-f3230e2382284f73b1da76132a1ecc6c-junk-12582912:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809251) initiated 2021-11-02T16:20:22.036727+0100 currently delayed

2021-11-02T16:20:31.707687+0100 osd.6 (osd.6) 27 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:31.815481+0100 osd.37 (osd.37) 22 : cluster [WRN] slow request osd_op(client.31809201.0:43989918 1.3aa 1:55cc9832:::b1-3e489b0e647e4230ab8d1b82f7d5fa47-junk-12582912:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:23.759194+0100 currently started

2021-11-02T16:20:31.996376+0100 osd.48 (osd.48) 59 : cluster [WRN] slow request osd_op(client.31809201.0:43990123 1.798 1:19e3fe5b:::b1-0a066a0f3b7d41efb53e2a055bca63b8-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.453072+0100 currently delayed

2021-11-02T16:20:31.996401+0100 osd.48 (osd.48) 60 : cluster [WRN] slow request osd_op(client.31809201.0:43990303 1.a57 1:ea561900:::b1-3789e303e41c4e36854a31df75629ba8-junk-0:head [writefull 0~3904047 in=3904047b] snapc 0=[] ondisk+write+known_if_redirected e1809254) initiated 2021-11-02T16:20:26.924563+0100 currently delayed

2021-11-02T16:20:32.184487+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10862 : cluster [DBG] pgmap v11373: 4129 pgs: 13 active+undersized+degraded, 87 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 92 MiB/s rd, 85 MiB/s wr, 128 op/s; 190417/23540316 objects degraded (0.809%)

2021-11-02T16:20:32.483368+0100 osd.113 (osd.113) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:32.332129+0100 osd.5 (osd.5) 13 : cluster [WRN] slow request osd_op(client.31809201.0:43989819 1.c6f 1:f63a5a5c:::b1-980575befee740f295b7e13e2a5c827d-junk-8388608:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.980535+0100 currently delayed

2021-11-02T16:20:32.658263+0100 osd.6 (osd.6) 28 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:32.993163+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7932 : cluster [WRN] Health check failed: 5 slow ops, oldest one blocked for 8 sec, daemons [osd.116,osd.22,osd.37,osd.48,osd.99] have slow ops. (SLOW_OPS)

2021-11-02T16:20:32.999000+0100 osd.48 (osd.48) 61 : cluster [WRN] slow request osd_op(client.31809201.0:43990123 1.798 1:19e3fe5b:::b1-0a066a0f3b7d41efb53e2a055bca63b8-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.453072+0100 currently delayed

2021-11-02T16:20:32.999010+0100 osd.48 (osd.48) 62 : cluster [WRN] slow request osd_op(client.31809201.0:43990303 1.a57 1:ea561900:::b1-3789e303e41c4e36854a31df75629ba8-junk-0:head [writefull 0~3904047 in=3904047b] snapc 0=[] ondisk+write+known_if_redirected e1809254) initiated 2021-11-02T16:20:26.924563+0100 currently delayed

2021-11-02T16:20:33.157436+0100 osd.118 (osd.118) 7 : cluster [WRN] slow request osd_op(client.31809201.0:43990413 1.4f0 1:0f27460a:::b1-47f9b09edb8348d9b15f889ac0505c21-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809254) initiated 2021-11-02T16:20:27.786066+0100 currently delayed

2021-11-02T16:20:33.472827+0100 osd.113 (osd.113) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:33.523213+0100 osd.31 (osd.31) 56 : cluster [WRN] slow request osd_op(client.31809201.0:43990422 1.267 1:e640d9c7:::b1-d4c08020ada24b12b9a22589e0ab8a98-index:head [writefull 0~12 in=12b] snapc 0=[] ondisk+write+known_if_redirected e1809254) initiated 2021-11-02T16:20:27.838596+0100 currently delayed

2021-11-02T16:20:33.686453+0100 osd.6 (osd.6) 29 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:33.844528+0100 osd.87 (osd.87) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43990425 1.1ac 1:358a504a:::b1-7f560319c43d417eb813b7aac1fb82e3-junk-8388608:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809254) initiated 2021-11-02T16:20:27.859943+0100 currently delayed

2021-11-02T16:20:33.961109+0100 osd.48 (osd.48) 63 : cluster [WRN] slow request osd_op(client.31809201.0:43990123 1.798 1:19e3fe5b:::b1-0a066a0f3b7d41efb53e2a055bca63b8-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809254) initiated 2021-11-02T16:20:25.453072+0100 currently delayed

2021-11-02T16:20:33.961119+0100 osd.48 (osd.48) 64 : cluster [WRN] slow request osd_op(client.31809201.0:43990303 1.a57 1:ea561900:::b1-3789e303e41c4e36854a31df75629ba8-junk-0:head [writefull 0~3904047 in=3904047b] snapc 0=[] ondisk+write+known_if_redirected e1809254) initiated 2021-11-02T16:20:26.924563+0100 currently delayed

2021-11-02T16:20:34.194294+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10863 : cluster [DBG] pgmap v11374: 4129 pgs: 24 active+undersized+degraded, 76 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 84 MiB/s rd, 81 MiB/s wr, 117 op/s; 190417/23540361 objects degraded (0.809%)

2021-11-02T16:20:34.496645+0100 osd.113 (osd.113) 12 : cluster [WRN] slow request osd_op(client.31809201.0:43989779 1.454 1:2a2730c5:::b1-c3b7c7a337b240f4aed3c270f9bc1d71-junk-12582912:head [read 0~4194304] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e1809252) initiated 2021-11-02T16:20:22.863094+0100 currently delayed

2021-11-02T16:20:34.680674+0100 osd.6 (osd.6) 30 : cluster [WRN] slow request osd_op(client.31809201.0:43990040 1.f77 1:eef2e7fe:::b1-f1309071a23946e6b1e1c0d1acea4827-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809253) initiated 2021-11-02T16:20:24.744216+0100 currently delayed

2021-11-02T16:20:34.967063+0100 osd.48 (osd.48) 65 : cluster [WRN] slow request osd_op(client.31809201.0:43990303 1.a57 1:ea561900:::b1-3789e303e41c4e36854a31df75629ba8-junk-0:head [writefull 0~3904047 in=3904047b] snapc 0=[] ondisk+write+known_if_redirected e1809254) initiated 2021-11-02T16:20:26.924563+0100 currently delayed

2021-11-02T16:20:35.016969+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7933 : cluster [WRN] Health check update: Degraded data redundancy: 190417/23540361 objects degraded (0.809%), 100 pgs degraded (PG_DEGRADED)

2021-11-02T16:20:36.198683+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10864 : cluster [DBG] pgmap v11375: 4129 pgs: 24 active+undersized+degraded, 76 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 75 MiB/s rd, 72 MiB/s wr, 104 op/s; 190417/23540361 objects degraded (0.809%)

2021-11-02T16:20:38.220968+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10865 : cluster [DBG] pgmap v11376: 4129 pgs: 98 active+undersized+degraded, 2 active+undersized+degraded+wait, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 111 MiB/s rd, 104 MiB/s wr, 150 op/s; 190414/23540286 objects degraded (0.809%)

2021-11-02T16:20:38.544003+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7938 : cluster [WRN] Health check update: 8 slow ops, oldest one blocked for 9 sec, daemons [osd.113,osd.116,osd.22,osd.35,osd.37,osd.48,osd.5,osd.6,osd.99] have slow ops. (SLOW_OPS)

2021-11-02T16:20:40.278690+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7939 : cluster [WRN] Health check update: Degraded data redundancy: 190415/23540247 objects degraded (0.809%), 100 pgs degraded (PG_DEGRADED)

2021-11-02T16:20:40.278753+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7940 : cluster [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 9 sec, osd.35 has slow ops)

2021-11-02T16:20:40.224348+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10866 : cluster [DBG] pgmap v11377: 4129 pgs: 100 active+undersized+degraded, 3 active+clean+scrubbing+deep+repair, 4026 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 79 MiB/s rd, 74 MiB/s wr, 108 op/s; 190415/23540247 objects degraded (0.809%)