|
2021-11-02T16:21:39.085277+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7976 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down)
|
|
2021-11-02T16:21:39.118930+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7977 : cluster [INF] osd.7 [v2:10.88.7.61:6840/3774830,v1:10.88.7.61:6843/3774830] boot
|
|
2021-11-02T16:21:39.119025+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7978 : cluster [DBG] osdmap e1809257: 125 total, 125 up, 125 in
|
|
2021-11-02T16:21:40.138175+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7979 : cluster [DBG] osdmap e1809258: 125 total, 125 up, 125 in
|
|
2021-11-02T16:21:40.576952+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10905 : cluster [DBG] pgmap v11411: 4129 pgs: 100 active+undersized+degraded, 1 active+clean+scrubbing+deep+repair, 4028 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 82 MiB/s rd, 78 MiB/s wr, 105 op/s; 190405/23539500 objects degraded (0.809%)
|
|
2021-11-02T16:21:40.590841+0100 osd.82 (osd.82) 13 : cluster [DBG] 1.908 scrub ok
|
|
2021-11-02T16:21:42.193275+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7980 : cluster [DBG] osdmap e1809259: 125 total, 125 up, 125 in
|
|
2021-11-02T16:21:42.582907+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10906 : cluster [DBG] pgmap v11413: 4129 pgs: 1 active+clean+scrubbing, 1 active+clean+laggy, 1 active+recovering+degraded, 36 active+recovery_wait+degraded, 40 active+undersized+degraded, 1 active+clean+scrubbing+deep+repair, 4049 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 93 MiB/s rd, 104 MiB/s wr, 130 op/s; 76936/23539503 objects degraded (0.327%); 4.3 MiB/s, 3 objects/s recovering
|
|
2021-11-02T16:21:42.943250+0100 osd.75 (osd.75) 14 : cluster [DBG] 1.ebb scrub ok
|
|
2021-11-02T16:21:43.630833+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7982 : cluster [WRN] Health check update: Degraded data redundancy: 76936/23539503 objects degraded (0.327%), 77 pgs degraded, 40 pgs undersized (PG_DEGRADED)
|
|
2021-11-02T16:21:44.285236+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7983 : cluster [DBG] osdmap e1809260: 125 total, 125 up, 125 in
|
|
2021-11-02T16:21:44.587305+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10907 : cluster [DBG] pgmap v11415: 4129 pgs: 1 active+clean+scrubbing+deep+repair, 14 active+clean+wait, 1 active+clean+scrubbing, 1 active+clean+laggy, 1 active+recovering+degraded, 41 active+recovery_wait+degraded, 18 active+recovery_wait+degraded+wait, 4052 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 78 MiB/s rd, 94 MiB/s wr, 114 op/s; 102/23539515 objects degraded (0.000%); 6.5 MiB/s, 6 objects/s recovering
|
|
2021-11-02T16:21:44.592064+0100 osd.118 (osd.118) 8 : cluster [DBG] 1.4f0 scrub ok
|
|
2021-11-02T16:21:44.771995+0100 osd.7 (osd.7) 1 : cluster [WRN] slow request osd_op(client.31809201.0:43999711 1.603 1:c0623462:::b1-bea5a4be768748369bf3aaa7b5280090-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.304199+0100 currently delayed
|
|
2021-11-02T16:21:44.772009+0100 osd.7 (osd.7) 2 : cluster [WRN] slow request osd_op(client.31809201.0:43999721 1.ea 1:570e0851:::b1-6c057c710a6246c5873ae150188b9df7-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.363418+0100 currently delayed
|
|
2021-11-02T16:21:44.774592+0100 osd.55 (osd.55) 48 : cluster [WRN] slow request osd_op(client.31809201.0:43999745 1.1e5 1:a78f5f83:::b1-3e23be04e57d4aee8d958f261c7e2dcf-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.542351+0100 currently delayed
|
|
2021-11-02T16:21:45.747868+0100 osd.7 (osd.7) 3 : cluster [WRN] slow request osd_op(client.31809201.0:43999711 1.603 1:c0623462:::b1-bea5a4be768748369bf3aaa7b5280090-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.304199+0100 currently delayed
|
|
2021-11-02T16:21:45.747887+0100 osd.7 (osd.7) 4 : cluster [WRN] slow request osd_op(client.31809201.0:43999721 1.ea 1:570e0851:::b1-6c057c710a6246c5873ae150188b9df7-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.363418+0100 currently delayed
|
|
2021-11-02T16:21:45.747910+0100 osd.7 (osd.7) 5 : cluster [WRN] slow request osd_op(client.31809201.0:43999787 1.603 1:c0675d60:::b1-dbcf55692f044960b250ec65cfd69b88-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.901891+0100 currently delayed
|
|
2021-11-02T16:21:45.755661+0100 osd.55 (osd.55) 49 : cluster [WRN] slow request osd_op(client.31809201.0:43999745 1.1e5 1:a78f5f83:::b1-3e23be04e57d4aee8d958f261c7e2dcf-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.542351+0100 currently delayed
|
|
2021-11-02T16:21:46.592034+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10908 : cluster [DBG] pgmap v11416: 4129 pgs: 1 active+clean+scrubbing+deep+repair, 14 active+clean+wait, 1 active+clean+scrubbing, 1 active+clean+laggy, 1 active+recovering+degraded, 41 active+recovery_wait+degraded, 18 active+recovery_wait+degraded+wait, 4052 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 66 MiB/s rd, 79 MiB/s wr, 97 op/s; 102/23539515 objects degraded (0.000%); 5.5 MiB/s, 5 objects/s recovering
|
|
2021-11-02T16:21:46.722632+0100 osd.55 (osd.55) 50 : cluster [WRN] slow request osd_op(client.31809201.0:43999745 1.1e5 1:a78f5f83:::b1-3e23be04e57d4aee8d958f261c7e2dcf-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.542351+0100 currently delayed
|
|
2021-11-02T16:21:46.786793+0100 osd.7 (osd.7) 6 : cluster [WRN] slow request osd_op(client.31809201.0:43999711 1.603 1:c0623462:::b1-bea5a4be768748369bf3aaa7b5280090-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.304199+0100 currently delayed
|
|
2021-11-02T16:21:46.786807+0100 osd.7 (osd.7) 7 : cluster [WRN] slow request osd_op(client.31809201.0:43999721 1.ea 1:570e0851:::b1-6c057c710a6246c5873ae150188b9df7-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.363418+0100 currently delayed
|
|
2021-11-02T16:21:46.786812+0100 osd.7 (osd.7) 8 : cluster [WRN] slow request osd_op(client.31809201.0:43999787 1.603 1:c0675d60:::b1-dbcf55692f044960b250ec65cfd69b88-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.901891+0100 currently delayed
|
|
2021-11-02T16:21:47.748323+0100 osd.7 (osd.7) 9 : cluster [WRN] slow request osd_op(client.31809201.0:43999711 1.603 1:c0623462:::b1-bea5a4be768748369bf3aaa7b5280090-junk-0:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.304199+0100 currently delayed
|
|
2021-11-02T16:21:47.748334+0100 osd.7 (osd.7) 10 : cluster [WRN] slow request osd_op(client.31809201.0:43999721 1.ea 1:570e0851:::b1-6c057c710a6246c5873ae150188b9df7-junk-4194304:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.363418+0100 currently delayed
|
|
2021-11-02T16:21:47.748341+0100 osd.7 (osd.7) 11 : cluster [WRN] slow request osd_op(client.31809201.0:43999787 1.603 1:c0675d60:::b1-dbcf55692f044960b250ec65cfd69b88-junk-0:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.901891+0100 currently delayed
|
|
2021-11-02T16:21:47.748359+0100 osd.7 (osd.7) 12 : cluster [WRN] slow request osd_op(client.31809201.0:44000067 1.6c7 1:e36465c5:::b1-3b4722626bf24f00ad0eb7f8f3638c50-index:head [writefull 0~21 in=21b] snapc 0=[] ondisk+write+known_if_redirected e1809258) initiated 2021-11-02T16:21:42.055739+0100 currently delayed
|
|
2021-11-02T16:21:47.748371+0100 osd.7 (osd.7) 13 : cluster [WRN] slow request osd_op(client.31809201.0:43999826 1.a97 1:e95ee9c5:::b1-fbfd73e628794560a54e4a7339c12fee-junk-16777216:head [writefull 0~2134455 in=2134455b] snapc 0=[] ondisk+write+known_if_redirected e1809258) initiated 2021-11-02T16:21:40.253941+0100 currently delayed
|
|
2021-11-02T16:21:47.766518+0100 osd.55 (osd.55) 51 : cluster [WRN] slow request osd_op(client.31809201.0:43999745 1.1e5 1:a78f5f83:::b1-3e23be04e57d4aee8d958f261c7e2dcf-junk-4194304:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected e1809257) initiated 2021-11-02T16:21:39.542351+0100 currently delayed
|
|
2021-11-02T16:21:48.523640+0100 osd.74 (osd.74) 12 : cluster [DBG] 1.bf8 repair ok, 0 fixed
|
|
2021-11-02T16:21:48.615960+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10909 : cluster [DBG] pgmap v11417: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 1 active+clean+laggy, 24 active+clean+wait, 2 active+recovery_wait+degraded, 1 active+recovering+degraded+wait, 7 active+recovery_wait+degraded+wait, 4092 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 109 MiB/s rd, 127 MiB/s wr, 158 op/s; 16/23539443 objects degraded (0.000%); 14 MiB/s, 14 objects/s recovering
|
|
2021-11-02T16:21:48.636008+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7985 : cluster [WRN] Health check update: Degraded data redundancy: 102/23539515 objects degraded (0.000%), 60 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:21:49.394929+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7987 : cluster [WRN] Health check failed: 6 slow ops, oldest one blocked for 8 sec, daemons [osd.55,osd.7] have slow ops. (SLOW_OPS)
|
|
2021-11-02T16:21:50.463970+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7988 : cluster [DBG] osdmap e1809261: 125 total, 125 up, 125 in
|
|
2021-11-02T16:21:50.621164+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10910 : cluster [DBG] pgmap v11419: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 1 active+clean+laggy, 24 active+clean+wait, 2 active+recovery_wait+degraded, 1 active+recovering+degraded+wait, 7 active+recovery_wait+degraded+wait, 4092 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 62 MiB/s rd, 72 MiB/s wr, 90 op/s; 16/23539440 objects degraded (0.000%); 12 MiB/s, 12 objects/s recovering
|
|
2021-11-02T16:21:52.644418+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10911 : cluster [DBG] pgmap v11420: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 24 active+clean+wait, 1 active+recovering+degraded+wait, 7 active+recovery_wait+degraded+wait, 4095 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 111 MiB/s rd, 114 MiB/s wr, 146 op/s; 14/23539452 objects degraded (0.000%); 9.4 MiB/s, 10 objects/s recovering
|
|
2021-11-02T16:21:53.641840+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7989 : cluster [WRN] Health check update: Degraded data redundancy: 14/23539452 objects degraded (0.000%), 8 pgs degraded (PG_DEGRADED)
|
|
2021-11-02T16:21:54.655606+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10912 : cluster [DBG] pgmap v11421: 4129 pgs: 1 active+clean+scrubbing+deep+repair, 4128 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 101 MiB/s rd, 105 MiB/s wr, 131 op/s; 9.1 MiB/s, 10 objects/s recovering
|
|
2021-11-02T16:21:54.720855+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7991 : cluster [WRN] Health check update: 0 slow ops, oldest one blocked for 12 sec, osd.7 has slow ops (SLOW_OPS)
|
|
2021-11-02T16:21:54.720950+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7992 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/23539452 objects degraded (0.000%), 8 pgs degraded)
|
|
2021-11-02T16:21:56.660459+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10913 : cluster [DBG] pgmap v11422: 4129 pgs: 1 active+clean+scrubbing+deep+repair, 4128 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 101 MiB/s rd, 105 MiB/s wr, 131 op/s; 9.1 MiB/s, 10 objects/s recovering
|
|
2021-11-02T16:21:58.695952+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10914 : cluster [DBG] pgmap v11423: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 4127 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 110 MiB/s rd, 113 MiB/s wr, 134 op/s; 1.2 MiB/s, 1 objects/s recovering
|
|
2021-11-02T16:21:59.663103+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7993 : cluster [INF] Health check cleared: SLOW_OPS (was: 0 slow ops, oldest one blocked for 12 sec, osd.7 has slow ops)
|
|
2021-11-02T16:21:59.663202+0100 mon.csdeveubs-u02c01mon01 (mon.0) 7994 : cluster [INF] Cluster is now healthy
|
|
2021-11-02T16:22:00.345157+0100 osd.92 (osd.92) 11 : cluster [DBG] 1.1c scrub ok
|
|
2021-11-02T16:22:00.700763+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10915 : cluster [DBG] pgmap v11424: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 4127 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 108 MiB/s rd, 111 MiB/s wr, 131 op/s; 1.2 MiB/s, 1 objects/s recovering
|
|
2021-11-02T16:22:02.728946+0100 mgr.csdeveubs-u02c01mon03 (mgr.33864150) 10918 : cluster [DBG] pgmap v11425: 4129 pgs: 2 active+clean+scrubbing+deep+repair, 4127 active+clean; 19 TiB data, 57 TiB used, 57 TiB / 114 TiB avail; 125 MiB/s rd, 138 MiB/s wr, 157 op/s; 1.0 MiB/s, 1 objects/s recovering
|