Actions
Bug #11177
closedmon: scrub messages to go ceph.default.log, not ceph.log
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
hammer:
root@mira054:/var/log/ceph# tail ceph.default.log 2015-03-19 16:21:59.777032 osd.34 10.214.135.108:6804/7308 260 : default [WRN] slow request 30.555227 seconds old, received at 2015-03-19 16:21:29.221697: osd_sub_op(mds.0.121:11363208 1.37 7d2b60b7/300.0007fc92/head//1 [] v 404991'3192517 snapset=0=[]:[] snapc=0=[]) currently commit_sent 2015-03-19 16:21:59.777037 osd.34 10.214.135.108:6804/7308 261 : default [WRN] slow request 30.554268 seconds old, received at 2015-03-19 16:21:29.222656: osd_sub_op(mds.0.121:11363209 1.37 7d2b60b7/300.0007fc92/head//1 [] v 404991'3192518 snapset=0=[]:[] snapc=0=[]) currently commit_sent 2015-03-19 16:21:59.777041 osd.34 10.214.135.108:6804/7308 262 : default [WRN] slow request 30.553102 seconds old, received at 2015-03-19 16:21:29.223821: osd_sub_op(mds.0.121:11363210 1.37 7d2b60b7/300.0007fc92/head//1 [] v 404991'3192519 snapset=0=[]:[] snapc=0=[]) currently commit_sent 2015-03-19 16:21:59.777047 osd.34 10.214.135.108:6804/7308 263 : default [WRN] slow request 30.575679 seconds old, received at 2015-03-19 16:21:29.201245: osd_op(mds.0.121:11363033 1000968dc3d.00000000 [delete] 0.bd4c6365 snapc 1=[] ondisk+write+known_if_redirected e404991) currently commit_sent 2015-03-19 16:22:52.882798 osd.37 10.214.136.136:6806/7853 66 : default [ERR] deep-scrub 0.4a7 d109f4a7/10009eb16aa.00000000/head//0 on disk data digest 0xb7a92750 != 0x767bf2aa 2015-03-19 16:22:52.882802 osd.37 10.214.136.136:6806/7853 67 : default [ERR] deep-scrub 0.4a7 d109f4a7/10009eb16aa.00000000/head//0 on disk omap digest 0x0 != 0xffffffff 2015-03-19 16:22:59.451691 osd.39 10.214.136.136:6811/7846 46 : default [ERR] 0.ed0 deep-scrub 14 errors 2015-03-19 16:23:16.798815 osd.37 10.214.136.136:6806/7853 68 : default [ERR] 0.4a7 deep-scrub 12 errors 2015-03-19 16:23:19.797619 osd.34 10.214.135.108:6804/7308 264 : default [WRN] 1 slow requests, 1 included below; oldest blocked for > 30.568839 secs 2015-03-19 16:23:19.797628 osd.34 10.214.135.108:6804/7308 265 : default [WRN] slow request 30.568839 seconds old, received at 2015-03-19 16:22:49.228729: osd_op(mds.0.121:11371184 10009695530.00000000 [delete] 0.428b84cf snapc 1=[] ondisk+write+known_if_redirected e404991) currently waiting for subops from 11,57 root@mira054:/var/log/ceph# tail ceph.log 2015-03-19 16:24:06.497779 mon.0 10.214.133.104:6789/0 17257 : cluster [INF] pgmap v26964342: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 13920 kB/s rd, 4741 kB/s wr, 85 op/s 2015-03-19 16:24:06.708698 mon.0 10.214.133.104:6789/0 17258 : cluster [INF] HEALTH_ERR; 12 pgs inconsistent; 2 near full osd(s); 164 scrub errors; mds0: Client 1756771 failing to respond to cache pressure; mds0: Client 9664616 failing to respond to cache pressure; mds0: Client slider failing to respond to cache pressure; noout flag(s) set 2015-03-19 16:24:08.511257 mon.0 10.214.133.104:6789/0 17259 : cluster [INF] pgmap v26964343: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 1590 kB/s wr, 6 op/s 2015-03-19 16:24:09.557189 mon.0 10.214.133.104:6789/0 17260 : cluster [INF] pgmap v26964344: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 6718 kB/s rd, 19454 kB/s wr, 62 op/s 2015-03-19 16:24:10.584664 mon.0 10.214.133.104:6789/0 17262 : cluster [INF] pgmap v26964345: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 32134 kB/s rd, 73135 kB/s wr, 182 op/s 2015-03-19 16:24:11.597511 mon.0 10.214.133.104:6789/0 17263 : cluster [INF] pgmap v26964346: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 24124 kB/s rd, 48551 kB/s wr, 105 op/s 2015-03-19 16:24:13.609214 mon.0 10.214.133.104:6789/0 17266 : cluster [INF] pgmap v26964347: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 1342 kB/s rd, 4080 kB/s wr, 20 op/s 2015-03-19 16:24:14.679500 mon.0 10.214.133.104:6789/0 17267 : cluster [INF] pgmap v26964348: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 13418 kB/s rd, 63927 kB/s wr, 192 op/s 2015-03-19 16:24:15.693020 mon.0 10.214.133.104:6789/0 17268 : cluster [INF] pgmap v26964349: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 34129 kB/s rd, 139 MB/s wr, 405 op/s 2015-03-19 16:24:16.738218 mon.0 10.214.133.104:6789/0 17270 : cluster [INF] pgmap v26964350: 8384 pgs: 8372 active+clean, 12 active+clean+inconsistent; 19895 GB data, 60432 GB used, 22340 GB / 82772 GB avail; 14123 kB/s rd, 49207 kB/s wr, 143 op/s
Actions