Project

General

Profile

Bug #1064 » ceph_debug_1.txt

Sergey Yudin, 05/05/2011 03:52 AM

 
2011-05-05 14:30:59.807013 mds e3354: 1/1/1 up {0=up:replay}, 10 up:standby
2011-05-05 14:30:59.990227 log 2011-05-05 14:30:59.744081 mon0 1.1.1.67:6789/0 58 : [INF] mds? 1.1.1.80:6804/8948 up:boot
2011-05-05 14:31:13.237626 mds e3355: 1/1/1 up {0=up:reconnect}, 10 up:standby
2011-05-05 14:31:13.420820 log 2011-05-05 14:31:13.207374 mon0 1.1.1.67:6789/0 59 : [INF] mds0 1.1.1.76:6800/9523 up:reconnect
2011-05-05 14:31:26.501615 log 2011-05-05 14:31:26.235591 mon12 1.1.1.71:6789/0 2 : [WRN] message from mon0 was stamped 0.010040s in the future, clocks not synchronized
2011-05-05 14:31:27.567620 log 2011-05-05 14:31:27.130201 mon12 1.1.1.71:6789/0 3 : [WRN] message from mon0 was stamped 0.010022s in the future, clocks not synchronized
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:13.363283 mds0 1.1.1.76:6800/9523 1 : [DBG] reconnect by client24516 1.1.1.66:0/23054 after 0.094388
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:15.867807 mds0 1.1.1.76:6800/9523 2 : [DBG] reconnect by client24641 1.1.1.69:0/7984 after 2.598935
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:18.153301 mds0 1.1.1.76:6800/9523 3 : [DBG] reconnect by client24574 1.1.1.68:0/7829 after 4.884428
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:20.527163 mds0 1.1.1.76:6800/9523 4 : [DBG] reconnect by client24566 1.1.1.70:0/9698 after 7.258289
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:25.283859 mds0 1.1.1.76:6800/9523 5 : [DBG] reconnect by client24593 1.1.1.71:0/7863 after 12.014984
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:27.842778 mon12 1.1.1.71:6789/0 4 : [WRN] message from mon0 was stamped 0.010058s in the future, clocks not synchronized
2011-05-05 14:31:28.675079 log 2011-05-05 14:31:28.173181 mon12 1.1.1.71:6789/0 5 : [WRN] message from mon0 was stamped 0.010026s in the future, clocks not synchronized
2011-05-05 14:31:29.715736 log 2011-05-05 14:31:29.235399 mon12 1.1.1.71:6789/0 6 : [WRN] message from mon0 was stamped 0.010074s in the future, clocks not synchronized
2011-05-05 14:31:31.339490 log 2011-05-05 14:31:31.053777 mon12 1.1.1.71:6789/0 7 : [WRN] message from mon0 was stamped 0.010018s in the future, clocks not synchronized
2011-05-05 14:31:31.938934 osd e984: 30 osds: 30 up, 30 in
2011-05-05 14:31:32.237375 pg v138746: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:32.829830 log 2011-05-05 14:31:32.547305 mon12 1.1.1.71:6789/0 8 : [WRN] message from mon0 was stamped 0.010162s in the future, clocks not synchronized
2011-05-05 14:31:33.519637 pg v138747: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:34.570088 log 2011-05-05 14:31:34.173430 mon12 1.1.1.71:6789/0 9 : [WRN] message from mon0 was stamped 0.010096s in the future, clocks not synchronized
2011-05-05 14:31:34.961734 pg v138748: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:35.988221 pg v138749: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:36.335175 log 2011-05-05 14:31:36.138375 mon12 1.1.1.71:6789/0 10 : [WRN] message from mon0 was stamped 0.010133s in the future, clocks not synchronized
2011-05-05 14:31:36.919837 pg v138750: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:37.984816 pg v138751: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:31:38.816532 log 2011-05-05 14:31:38.594262 mon12 1.1.1.71:6789/0 11 : [WRN] message from mon0 was stamped 0.010183s in the future, clocks not synchronized
2011-05-05 14:31:41.072947 log 2011-05-05 14:31:40.867238 mon12 1.1.1.71:6789/0 12 : [WRN] message from mon0 was stamped 0.010263s in the future, clocks not synchronized
2011-05-05 14:31:41.072947 log 2011-05-05 14:31:40.871735 mon12 1.1.1.71:6789/0 13 : [WRN] message from mon0 was stamped 0.010288s in the future, clocks not synchronized
2011-05-05 14:31:44.916462 log 2011-05-05 14:31:44.005915 mon12 1.1.1.71:6789/0 14 : [WRN] message from mon0 was stamped 0.010374s in the future, clocks not synchronized
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:27.808017 mds0 1.1.1.76:6800/9523 6 : [DBG] reconnect by client24570 1.1.1.83:0/7912 after 14.539147
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:30.437703 mds0 1.1.1.76:6800/9523 7 : [DBG] reconnect by client24532 1.1.1.73:0/8326 after 17.168817
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:34.433514 mds0 1.1.1.76:6800/9523 8 : [DBG] reconnect by client24594 1.1.1.82:0/8248 after 21.164639
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:37.111324 mds0 1.1.1.76:6800/9523 9 : [DBG] reconnect by client24542 1.1.1.77:0/7501 after 23.842448
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:39.874615 mds0 1.1.1.76:6800/9523 10 : [DBG] reconnect by client24607 1.1.1.79:0/7981 after 26.605738
2011-05-05 14:31:45.877298 log 2011-05-05 14:31:42.746612 mds0 1.1.1.76:6800/9523 11 : [DBG] reconnect by client24576 1.1.1.96:0/6823 after 29.477726
2011-05-05 14:31:52.013770 log 2011-05-05 14:31:51.810129 mon12 1.1.1.71:6789/0 15 : [WRN] message from mon0 was stamped 0.010582s in the future, clocks not synchronized
2011-05-05 14:31:59.848935 log 2011-05-05 14:31:59.594693 mon12 1.1.1.71:6789/0 16 : [WRN] message from mon0 was stamped 0.010793s in the future, clocks not synchronized
2011-05-05 14:32:01.487420 log 2011-05-05 14:32:01.174105 mon12 1.1.1.71:6789/0 17 : [WRN] message from mon0 was stamped 0.010711s in the future, clocks not synchronized
2011-05-05 14:32:02.255337 osd e985: 30 osds: 30 up, 30 in
2011-05-05 14:32:02.571881 mds e3356: 1/1/1 up {0=up:replay}, 9 up:standby
2011-05-05 14:32:02.904768 pg v138752: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:03.369651 log 2011-05-05 14:32:03.139008 mon12 1.1.1.71:6789/0 18 : [WRN] message from mon0 was stamped 0.010703s in the future, clocks not synchronized
2011-05-05 14:32:04.135900 pg v138753: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:04.470049 log 2011-05-05 14:31:45.693214 mds0 1.1.1.76:6800/9523 12 : [DBG] reconnect by client24554 1.1.1.97:0/7606 after 32.424339
2011-05-05 14:32:04.470049 log 2011-05-05 14:31:48.710377 mds0 1.1.1.76:6800/9523 13 : [DBG] reconnect by client24632 1.1.1.90:0/8507 after 35.441499
2011-05-05 14:32:04.470049 log 2011-05-05 14:31:53.969673 mds0 1.1.1.76:6800/9523 14 : [DBG] reconnect by client24543 1.1.1.88:0/8437 after 40.700789
2011-05-05 14:32:04.470049 log 2011-05-05 14:31:57.126801 mds0 1.1.1.76:6800/9523 15 : [DBG] reconnect by client24583 1.1.1.89:0/7791 after 43.857923
2011-05-05 14:32:04.470049 log 2011-05-05 14:32:00.356149 mds0 1.1.1.76:6800/9523 16 : [DBG] reconnect by client24560 1.1.1.72:0/8293 after 47.087271
2011-05-05 14:32:05.318580 pg v138754: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:06.027103 log 2011-05-05 14:32:05.813610 mon12 1.1.1.71:6789/0 19 : [WRN] message from mon0 was stamped 0.010655s in the future, clocks not synchronized
2011-05-05 14:32:06.498172 pg v138755: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:07.259448 osd e986: 30 osds: 30 up, 30 in
2011-05-05 14:32:07.550801 pg v138756: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:07.917202 mds e3357: 1/1/1 up {0=up:replay}, 10 up:standby
2011-05-05 14:32:08.100373 log 2011-05-05 14:32:07.900430 mon0 1.1.1.67:6789/0 60 : [INF] mds? 1.1.1.76:6804/9524 up:boot
2011-05-05 14:32:08.791628 pg v138757: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:09.947310 pg v138758: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:10.373791 log 2011-05-05 14:32:10.173760 mon12 1.1.1.71:6789/0 20 : [WRN] message from mon0 was stamped 0.010755s in the future, clocks not synchronized
2011-05-05 14:32:11.035025 pg v138759: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:12.131234 pg v138760: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:13.266255 pg v138761: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:14.312681 pg v138762: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:17.425857 log 2011-05-05 14:32:17.200857 mon12 1.1.1.71:6789/0 21 : [WRN] message from mon0 was stamped 0.010777s in the future, clocks not synchronized
2011-05-05 14:32:20.598292 log 2011-05-05 14:32:20.364639 mon12 1.1.1.71:6789/0 22 : [WRN] message from mon0 was stamped 0.010787s in the future, clocks not synchronized
2011-05-05 14:32:25.127840 mds e3358: 1/1/1 up {0=up:reconnect}, 10 up:standby
2011-05-05 14:32:25.622156 log 2011-05-05 14:32:25.101514 mon0 1.1.1.67:6789/0 61 : [INF] mds0 1.1.1.97:6800/7791 up:reconnect
2011-05-05 14:32:30.281677 log 2011-05-05 14:32:25.243418 mds0 1.1.1.97:6800/7791 1 : [DBG] reconnect by client24554 1.1.1.97:0/7606 after 0.092653
2011-05-05 14:32:37.334113 osd e987: 30 osds: 30 up, 30 in
2011-05-05 14:32:38.017054 pg v138763: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:39.117693 pg v138764: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:40.227280 pg v138765: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:41.418455 pg v138766: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:43.281322 pg v138767: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:44.206145 pg v138768: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:47.433853 osd e988: 30 osds: 30 up, 30 in
2011-05-05 14:32:47.717092 mds e3359: 1/1/1 up {0=up:replay}, 9 up:standby
2011-05-05 14:32:47.816966 pg v138769: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:49.118380 pg v138770: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:50.066599 pg v138771: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:51.243367 pg v138772: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:52.948616 pg v138773: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:53.404056 mds e3360: 1/1/1 up {0=up:replay}, 10 up:standby
2011-05-05 14:32:54.053503 log 2011-05-05 14:32:53.426124 mon0 1.1.1.67:6789/0 62 : [INF] mds? 1.1.1.97:6804/7792 up:boot
2011-05-05 14:32:54.681016 pg v138774: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:32:57.317354 log 2011-05-05 14:32:57.115910 mon18 1.1.1.84:6789/0 2 : [WRN] message from mon0 was stamped 0.010005s in the future, clocks not synchronized
2011-05-05 14:33:02.870221 osd e989: 30 osds: 30 up, 30 in
2011-05-05 14:33:03.301958 pg v138775: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:04.349191 pg v138776: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:05.586934 pg v138777: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:06.625903 log 2011-05-05 14:33:06.116003 mon18 1.1.1.84:6789/0 3 : [WRN] message from mon0 was stamped 0.010052s in the future, clocks not synchronized
2011-05-05 14:33:07.295443 pg v138778: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:07.617160 mds e3361: 1/1/1 up {0=up:reconnect}, 10 up:standby
2011-05-05 14:33:07.933366 log 2011-05-05 14:33:07.058144 mon18 1.1.1.84:6789/0 4 : [WRN] message from mon0 was stamped 0.010012s in the future, clocks not synchronized
2011-05-05 14:33:08.344678 pg v138779: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:08.981437 log 2011-05-05 14:33:07.504839 mon0 1.1.1.67:6789/0 63 : [INF] mds0 1.1.1.77:6804/7154 up:reconnect
2011-05-05 14:33:09.973427 pg v138780: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:10.272911 log 2011-05-05 14:33:08.859723 mon18 1.1.1.84:6789/0 5 : [WRN] message from mon0 was stamped 0.010027s in the future, clocks not synchronized
2011-05-05 14:33:11.039737 pg v138781: 7920 pgs: 7920 active+clean; 489 GB data, 990 GB used, 54832 GB / 55890 GB avail
2011-05-05 14:33:12.471060 log 2011-05-05 14:33:12.193881 mon18 1.1.1.84:6789/0 6 : [WRN] message from mon0 was stamped 0.010108s in the future, clocks not synchronized
(1-1/2)