Bug #7635
closedfailed to recover before timeout expired
0%
Description
http://pulpito.ceph.com/teuthology-2014-03-06_02:30:01-rados-firefly-distro-basic-plana/118391/
Strange it looks like there is 1 pg in active+recovery for 20 minutes, but it didn't decide to fail for another 20 minutes with no more status output. I show the first and last ceph status. There are no cores present. I don't recognize anything at end of any of ceph-osd logs that would indicate a problem.
2014-03-06T05:09:51.324 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'b', u'rank': 0, u'addr': u'10.214.132.32:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.132.35:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.132.35:6790/0'}], u'modified': u'2014-03-06 05:07:54.397557', u'fsid': u'4d3e5e97-e4f3-41ef-ace4-fab290897535', u'created': u'2014-03-06 05:07:54.397557'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 2, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.063286', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.135932', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-03-06 05:09:04.996692', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436908544, u'health': u'HEALTH_OK', u'kb_used': 11436824, u'store_stats': {u'bytes_total': 2098972, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}, {u'last_updated': u'2014-03-06 05:09:05.104731', u'name': u'a', u'avail_percent': 91, u'kb_total': 472345880, u'kb_avail': 430407420, u'health': u'HEALTH_OK', u'kb_used': 17937948, u'store_stats': {u'bytes_total': 2098972, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}, {u'last_updated': u'2014-03-06 05:09:05.123374', u'name': u'c', u'avail_percent': 91, u'kb_total': 472345880, u'kb_avail': 430407420, u'health': u'HEALTH_OK', u'kb_used': 17937948, u'store_stats': {u'bytes_total': 2098972, u'bytes_log': 2031616, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 842/2639 objects degraded (31.906%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2000431448064, u'recovering_bytes_per_sec': 13, u'degraded_objects': 842, u'num_pgs': 103, u'recovering_keys_per_sec': 0, u'data_bytes': 3611298148, u'degraded_total': 2639, u'bytes_used': 4099022848, u'recovering_objects_per_sec': 0, u'version': 53, u'pgs_by_state': [{u'count': 102, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'31.906', u'bytes_avail': 1994147061760}, u'quorum_names': [u'b', u'a', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 26, u'num_in_osds': 5}}, u'fsid': u'4d3e5e97-e4f3-41ef-ace4-fab290897535'}
.......
2014-03-06T05:29:51.377 INFO:teuthology.task.thrashosds.ceph_manager:{u'election_epoch': 6, u'quorum': [0, 1, 2], u'mdsmap': {u'max': 1, u'epoch': 5, u'by_rank': [{u'status': u'up:active', u'name': u'a', u'rank': 0}], u'up': 1, u'in': 1}, u'monmap': {u'epoch': 1, u'mons': [{u'name': u'b', u'rank': 0, u'addr': u'10.214.132.32:6789/0'}, {u'name': u'a', u'rank': 1, u'addr': u'10.214.132.35:6789/0'}, {u'name': u'c', u'rank': 2, u'addr': u'10.214.132.35:6790/0'}], u'modified': u'2014-03-06 05:07:54.397557', u'fsid': u'4d3e5e97-e4f3-41ef-ace4-fab290897535', u'created': u'2014-03-06 05:07:54.397557'}, u'health': {u'detail': [], u'timechecks': {u'round_status': u'finished', u'epoch': 6, u'round': 10, u'mons': [{u'latency': u'0.000000', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'b'}, {u'latency': u'0.026384', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'a'}, {u'latency': u'0.056143', u'skew': u'0.000000', u'health': u'HEALTH_OK', u'name': u'c'}]}, u'health': {u'health_services': [{u'mons': [{u'last_updated': u'2014-03-06 05:29:05.008870', u'name': u'b', u'avail_percent': 92, u'kb_total': 472345880, u'kb_avail': 436164060, u'health': u'HEALTH_OK', u'kb_used': 12181308, u'store_stats': {u'bytes_total': 17827612, u'bytes_log': 17760256, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}, {u'last_updated': u'2014-03-06 05:29:05.109928', u'name': u'a', u'avail_percent': 91, u'kb_total': 472345880, u'kb_avail': 430298320, u'health': u'HEALTH_OK', u'kb_used': 18047048, u'store_stats': {u'bytes_total': 26216220, u'bytes_log': 26148864, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}, {u'last_updated': u'2014-03-06 05:29:05.128563', u'name': u'c', u'avail_percent': 91, u'kb_total': 472345880, u'kb_avail': 430298316, u'health': u'HEALTH_OK', u'kb_used': 18047052, u'store_stats': {u'bytes_total': 26216220, u'bytes_log': 26148864, u'last_updated': u'0.000000', u'bytes_misc': 65552, u'bytes_sst': 1804}}]}]}, u'overall_status': u'HEALTH_WARN', u'summary': [{u'severity': u'HEALTH_WARN', u'summary': u'1 pgs recovering'}, {u'severity': u'HEALTH_WARN', u'summary': u'1 pgs stuck unclean'}, {u'severity': u'HEALTH_WARN', u'summary': u'58 requests are blocked > 32 sec'}, {u'severity': u'HEALTH_WARN', u'summary': u'recovery 4116/7805 objects degraded (52.735%)'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool data pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool metadata pg_num 34 > pgp_num 24'}, {u'severity': u'HEALTH_WARN', u'summary': u'pool rbd pg_num 34 > pgp_num 24'}]}, u'pgmap': {u'bytes_total': 2500539310080, u'degraded_objects': 4116, u'num_pgs': 103, u'data_bytes': 10833889636, u'degraded_total': 7805, u'bytes_used': 15092121600, u'op_per_sec': 1, u'write_bytes_sec': 6201483, u'version': 325, u'pgs_by_state': [{u'count': 102, u'state_name': u'active+clean'}, {u'count': 1, u'state_name': u'active+recovering'}], u'degraded_ratio': u'52.735', u'bytes_avail': 2481132896256}, u'quorum_names': [u'b', u'a', u'c'], u'osdmap': {u'osdmap': {u'full': u'false', u'nearfull': u'false', u'num_osds': 6, u'num_up_osds': 6, u'epoch': 26, u'num_in_osds': 5}}, u'fsid': u'4d3e5e97-e4f3-41ef-ace4-fab290897535'}
.......
2014-03-06T05:41:39.124 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.132.35]: 2014-03-06 05:41:39.123820 7f20462f0700 0 -- 10.214.132.35:0/1027349 >> 10.214.132.32:6805/7736 pipe(0x7f204001fb90 sd=10 :41719 s=2 pgs=24 cs=1 l=1 c=0x7f204001fec0).injecting socket failure
2014-03-06T05:41:46.985 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.132.35]: 2014-03-06 05:41:46.984511 7f20462f0700 0 -- 10.214.132.35:0/1027349 >> 10.214.132.32:6805/7736 pipe(0x7f204001dce0 sd=10 :41724 s=2 pgs=29 cs=1 l=1 c=0x7f204001a160).injecting socket failure
2014-03-06T05:41:49.184 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.132.35]: 2014-03-06 05:41:49.183615 7f2047fff700 0 -- 10.214.132.35:0/1027349 >> 10.214.132.32:6805/7736 pipe(0x7f204001dce0 sd=10 :41726 s=2 pgs=30 cs=1 l=1 c=0x7f204001f540).injecting socket failure
2014-03-06T05:41:52.624 INFO:teuthology.task.radosbench.radosbench.0.err:[10.214.132.35]: 2014-03-06 05:41:52.623492 7f20462f0700 0 -- 10.214.132.35:0/1027349 >> 10.214.132.32:6805/7736 pipe(0x7f204000cd60 sd=10 :41729 s=2 pgs=33 cs=1 l=1 c=0x7f204001ab20).injecting socket failure
2014-03-06T05:41:57.346 INFO:teuthology.task.thrashosds.ceph_manager:creating pool_name unique_pool_0
2014-03-06T05:41:57.347 DEBUG:teuthology.orchestra.run:Running [10.214.132.35]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados rmpool unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it'
2014-03-06T05:41:58.110 INFO:teuthology.orchestra.run.out:[10.214.132.35]: successfully deleted pool unique_pool_0
2014-03-06T05:41:58.112 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds
2014-03-06T05:41:58.112 INFO:teuthology.task.thrashosds:joining thrashosds
2014-03-06T05:41:58.112 ERROR:teuthology.run_tasks:Manager failed: thrashosds
Traceback (most recent call last):
File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 84, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in exit
self.gen.next()
File "/home/teuthworker/teuthology-firefly/teuthology/task/thrashosds.py", line 172, in task
thrash_proc.do_join()
File "/home/teuthworker/teuthology-firefly/teuthology/task/ceph_manager.py", line 153, in do_join
self.thread.get()
File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
raise self._exception
AssertionError: failed to recover before timeout expired
Updated by Samuel Just about 10 years ago
It did eventually recover, as it turns out.
2014-03-06 05:40:41.795123 mon.0 10.214.132.32:6789/0 521 : [INF] pgmap v474: 103 pgs: 102 active+clean, 1 active+recovering; 11884 MB data, 19850 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 4 op/s; 4673/8975 objects degraded (52.067%); 19658 kB/s, 4 objects/s recovering
2014-03-06 05:40:46.787480 mon.0 10.214.132.32:6789/0 522 : [INF] pgmap v475: 103 pgs: 102 active+clean, 1 active+recovering; 11812 MB data, 19902 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 4 op/s; 4618/8921 objects degraded (51.765%); 19657 kB/s, 4 objects/s recovering
2014-03-06 05:40:46.787480 mon.0 10.214.132.32:6789/0 522 : [INF] pgmap v475: 103 pgs: 102 active+clean, 1 active+recovering; 11812 MB data, 19902 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 4 op/s; 4618/8921 objects degraded (51.765%); 19657 kB/s, 4 objects/s recovering
2014-03-06 05:40:51.796800 mon.0 10.214.132.32:6789/0 523 : [INF] pgmap v476: 103 pgs: 102 active+clean, 1 active+recovering; 11700 MB data, 19897 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 4 op/s; 4532/8837 objects degraded (51.284%); 20067 kB/s, 4 objects/s recovering
2014-03-06 05:40:51.796800 mon.0 10.214.132.32:6789/0 523 : [INF] pgmap v476: 103 pgs: 102 active+clean, 1 active+recovering; 11700 MB data, 19897 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 4 op/s; 4532/8837 objects degraded (51.284%); 20067 kB/s, 4 objects/s recovering
2014-03-06 05:40:56.780646 mon.0 10.214.132.32:6789/0 524 : [INF] pgmap v477: 103 pgs: 102 active+clean, 1 active+recovering; 11592 MB data, 20012 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4449/8756 objects degraded (50.811%); 24162 kB/s, 5 objects/s recovering
2014-03-06 05:40:56.780646 mon.0 10.214.132.32:6789/0 524 : [INF] pgmap v477: 103 pgs: 102 active+clean, 1 active+recovering; 11592 MB data, 20012 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4449/8756 objects degraded (50.811%); 24162 kB/s, 5 objects/s recovering
2014-03-06 05:41:01.798240 mon.0 10.214.132.32:6789/0 525 : [INF] pgmap v478: 103 pgs: 102 active+clean, 1 active+recovering; 11488 MB data, 20065 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4373/8678 objects degraded (50.392%); 21704 kB/s, 5 objects/s recovering
2014-03-06 05:41:01.798240 mon.0 10.214.132.32:6789/0 525 : [INF] pgmap v478: 103 pgs: 102 active+clean, 1 active+recovering; 11488 MB data, 20065 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4373/8678 objects degraded (50.392%); 21704 kB/s, 5 objects/s recovering
2014-03-06 05:41:05.107150 mon.0 10.214.132.32:6789/0 526 : [INF] pgmap v479: 103 pgs: 102 active+clean, 1 active+recovering; 11488 MB data, 20065 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 3 op/s; 4373/8678 objects degraded (50.392%); 11811 kB/s, 2 objects/s recovering
2014-03-06 05:41:05.107150 mon.0 10.214.132.32:6789/0 526 : [INF] pgmap v479: 103 pgs: 102 active+clean, 1 active+recovering; 11488 MB data, 20065 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 3 op/s; 4373/8678 objects degraded (50.392%); 11811 kB/s, 2 objects/s recovering
2014-03-06 05:41:07.266020 mon.0 10.214.132.32:6789/0 527 : [INF] pgmap v480: 103 pgs: 102 active+clean, 1 active+recovering; 11364 MB data, 20112 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4283/8585 objects degraded (49.889%); 20985 kB/s, 5 objects/s recovering
2014-03-06 05:41:07.266020 mon.0 10.214.132.32:6789/0 527 : [INF] pgmap v480: 103 pgs: 102 active+clean, 1 active+recovering; 11364 MB data, 20112 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 5 op/s; 4283/8585 objects degraded (49.889%); 20985 kB/s, 5 objects/s recovering
2014-03-06 05:41:11.826639 mon.0 10.214.132.32:6789/0 528 : [INF] pgmap v481: 103 pgs: 102 active+clean, 1 active+recovering; 11236 MB data, 20155 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 9 op/s; 4187/8489 objects degraded (49.323%); 36792 kB/s, 8 objects/s recovering
2014-03-06 05:41:11.826639 mon.0 10.214.132.32:6789/0 528 : [INF] pgmap v481: 103 pgs: 102 active+clean, 1 active+recovering; 11236 MB data, 20155 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 9 op/s; 4187/8489 objects degraded (49.323%); 36792 kB/s, 8 objects/s recovering
2014-03-06 05:41:16.802233 mon.0 10.214.132.32:6789/0 529 : [INF] pgmap v482: 103 pgs: 102 active+clean, 1 active+recovering; 11092 MB data, 20256 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 7 op/s; 4068/8381 objects degraded (48.538%); 33924 kB/s, 8 objects/s recovering
2014-03-06 05:41:16.802233 mon.0 10.214.132.32:6789/0 529 : [INF] pgmap v482: 103 pgs: 102 active+clean, 1 active+recovering; 11092 MB data, 20256 MB used, 2305 GB / 2328 GB avail; 0 B/s wr, 7 op/s; 4068/8381 objects degraded (48.538%); 33924 kB/s, 8 objects/s recovering
2014-03-06 05:41:21.795085 mon.0 10.214.132.32:6789/0 530 : [INF] pgmap v483: 103 pgs: 102 active+clean, 1 active+recovering; 10892 MB data, 20366 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 8 op/s; 3915/8231 objects degraded (47.564%); 40954 kB/s, 9 objects/s recovering
2014-03-06 05:41:21.795085 mon.0 10.214.132.32:6789/0 530 : [INF] pgmap v483: 103 pgs: 102 active+clean, 1 active+recovering; 10892 MB data, 20366 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 8 op/s; 3915/8231 objects degraded (47.564%); 40954 kB/s, 9 objects/s recovering
2014-03-06 05:41:26.795848 mon.0 10.214.132.32:6789/0 531 : [INF] pgmap v484: 103 pgs: 102 active+clean, 1 active+recovering; 10652 MB data, 20479 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 10 op/s; 3744/8051 objects degraded (46.504%); 42592 kB/s, 10 objects/s recovering
2014-03-06 05:41:26.795848 mon.0 10.214.132.32:6789/0 531 : [INF] pgmap v484: 103 pgs: 102 active+clean, 1 active+recovering; 10652 MB data, 20479 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 10 op/s; 3744/8051 objects degraded (46.504%); 42592 kB/s, 10 objects/s recovering
2014-03-06 05:41:31.788068 mon.0 10.214.132.32:6789/0 532 : [INF] pgmap v485: 103 pgs: 102 active+clean, 1 active+recovering; 10424 MB data, 20633 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 11 op/s; 3578/7880 objects degraded (45.406%); 42178 kB/s, 10 objects/s recovering
2014-03-06 05:41:31.788068 mon.0 10.214.132.32:6789/0 532 : [INF] pgmap v485: 103 pgs: 102 active+clean, 1 active+recovering; 10424 MB data, 20633 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 11 op/s; 3578/7880 objects degraded (45.406%); 42178 kB/s, 10 objects/s recovering
2014-03-06 05:41:36.788677 mon.0 10.214.132.32:6789/0 533 : [INF] pgmap v486: 103 pgs: 102 active+clean, 1 active+recovering; 10056 MB data, 20734 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 14 op/s; 3319/7604 objects degraded (43.648%); 52011 kB/s, 12 objects/s recovering
2014-03-06 05:41:36.788677 mon.0 10.214.132.32:6789/0 533 : [INF] pgmap v486: 103 pgs: 102 active+clean, 1 active+recovering; 10056 MB data, 20734 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 14 op/s; 3319/7604 objects degraded (43.648%); 52011 kB/s, 12 objects/s recovering
2014-03-06 05:41:41.798212 mon.0 10.214.132.32:6789/0 534 : [INF] pgmap v487: 103 pgs: 103 active+clean; 7712 MB data, 20788 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 67 op/s; 34402 kB/s, 8 objects/s recovering
2014-03-06 05:41:41.798212 mon.0 10.214.132.32:6789/0 534 : [INF] pgmap v487: 103 pgs: 103 active+clean; 7712 MB data, 20788 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 67 op/s; 34402 kB/s, 8 objects/s recovering
2014-03-06 05:41:46.781812 mon.0 10.214.132.32:6789/0 535 : [INF] pgmap v488: 103 pgs: 103 active+clean; 5368 MB data, 20801 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 117 op/s; 3685 kB/s, 0 objects/s recovering
2014-03-06 05:41:46.781812 mon.0 10.214.132.32:6789/0 535 : [INF] pgmap v488: 103 pgs: 103 active+clean; 5368 MB data, 20801 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 117 op/s; 3685 kB/s, 0 objects/s recovering
2014-03-06 05:41:47.957663 mon.0 10.214.132.32:6789/0 536 : [INF] pgmap v489: 103 pgs: 103 active+clean; 5368 MB data, 20809 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 96 op/s
2014-03-06 05:41:47.957663 mon.0 10.214.132.32:6789/0 536 : [INF] pgmap v489: 103 pgs: 103 active+clean; 5368 MB data, 20809 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 96 op/s
2014-03-06 05:41:51.493038 mon.0 10.214.132.32:6789/0 537 : [INF] pgmap v490: 103 pgs: 103 active+clean; 5368 MB data, 20817 MB used, 2304 GB / 2328 GB avail
2014-03-06 05:41:51.493038 mon.0 10.214.132.32:6789/0 537 : [INF] pgmap v490: 103 pgs: 103 active+clean; 5368 MB data, 20817 MB used, 2304 GB / 2328 GB avail
2014-03-06 05:41:52.701672 mon.0 10.214.132.32:6789/0 538 : [INF] pgmap v491: 103 pgs: 103 active+clean; 2800 MB data, 20807 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 136 op/s
2014-03-06 05:41:52.701672 mon.0 10.214.132.32:6789/0 538 : [INF] pgmap v491: 103 pgs: 103 active+clean; 2800 MB data, 20807 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 136 op/s
2014-03-06 05:41:56.793956 mon.0 10.214.132.32:6789/0 539 : [INF] pgmap v492: 103 pgs: 103 active+clean; 304 MB data, 20811 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 238 op/s
2014-03-06 05:41:56.793956 mon.0 10.214.132.32:6789/0 539 : [INF] pgmap v492: 103 pgs: 103 active+clean; 304 MB data, 20811 MB used, 2304 GB / 2328 GB avail; 0 B/s wr, 238 op/s
2014-03-06 05:41:58.019405 mon.0 10.214.132.32:6789/0 540 : [INF] osdmap e27: 6 osds: 6 up, 5 in