Project

General

Profile

Actions

Bug #12661

closed

radosgw-agent datasync S3ResponseError: 403 Forbidden dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')

Added by Vasu Kulkarni over 8 years ago. Updated over 8 years ago.

Status:
Duplicate
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

frontend is apache

2015-08-10T15:27:08.645 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,619 17938 [radosgw_agent.sync][INFO  ] 5/5 items processed
2015-08-10T15:27:08.684 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:08] "POST /metadata/full HTTP/1.1" 200 0
2015-08-10T15:27:08.685 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:08] "POST /metadata/full HTTP/1.1" 200 -
2015-08-10T15:27:08.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,717 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:27:08.728 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,718 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:27:08.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,719 17938 [radosgw_agent.sync][INFO  ] preparing to do a full data sync
2015-08-10T15:27:08.732 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,724 17938 [radosgw_agent.sync][DEBUG ] 128 shards to check
2015-08-10T15:27:09.306 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,297 17938 [radosgw_agent.sync][DEBUG ] getting bucket list
2015-08-10T15:27:09.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,307 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:27:09.317 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,307 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:27:09.358 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,309 17938 [radosgw_agent.sync][INFO  ] waiting to make sure bucket log is consistent
2015-08-10T15:27:39.350 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,341 17938 [radosgw_agent.sync][INFO  ] Starting sync
2015-08-10T15:27:39.465 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][DEBUG ] bucket instance is "myfoodata:r0z0.4146.4" with marker 0#00000000002.2.3
2015-08-10T15:27:39.495 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-10T15:27:39.496 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][INFO  ] syncing bucket "myfoodata" 
2015-08-10T15:27:39.537 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,467 20130 [radosgw_agent.worker][DEBUG ] syncing object myfoodata/tiny_file
2015-08-10T15:27:40.261 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] synced 1 objects
2015-08-10T15:27:40.266 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] completed syncing bucket "myfoodata" 
2015-08-10T15:27:40.267 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-10T15:27:40.421 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 20130 [radosgw_agent.worker][INFO  ] finished syncing shard 124
2015-08-10T15:27:40.435 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 17938 [radosgw_agent.sync][DEBUG ] synced item (124, []) successfully
2015-08-10T15:27:40.436 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 20130 [radosgw_agent.worker][INFO  ] No more entries in queue, exiting
2015-08-10T15:27:40.491 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,479 17938 [radosgw_agent.sync][INFO  ] 1/1 items processed
2015-08-10T15:27:40.519 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:40] "POST /data/full HTTP/1.1" 200 0
2015-08-10T15:27:40.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:40] "POST /data/full HTTP/1.1" 200 -
2015-08-10T15:27:40.748 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/radosgw_admin.py", line 472, in task
    dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

2015-08-10T15:27:40.770 INFO:tasks.radosgw_agent:shutting down sync agent on client.0
2015-08-10T15:27:40.782 INFO:teuthology.orchestra.run.magna087:Running: 'rm -rf /home/ubuntu/cephtest/radosgw-agent.client.0'
2015-08-10T15:27:40.904 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/vasu/ds/teuthology/teuthology/contextutil.py", line 30, in nested
    yield vars
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/rgw.py", line 846, in task
    yield
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/radosgw_admin.py", line 472, in task
    dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

Full Logs:

2015-08-10T15:24:17.305 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 17938 [radosgw_agent.sync][DEBUG ] synced item (34, []) successfully
2015-08-10T15:24:17.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 35
2015-08-10T15:24:17.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 17938 [radosgw_agent.sync][INFO  ] 35/64 items processed
2015-08-10T15:24:17.436 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 19792 [radosgw_agent.worker][INFO  ] finished processing shard 35
2015-08-10T15:24:17.438 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 17938 [radosgw_agent.sync][DEBUG ] synced item (35, []) successfully
2015-08-10T15:24:17.479 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 36
2015-08-10T15:24:17.480 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 17938 [radosgw_agent.sync][INFO  ] 36/64 items processed
2015-08-10T15:24:17.712 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 19792 [radosgw_agent.worker][INFO  ] finished processing shard 36
2015-08-10T15:24:17.724 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 17938 [radosgw_agent.sync][DEBUG ] synced item (36, []) successfully
2015-08-10T15:24:17.725 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 37
2015-08-10T15:24:17.765 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 17938 [radosgw_agent.sync][INFO  ] 37/64 items processed
2015-08-10T15:24:17.955 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 19792 [radosgw_agent.worker][INFO  ] finished processing shard 37
2015-08-10T15:24:17.976 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 17938 [radosgw_agent.sync][DEBUG ] synced item (37, []) successfully
2015-08-10T15:24:17.977 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 38
2015-08-10T15:24:18.017 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,946 17938 [radosgw_agent.sync][INFO  ] 38/64 items processed
2015-08-10T15:24:18.130 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 19792 [radosgw_agent.worker][INFO  ] finished processing shard 38
2015-08-10T15:24:18.141 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 17938 [radosgw_agent.sync][DEBUG ] synced item (38, []) successfully
2015-08-10T15:24:18.142 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 39
2015-08-10T15:24:18.183 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,122 17938 [radosgw_agent.sync][INFO  ] 39/64 items processed
2015-08-10T15:24:18.306 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 19792 [radosgw_agent.worker][INFO  ] finished processing shard 39
2015-08-10T15:24:18.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 17938 [radosgw_agent.sync][DEBUG ] synced item (39, []) successfully
2015-08-10T15:24:18.309 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 40
2015-08-10T15:24:18.350 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 17938 [radosgw_agent.sync][INFO  ] 40/64 items processed
2015-08-10T15:24:18.499 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 19792 [radosgw_agent.worker][INFO  ] finished processing shard 40
2015-08-10T15:24:18.518 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 17938 [radosgw_agent.sync][DEBUG ] synced item (40, []) successfully
2015-08-10T15:24:18.519 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 41
2015-08-10T15:24:18.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 17938 [radosgw_agent.sync][INFO  ] 41/64 items processed
2015-08-10T15:24:18.675 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 19792 [radosgw_agent.worker][INFO  ] finished processing shard 41
2015-08-10T15:24:18.685 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 17938 [radosgw_agent.sync][DEBUG ] synced item (41, []) successfully
2015-08-10T15:24:18.686 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 42
2015-08-10T15:24:18.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 17938 [radosgw_agent.sync][INFO  ] 42/64 items processed
2015-08-10T15:24:18.835 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 19792 [radosgw_agent.worker][INFO  ] finished processing shard 42
2015-08-10T15:24:18.871 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 17938 [radosgw_agent.sync][DEBUG ] synced item (42, []) successfully
2015-08-10T15:24:18.912 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 43
2015-08-10T15:24:18.913 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,827 17938 [radosgw_agent.sync][INFO  ] 43/64 items processed
2015-08-10T15:24:18.979 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 19792 [radosgw_agent.worker][INFO  ] finished processing shard 43
2015-08-10T15:24:18.994 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 17938 [radosgw_agent.sync][DEBUG ] synced item (43, []) successfully
2015-08-10T15:24:18.995 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 44
2015-08-10T15:24:19.036 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 17938 [radosgw_agent.sync][INFO  ] 44/64 items processed
2015-08-10T15:24:19.120 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 19792 [radosgw_agent.worker][INFO  ] finished processing shard 44
2015-08-10T15:24:19.127 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 17938 [radosgw_agent.sync][DEBUG ] synced item (44, []) successfully
2015-08-10T15:24:19.127 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 45
2015-08-10T15:24:19.128 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 17938 [radosgw_agent.sync][INFO  ] 45/64 items processed
2015-08-10T15:24:23.249 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 19792 [radosgw_agent.worker][INFO  ] finished processing shard 45
2015-08-10T15:24:23.251 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 46
2015-08-10T15:24:23.252 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 17938 [radosgw_agent.sync][DEBUG ] synced item (45, []) successfully
2015-08-10T15:24:23.292 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 17938 [radosgw_agent.sync][INFO  ] 46/64 items processed
2015-08-10T15:24:24.283 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,274 19792 [radosgw_agent.worker][INFO  ] finished processing shard 46
2015-08-10T15:24:24.303 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 17938 [radosgw_agent.sync][DEBUG ] synced item (46, []) successfully
2015-08-10T15:24:24.304 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 47
2015-08-10T15:24:24.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 17938 [radosgw_agent.sync][INFO  ] 47/64 items processed
2015-08-10T15:24:24.510 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,500 19792 [radosgw_agent.worker][INFO  ] finished processing shard 47
2015-08-10T15:24:24.512 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 17938 [radosgw_agent.sync][DEBUG ] synced item (47, []) successfully
2015-08-10T15:24:24.513 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 48
2015-08-10T15:24:24.554 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 17938 [radosgw_agent.sync][INFO  ] 48/64 items processed
2015-08-10T15:24:24.645 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 19792 [radosgw_agent.worker][INFO  ] finished processing shard 48
2015-08-10T15:24:24.678 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 17938 [radosgw_agent.sync][DEBUG ] synced item (48, []) successfully
2015-08-10T15:24:24.679 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 49
2015-08-10T15:24:24.680 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 17938 [radosgw_agent.sync][INFO  ] 49/64 items processed
2015-08-10T15:24:25.089 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,079 19792 [radosgw_agent.worker][INFO  ] finished processing shard 49
2015-08-10T15:24:25.190 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 50
2015-08-10T15:24:25.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 17938 [radosgw_agent.sync][DEBUG ] synced item (49, []) successfully
2015-08-10T15:24:25.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 17938 [radosgw_agent.sync][INFO  ] 50/64 items processed
2015-08-10T15:24:25.256 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 19792 [radosgw_agent.worker][INFO  ] finished processing shard 50
2015-08-10T15:24:25.273 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 17938 [radosgw_agent.sync][DEBUG ] synced item (50, []) successfully
2015-08-10T15:24:25.274 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 51
2015-08-10T15:24:25.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 17938 [radosgw_agent.sync][INFO  ] 51/64 items processed
2015-08-10T15:24:25.440 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 19792 [radosgw_agent.worker][INFO  ] finished processing shard 51
2015-08-10T15:24:25.445 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 17938 [radosgw_agent.sync][DEBUG ] synced item (51, []) successfully
2015-08-10T15:24:25.446 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 52
2015-08-10T15:24:25.487 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 17938 [radosgw_agent.sync][INFO  ] 52/64 items processed
2015-08-10T15:24:25.592 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 19792 [radosgw_agent.worker][INFO  ] finished processing shard 52
2015-08-10T15:24:25.613 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 17938 [radosgw_agent.sync][DEBUG ] synced item (52, []) successfully
2015-08-10T15:24:25.614 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 53
2015-08-10T15:24:25.655 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 17938 [radosgw_agent.sync][INFO  ] 53/64 items processed
2015-08-10T15:24:25.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,716 19792 [radosgw_agent.worker][INFO  ] finished processing shard 53
2015-08-10T15:24:25.738 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,716 17938 [radosgw_agent.sync][DEBUG ] synced item (53, []) successfully
2015-08-10T15:24:25.739 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,717 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 54
2015-08-10T15:24:25.780 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,717 17938 [radosgw_agent.sync][INFO  ] 54/64 items processed
2015-08-10T15:24:25.910 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 19792 [radosgw_agent.worker][INFO  ] finished processing shard 54
2015-08-10T15:24:25.912 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 17938 [radosgw_agent.sync][DEBUG ] synced item (54, []) successfully
2015-08-10T15:24:25.952 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 55
2015-08-10T15:24:25.953 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 17938 [radosgw_agent.sync][INFO  ] 55/64 items processed
2015-08-10T15:24:26.010 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,001 19792 [radosgw_agent.worker][DEBUG ] syncing metadata type bucket key "mybar" 
2015-08-10T15:24:26.288 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,279 19792 [radosgw_agent.worker][INFO  ] finished processing shard 55
2015-08-10T15:24:26.289 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,279 17938 [radosgw_agent.sync][DEBUG ] synced item (55, []) successfully
2015-08-10T15:24:26.330 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,280 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 56
2015-08-10T15:24:26.431 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,422 17938 [radosgw_agent.sync][INFO  ] 56/64 items processed
2015-08-10T15:24:30.377 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,367 19792 [radosgw_agent.worker][INFO  ] finished processing shard 56
2015-08-10T15:24:30.408 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,368 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 57
2015-08-10T15:24:30.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,367 17938 [radosgw_agent.sync][DEBUG ] synced item (56, []) successfully
2015-08-10T15:24:30.450 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,368 17938 [radosgw_agent.sync][INFO  ] 57/64 items processed
2015-08-10T15:24:31.361 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,351 19792 [radosgw_agent.worker][INFO  ] finished processing shard 57
2015-08-10T15:24:31.367 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 17938 [radosgw_agent.sync][DEBUG ] synced item (57, []) successfully
2015-08-10T15:24:31.368 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 58
2015-08-10T15:24:31.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 17938 [radosgw_agent.sync][INFO  ] 58/64 items processed
2015-08-10T15:24:31.587 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,577 19792 [radosgw_agent.worker][INFO  ] finished processing shard 58
2015-08-10T15:24:31.618 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 17938 [radosgw_agent.sync][DEBUG ] synced item (58, []) successfully
2015-08-10T15:24:31.619 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 59
2015-08-10T15:24:31.659 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 17938 [radosgw_agent.sync][INFO  ] 59/64 items processed
2015-08-10T15:24:31.756 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,746 19792 [radosgw_agent.worker][INFO  ] finished processing shard 59
2015-08-10T15:24:31.782 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 17938 [radosgw_agent.sync][DEBUG ] synced item (59, []) successfully
2015-08-10T15:24:31.783 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 60
2015-08-10T15:24:31.824 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 17938 [radosgw_agent.sync][INFO  ] 60/64 items processed
2015-08-10T15:24:32.173 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 19792 [radosgw_agent.worker][INFO  ] finished processing shard 60
2015-08-10T15:24:32.173 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 17938 [radosgw_agent.sync][DEBUG ] synced item (60, []) successfully
2015-08-10T15:24:32.214 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 61
2015-08-10T15:24:32.215 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 17938 [radosgw_agent.sync][INFO  ] 61/64 items processed
2015-08-10T15:24:32.349 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 19792 [radosgw_agent.worker][INFO  ] finished processing shard 61
2015-08-10T15:24:32.351 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 17938 [radosgw_agent.sync][DEBUG ] synced item (61, []) successfully
2015-08-10T15:24:32.391 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 62
2015-08-10T15:24:32.392 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 17938 [radosgw_agent.sync][INFO  ] 62/64 items processed
2015-08-10T15:24:32.525 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 19792 [radosgw_agent.worker][INFO  ] finished processing shard 62
2015-08-10T15:24:32.550 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 17938 [radosgw_agent.sync][DEBUG ] synced item (62, []) successfully
2015-08-10T15:24:32.550 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 63
2015-08-10T15:24:32.591 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 17938 [radosgw_agent.sync][INFO  ] 63/64 items processed
2015-08-10T15:24:32.692 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,683 19792 [radosgw_agent.worker][INFO  ] finished processing shard 63
2015-08-10T15:24:32.693 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,683 17938 [radosgw_agent.sync][DEBUG ] synced item (63, []) successfully
2015-08-10T15:24:32.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,684 17938 [radosgw_agent.sync][INFO  ] 64/64 items processed
2015-08-10T15:24:32.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:24:32] "POST /metadata/incremental HTTP/1.1" 200 0
2015-08-10T15:24:32.775 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:24:32] "POST /metadata/incremental HTTP/1.1" 200 -
2015-08-10T15:24:32.777 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:32.816 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:32.817 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 metadata get bucket:mybar'
2015-08-10T15:24:34.731 INFO:tasks.util.rgw: json result: {u'data': {u'owner': u'fud', u'creation_time': 1439234556, u'has_bucket_info': u'false', u'bucket': {u'name': u'mybar', u'bucket_id': u'r0z0.4146.3', u'marker': u'r0z0.4146.3', u'data_extra_pool': u'.region0.r0z0.data_extra_pool', u'pool': u'.region0.r0z0.data_pool', u'index_pool': u'.region0.r0z0.index_pool'}, u'linked': u'true'}, u'ver': {u'tag': u'_7e_pwbuFp-pohyTXyMynUDl', u'ver': 1}, u'key': u'bucket:mybar', u'mtime': 1439234256}
2015-08-10T15:24:35.156 INFO:tasks.util.rgw:rgwadmin: client.1 : ['metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:35.157 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', 'metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:35.159 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 metadata get bucket:mybar'
2015-08-10T15:24:37.563 INFO:tasks.util.rgw: json result: {u'data': {u'owner': u'fud', u'creation_time': 1439234556, u'has_bucket_info': u'false', u'bucket': {u'name': u'mybar', u'bucket_id': u'r0z0.4146.3', u'marker': u'r0z0.4146.3', u'data_extra_pool': u'.region0.r0z0.data_extra_pool', u'pool': u'.region0.r0z0.data_pool', u'index_pool': u'.region0.r0z0.index_pool'}, u'linked': u'true'}, u'ver': {u'tag': u'_7e_pwbuFp-pohyTXyMynUDl', u'ver': 1}, u'key': u'bucket:mybar', u'mtime': 1439234256}
2015-08-10T15:24:38.331 INFO:tasks.util.rgw:sync agent magna087.ceph.redhat.com:8000
2015-08-10T15:24:38.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,335 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:24:38.381 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,336 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:24:38.422 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,341 17938 [radosgw_agent.sync][DEBUG ] 64 shards to check
2015-08-10T15:24:53.398 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,389 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:24:53.400 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,390 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:24:53.402 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,392 17938 [radosgw_agent.sync][INFO  ] Starting sync
2015-08-10T15:24:53.403 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,393 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 0
2015-08-10T15:24:53.573 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 19923 [radosgw_agent.worker][INFO  ] finished processing shard 0
2015-08-10T15:24:53.574 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 17938 [radosgw_agent.sync][DEBUG ] synced item (0, []) successfully
2015-08-10T15:24:53.575 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 1
2015-08-10T15:24:53.576 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 17938 [radosgw_agent.sync][INFO  ] 1/64 items processed
2015-08-10T15:24:53.716 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 19923 [radosgw_agent.worker][INFO  ] finished processing shard 1
2015-08-10T15:24:53.717 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 17938 [radosgw_agent.sync][DEBUG ] synced item (1, []) successfully
2015-08-10T15:24:53.718 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 2
2015-08-10T15:24:53.719 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,707 17938 [radosgw_agent.sync][INFO  ] 2/64 items processed
2015-08-10T15:24:53.908 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 19923 [radosgw_agent.worker][INFO  ] finished processing shard 2
2015-08-10T15:24:53.915 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 17938 [radosgw_agent.sync][DEBUG ] synced item (2, []) successfully
2015-08-10T15:24:53.916 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 3
2015-08-10T15:24:53.957 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 17938 [radosgw_agent.sync][INFO  ] 3/64 items processed
2015-08-10T15:24:54.094 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,084 19923 [radosgw_agent.worker][INFO  ] finished processing shard 3
2015-08-10T15:24:54.095 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 17938 [radosgw_agent.sync][DEBUG ] synced item (3, []) successfully
2015-08-10T15:24:54.096 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 4
2015-08-10T15:24:54.097 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 17938 [radosgw_agent.sync][INFO  ] 4/64 items processed
2015-08-10T15:24:54.228 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 19923 [radosgw_agent.worker][INFO  ] finished processing shard 4
2015-08-10T15:24:54.229 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 17938 [radosgw_agent.sync][DEBUG ] synced item (4, []) successfully
2015-08-10T15:24:54.230 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 5
2015-08-10T15:24:54.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 17938 [radosgw_agent.sync][INFO  ] 5/64 items processed
2015-08-10T15:24:54.379 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 19923 [radosgw_agent.worker][INFO  ] finished processing shard 5
2015-08-10T15:24:54.380 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 17938 [radosgw_agent.sync][DEBUG ] synced item (5, []) successfully
2015-08-10T15:24:54.381 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 6
2015-08-10T15:24:54.382 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 17938 [radosgw_agent.sync][INFO  ] 6/64 items processed
2015-08-10T15:24:54.546 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,537 19923 [radosgw_agent.worker][INFO  ] finished processing shard 6
2015-08-10T15:24:54.547 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,537 17938 [radosgw_agent.sync][DEBUG ] synced item (6, []) successfully
2015-08-10T15:24:54.548 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,538 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 7
2015-08-10T15:24:54.549 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,538 17938 [radosgw_agent.sync][INFO  ] 7/64 items processed
2015-08-10T15:24:54.731 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 19923 [radosgw_agent.worker][INFO  ] finished processing shard 7
2015-08-10T15:24:55.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 17938 [radosgw_agent.sync][DEBUG ] synced item (7, []) successfully
2015-08-10T15:24:55.272 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 8
2015-08-10T15:24:55.273 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 17938 [radosgw_agent.sync][INFO  ] 8/64 items processed
2015-08-10T15:25:01.562 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,552 19923 [radosgw_agent.worker][INFO  ] finished processing shard 8
2015-08-10T15:25:01.563 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 9
2015-08-10T15:25:01.564 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 17938 [radosgw_agent.sync][DEBUG ] synced item (8, []) successfully
2015-08-10T15:25:01.565 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 17938 [radosgw_agent.sync][INFO  ] 9/64 items processed
2015-08-10T15:25:01.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 19923 [radosgw_agent.worker][INFO  ] finished processing shard 9
2015-08-10T15:25:01.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 17938 [radosgw_agent.sync][DEBUG ] synced item (9, []) successfully
2015-08-10T15:25:01.730 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 10
2015-08-10T15:25:01.730 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 17938 [radosgw_agent.sync][INFO  ] 10/64 items processed
2015-08-10T15:25:02.154 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,144 19923 [radosgw_agent.worker][INFO  ] finished processing shard 10
2015-08-10T15:25:02.155 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 11
2015-08-10T15:25:02.194 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 17938 [radosgw_agent.sync][DEBUG ] synced item (10, []) successfully
2015-08-10T15:25:02.195 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 17938 [radosgw_agent.sync][INFO  ] 11/64 items processed
2015-08-10T15:25:02.374 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 19923 [radosgw_agent.worker][INFO  ] finished processing shard 11
2015-08-10T15:25:02.414 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 17938 [radosgw_agent.sync][DEBUG ] synced item (11, []) successfully
2015-08-10T15:25:02.415 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 12
2015-08-10T15:25:02.455 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 17938 [radosgw_agent.sync][INFO  ] 12/64 items processed
2015-08-10T15:25:02.534 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 19923 [radosgw_agent.worker][INFO  ] finished processing shard 12
2015-08-10T15:25:02.539 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 17938 [radosgw_agent.sync][DEBUG ] synced item (12, []) successfully
2015-08-10T15:25:02.540 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 13
2015-08-10T15:25:02.580 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 17938 [radosgw_agent.sync][INFO  ] 13/64 items processed
2015-08-10T15:25:02.702 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 19923 [radosgw_agent.worker][INFO  ] finished processing shard 13
2015-08-10T15:25:02.704 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 17938 [radosgw_agent.sync][DEBUG ] synced item (13, []) successfully
2015-08-10T15:25:02.708 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 14
2015-08-10T15:25:02.710 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 17938 [radosgw_agent.sync][INFO  ] 14/64 items processed
2015-08-10T15:25:02.869 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,859 19923 [radosgw_agent.worker][INFO  ] finished processing shard 14
2015-08-10T15:25:02.877 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 17938 [radosgw_agent.sync][DEBUG ] synced item (14, []) successfully
2015-08-10T15:25:02.878 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 15
2015-08-10T15:25:02.882 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 17938 [radosgw_agent.sync][INFO  ] 15/64 items processed
2015-08-10T15:25:03.038 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,028 19923 [radosgw_agent.worker][INFO  ] finished processing shard 15
2015-08-10T15:25:03.040 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,028 17938 [radosgw_agent.sync][DEBUG ] synced item (15, []) successfully
2015-08-10T15:25:03.081 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,029 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 16
2015-08-10T15:25:03.081 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,029 17938 [radosgw_agent.sync][INFO  ] 16/64 items processed
2015-08-10T15:25:03.188 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 19923 [radosgw_agent.worker][INFO  ] finished processing shard 16
2015-08-10T15:25:03.220 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 17938 [radosgw_agent.sync][DEBUG ] synced item (16, []) successfully
2015-08-10T15:25:03.221 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 17
2015-08-10T15:25:03.222 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 17938 [radosgw_agent.sync][INFO  ] 17/64 items processed
2015-08-10T15:25:03.356 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 19923 [radosgw_agent.worker][INFO  ] finished processing shard 17
2015-08-10T15:25:03.368 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 17938 [radosgw_agent.sync][DEBUG ] synced item (17, []) successfully
2015-08-10T15:25:03.369 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 18
2015-08-10T15:25:03.371 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 17938 [radosgw_agent.sync][INFO  ] 18/64 items processed
2015-08-10T15:25:03.549 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,539 19923 [radosgw_agent.worker][INFO  ] finished processing shard 18
2015-08-10T15:25:03.552 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 17938 [radosgw_agent.sync][DEBUG ] synced item (18, []) successfully
2015-08-10T15:25:03.593 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 19
2015-08-10T15:25:03.594 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 17938 [radosgw_agent.sync][INFO  ] 19/64 items processed
2015-08-10T15:25:03.700 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 19923 [radosgw_agent.worker][INFO  ] finished processing shard 19
2015-08-10T15:25:03.731 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 17938 [radosgw_agent.sync][DEBUG ] synced item (19, []) successfully
2015-08-10T15:25:03.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 20
2015-08-10T15:25:03.775 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 17938 [radosgw_agent.sync][INFO  ] 20/64 items processed
2015-08-10T15:25:06.543 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 19923 [radosgw_agent.worker][INFO  ] finished processing shard 20
2015-08-10T15:25:06.544 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 17938 [radosgw_agent.sync][DEBUG ] synced item (20, []) successfully
2015-08-10T15:25:06.545 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 21
2015-08-10T15:25:06.585 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 17938 [radosgw_agent.sync][INFO  ] 21/64 items processed
2015-08-10T15:25:09.023 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 19923 [radosgw_agent.worker][INFO  ] finished processing shard 21
2015-08-10T15:25:09.024 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 22
2015-08-10T15:25:09.025 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 17938 [radosgw_agent.sync][DEBUG ] synced item (21, []) successfully
2015-08-10T15:25:09.026 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 17938 [radosgw_agent.sync][INFO  ] 22/64 items processed
2015-08-10T15:25:09.174 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 19923 [radosgw_agent.worker][INFO  ] finished processing shard 22
2015-08-10T15:25:09.175 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 17938 [radosgw_agent.sync][DEBUG ] synced item (22, []) successfully
2015-08-10T15:25:09.215 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 23
2015-08-10T15:25:09.216 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 17938 [radosgw_agent.sync][INFO  ] 23/64 items processed
2015-08-10T15:25:09.342 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,332 19923 [radosgw_agent.worker][INFO  ] finished processing shard 23
2015-08-10T15:25:09.343 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 17938 [radosgw_agent.sync][DEBUG ] synced item (23, []) successfully
2015-08-10T15:25:09.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 24
2015-08-10T15:25:09.345 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 17938 [radosgw_agent.sync][INFO  ] 24/64 items processed
2015-08-10T15:25:09.559 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 19923 [radosgw_agent.worker][INFO  ] finished processing shard 24
2015-08-10T15:25:09.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 17938 [radosgw_agent.sync][DEBUG ] synced item (24, []) successfully
2015-08-10T15:25:09.561 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 25
2015-08-10T15:25:09.562 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 17938 [radosgw_agent.sync][INFO  ] 25/64 items processed
2015-08-10T15:25:09.718 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 19923 [radosgw_agent.worker][INFO  ] finished processing shard 25
2015-08-10T15:25:10.157 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 17938 [radosgw_agent.sync][DEBUG ] synced item (25, []) successfully
2015-08-10T15:25:10.158 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 26
2015-08-10T15:25:10.199 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,710 17938 [radosgw_agent.sync][INFO  ] 26/64 items processed
2015-08-10T15:25:10.202 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 19923 [radosgw_agent.worker][INFO  ] finished processing shard 26
2015-08-10T15:25:10.241 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 17938 [radosgw_agent.sync][DEBUG ] synced item (26, []) successfully
2015-08-10T15:25:10.242 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 27
2015-08-10T15:25:10.283 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 17938 [radosgw_agent.sync][INFO  ] 27/64 items processed
2015-08-10T15:25:10.284 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,044 19923 [radosgw_agent.worker][INFO  ] finished processing shard 27
2015-08-10T15:25:10.325 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 17938 [radosgw_agent.sync][DEBUG ] synced item (27, []) successfully
2015-08-10T15:25:10.326 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 28
2015-08-10T15:25:10.367 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 17938 [radosgw_agent.sync][INFO  ] 28/64 items processed
2015-08-10T15:25:10.369 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 19923 [radosgw_agent.worker][INFO  ] finished processing shard 28
2015-08-10T15:25:10.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 17938 [radosgw_agent.sync][DEBUG ] synced item (28, []) successfully
2015-08-10T15:25:10.410 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 29
2015-08-10T15:25:10.450 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 17938 [radosgw_agent.sync][INFO  ] 29/64 items processed
2015-08-10T15:25:10.452 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 19923 [radosgw_agent.worker][INFO  ] finished processing shard 29
2015-08-10T15:25:10.491 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 17938 [radosgw_agent.sync][DEBUG ] synced item (29, []) successfully
2015-08-10T15:25:10.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 30
2015-08-10T15:25:10.532 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 17938 [radosgw_agent.sync][INFO  ] 30/64 items processed
2015-08-10T15:25:10.541 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 19923 [radosgw_agent.worker][INFO  ] finished processing shard 30
2015-08-10T15:25:10.574 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 17938 [radosgw_agent.sync][DEBUG ] synced item (30, []) successfully
2015-08-10T15:25:10.575 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 31
2015-08-10T15:25:10.616 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 17938 [radosgw_agent.sync][INFO  ] 31/64 items processed
2015-08-10T15:25:10.725 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 19923 [radosgw_agent.worker][INFO  ] finished processing shard 31
2015-08-10T15:25:10.740 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 17938 [radosgw_agent.sync][DEBUG ] synced item (31, []) successfully
2015-08-10T15:25:10.741 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 32
2015-08-10T15:25:10.781 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 17938 [radosgw_agent.sync][INFO  ] 32/64 items processed
2015-08-10T15:25:10.893 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,883 19923 [radosgw_agent.worker][INFO  ] finished processing shard 32
2015-08-10T15:25:10.907 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 17938 [radosgw_agent.sync][DEBUG ] synced item (32, []) successfully
2015-08-10T15:25:10.908 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 33
2015-08-10T15:25:10.949 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 17938 [radosgw_agent.sync][INFO  ] 33/64 items processed
2015-08-10T15:25:15.756 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 19923 [radosgw_agent.worker][INFO  ] finished processing shard 33
2015-08-10T15:25:15.793 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 17938 [radosgw_agent.sync][DEBUG ] synced item (33, []) successfully
2015-08-10T15:25:15.794 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 34
2015-08-10T15:25:15.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,747 17938 [radosgw_agent.sync][INFO  ] 34/64 items processed
2015-08-10T15:25:20.193 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,183 19923 [radosgw_agent.worker][INFO  ] finished processing shard 34
2015-08-10T15:25:20.223 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,183 17938 [radosgw_agent.sync][DEBUG ] synced item (34, []) successfully
2015-08-10T15:25:20.224 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,184 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 35
2015-08-10T15:25:20.225 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,184 17938 [radosgw_agent.sync][INFO  ] 35/64 items processed
2015-08-10T15:25:20.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 19923 [radosgw_agent.worker][INFO  ] finished processing shard 35
2015-08-10T15:25:20.379 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 17938 [radosgw_agent.sync][DEBUG ] synced item (35, []) successfully
2015-08-10T15:25:20.380 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 36
2015-08-10T15:25:20.391 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 17938 [radosgw
Actions

Also available in: Atom PDF