INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.sync:synced shard 59
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:radosgw_agent.worker:30748 is processing shard number 60
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:radosgw_agent.sync:60/64 shards processed
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.lock:set_shard to 60
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.lock:acquire lock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:path=//admin/log?lock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:auth_path=//admin/log?lock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: POST
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: application/json; charset=UTF-8
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/log
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira037.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/log?lock&locker-id=mira03730462&length=60&zone-id=r0z1&type=metadata&id=60 HTTP/1.1" 200 None
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:path=//admin/replica_log?bounds
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:auth_path=//admin/replica_log?bounds
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: GET
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/replica_log
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira053.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/replica_log?bounds&type=metadata&id=60 HTTP/1.1" 404 20
INFO:teuthology.task.rgw.client.1.out:[10.214.133.106]: 2013-08-18 19:11:11.612117 7ff8f97b2700 -1 ERROR: could not open rados pool .region1.r1z1.log_pool
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: GET
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/log
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira037.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/log?marker=&type=metadata&id=60&max-entries=1000 HTTP/1.1" 200 None
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:radosgw_agent.worker:shard 60 has 6 entries after ''
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.worker:syncing metadata type bucket.instance key u'myfoo:r0z1.4137.1'
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: GET
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/metadata/bucket.instance
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira037.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/metadata/bucket.instance?key=myfoo%3Ar0z1.4137.1 HTTP/1.1" 200 None
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: PUT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: application/json; charset=UTF-8
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/metadata/bucket.instance
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira053.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"PUT /admin/metadata/bucket.instance?key=myfoo%3Ar0z1.4137.1 HTTP/1.1" 204 0
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.worker:syncing metadata type bucket key u'myfoo'
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: GET
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/metadata/bucket
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira037.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"GET /admin/metadata/bucket?key=myfoo HTTP/1.1" 404 20
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.worker:bucket u'myfoo' not found on master, deleting from secondary
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DELETE
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/metadata/bucket
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira053.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"DELETE /admin/metadata/bucket?key=myfoo HTTP/1.1" 200 None
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:path=//admin/replica_log?work_bound
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:auth_path=//admin/replica_log?work_bound
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: POST
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: application/json; charset=UTF-8
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/replica_log
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira053.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/replica_log?work_bound&marker=1_1376878269.389829_86.1&time=2013-08-19+02%3A11%3A09.389829Z&type=metadata&id=60&daemon_id=ubuntu%40mira037.front.sepia.ceph.com.8000.syncdaemon HTTP/1.1" 404 20
INFO:teuthology.task.rgw.client.1.out:[10.214.133.106]: 2013-08-18 19:11:11.676214 7ff90efdd700 -1 ERROR: could not open rados pool .region1.r1z1.log_pool
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: ERROR:radosgw_agent.worker:error setting worker bound, may duplicate some work later
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Traceback (most recent call last):
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:   File "/home/ubuntu/cephtest/bu1308181909/radosgw-agent.client.0/radosgw_agent/worker.py", line 114, in _get_and_process_entries
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:     self.daemon_id)
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:   File "/home/ubuntu/cephtest/bu1308181909/radosgw-agent.client.0/radosgw_agent/client.py", line 157, in set_worker_bound
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:     special_first_param='work_bound',
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:   File "/home/ubuntu/cephtest/bu1308181909/radosgw-agent.client.0/radosgw_agent/client.py", line 72, in request
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]:     HttpError)(result.status_code, result.content)
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: NotFound: Http error code 404 content {"Code":"NoSuchKey"}
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:radosgw_agent.lock:release and clear lock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:path=//admin/log?unlock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:auth_path=//admin/log?unlock
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:boto:StringToSign:
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: POST
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: 
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: application/json; charset=UTF-8
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: Mon, 19 Aug 2013 02:11:11 GMT
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: /admin/log
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): mira037.front.sepia.ceph.com
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/log?unlock&locker-id=mira03730462&type=metadata&id=60&zone-id=r0z1 HTTP/1.1" 200 None
INFO:teuthology.task.radosgw-agent.ubuntu@mira037.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.133.140]: INFO:radosgw_agent.worker:finished processing shard 60