Project

General

Profile

Actions
Copy link

Bug #12661

closed

radosgw-agent datasync S3ResponseError: 403 Forbidden dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')

Added by Vasu Kulkarni over 8 years ago. Updated over 8 years ago.

Status:
Duplicate
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

frontend is apache

2015-08-10T15:27:08.645 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,619 17938 [radosgw_agent.sync][INFO  ] 5/5 items processed
2015-08-10T15:27:08.684 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:08] "POST /metadata/full HTTP/1.1" 200 0
2015-08-10T15:27:08.685 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:08] "POST /metadata/full HTTP/1.1" 200 -
2015-08-10T15:27:08.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,717 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:27:08.728 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,718 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:27:08.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,719 17938 [radosgw_agent.sync][INFO  ] preparing to do a full data sync
2015-08-10T15:27:08.732 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:08,724 17938 [radosgw_agent.sync][DEBUG ] 128 shards to check
2015-08-10T15:27:09.306 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,297 17938 [radosgw_agent.sync][DEBUG ] getting bucket list
2015-08-10T15:27:09.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,307 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:27:09.317 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,307 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:27:09.358 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:09,309 17938 [radosgw_agent.sync][INFO  ] waiting to make sure bucket log is consistent
2015-08-10T15:27:39.350 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,341 17938 [radosgw_agent.sync][INFO  ] Starting sync
2015-08-10T15:27:39.465 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][DEBUG ] bucket instance is "myfoodata:r0z0.4146.4" with marker 0#00000000002.2.3
2015-08-10T15:27:39.495 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-10T15:27:39.496 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,456 20130 [radosgw_agent.worker][INFO  ] syncing bucket "myfoodata" 
2015-08-10T15:27:39.537 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:39,467 20130 [radosgw_agent.worker][DEBUG ] syncing object myfoodata/tiny_file
2015-08-10T15:27:40.261 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] synced 1 objects
2015-08-10T15:27:40.266 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] completed syncing bucket "myfoodata" 
2015-08-10T15:27:40.267 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,252 20130 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-10T15:27:40.421 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 20130 [radosgw_agent.worker][INFO  ] finished syncing shard 124
2015-08-10T15:27:40.435 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 17938 [radosgw_agent.sync][DEBUG ] synced item (124, []) successfully
2015-08-10T15:27:40.436 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,412 20130 [radosgw_agent.worker][INFO  ] No more entries in queue, exiting
2015-08-10T15:27:40.491 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:27:40,479 17938 [radosgw_agent.sync][INFO  ] 1/1 items processed
2015-08-10T15:27:40.519 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:40] "POST /data/full HTTP/1.1" 200 0
2015-08-10T15:27:40.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:27:40] "POST /data/full HTTP/1.1" 200 -
2015-08-10T15:27:40.748 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/radosgw_admin.py", line 472, in task
    dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

2015-08-10T15:27:40.770 INFO:tasks.radosgw_agent:shutting down sync agent on client.0
2015-08-10T15:27:40.782 INFO:teuthology.orchestra.run.magna087:Running: 'rm -rf /home/ubuntu/cephtest/radosgw-agent.client.0'
2015-08-10T15:27:40.904 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/vasu/ds/teuthology/teuthology/contextutil.py", line 30, in nested
    yield vars
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/rgw.py", line 846, in task
    yield
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/vasu/ds/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/ceph-qa-suite_rh130/tasks/radosgw_admin.py", line 472, in task
    dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/home/vasu/ds/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
S3ResponseError: S3ResponseError: 403 Forbidden

Full Logs:

2015-08-10T15:24:17.305 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 17938 [radosgw_agent.sync][DEBUG ] synced item (34, []) successfully
2015-08-10T15:24:17.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 35
2015-08-10T15:24:17.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,293 17938 [radosgw_agent.sync][INFO  ] 35/64 items processed
2015-08-10T15:24:17.436 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 19792 [radosgw_agent.worker][INFO  ] finished processing shard 35
2015-08-10T15:24:17.438 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 17938 [radosgw_agent.sync][DEBUG ] synced item (35, []) successfully
2015-08-10T15:24:17.479 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 36
2015-08-10T15:24:17.480 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,427 17938 [radosgw_agent.sync][INFO  ] 36/64 items processed
2015-08-10T15:24:17.712 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 19792 [radosgw_agent.worker][INFO  ] finished processing shard 36
2015-08-10T15:24:17.724 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 17938 [radosgw_agent.sync][DEBUG ] synced item (36, []) successfully
2015-08-10T15:24:17.725 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 37
2015-08-10T15:24:17.765 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,703 17938 [radosgw_agent.sync][INFO  ] 37/64 items processed
2015-08-10T15:24:17.955 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 19792 [radosgw_agent.worker][INFO  ] finished processing shard 37
2015-08-10T15:24:17.976 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 17938 [radosgw_agent.sync][DEBUG ] synced item (37, []) successfully
2015-08-10T15:24:17.977 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,945 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 38
2015-08-10T15:24:18.017 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:17,946 17938 [radosgw_agent.sync][INFO  ] 38/64 items processed
2015-08-10T15:24:18.130 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 19792 [radosgw_agent.worker][INFO  ] finished processing shard 38
2015-08-10T15:24:18.141 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 17938 [radosgw_agent.sync][DEBUG ] synced item (38, []) successfully
2015-08-10T15:24:18.142 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,121 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 39
2015-08-10T15:24:18.183 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,122 17938 [radosgw_agent.sync][INFO  ] 39/64 items processed
2015-08-10T15:24:18.306 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 19792 [radosgw_agent.worker][INFO  ] finished processing shard 39
2015-08-10T15:24:18.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 17938 [radosgw_agent.sync][DEBUG ] synced item (39, []) successfully
2015-08-10T15:24:18.309 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 40
2015-08-10T15:24:18.350 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,297 17938 [radosgw_agent.sync][INFO  ] 40/64 items processed
2015-08-10T15:24:18.499 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 19792 [radosgw_agent.worker][INFO  ] finished processing shard 40
2015-08-10T15:24:18.518 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 17938 [radosgw_agent.sync][DEBUG ] synced item (40, []) successfully
2015-08-10T15:24:18.519 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 41
2015-08-10T15:24:18.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,490 17938 [radosgw_agent.sync][INFO  ] 41/64 items processed
2015-08-10T15:24:18.675 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 19792 [radosgw_agent.worker][INFO  ] finished processing shard 41
2015-08-10T15:24:18.685 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 17938 [radosgw_agent.sync][DEBUG ] synced item (41, []) successfully
2015-08-10T15:24:18.686 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 42
2015-08-10T15:24:18.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,666 17938 [radosgw_agent.sync][INFO  ] 42/64 items processed
2015-08-10T15:24:18.835 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 19792 [radosgw_agent.worker][INFO  ] finished processing shard 42
2015-08-10T15:24:18.871 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 17938 [radosgw_agent.sync][DEBUG ] synced item (42, []) successfully
2015-08-10T15:24:18.912 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,826 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 43
2015-08-10T15:24:18.913 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,827 17938 [radosgw_agent.sync][INFO  ] 43/64 items processed
2015-08-10T15:24:18.979 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 19792 [radosgw_agent.worker][INFO  ] finished processing shard 43
2015-08-10T15:24:18.994 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 17938 [radosgw_agent.sync][DEBUG ] synced item (43, []) successfully
2015-08-10T15:24:18.995 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 44
2015-08-10T15:24:19.036 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:18,969 17938 [radosgw_agent.sync][INFO  ] 44/64 items processed
2015-08-10T15:24:19.120 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 19792 [radosgw_agent.worker][INFO  ] finished processing shard 44
2015-08-10T15:24:19.127 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 17938 [radosgw_agent.sync][DEBUG ] synced item (44, []) successfully
2015-08-10T15:24:19.127 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 45
2015-08-10T15:24:19.128 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:19,112 17938 [radosgw_agent.sync][INFO  ] 45/64 items processed
2015-08-10T15:24:23.249 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 19792 [radosgw_agent.worker][INFO  ] finished processing shard 45
2015-08-10T15:24:23.251 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 46
2015-08-10T15:24:23.252 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 17938 [radosgw_agent.sync][DEBUG ] synced item (45, []) successfully
2015-08-10T15:24:23.292 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:23,240 17938 [radosgw_agent.sync][INFO  ] 46/64 items processed
2015-08-10T15:24:24.283 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,274 19792 [radosgw_agent.worker][INFO  ] finished processing shard 46
2015-08-10T15:24:24.303 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 17938 [radosgw_agent.sync][DEBUG ] synced item (46, []) successfully
2015-08-10T15:24:24.304 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 47
2015-08-10T15:24:24.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,275 17938 [radosgw_agent.sync][INFO  ] 47/64 items processed
2015-08-10T15:24:24.510 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,500 19792 [radosgw_agent.worker][INFO  ] finished processing shard 47
2015-08-10T15:24:24.512 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 17938 [radosgw_agent.sync][DEBUG ] synced item (47, []) successfully
2015-08-10T15:24:24.513 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 48
2015-08-10T15:24:24.554 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,501 17938 [radosgw_agent.sync][INFO  ] 48/64 items processed
2015-08-10T15:24:24.645 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 19792 [radosgw_agent.worker][INFO  ] finished processing shard 48
2015-08-10T15:24:24.678 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 17938 [radosgw_agent.sync][DEBUG ] synced item (48, []) successfully
2015-08-10T15:24:24.679 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 49
2015-08-10T15:24:24.680 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:24,636 17938 [radosgw_agent.sync][INFO  ] 49/64 items processed
2015-08-10T15:24:25.089 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,079 19792 [radosgw_agent.worker][INFO  ] finished processing shard 49
2015-08-10T15:24:25.190 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 50
2015-08-10T15:24:25.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 17938 [radosgw_agent.sync][DEBUG ] synced item (49, []) successfully
2015-08-10T15:24:25.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,080 17938 [radosgw_agent.sync][INFO  ] 50/64 items processed
2015-08-10T15:24:25.256 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 19792 [radosgw_agent.worker][INFO  ] finished processing shard 50
2015-08-10T15:24:25.273 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 17938 [radosgw_agent.sync][DEBUG ] synced item (50, []) successfully
2015-08-10T15:24:25.274 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 51
2015-08-10T15:24:25.315 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,247 17938 [radosgw_agent.sync][INFO  ] 51/64 items processed
2015-08-10T15:24:25.440 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 19792 [radosgw_agent.worker][INFO  ] finished processing shard 51
2015-08-10T15:24:25.445 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 17938 [radosgw_agent.sync][DEBUG ] synced item (51, []) successfully
2015-08-10T15:24:25.446 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 52
2015-08-10T15:24:25.487 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,431 17938 [radosgw_agent.sync][INFO  ] 52/64 items processed
2015-08-10T15:24:25.592 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 19792 [radosgw_agent.worker][INFO  ] finished processing shard 52
2015-08-10T15:24:25.613 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 17938 [radosgw_agent.sync][DEBUG ] synced item (52, []) successfully
2015-08-10T15:24:25.614 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 53
2015-08-10T15:24:25.655 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,582 17938 [radosgw_agent.sync][INFO  ] 53/64 items processed
2015-08-10T15:24:25.726 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,716 19792 [radosgw_agent.worker][INFO  ] finished processing shard 53
2015-08-10T15:24:25.738 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,716 17938 [radosgw_agent.sync][DEBUG ] synced item (53, []) successfully
2015-08-10T15:24:25.739 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,717 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 54
2015-08-10T15:24:25.780 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,717 17938 [radosgw_agent.sync][INFO  ] 54/64 items processed
2015-08-10T15:24:25.910 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 19792 [radosgw_agent.worker][INFO  ] finished processing shard 54
2015-08-10T15:24:25.912 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 17938 [radosgw_agent.sync][DEBUG ] synced item (54, []) successfully
2015-08-10T15:24:25.952 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 55
2015-08-10T15:24:25.953 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:25,901 17938 [radosgw_agent.sync][INFO  ] 55/64 items processed
2015-08-10T15:24:26.010 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,001 19792 [radosgw_agent.worker][DEBUG ] syncing metadata type bucket key "mybar" 
2015-08-10T15:24:26.288 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,279 19792 [radosgw_agent.worker][INFO  ] finished processing shard 55
2015-08-10T15:24:26.289 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,279 17938 [radosgw_agent.sync][DEBUG ] synced item (55, []) successfully
2015-08-10T15:24:26.330 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,280 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 56
2015-08-10T15:24:26.431 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:26,422 17938 [radosgw_agent.sync][INFO  ] 56/64 items processed
2015-08-10T15:24:30.377 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,367 19792 [radosgw_agent.worker][INFO  ] finished processing shard 56
2015-08-10T15:24:30.408 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,368 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 57
2015-08-10T15:24:30.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,367 17938 [radosgw_agent.sync][DEBUG ] synced item (56, []) successfully
2015-08-10T15:24:30.450 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:30,368 17938 [radosgw_agent.sync][INFO  ] 57/64 items processed
2015-08-10T15:24:31.361 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,351 19792 [radosgw_agent.worker][INFO  ] finished processing shard 57
2015-08-10T15:24:31.367 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 17938 [radosgw_agent.sync][DEBUG ] synced item (57, []) successfully
2015-08-10T15:24:31.368 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 58
2015-08-10T15:24:31.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,352 17938 [radosgw_agent.sync][INFO  ] 58/64 items processed
2015-08-10T15:24:31.587 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,577 19792 [radosgw_agent.worker][INFO  ] finished processing shard 58
2015-08-10T15:24:31.618 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 17938 [radosgw_agent.sync][DEBUG ] synced item (58, []) successfully
2015-08-10T15:24:31.619 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 59
2015-08-10T15:24:31.659 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,578 17938 [radosgw_agent.sync][INFO  ] 59/64 items processed
2015-08-10T15:24:31.756 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,746 19792 [radosgw_agent.worker][INFO  ] finished processing shard 59
2015-08-10T15:24:31.782 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 17938 [radosgw_agent.sync][DEBUG ] synced item (59, []) successfully
2015-08-10T15:24:31.783 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 60
2015-08-10T15:24:31.824 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:31,747 17938 [radosgw_agent.sync][INFO  ] 60/64 items processed
2015-08-10T15:24:32.173 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 19792 [radosgw_agent.worker][INFO  ] finished processing shard 60
2015-08-10T15:24:32.173 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 17938 [radosgw_agent.sync][DEBUG ] synced item (60, []) successfully
2015-08-10T15:24:32.214 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 61
2015-08-10T15:24:32.215 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,164 17938 [radosgw_agent.sync][INFO  ] 61/64 items processed
2015-08-10T15:24:32.349 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 19792 [radosgw_agent.worker][INFO  ] finished processing shard 61
2015-08-10T15:24:32.351 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 17938 [radosgw_agent.sync][DEBUG ] synced item (61, []) successfully
2015-08-10T15:24:32.391 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 62
2015-08-10T15:24:32.392 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,340 17938 [radosgw_agent.sync][INFO  ] 62/64 items processed
2015-08-10T15:24:32.525 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 19792 [radosgw_agent.worker][INFO  ] finished processing shard 62
2015-08-10T15:24:32.550 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 17938 [radosgw_agent.sync][DEBUG ] synced item (62, []) successfully
2015-08-10T15:24:32.550 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 19792 [radosgw_agent.worker][INFO  ] 19792 is processing shard number 63
2015-08-10T15:24:32.591 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,516 17938 [radosgw_agent.sync][INFO  ] 63/64 items processed
2015-08-10T15:24:32.692 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,683 19792 [radosgw_agent.worker][INFO  ] finished processing shard 63
2015-08-10T15:24:32.693 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,683 17938 [radosgw_agent.sync][DEBUG ] synced item (63, []) successfully
2015-08-10T15:24:32.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:32,684 17938 [radosgw_agent.sync][INFO  ] 64/64 items processed
2015-08-10T15:24:32.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:24:32] "POST /metadata/incremental HTTP/1.1" 200 0
2015-08-10T15:24:32.775 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:10.8.128.2 - - [10/Aug/2015 15:24:32] "POST /metadata/incremental HTTP/1.1" 200 -
2015-08-10T15:24:32.777 INFO:tasks.util.rgw:rgwadmin: client.0 : ['metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:32.816 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:32.817 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 metadata get bucket:mybar'
2015-08-10T15:24:34.731 INFO:tasks.util.rgw: json result: {u'data': {u'owner': u'fud', u'creation_time': 1439234556, u'has_bucket_info': u'false', u'bucket': {u'name': u'mybar', u'bucket_id': u'r0z0.4146.3', u'marker': u'r0z0.4146.3', u'data_extra_pool': u'.region0.r0z0.data_extra_pool', u'pool': u'.region0.r0z0.data_pool', u'index_pool': u'.region0.r0z0.index_pool'}, u'linked': u'true'}, u'ver': {u'tag': u'_7e_pwbuFp-pohyTXyMynUDl', u'ver': 1}, u'key': u'bucket:mybar', u'mtime': 1439234256}
2015-08-10T15:24:35.156 INFO:tasks.util.rgw:rgwadmin: client.1 : ['metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:35.157 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', 'metadata', 'get', 'bucket:mybar']
2015-08-10T15:24:35.159 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 metadata get bucket:mybar'
2015-08-10T15:24:37.563 INFO:tasks.util.rgw: json result: {u'data': {u'owner': u'fud', u'creation_time': 1439234556, u'has_bucket_info': u'false', u'bucket': {u'name': u'mybar', u'bucket_id': u'r0z0.4146.3', u'marker': u'r0z0.4146.3', u'data_extra_pool': u'.region0.r0z0.data_extra_pool', u'pool': u'.region0.r0z0.data_pool', u'index_pool': u'.region0.r0z0.index_pool'}, u'linked': u'true'}, u'ver': {u'tag': u'_7e_pwbuFp-pohyTXyMynUDl', u'ver': 1}, u'key': u'bucket:mybar', u'mtime': 1439234256}
2015-08-10T15:24:38.331 INFO:tasks.util.rgw:sync agent magna087.ceph.redhat.com:8000
2015-08-10T15:24:38.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,335 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:24:38.381 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,336 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:24:38.422 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:38,341 17938 [radosgw_agent.sync][DEBUG ] 64 shards to check
2015-08-10T15:24:53.398 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,389 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10T15:24:53.400 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,390 17938 [radosgw_agent.client][INFO  ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10T15:24:53.402 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,392 17938 [radosgw_agent.sync][INFO  ] Starting sync
2015-08-10T15:24:53.403 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,393 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 0
2015-08-10T15:24:53.573 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 19923 [radosgw_agent.worker][INFO  ] finished processing shard 0
2015-08-10T15:24:53.574 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 17938 [radosgw_agent.sync][DEBUG ] synced item (0, []) successfully
2015-08-10T15:24:53.575 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 1
2015-08-10T15:24:53.576 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,564 17938 [radosgw_agent.sync][INFO  ] 1/64 items processed
2015-08-10T15:24:53.716 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 19923 [radosgw_agent.worker][INFO  ] finished processing shard 1
2015-08-10T15:24:53.717 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 17938 [radosgw_agent.sync][DEBUG ] synced item (1, []) successfully
2015-08-10T15:24:53.718 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,706 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 2
2015-08-10T15:24:53.719 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,707 17938 [radosgw_agent.sync][INFO  ] 2/64 items processed
2015-08-10T15:24:53.908 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 19923 [radosgw_agent.worker][INFO  ] finished processing shard 2
2015-08-10T15:24:53.915 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 17938 [radosgw_agent.sync][DEBUG ] synced item (2, []) successfully
2015-08-10T15:24:53.916 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 3
2015-08-10T15:24:53.957 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:53,899 17938 [radosgw_agent.sync][INFO  ] 3/64 items processed
2015-08-10T15:24:54.094 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,084 19923 [radosgw_agent.worker][INFO  ] finished processing shard 3
2015-08-10T15:24:54.095 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 17938 [radosgw_agent.sync][DEBUG ] synced item (3, []) successfully
2015-08-10T15:24:54.096 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 4
2015-08-10T15:24:54.097 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,085 17938 [radosgw_agent.sync][INFO  ] 4/64 items processed
2015-08-10T15:24:54.228 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 19923 [radosgw_agent.worker][INFO  ] finished processing shard 4
2015-08-10T15:24:54.229 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 17938 [radosgw_agent.sync][DEBUG ] synced item (4, []) successfully
2015-08-10T15:24:54.230 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 5
2015-08-10T15:24:54.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,219 17938 [radosgw_agent.sync][INFO  ] 5/64 items processed
2015-08-10T15:24:54.379 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 19923 [radosgw_agent.worker][INFO  ] finished processing shard 5
2015-08-10T15:24:54.380 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 17938 [radosgw_agent.sync][DEBUG ] synced item (5, []) successfully
2015-08-10T15:24:54.381 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 6
2015-08-10T15:24:54.382 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,370 17938 [radosgw_agent.sync][INFO  ] 6/64 items processed
2015-08-10T15:24:54.546 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,537 19923 [radosgw_agent.worker][INFO  ] finished processing shard 6
2015-08-10T15:24:54.547 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,537 17938 [radosgw_agent.sync][DEBUG ] synced item (6, []) successfully
2015-08-10T15:24:54.548 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,538 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 7
2015-08-10T15:24:54.549 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,538 17938 [radosgw_agent.sync][INFO  ] 7/64 items processed
2015-08-10T15:24:54.731 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 19923 [radosgw_agent.worker][INFO  ] finished processing shard 7
2015-08-10T15:24:55.231 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 17938 [radosgw_agent.sync][DEBUG ] synced item (7, []) successfully
2015-08-10T15:24:55.272 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 8
2015-08-10T15:24:55.273 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:24:54,722 17938 [radosgw_agent.sync][INFO  ] 8/64 items processed
2015-08-10T15:25:01.562 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,552 19923 [radosgw_agent.worker][INFO  ] finished processing shard 8
2015-08-10T15:25:01.563 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 9
2015-08-10T15:25:01.564 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 17938 [radosgw_agent.sync][DEBUG ] synced item (8, []) successfully
2015-08-10T15:25:01.565 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,553 17938 [radosgw_agent.sync][INFO  ] 9/64 items processed
2015-08-10T15:25:01.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 19923 [radosgw_agent.worker][INFO  ] finished processing shard 9
2015-08-10T15:25:01.729 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 17938 [radosgw_agent.sync][DEBUG ] synced item (9, []) successfully
2015-08-10T15:25:01.730 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 10
2015-08-10T15:25:01.730 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:01,720 17938 [radosgw_agent.sync][INFO  ] 10/64 items processed
2015-08-10T15:25:02.154 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,144 19923 [radosgw_agent.worker][INFO  ] finished processing shard 10
2015-08-10T15:25:02.155 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 11
2015-08-10T15:25:02.194 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 17938 [radosgw_agent.sync][DEBUG ] synced item (10, []) successfully
2015-08-10T15:25:02.195 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,145 17938 [radosgw_agent.sync][INFO  ] 11/64 items processed
2015-08-10T15:25:02.374 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 19923 [radosgw_agent.worker][INFO  ] finished processing shard 11
2015-08-10T15:25:02.414 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 17938 [radosgw_agent.sync][DEBUG ] synced item (11, []) successfully
2015-08-10T15:25:02.415 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 12
2015-08-10T15:25:02.455 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,365 17938 [radosgw_agent.sync][INFO  ] 12/64 items processed
2015-08-10T15:25:02.534 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 19923 [radosgw_agent.worker][INFO  ] finished processing shard 12
2015-08-10T15:25:02.539 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 17938 [radosgw_agent.sync][DEBUG ] synced item (12, []) successfully
2015-08-10T15:25:02.540 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 13
2015-08-10T15:25:02.580 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,524 17938 [radosgw_agent.sync][INFO  ] 13/64 items processed
2015-08-10T15:25:02.702 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 19923 [radosgw_agent.worker][INFO  ] finished processing shard 13
2015-08-10T15:25:02.704 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 17938 [radosgw_agent.sync][DEBUG ] synced item (13, []) successfully
2015-08-10T15:25:02.708 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 14
2015-08-10T15:25:02.710 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,692 17938 [radosgw_agent.sync][INFO  ] 14/64 items processed
2015-08-10T15:25:02.869 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,859 19923 [radosgw_agent.worker][INFO  ] finished processing shard 14
2015-08-10T15:25:02.877 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 17938 [radosgw_agent.sync][DEBUG ] synced item (14, []) successfully
2015-08-10T15:25:02.878 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 15
2015-08-10T15:25:02.882 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:02,860 17938 [radosgw_agent.sync][INFO  ] 15/64 items processed
2015-08-10T15:25:03.038 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,028 19923 [radosgw_agent.worker][INFO  ] finished processing shard 15
2015-08-10T15:25:03.040 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,028 17938 [radosgw_agent.sync][DEBUG ] synced item (15, []) successfully
2015-08-10T15:25:03.081 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,029 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 16
2015-08-10T15:25:03.081 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,029 17938 [radosgw_agent.sync][INFO  ] 16/64 items processed
2015-08-10T15:25:03.188 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 19923 [radosgw_agent.worker][INFO  ] finished processing shard 16
2015-08-10T15:25:03.220 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 17938 [radosgw_agent.sync][DEBUG ] synced item (16, []) successfully
2015-08-10T15:25:03.221 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 17
2015-08-10T15:25:03.222 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,179 17938 [radosgw_agent.sync][INFO  ] 17/64 items processed
2015-08-10T15:25:03.356 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 19923 [radosgw_agent.worker][INFO  ] finished processing shard 17
2015-08-10T15:25:03.368 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 17938 [radosgw_agent.sync][DEBUG ] synced item (17, []) successfully
2015-08-10T15:25:03.369 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 18
2015-08-10T15:25:03.371 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,347 17938 [radosgw_agent.sync][INFO  ] 18/64 items processed
2015-08-10T15:25:03.549 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,539 19923 [radosgw_agent.worker][INFO  ] finished processing shard 18
2015-08-10T15:25:03.552 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 17938 [radosgw_agent.sync][DEBUG ] synced item (18, []) successfully
2015-08-10T15:25:03.593 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 19
2015-08-10T15:25:03.594 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,540 17938 [radosgw_agent.sync][INFO  ] 19/64 items processed
2015-08-10T15:25:03.700 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 19923 [radosgw_agent.worker][INFO  ] finished processing shard 19
2015-08-10T15:25:03.731 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 17938 [radosgw_agent.sync][DEBUG ] synced item (19, []) successfully
2015-08-10T15:25:03.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 20
2015-08-10T15:25:03.775 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:03,691 17938 [radosgw_agent.sync][INFO  ] 20/64 items processed
2015-08-10T15:25:06.543 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 19923 [radosgw_agent.worker][INFO  ] finished processing shard 20
2015-08-10T15:25:06.544 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 17938 [radosgw_agent.sync][DEBUG ] synced item (20, []) successfully
2015-08-10T15:25:06.545 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 21
2015-08-10T15:25:06.585 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:06,534 17938 [radosgw_agent.sync][INFO  ] 21/64 items processed
2015-08-10T15:25:09.023 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 19923 [radosgw_agent.worker][INFO  ] finished processing shard 21
2015-08-10T15:25:09.024 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 22
2015-08-10T15:25:09.025 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 17938 [radosgw_agent.sync][DEBUG ] synced item (21, []) successfully
2015-08-10T15:25:09.026 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,014 17938 [radosgw_agent.sync][INFO  ] 22/64 items processed
2015-08-10T15:25:09.174 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 19923 [radosgw_agent.worker][INFO  ] finished processing shard 22
2015-08-10T15:25:09.175 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 17938 [radosgw_agent.sync][DEBUG ] synced item (22, []) successfully
2015-08-10T15:25:09.215 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 23
2015-08-10T15:25:09.216 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,165 17938 [radosgw_agent.sync][INFO  ] 23/64 items processed
2015-08-10T15:25:09.342 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,332 19923 [radosgw_agent.worker][INFO  ] finished processing shard 23
2015-08-10T15:25:09.343 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 17938 [radosgw_agent.sync][DEBUG ] synced item (23, []) successfully
2015-08-10T15:25:09.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 24
2015-08-10T15:25:09.345 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,333 17938 [radosgw_agent.sync][INFO  ] 24/64 items processed
2015-08-10T15:25:09.559 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 19923 [radosgw_agent.worker][INFO  ] finished processing shard 24
2015-08-10T15:25:09.560 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 17938 [radosgw_agent.sync][DEBUG ] synced item (24, []) successfully
2015-08-10T15:25:09.561 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 25
2015-08-10T15:25:09.562 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,550 17938 [radosgw_agent.sync][INFO  ] 25/64 items processed
2015-08-10T15:25:09.718 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 19923 [radosgw_agent.worker][INFO  ] finished processing shard 25
2015-08-10T15:25:10.157 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 17938 [radosgw_agent.sync][DEBUG ] synced item (25, []) successfully
2015-08-10T15:25:10.158 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,709 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 26
2015-08-10T15:25:10.199 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,710 17938 [radosgw_agent.sync][INFO  ] 26/64 items processed
2015-08-10T15:25:10.202 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 19923 [radosgw_agent.worker][INFO  ] finished processing shard 26
2015-08-10T15:25:10.241 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 17938 [radosgw_agent.sync][DEBUG ] synced item (26, []) successfully
2015-08-10T15:25:10.242 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 27
2015-08-10T15:25:10.283 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:09,877 17938 [radosgw_agent.sync][INFO  ] 27/64 items processed
2015-08-10T15:25:10.284 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,044 19923 [radosgw_agent.worker][INFO  ] finished processing shard 27
2015-08-10T15:25:10.325 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 17938 [radosgw_agent.sync][DEBUG ] synced item (27, []) successfully
2015-08-10T15:25:10.326 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 28
2015-08-10T15:25:10.367 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,045 17938 [radosgw_agent.sync][INFO  ] 28/64 items processed
2015-08-10T15:25:10.369 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 19923 [radosgw_agent.worker][INFO  ] finished processing shard 28
2015-08-10T15:25:10.409 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 17938 [radosgw_agent.sync][DEBUG ] synced item (28, []) successfully
2015-08-10T15:25:10.410 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 29
2015-08-10T15:25:10.450 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,204 17938 [radosgw_agent.sync][INFO  ] 29/64 items processed
2015-08-10T15:25:10.452 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 19923 [radosgw_agent.worker][INFO  ] finished processing shard 29
2015-08-10T15:25:10.491 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 17938 [radosgw_agent.sync][DEBUG ] synced item (29, []) successfully
2015-08-10T15:25:10.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 30
2015-08-10T15:25:10.532 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,381 17938 [radosgw_agent.sync][INFO  ] 30/64 items processed
2015-08-10T15:25:10.541 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 19923 [radosgw_agent.worker][INFO  ] finished processing shard 30
2015-08-10T15:25:10.574 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 17938 [radosgw_agent.sync][DEBUG ] synced item (30, []) successfully
2015-08-10T15:25:10.575 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 31
2015-08-10T15:25:10.616 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,532 17938 [radosgw_agent.sync][INFO  ] 31/64 items processed
2015-08-10T15:25:10.725 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 19923 [radosgw_agent.worker][INFO  ] finished processing shard 31
2015-08-10T15:25:10.740 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 17938 [radosgw_agent.sync][DEBUG ] synced item (31, []) successfully
2015-08-10T15:25:10.741 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 32
2015-08-10T15:25:10.781 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,716 17938 [radosgw_agent.sync][INFO  ] 32/64 items processed
2015-08-10T15:25:10.893 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,883 19923 [radosgw_agent.worker][INFO  ] finished processing shard 32
2015-08-10T15:25:10.907 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 17938 [radosgw_agent.sync][DEBUG ] synced item (32, []) successfully
2015-08-10T15:25:10.908 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 33
2015-08-10T15:25:10.949 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:10,884 17938 [radosgw_agent.sync][INFO  ] 33/64 items processed
2015-08-10T15:25:15.756 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 19923 [radosgw_agent.worker][INFO  ] finished processing shard 33
2015-08-10T15:25:15.793 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 17938 [radosgw_agent.sync][DEBUG ] synced item (33, []) successfully
2015-08-10T15:25:15.794 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,746 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 34
2015-08-10T15:25:15.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:15,747 17938 [radosgw_agent.sync][INFO  ] 34/64 items processed
2015-08-10T15:25:20.193 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,183 19923 [radosgw_agent.worker][INFO  ] finished processing shard 34
2015-08-10T15:25:20.223 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,183 17938 [radosgw_agent.sync][DEBUG ] synced item (34, []) successfully
2015-08-10T15:25:20.224 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,184 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 35
2015-08-10T15:25:20.225 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,184 17938 [radosgw_agent.sync][INFO  ] 35/64 items processed
2015-08-10T15:25:20.344 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 19923 [radosgw_agent.worker][INFO  ] finished processing shard 35
2015-08-10T15:25:20.379 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 17938 [radosgw_agent.sync][DEBUG ] synced item (35, []) successfully
2015-08-10T15:25:20.380 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 19923 [radosgw_agent.worker][INFO  ] 19923 is processing shard number 36
2015-08-10T15:25:20.391 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 15:25:20,334 17938 [radosgw
Actions
Copy link
 #1

Updated by Vasu Kulkarni over 8 years ago

looks like some config issue , the key gets listed if one gets bucket first and iterates over key

>>> dest_k = dest_connection.get_bucket(bucket_name + 'data')
>>> buckets = dest_k.list()
>>> for key in buckets:
...   print key.name
... 
tiny_file
>>> dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/vasu/venv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 192, in get_key
    key, resp = self._get_key_internal(key_name, headers, query_args_l)
  File "/home/vasu/venv/local/lib/python2.7/site-packages/boto/s3/bucket.py", line 230, in _get_key_internal
    response.status, response.reason, '')
boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden

>>>
Actions
Copy link
 #2

Updated by Vasu Kulkarni over 8 years ago

I see the following crash on Ubuntu not seen in RHEL run, this is on next branch and fails during the same tiny_file sync

FO  ] creating connection to endpoint: http://vpm174.front.sepia.ceph.com:7281
2015-08-12 03:47:22,273.273 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:22,268 12523 [radosgw_agent.sync][INFO  ] waiting to make sure bucket log is consistent
2015-08-12 03:47:52,309.309 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:52,303 12523 [radosgw_agent.sync][INFO  ] Starting sync
2015-08-12 03:47:52,387.387 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:52,383 13877 [radosgw_agent.worker][DEBUG ] bucket instance is "myfoodata:r0z0.4144.4" with marker 0#00000000002.2.3
2015-08-12 03:47:52,388.388 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:52,383 13877 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-12 03:47:52,388.388 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:52,383 13877 [radosgw_agent.worker][INFO  ] syncing bucket "myfoodata" 
2015-08-12 03:47:52,451.451 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:52,446 13877 [radosgw_agent.worker][DEBUG ] syncing object myfoodata/tiny_file
2015-08-12 03:47:52,467.467 INFO:tasks.rgw.client.1.vpm174.stdout:*** Caught signal (Segmentation fault) **
2015-08-12 03:47:52,467.467 INFO:tasks.rgw.client.1.vpm174.stdout: in thread 7f76937fe700
2015-08-12 03:47:52,470.470 INFO:tasks.rgw.client.1.vpm174.stdout: ceph version 9.0.2-1242-g79e7768 (79e776865e475b492d17b2ef197a95be025f8db9)
2015-08-12 03:47:52,470.470 INFO:tasks.rgw.client.1.vpm174.stdout: 1: (()+0x2ef91a) [0x7f76ca8e691a]
2015-08-12 03:47:52,470.470 INFO:tasks.rgw.client.1.vpm174.stdout: 2: (()+0x10340) [0x7f76c6ca3340]
2015-08-12 03:47:52,470.470 INFO:tasks.rgw.client.1.vpm174.stdout: 3: (strlen()+0x2a) [0x7f76c5382aea]
2015-08-12 03:47:52,470.470 INFO:tasks.rgw.client.1.vpm174.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f76ca86b6f3]
2015-08-12 03:47:52,471.471 INFO:tasks.rgw.client.1.vpm174.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f76ca8a3859]
2015-08-12 03:47:52,471.471 INFO:tasks.rgw.client.1.vpm174.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f76ca84ab5b]
2015-08-12 03:47:52,471.471 INFO:tasks.rgw.client.1.vpm174.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x40c) [0x7f76ca82bedc]
2015-08-12 03:47:52,471.471 INFO:tasks.rgw.client.1.vpm174.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x395) [0x7f76ca82d005]
2015-08-12 03:47:52,471.471 INFO:tasks.rgw.client.1.vpm174.stdout: 9: (RGWCopyObj::execute()+0x1ae) [0x7f76ca8563de]
2015-08-12 03:47:52,472.472 INFO:tasks.rgw.client.1.vpm174.stdout: 10: (()+0x1abd10) [0x7f76ca7a2d10]
2015-08-12 03:47:52,472.472 INFO:tasks.rgw.client.1.vpm174.stdout: 11: (()+0x1ac928) [0x7f76ca7a3928]
2015-08-12 03:47:52,472.472 INFO:tasks.rgw.client.1.vpm174.stdout: 12: (()+0x2b711f) [0x7f76ca8ae11f]
2015-08-12 03:47:52,472.472 INFO:tasks.rgw.client.1.vpm174.stdout: 13: (()+0x2b90ee) [0x7f76ca8b00ee]
2015-08-12 03:47:52,472.472 INFO:tasks.rgw.client.1.vpm174.stdout: 14: (()+0x8182) [0x7f76c6c9b182]
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout: 15: (clone()+0x6d) [0x7f76c53f447d]
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout:2015-08-12 03:47:52.465872 7f76937fe700 -1 *** Caught signal (Segmentation fault) **
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout: in thread 7f76937fe700
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout:
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout: ceph version 9.0.2-1242-g79e7768 (79e776865e475b492d17b2ef197a95be025f8db9)
2015-08-12 03:47:52,473.473 INFO:tasks.rgw.client.1.vpm174.stdout: 1: (()+0x2ef91a) [0x7f76ca8e691a]
2015-08-12 03:47:52,474.474 INFO:tasks.rgw.client.1.vpm174.stdout: 2: (()+0x10340) [0x7f76c6ca3340]
2015-08-12 03:47:52,474.474 INFO:tasks.rgw.client.1.vpm174.stdout: 3: (strlen()+0x2a) [0x7f76c5382aea]
2015-08-12 03:47:52,474.474 INFO:tasks.rgw.client.1.vpm174.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f76ca86b6f3]
2015-08-12 03:47:52,474.474 INFO:tasks.rgw.client.1.vpm174.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f76ca8a3859]
2015-08-12 03:47:52,474.474 INFO:tasks.rgw.client.1.vpm174.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f76ca84ab5b]
2015-08-12 03:47:52,475.475 INFO:tasks.rgw.client.1.vpm174.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x40c) [0x7f76ca82bedc]
2015-08-12 03:47:52,475.475 INFO:tasks.rgw.client.1.vpm174.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x395) [0x7f76ca82d005]
2015-08-12 03:47:52,475.475 INFO:tasks.rgw.client.1.vpm174.stdout: 9: (RGWCopyObj::execute()+0x1ae) [0x7f76ca8563de]
2015-08-12 03:47:52,475.475 INFO:tasks.rgw.client.1.vpm174.stdout: 10: (()+0x1abd10) [0x7f76ca7a2d10]
2015-08-12 03:47:52,475.475 INFO:tasks.rgw.client.1.vpm174.stdout: 11: (()+0x1ac928) [0x7f76ca7a3928]
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout: 12: (()+0x2b711f) [0x7f76ca8ae11f]
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout: 13: (()+0x2b90ee) [0x7f76ca8b00ee]
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout: 14: (()+0x8182) [0x7f76c6c9b182]
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout: 15: (clone()+0x6d) [0x7f76c53f447d]
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12 03:47:52,476.476 INFO:tasks.rgw.client.1.vpm174.stdout:
2015-08-12 03:47:52,540.540 INFO:tasks.rgw.client.1.vpm174.stdout:     0> 2015-08-12 03:47:52.465872 7f76937fe700 -1 *** Caught signal (Segmentation fault) **
2015-08-12 03:47:52,540.540 INFO:tasks.rgw.client.1.vpm174.stdout: in thread 7f76937fe700
2015-08-12 03:47:52,541.541 INFO:tasks.rgw.client.1.vpm174.stdout:
2015-08-12 03:47:52,541.541 INFO:tasks.rgw.client.1.vpm174.stdout: ceph version 9.0.2-1242-g79e7768 (79e776865e475b492d17b2ef197a95be025f8db9)
2015-08-12 03:47:52,541.541 INFO:tasks.rgw.client.1.vpm174.stdout: 1: (()+0x2ef91a) [0x7f76ca8e691a]
2015-08-12 03:47:52,541.541 INFO:tasks.rgw.client.1.vpm174.stdout: 2: (()+0x10340) [0x7f76c6ca3340]
2015-08-12 03:47:52,541.541 INFO:tasks.rgw.client.1.vpm174.stdout: 3: (strlen()+0x2a) [0x7f76c5382aea]
2015-08-12 03:47:52,542.542 INFO:tasks.rgw.client.1.vpm174.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f76ca86b6f3]
2015-08-12 03:47:52,542.542 INFO:tasks.rgw.client.1.vpm174.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f76ca8a3859]
2015-08-12 03:47:52,542.542 INFO:tasks.rgw.client.1.vpm174.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f76ca84ab5b]
2015-08-12 03:47:52,542.542 INFO:tasks.rgw.client.1.vpm174.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x40c) [0x7f76ca82bedc]
2015-08-12 03:47:52,542.542 INFO:tasks.rgw.client.1.vpm174.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x395) [0x7f76ca82d005]
2015-08-12 03:47:52,543.543 INFO:tasks.rgw.client.1.vpm174.stdout: 9: (RGWCopyObj::execute()+0x1ae) [0x7f76ca8563de]
2015-08-12 03:47:52,543.543 INFO:tasks.rgw.client.1.vpm174.stdout: 10: (()+0x1abd10) [0x7f76ca7a2d10]
2015-08-12 03:47:52,543.543 INFO:tasks.rgw.client.1.vpm174.stdout: 11: (()+0x1ac928) [0x7f76ca7a3928]
2015-08-12 03:47:52,543.543 INFO:tasks.rgw.client.1.vpm174.stdout: 12: (()+0x2b711f) [0x7f76ca8ae11f]
2015-08-12 03:47:52,543.543 INFO:tasks.rgw.client.1.vpm174.stdout: 13: (()+0x2b90ee) [0x7f76ca8b00ee]
2015-08-12 03:47:52,544.544 INFO:tasks.rgw.client.1.vpm174.stdout: 14: (()+0x8182) [0x7f76c6c9b182]
2015-08-12 03:47:52,544.544 INFO:tasks.rgw.client.1.vpm174.stdout: 15: (clone()+0x6d) [0x7f76c53f447d]
2015-08-12 03:47:52,544.544 INFO:tasks.rgw.client.1.vpm174.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-08-12 03:47:52,544.544 INFO:tasks.rgw.client.1.vpm174.stdout:
2015-08-12 03:47:52,746.746 INFO:tasks.rgw.client.1.vpm174.stderr:daemon-helper: command crashed with signal 11
2015-08-12 03:47:57,372.372 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:47:57,368 13877 [radosgw_agent.worker][WARNING] encountered an error during sync: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:03,786.786 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:03,781 13877 [radosgw_agent.worker][DEBUG ] error geting op state: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:03,786.786 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:03,786.786 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 273, in wait_for_object
2015-08-12 03:48:03,787.787 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    bucket, obj)
2015-08-12 03:48:03,787.787 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 250, in get_op_state
2015-08-12 03:48:03,787.787 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    'client-id': client_id,
2015-08-12 03:48:03,787.787 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 207, in request
2015-08-12 03:48:03,788.788 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise NetworkError(msg)
2015-08-12 03:48:03,788.788 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:NetworkError: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:07,371.371 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:07,367 13877 [radosgw_agent.worker][DEBUG ] error geting op state: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:07,372.372 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:07,372.372 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 273, in wait_for_object
2015-08-12 03:48:07,372.372 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    bucket, obj)
2015-08-12 03:48:07,373.373 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 250, in get_op_state
2015-08-12 03:48:07,373.373 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    'client-id': client_id,
2015-08-12 03:48:07,373.373 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 207, in request
2015-08-12 03:48:07,373.373 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise NetworkError(msg)
2015-08-12 03:48:07,374.374 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:NetworkError: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:14,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:14,786 13877 [radosgw_agent.worker][DEBUG ] error geting op state: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:14,792.792 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:14,792.792 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 273, in wait_for_object
2015-08-12 03:48:14,792.792 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    bucket, obj)
2015-08-12 03:48:14,793.793 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 250, in get_op_state
2015-08-12 03:48:14,793.793 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    'client-id': client_id,
2015-08-12 03:48:14,793.793 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 207, in request
2015-08-12 03:48:14,793.793 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise NetworkError(msg)
2015-08-12 03:48:14,793.793 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:NetworkError: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:22,514.514 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:22,509 13877 [radosgw_agent.worker][DEBUG ] error geting op state: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:22,514.514 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:22,515.515 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 273, in wait_for_object
2015-08-12 03:48:22,515.515 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    bucket, obj)
2015-08-12 03:48:22,515.515 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 250, in get_op_state
2015-08-12 03:48:22,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    'client-id': client_id,
2015-08-12 03:48:22,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 207, in request
2015-08-12 03:48:22,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise NetworkError(msg)
2015-08-12 03:48:22,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:NetworkError: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:23,515.515 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,511 13877 [radosgw_agent.worker][ERROR ] failed to sync object myfoodata/tiny_file:
2015-08-12 03:48:23,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,511 13877 [radosgw_agent.worker][WARNING] will retry sync of failed object at next incremental sync
2015-08-12 03:48:23,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,512 13877 [radosgw_agent.worker][INFO  ] synced 0 objects
2015-08-12 03:48:23,516.516 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,512 13877 [radosgw_agent.worker][INFO  ] completed syncing bucket "myfoodata" 
2015-08-12 03:48:23,517.517 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,513 13877 [radosgw_agent.worker][INFO  ] ********************************************************************************
2015-08-12 03:48:23,521.521 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:23,513 13877 [radosgw_agent.worker][WARNING] error setting worker bound for key "myfoodata:r0z0.4144.4", may duplicate some work later. Traceback:
2015-08-12 03:48:23,521.521 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:23,522.522 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 87, in set_bound
2015-08-12 03:48:23,522.522 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    data=data)
2015-08-12 03:48:23,522.522 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 493, in set_worker_bound
2015-08-12 03:48:23,522.522 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    data=json.dumps(data),
2015-08-12 03:48:23,522.522 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
2015-08-12 03:48:23,523.523 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    return _default_encoder.encode(obj)
2015-08-12 03:48:23,523.523 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
2015-08-12 03:48:23,523.523 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    chunks = self.iterencode(o, _one_shot=True)
2015-08-12 03:48:23,523.523 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
2015-08-12 03:48:23,523.523 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    return _iterencode(o, 0)
2015-08-12 03:48:23,524.524 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
2015-08-12 03:48:23,524.524 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise TypeError(repr(o) + " is not JSON serializable")
2015-08-12 03:48:23,524.524 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:TypeError: <boto.s3.user.User object at 0x7f83c027fed0> is not JSON serializable
2015-08-12 03:48:27,651.651 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Exception in thread Thread-1:
2015-08-12 03:48:27,652.652 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:27,652.652 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
2015-08-12 03:48:27,652.652 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    self.run()
2015-08-12 03:48:27,652.652 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/lock.py", line 109, in run
2015-08-12 03:48:27,652.652 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    except client.HttpError as e:
2015-08-12 03:48:27,653.653 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:AttributeError: 'module' object has no attribute 'HttpError'
2015-08-12 03:48:27,653.653 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:
2015-08-12 03:48:37,789.789 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:37,782 13877 [radosgw_agent.worker][WARNING] error unlocking log, continuing anyway since lock will timeout. Traceback:
2015-08-12 03:48:37,789.789 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:37,790.790 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/worker.py", line 65, in unlock_shard
2015-08-12 03:48:37,790.790 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    self.lock.release_and_clear()
2015-08-12 03:48:37,790.790 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/lock.py", line 98, in release_and_clear
2015-08-12 03:48:37,790.790 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    except client.HttpError as e:
2015-08-12 03:48:37,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:AttributeError: 'module' object has no attribute 'HttpError'
2015-08-12 03:48:37,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:37,783 13877 [radosgw_agent.worker][INFO  ] finished syncing shard 124
2015-08-12 03:48:37,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:37,783 13877 [radosgw_agent.worker][INFO  ] incremental sync will need to retry buckets: [u'myfoodata']
2015-08-12 03:48:37,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:37,784 13877 [radosgw_agent.worker][INFO  ] No more entries in queue, exiting
2015-08-12 03:48:37,791.791 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:37,784 12523 [radosgw_agent.sync][DEBUG ] synced item (124, [u'myfoodata']) successfully
2015-08-12 03:48:43,303.303 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:43,297 12523 [radosgw_agent.sync][WARNING] could not set worker bounds, may repeat some work.Traceback:
2015-08-12 03:48:43,303.303 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:Traceback (most recent call last):
2015-08-12 03:48:43,303.303 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/sync.py", line 119, in complete_item
2015-08-12 03:48:43,304.304 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    data)
2015-08-12 03:48:43,304.304 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 494, in set_worker_bound
2015-08-12 03:48:43,304.304 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    special_first_param='work_bound',
2015-08-12 03:48:43,305.305 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:  File "/home/ubuntu/cephtest/radosgw-agent.client.0/radosgw_agent/client.py", line 207, in request
2015-08-12 03:48:43,305.305 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:    raise NetworkError(msg)
2015-08-12 03:48:43,305.305 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:NetworkError: unable to connect to vpm174.front.sepia.ceph.com:7281 [Errno 111] Connection refused
2015-08-12 03:48:43,305.305 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:43,298 12523 [radosgw_agent.sync][INFO  ] 1/1 items processed
2015-08-12 03:48:43,307.307 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:2015-08-12 03:48:43,299 12523 [radosgw_agent.sync][ERROR ] Encountered errors syncing these 1 shards: [u'myfoodata']
2015-08-12 03:48:43,307.307 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:10.214.140.50 - - [12/Aug/2015 03:48:43] "POST /data/full HTTP/1.1" 200 0
2015-08-12 03:48:43,307.307 INFO:tasks.radosgw_agent.ubuntu@vpm174.front.sepia.ceph.com.8000.syncdaemon.vpm174.stderr:10.214.140.50 - - [12/Aug/2015 03:48:43] "POST /data/full HTTP/1.1" 200 -
2015-08-12 03:49:51,207.207 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/ubuntu/teuthology/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/ubuntu/src/ceph-qa-suite_master/tasks/radosgw_admin.py", line 472, in task
    dest_k = dest_connection.get_bucket(bucket_name + 'data').get_key('tiny_file')
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 502, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 521, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
    retry_handler=retry_handler
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/home/ubuntu/teuthology/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
    raise ex
error: [Errno 111] Connection refused
Actions
Copy link
 #3

Updated by Vasu Kulkarni over 8 years ago

With recent master:
http://qa-proxy.ceph.com/teuthology/teuthology-2015-09-22_23:02:02-rgw-master---basic-multi/1066221/teuthology.log

2015-09-24T23:37:09.628 INFO:tasks.radosgw_agent.ubuntu@plana24.front.sepia.ceph.com.8000.syncdaemon.plana24.stderr:2015-09-24 23:37:09,631 1583 [radosgw_agent.worker][DEBUG ] syncing object myfoodata/tiny_file
2015-09-24T23:37:09.759 INFO:tasks.rgw.client.1.plana24.stdout:*** Caught signal (Segmentation fault) **
2015-09-24T23:37:09.760 INFO:tasks.rgw.client.1.plana24.stdout: in thread 7f2ef8f69700
2015-09-24T23:37:09.774 INFO:tasks.rgw.client.1.plana24.stdout: ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
2015-09-24T23:37:09.775 INFO:tasks.rgw.client.1.plana24.stdout: 1: (()+0x2faefa) [0x7f3073572efa]
2015-09-24T23:37:09.775 INFO:tasks.rgw.client.1.plana24.stdout: 2: (()+0x10340) [0x7f306f91e340]
2015-09-24T23:37:09.775 INFO:tasks.rgw.client.1.plana24.stdout: 3: (strlen()+0x2a) [0x7f306e20356a]
2015-09-24T23:37:09.775 INFO:tasks.rgw.client.1.plana24.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f30734f27b3]
2015-09-24T23:37:09.775 INFO:tasks.rgw.client.1.plana24.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f307352cb59]
2015-09-24T23:37:09.776 INFO:tasks.rgw.client.1.plana24.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f30734d120b]
2015-09-24T23:37:09.776 INFO:tasks.rgw.client.1.plana24.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x412) [0x7f30734b20e2]
2015-09-24T23:37:09.776 INFO:tasks.rgw.client.1.plana24.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x3a4) [0x7f30734b3474]
2015-09-24T23:37:09.776 INFO:tasks.rgw.client.1.plana24.stdout: 9: (RGWCopyObj::execute()+0x1ce) [0x7f30734e743e]
2015-09-24T23:37:09.776 INFO:tasks.rgw.client.1.plana24.stdout: 10: (()+0x1afdc0) [0x7f3073427dc0]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 11: (RGWFCGXProcess::handle_request(RGWRequest*)+0x7f) [0x7f307342844f]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 12: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x30) [0x7f307342aca0]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0x7f3073599f96]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 14: (ThreadPool::WorkThread::entry()+0x10) [0x7f307359aea0]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 15: (()+0x8182) [0x7f306f916182]
2015-09-24T23:37:09.777 INFO:tasks.rgw.client.1.plana24.stdout: 16: (clone()+0x6d) [0x7f306e274fbd]
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout:2015-09-24 23:37:09.777451 7f2ef8f69700 -1 *** Caught signal (Segmentation fault) **
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout: in thread 7f2ef8f69700
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout:
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout: ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout: 1: (()+0x2faefa) [0x7f3073572efa]
2015-09-24T23:37:09.778 INFO:tasks.rgw.client.1.plana24.stdout: 2: (()+0x10340) [0x7f306f91e340]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 3: (strlen()+0x2a) [0x7f306e20356a]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f30734f27b3]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f307352cb59]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f30734d120b]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x412) [0x7f30734b20e2]
2015-09-24T23:37:09.779 INFO:tasks.rgw.client.1.plana24.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x3a4) [0x7f30734b3474]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 9: (RGWCopyObj::execute()+0x1ce) [0x7f30734e743e]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 10: (()+0x1afdc0) [0x7f3073427dc0]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 11: (RGWFCGXProcess::handle_request(RGWRequest*)+0x7f) [0x7f307342844f]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 12: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x30) [0x7f307342aca0]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0x7f3073599f96]
2015-09-24T23:37:09.780 INFO:tasks.rgw.client.1.plana24.stdout: 14: (ThreadPool::WorkThread::entry()+0x10) [0x7f307359aea0]
2015-09-24T23:37:09.781 INFO:tasks.rgw.client.1.plana24.stdout: 15: (()+0x8182) [0x7f306f916182]
2015-09-24T23:37:09.781 INFO:tasks.rgw.client.1.plana24.stdout: 16: (clone()+0x6d) [0x7f306e274fbd]
2015-09-24T23:37:09.781 INFO:tasks.rgw.client.1.plana24.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-09-24T23:37:09.781 INFO:tasks.rgw.client.1.plana24.stdout:
2015-09-24T23:37:09.875 INFO:tasks.rgw.client.1.plana24.stdout:     0> 2015-09-24 23:37:09.777451 7f2ef8f69700 -1 *** Caught signal (Segmentation fault) **
2015-09-24T23:37:09.875 INFO:tasks.rgw.client.1.plana24.stdout: in thread 7f2ef8f69700
2015-09-24T23:37:09.876 INFO:tasks.rgw.client.1.plana24.stdout:
2015-09-24T23:37:09.876 INFO:tasks.rgw.client.1.plana24.stdout: ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
2015-09-24T23:37:09.876 INFO:tasks.rgw.client.1.plana24.stdout: 1: (()+0x2faefa) [0x7f3073572efa]
2015-09-24T23:37:09.877 INFO:tasks.rgw.client.1.plana24.stdout: 2: (()+0x10340) [0x7f306f91e340]
2015-09-24T23:37:09.877 INFO:tasks.rgw.client.1.plana24.stdout: 3: (strlen()+0x2a) [0x7f306e20356a]
2015-09-24T23:37:09.877 INFO:tasks.rgw.client.1.plana24.stdout: 4: (req_info::req_info(CephContext*, RGWEnv*)+0x253) [0x7f30734f27b3]
2015-09-24T23:37:09.877 INFO:tasks.rgw.client.1.plana24.stdout: 5: (RGWRESTStreamReadRequest::get_obj(RGWAccessKey&, std::map<std::string, std::string, std::less<std::string>, std::allocator<std::pair<std::string const, std::string> > >&, rgw_obj&)+0x179) [0x7f307352cb59]
2015-09-24T23:37:09.878 INFO:tasks.rgw.client.1.plana24.stdout: 6: (RGWRESTConn::get_obj(std::string const&, req_info*, rgw_obj&, bool, RGWGetDataCB*, RGWRESTStreamReadRequest**)+0x70b) [0x7f30734d120b]
2015-09-24T23:37:09.878 INFO:tasks.rgw.client.1.plana24.stdout: 7: (RGWRados::fetch_remote_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x412) [0x7f30734b20e2]
2015-09-24T23:37:09.878 INFO:tasks.rgw.client.1.plana24.stdout: 8: (RGWRados::copy_obj(RGWObjectCtx&, std::string const&, std::string const&, std::string const&, req_info*, std::string const&, rgw_obj&, rgw_obj&, RGWBucketInfo&, RGWBucketInfo&, long*, long*, long const*, long const*, char const*, char const*, RGWRados::AttrsMod, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&, RGWObjCategory, unsigned long, long, std::string*, std::string*, std::string*, rgw_err*, void (*)(long, void*), void*)+0x3a4) [0x7f30734b3474]
2015-09-24T23:37:09.878 INFO:tasks.rgw.client.1.plana24.stdout: 9: (RGWCopyObj::execute()+0x1ce) [0x7f30734e743e]
2015-09-24T23:37:09.878 INFO:tasks.rgw.client.1.plana24.stdout: 10: (()+0x1afdc0) [0x7f3073427dc0]
2015-09-24T23:37:09.879 INFO:tasks.rgw.client.1.plana24.stdout: 11: (RGWFCGXProcess::handle_request(RGWRequest*)+0x7f) [0x7f307342844f]
2015-09-24T23:37:09.879 INFO:tasks.rgw.client.1.plana24.stdout: 12: (RGWProcess::RGWWQ::_process(RGWRequest*)+0x30) [0x7f307342aca0]
2015-09-24T23:37:09.879 INFO:tasks.rgw.client.1.plana24.stdout: 13: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa56) [0x7f3073599f96]
2015-09-24T23:37:09.879 INFO:tasks.rgw.client.1.plana24.stdout: 14: (ThreadPool::WorkThread::entry()+0x10) [0x7f307359aea0]
2015-09-24T23:37:09.879 INFO:tasks.rgw.client.1.plana24.stdout: 15: (()+0x8182) [0x7f306f916182]
2015-09-24T23:37:09.880 INFO:tasks.rgw.client.1.plana24.stdout: 16: (clone()+0x6d) [0x7f306e274fbd]
2015-09-24T23:37:09.880 INFO:tasks.rgw.client.1.plana24.stdout: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-09-24T23:37:09.880 INFO:tasks.rgw.client.1.plana24.stdout:
2015-09-24T23:37:10.211 INFO:tasks.rgw.client.1.plana24.stderr:daemon-helper: command crashed with signal 11
2015-09-24T23:37:17.580 INFO:tasks.radosgw_agent.ubuntu@plana24.front.sepia.ceph.com.8000.syncdaemon.plana24.stderr:2015-09-24 23:37:17,583 1583 [radosgw_agent.worker][WARNING] encountered an error during sync: Http error code 500 content Internal Server Error
2015-09-24T23:37:27.649 INFO:tasks.radosgw_agent.ubuntu@plana24.front.sepia.ceph.com.8000.syncdaemon.plana24.stderr:2015-09-24 23:37:27,652 1583 [radosgw_agent.worker][DEBUG ] error geting op state: Http error code 500 content Internal Server Error
Actions
Copy link
 #4

Updated by Vasu Kulkarni over 8 years ago

  • Priority changed from Normal to High
Actions
Copy link
 #5

Updated by Yehuda Sadeh over 8 years ago

  • Status changed from New to Duplicate

Duplicate of #13239

Actions
Copy link

Also available in: Atom PDF