Actions
Bug #12470
closedradosgw-agent bootstrap failure (ubuntu 12.04)
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Steps to reproduce
- docker run -t -i ubuntu:12.04 bash
- apt-get install git python-virtualenv
- git clone https://github.com/ceph/radosgw-agent
- cd radosgw-agent
- ./bootstrap
Expected output:
Successfully installed PyYAML-3.11 funcsigs-0.4 mock-1.3.0 pbr-1.3.0 pluggy-0.3.0 py-1.4.30 pytest-2.7.2 six-1.9.0 tox-2.1.1 virtualenv-13.1.0
What happens instead
Downloading/unpacking pytest>=2.1.3 (from -r requirements-dev.txt (line 1))
Downloading pytest-2.7.2.tar.gz (532Kb): 532Kb downloaded
Running setup.py egg_info for package pytest
Downloading/unpacking mock>=1.0 (from -r requirements-dev.txt (line 2))
Downloading mock-1.3.0.tar.gz (70Kb): 70Kb downloaded
Running setup.py egg_info for package mock
mock requires setuptools>=17.1. Aborting installation
Complete output from command python setup.py egg_info:
mock requires setuptools>=17.1. Aborting installation
----------------------------------------
Command python setup.py egg_info failed with error code 1 in /radosgw-agent/virtualenv/build/mock
Storing complete log in /root/.pip/pip.log
Updated by Loïc Dachary over 8 years ago
Trying locally on Ubuntu 14.04, ./bootstrap works.
Updated by Loïc Dachary over 8 years ago
- Subject changed from radosgw-agent install failure in hammer backports run to radosgw-agent install failure in hammer backports run (ubuntu 12.04)
Updated by Loïc Dachary over 8 years ago
- Subject changed from radosgw-agent install failure in hammer backports run (ubuntu 12.04) to radosgw-agent bootstrap failure (ubuntu 12.04)
- Description updated (diff)
- Status changed from New to 12
- Priority changed from Normal to High
Updated by Vasu Kulkarni over 8 years ago
This is due to boto's egg dependency on setuptools 0.9.8 and mock 1.3.0 requires 17.1
Processing dependencies for radosgw-agent==1.2.3 Searching for boto==2.38.0 Best match: boto 2.38.0 Processing boto-2.38.0-py2.7.egg boto 2.38.0 is already the active version in easy-install.pth Installing taskadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing dynamodb_load script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing list_instances script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing fetch_file script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing instance_events script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing cq script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing asadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing cfadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing elbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing route53 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing mturk script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing kill_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing sdbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing bundle_image script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing cwutil script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing dynamodb_dump script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing glacier script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing pyami_sendmail script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing launch_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing lss3 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Installing s3put script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin Using /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg Searching for setuptools==0.9.8 Best match: setuptools 0.9.8
Updated by Ken Dreyer over 8 years ago
It's weird because I don't see "setuptools==0.9.8" @ https://github.com/boto/boto/blob/2.38.0/requirements.txt
Updated by Vasu Kulkarni over 8 years ago
- Assignee set to Alfredo Deza
Alfredo found the root cause in setup.py, Assigning to him.
Updated by Alfredo Deza over 8 years ago
- Status changed from 12 to Fix Under Review
Pull request opened https://github.com/ceph/radosgw-agent/pull/35
Updated by Alfredo Deza over 8 years ago
- Status changed from Fix Under Review to Resolved
merged commit fc3985a into master
Updated by Vasu Kulkarni over 8 years ago
Thanks Alfredo, this is working fine, Partial logs from test run
2015-08-10 14:29:18,398.398 INFO:teuthology.orchestra.run.magna087:Running: 'cd /home/ubuntu/cephtest && git clone -b master git://git.ceph.com/radosgw-agent.git radosgw-agent.client.0'
2015-08-10 14:29:19,107.107 INFO:teuthology.orchestra.run.magna087.stdout:Cloning into 'radosgw-agent.client.0'...
2015-08-10 14:29:22,412.412 INFO:teuthology.orchestra.run.magna087:Running: 'cd /home/ubuntu/cephtest && cd radosgw-agent.client.0 && ./bootstrap'
2015-08-10 14:29:22,631.631 INFO:teuthology.orchestra.run.magna087.stdout:New python executable in virtualenv/bin/python
2015-08-10 14:29:23,007.007 INFO:teuthology.orchestra.run.magna087.stdout:Installing Setuptools..............................................................................................................................................................................................................................done.
2015-08-10 14:29:23,471.471 INFO:teuthology.orchestra.run.magna087.stdout:Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
2015-08-10 14:29:23,552.552 INFO:teuthology.orchestra.run.magna087.stderr:/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_requires'
2015-08-10 14:29:23,553.553 INFO:teuthology.orchestra.run.magna087.stderr: warnings.warn(msg)
2015-08-10 14:29:23,567.567 INFO:teuthology.orchestra.run.magna087.stdout:running develop
2015-08-10 14:29:23,577.577 INFO:teuthology.orchestra.run.magna087.stdout:running egg_info
2015-08-10 14:29:23,577.577 INFO:teuthology.orchestra.run.magna087.stdout:creating radosgw_agent.egg-info
2015-08-10 14:29:23,577.577 INFO:teuthology.orchestra.run.magna087.stdout:writing requirements to radosgw_agent.egg-info/requires.txt
2015-08-10 14:29:23,577.577 INFO:teuthology.orchestra.run.magna087.stdout:writing radosgw_agent.egg-info/PKG-INFO
2015-08-10 14:29:23,578.578 INFO:teuthology.orchestra.run.magna087.stdout:writing top-level names to radosgw_agent.egg-info/top_level.txt
2015-08-10 14:29:23,578.578 INFO:teuthology.orchestra.run.magna087.stdout:writing dependency_links to radosgw_agent.egg-info/dependency_links.txt
2015-08-10 14:29:23,578.578 INFO:teuthology.orchestra.run.magna087.stdout:writing entry points to radosgw_agent.egg-info/entry_points.txt
2015-08-10 14:29:23,578.578 INFO:teuthology.orchestra.run.magna087.stdout:writing manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-08-10 14:29:23,585.585 INFO:teuthology.orchestra.run.magna087.stdout:reading manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-08-10 14:29:23,586.586 INFO:teuthology.orchestra.run.magna087.stdout:reading manifest template 'MANIFEST.in'
2015-08-10 14:29:23,590.590 INFO:teuthology.orchestra.run.magna087.stdout:writing manifest file 'radosgw_agent.egg-info/SOURCES.txt'
2015-08-10 14:29:23,592.592 INFO:teuthology.orchestra.run.magna087.stdout:running build_ext
2015-08-10 14:29:23,593.593 INFO:teuthology.orchestra.run.magna087.stdout:Creating /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/radosgw-agent.egg-link (link to .)
2015-08-10 14:29:23,593.593 INFO:teuthology.orchestra.run.magna087.stdout:Adding radosgw-agent 1.2.3 to easy-install.pth file
2015-08-10 14:29:23,594.594 INFO:teuthology.orchestra.run.magna087.stdout:Installing radosgw-agent script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:23,594.594 INFO:teuthology.orchestra.run.magna087.stdout:
2015-08-10 14:29:23,594.594 INFO:teuthology.orchestra.run.magna087.stdout:Installed /home/ubuntu/cephtest/radosgw-agent.client.0
2015-08-10 14:29:23,594.594 INFO:teuthology.orchestra.run.magna087.stdout:Processing dependencies for radosgw-agent==1.2.3
2015-08-10 14:29:23,595.595 INFO:teuthology.orchestra.run.magna087.stdout:Searching for PyYAML
2015-08-10 14:29:23,595.595 INFO:teuthology.orchestra.run.magna087.stdout:Reading https://pypi.python.org/simple/PyYAML/
2015-08-10 14:29:23,999.999 INFO:teuthology.orchestra.run.magna087.stdout:Best match: PyYAML 3.11
2015-08-10 14:29:23,999.999 INFO:teuthology.orchestra.run.magna087.stdout:Downloading https://pypi.python.org/packages/source/P/PyYAML/PyYAML-3.11.zip#md5=89cbc92cda979042533b640b76e6e055
2015-08-10 14:29:24,276.276 INFO:teuthology.orchestra.run.magna087.stdout:Processing PyYAML-3.11.zip
2015-08-10 14:29:24,406.406 INFO:teuthology.orchestra.run.magna087.stdout:Writing /tmp/easy_install-fv4bJo/PyYAML-3.11/setup.cfg
2015-08-10 14:29:24,407.407 INFO:teuthology.orchestra.run.magna087.stdout:Running PyYAML-3.11/setup.py -q bdist_egg --dist-dir /tmp/easy_install-fv4bJo/PyYAML-3.11/egg-dist-tmp-ldDYOh
2015-08-10 14:29:24,447.447 INFO:teuthology.orchestra.run.magna087.stderr:build/temp.linux-x86_64-2.7/check_libyaml.c:2:18: fatal error: yaml.h: No such file or directory
2015-08-10 14:29:24,448.448 INFO:teuthology.orchestra.run.magna087.stderr: #include <yaml.h>
2015-08-10 14:29:24,448.448 INFO:teuthology.orchestra.run.magna087.stderr: ^
2015-08-10 14:29:24,448.448 INFO:teuthology.orchestra.run.magna087.stderr:compilation terminated.
2015-08-10 14:29:24,466.466 INFO:teuthology.orchestra.run.magna087.stderr:
2015-08-10 14:29:24,467.467 INFO:teuthology.orchestra.run.magna087.stderr:libyaml is not found or a compiler error: forcing --without-libyaml
2015-08-10 14:29:24,467.467 INFO:teuthology.orchestra.run.magna087.stderr:(if libyaml is installed correctly, you may need to
2015-08-10 14:29:24,467.467 INFO:teuthology.orchestra.run.magna087.stderr: specify the option --include-dirs or uncomment and
2015-08-10 14:29:24,467.467 INFO:teuthology.orchestra.run.magna087.stderr: modify the parameter include_dirs in setup.cfg)
2015-08-10 14:29:24,520.520 INFO:teuthology.orchestra.run.magna087.stderr:zip_safe flag not set; analyzing archive contents...
2015-08-10 14:29:24,556.556 INFO:teuthology.orchestra.run.magna087.stdout:Adding PyYAML 3.11 to easy-install.pth file
2015-08-10 14:29:24,557.557 INFO:teuthology.orchestra.run.magna087.stdout:
2015-08-10 14:29:24,557.557 INFO:teuthology.orchestra.run.magna087.stdout:Installed /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/PyYAML-3.11-py2.7-linux-x86_64.egg
2015-08-10 14:29:26,019.019 INFO:teuthology.orchestra.run.magna087.stdout:Searching for boto>=2.2.2,<3.0.0
2015-08-10 14:29:26,019.019 INFO:teuthology.orchestra.run.magna087.stdout:Reading https://pypi.python.org/simple/boto/
2015-08-10 14:29:26,352.352 INFO:teuthology.orchestra.run.magna087.stdout:Best match: boto 2.38.0
2015-08-10 14:29:26,353.353 INFO:teuthology.orchestra.run.magna087.stdout:Downloading https://pypi.python.org/packages/source/b/boto/boto-2.38.0.tar.gz#md5=28112f29e9c7b10e12b6917a325e70ce
2015-08-10 14:29:26,839.839 INFO:teuthology.orchestra.run.magna087.stdout:Processing boto-2.38.0.tar.gz
2015-08-10 14:29:27,218.218 INFO:teuthology.orchestra.run.magna087.stdout:Writing /tmp/easy_install-en4pIJ/boto-2.38.0/setup.cfg
2015-08-10 14:29:27,218.218 INFO:teuthology.orchestra.run.magna087.stdout:Running boto-2.38.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-en4pIJ/boto-2.38.0/egg-dist-tmp-k_8ECP
2015-08-10 14:29:27,326.326 INFO:teuthology.orchestra.run.magna087.stderr:warning: no files found matching 'boto/mturk/test/*.doctest'
2015-08-10 14:29:27,327.327 INFO:teuthology.orchestra.run.magna087.stderr:warning: no files found matching 'boto/mturk/test/.gitignore'
2015-08-10 14:29:28,392.392 INFO:teuthology.orchestra.run.magna087.stderr:zip_safe flag not set; analyzing archive contents...
2015-08-10 14:29:28,397.397 INFO:teuthology.orchestra.run.magna087.stderr:boto.connection: module references __file__
2015-08-10 14:29:28,398.398 INFO:teuthology.orchestra.run.magna087.stderr:boto.__init__: module references __file__
2015-08-10 14:29:28,401.401 INFO:teuthology.orchestra.run.magna087.stderr:boto.vendored.six: module references __path__
2015-08-10 14:29:29,881.881 INFO:teuthology.orchestra.run.magna087.stdout:Adding boto 2.38.0 to easy-install.pth file
2015-08-10 14:29:29,887.887 INFO:teuthology.orchestra.run.magna087.stdout:Installing taskadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,888.888 INFO:teuthology.orchestra.run.magna087.stdout:Installing dynamodb_load script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,889.889 INFO:teuthology.orchestra.run.magna087.stdout:Installing list_instances script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,890.890 INFO:teuthology.orchestra.run.magna087.stdout:Installing fetch_file script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,892.892 INFO:teuthology.orchestra.run.magna087.stdout:Installing instance_events script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,893.893 INFO:teuthology.orchestra.run.magna087.stdout:Installing cq script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,895.895 INFO:teuthology.orchestra.run.magna087.stdout:Installing asadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,896.896 INFO:teuthology.orchestra.run.magna087.stdout:Installing cfadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,898.898 INFO:teuthology.orchestra.run.magna087.stdout:Installing elbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,900.900 INFO:teuthology.orchestra.run.magna087.stdout:Installing route53 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,904.904 INFO:teuthology.orchestra.run.magna087.stdout:Installing mturk script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,905.905 INFO:teuthology.orchestra.run.magna087.stdout:Installing kill_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,907.907 INFO:teuthology.orchestra.run.magna087.stdout:Installing sdbadmin script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,907.907 INFO:teuthology.orchestra.run.magna087.stdout:Installing bundle_image script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,909.909 INFO:teuthology.orchestra.run.magna087.stdout:Installing cwutil script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,910.910 INFO:teuthology.orchestra.run.magna087.stdout:Installing dynamodb_dump script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,911.911 INFO:teuthology.orchestra.run.magna087.stdout:Installing glacier script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,912.912 INFO:teuthology.orchestra.run.magna087.stdout:Installing pyami_sendmail script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,914.914 INFO:teuthology.orchestra.run.magna087.stdout:Installing launch_instance script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,916.916 INFO:teuthology.orchestra.run.magna087.stdout:Installing lss3 script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,918.918 INFO:teuthology.orchestra.run.magna087.stdout:Installing s3put script to /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/bin
2015-08-10 14:29:29,919.919 INFO:teuthology.orchestra.run.magna087.stdout:
2015-08-10 14:29:29,919.919 INFO:teuthology.orchestra.run.magna087.stdout:Installed /home/ubuntu/cephtest/radosgw-agent.client.0/virtualenv/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg
2015-08-10 14:29:29,968.968 INFO:teuthology.orchestra.run.magna087.stdout:Finished processing dependencies for radosgw-agent==1.2.3
2015-08-10 14:29:29,981.981 INFO:tasks.util.rgw:rgwadmin: client.0 : ['-n', 'client.0', 'region-map', 'get']
2015-08-10 14:29:29,981.981 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '-n', 'client.0', 'region-map', 'get']
2015-08-10 14:29:29,982.982 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 -n client.0 region-map get'
2015-08-10 14:29:30,090.090 INFO:tasks.util.rgw: json result: {u'regions': [{u'val': {u'zones': [{u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7280/'], u'name': u'r0z0', u'log_meta': u'true', u'log_data': u'true'}, {u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7281/'], u'name': u'r0z1', u'log_meta': u'false', u'log_data': u'false'}], u'name': u'region0', u'default_placement': u'default_placement', u'placement_targets': [{u'name': u'default_placement', u'tags': []}], u'master_zone': u'r0z0', u'api_name': u'api1', u'hostnames': [], u'is_master': u'true', u'endpoints': [u'http://magna087.ceph.redhat.com:7281/', u'http://magna087.ceph.redhat.com:7280/']}, u'key': u'region0'}], u'master_region': u'region0', u'user_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}, u'bucket_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}}
2015-08-10 14:29:30,090.090 INFO:tasks.util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'region-map', 'get']
2015-08-10 14:29:30,090.090 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'region-map', 'get']
2015-08-10 14:29:30,091.091 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region-map get'
2015-08-10 14:29:30,197.197 INFO:tasks.util.rgw: json result: {u'regions': [{u'val': {u'zones': [{u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7280/'], u'name': u'r0z0', u'log_meta': u'true', u'log_data': u'true'}, {u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7281/'], u'name': u'r0z1', u'log_meta': u'false', u'log_data': u'false'}], u'name': u'region0', u'default_placement': u'default_placement', u'placement_targets': [{u'name': u'default_placement', u'tags': []}], u'master_zone': u'r0z0', u'api_name': u'api1', u'hostnames': [], u'is_master': u'true', u'endpoints': [u'http://magna087.ceph.redhat.com:7281/', u'http://magna087.ceph.redhat.com:7280/']}, u'key': u'region0'}], u'master_region': u'region0', u'user_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}, u'bucket_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}}
2015-08-10 14:29:30,198.198 INFO:tasks.util.rgw:rgwadmin: client.0 : ['-n', 'client.0', 'zone', 'get', '--rgw-zone', 'r0z0']
2015-08-10 14:29:30,198.198 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '-n', 'client.0', 'zone', 'get', '--rgw-zone', 'r0z0']
2015-08-10 14:29:30,198.198 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 -n client.0 zone get --rgw-zone r0z0'
2015-08-10 14:29:30,308.308 INFO:tasks.util.rgw: json result: {u'control_pool': u'.region0.r0z0.control_pool', u'user_swift_pool': u'.region0.r0z0.user_swift_pool', u'user_uid_pool': u'.users.uid.0', u'gc_pool': u'.rgw.gc.0', u'intent_log_pool': u'.region0.r0z0.intent_log_pool', u'user_email_pool': u'.region0.r0z0.user_email_pool', u'log_pool': u'.region0.r0z0.log_pool', u'user_keys_pool': u'.users.0', u'domain_root': u'.region0.r0z0.domain_root', u'system_key': {u'access_key': u'0te6NH5mcdcq0Tc5i8i2', u'secret_key': u'Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv'}, u'usage_log_pool': u'.region0.r0z0.usage_log_pool', u'placement_pools': [{u'val': {u'data_pool': u'.region0.r0z0.data_pool', u'data_extra_pool': u'.region0.r0z0.data_extra_pool', u'index_pool': u'.region0.r0z0.index_pool'}, u'key': u'default_placement'}]}
2015-08-10 14:29:30,308.308 INFO:tasks.util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'zone', 'get', '--rgw-zone', 'r0z1']
2015-08-10 14:29:30,308.308 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'zone', 'get', '--rgw-zone', 'r0z1']
2015-08-10 14:29:30,309.309 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 zone get --rgw-zone r0z1'
2015-08-10 14:29:30,418.418 INFO:tasks.util.rgw: json result: {u'control_pool': u'.region0.r0z1.control_pool', u'user_swift_pool': u'.region0.r0z1.user_swift_pool', u'user_uid_pool': u'.users.uid.1', u'gc_pool': u'.rgw.gc.1', u'intent_log_pool': u'.region0.r0z1.intent_log_pool', u'user_email_pool': u'.region0.r0z1.user_email_pool', u'log_pool': u'.region0.r0z1.log_pool', u'user_keys_pool': u'.users.1', u'domain_root': u'.region0.r0z1.domain_root', u'system_key': {u'access_key': u'1te6NH5mcdcq0Tc5i8i3', u'secret_key': u'Py4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXw'}, u'usage_log_pool': u'.region0.r0z1.usage_log_pool', u'placement_pools': [{u'val': {u'data_pool': u'.region0.r0z1.data_pool', u'data_extra_pool': u'.region0.r0z1.data_extra_pool', u'index_pool': u'.region0.r0z1.index_pool'}, u'key': u'default_placement'}]}
2015-08-10 14:29:30,419.419 INFO:teuthology.orchestra.run.magna087:Running: u'daemon-helper kill /home/ubuntu/cephtest/radosgw-agent.client.0/radosgw-agent -v --src-access-key 0te6NH5mcdcq0Tc5i8i2 --src-secret-key Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv --source http://magna087.ceph.redhat.com:7280 --dest-access-key 1te6NH5mcdcq0Tc5i8i3 --dest-secret-key Py4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXw --max-entries 10 --log-file /home/ubuntu/cephtest/archive/rgw_sync_agent.client.0.log --object-sync-timeout 30 --test-server-host 0.0.0.0 --test-server-port 8000 http://magna087.ceph.redhat.com:7281'
2015-08-10 14:29:30,421.421 INFO:teuthology.run_tasks:Running task radosgw-admin...
2015-08-10 14:29:30,657.657 INFO:tasks.util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'region-map', 'get']
2015-08-10 14:29:30,657.657 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'region-map', 'get']
2015-08-10 14:29:30,657.657 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region-map get'
2015-08-10 14:29:30,661.661 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,633 5377 [radosgw_agent][INFO ] __ __ __ ___ ___
2015-08-10 14:29:30,661.661 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] /__` \ / |\ | / ` /\ / _` |__ |\ | |
2015-08-10 14:29:30,661.661 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] .__/ | | \| \__, /~~\ \__> |___ | \| |
2015-08-10 14:29:30,661.661 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] v1.2.3
2015-08-10 14:29:30,661.661 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] agent options:
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] args:
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] conf : None
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,634 5377 [radosgw_agent][INFO ] dest_access_key : ****************
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] dest_secret_key : ****************
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] destination : http://magna087.ceph.redhat.com:7281
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] incremental_sync_delay : 30
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] lock_timeout : 60
2015-08-10 14:29:30,662.662 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] log_file : /home/ubuntu/cephtest/archive/rgw_sync_agent.client.0.log
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] log_lock_time : 20
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] max_entries : 10
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,635 5377 [radosgw_agent][INFO ] metadata_only : False
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] num_workers : 1
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] object_sync_timeout : 30
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] prepare_error_delay : 10
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] quiet : False
2015-08-10 14:29:30,663.663 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] rgw_data_log_window : 30
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] source : http://magna087.ceph.redhat.com:7280
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] src_access_key : ****************
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,636 5377 [radosgw_agent][INFO ] src_secret_key : ****************
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] src_zone : None
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] sync_scope : incremental
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] test_server_host : 0.0.0.0
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] test_server_port : 8000
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] verbose : True
2015-08-10 14:29:30,664.664 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent][INFO ] versioned : False
2015-08-10 14:29:30,665.665 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,637 5377 [radosgw_agent.client][INFO ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10 14:29:30,772.772 INFO:tasks.util.rgw: json result: {u'regions': [{u'val': {u'zones': [{u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7280/'], u'name': u'r0z0', u'log_meta': u'true', u'log_data': u'true'}, {u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7281/'], u'name': u'r0z1', u'log_meta': u'false', u'log_data': u'false'}], u'name': u'region0', u'default_placement': u'default_placement', u'placement_targets': [{u'name': u'default_placement', u'tags': []}], u'master_zone': u'r0z0', u'api_name': u'api1', u'hostnames': [], u'is_master': u'true', u'endpoints': [u'http://magna087.ceph.redhat.com:7281/', u'http://magna087.ceph.redhat.com:7280/']}, u'key': u'region0'}], u'master_region': u'region0', u'user_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}, u'bucket_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}}
2015-08-10 14:29:30,773.773 INFO:tasks.util.rgw:region={u'val': {u'zones': [{u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7280/'], u'name': u'r0z0', u'log_meta': u'true', u'log_data': u'true'}, {u'bucket_index_max_shards': 0, u'endpoints': [u'http://magna087.ceph.redhat.com:7281/'], u'name': u'r0z1', u'log_meta': u'false', u'log_data': u'false'}], u'name': u'region0', u'default_placement': u'default_placement', u'placement_targets': [{u'name': u'default_placement', u'tags': []}], u'master_zone': u'r0z0', u'api_name': u'api1', u'hostnames': [], u'is_master': u'true', u'endpoints': [u'http://magna087.ceph.redhat.com:7281/', u'http://magna087.ceph.redhat.com:7280/']}, u'key': u'region0'} is_master=True
2015-08-10 14:29:30,773.773 INFO:tasks.util.rgw:master_zone=r0z0
2015-08-10 14:29:30,774.774 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo']
2015-08-10 14:29:30,775.775 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'user', 'info', '--uid', 'foo']
2015-08-10 14:29:30,775.775 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 user info --uid foo'
2015-08-10 14:29:30,894.894 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,887 5377 [radosgw_agent][INFO ] http://magna087.ceph.redhat.com:7280 endpoint does not support versioning
2015-08-10 14:29:30,895.895 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,887 5377 [radosgw_agent][WARNING] encountered issues reaching to endpoint http://magna087.ceph.redhat.com:7280
2015-08-10 14:29:30,895.895 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,887 5377 [radosgw_agent][WARNING] HTTP Error 403: Forbidden
2015-08-10 14:29:30,895.895 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:30,887 5377 [radosgw_agent][WARNING] TEST MODE - do not run unless you are testing this program
2015-08-10 14:29:35,271.271 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4']
2015-08-10 14:29:35,271.271 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'user', 'create', '--uid', 'foo', '--display-name', 'Foo', '--email', 'foo@foo.com', '--access-key', '9te6NH5mcdcq0Tc5i8i1', '--secret', 'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', '--max-buckets', '4']
2015-08-10 14:29:35,271.271 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 user create --uid foo --display-name Foo --email foo@foo.com --access-key 9te6NH5mcdcq0Tc5i8i1 --secret Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu --max-buckets 4'
2015-08-10 14:29:40,822.822 INFO:tasks.util.rgw: json result: {u'auid': 0, u'display_name': u'Foo', u'keys': [{u'access_key': u'9te6NH5mcdcq0Tc5i8i1', u'secret_key': u'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', u'user': u'foo'}], u'default_placement': u'', u'temp_url_keys': [], u'caps': [], u'max_buckets': 4, u'swift_keys': [], u'user_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}, u'placement_tags': [], u'suspended': 0, u'op_mask': u'read, write, delete', u'user_id': u'foo', u'email': u'foo@foo.com', u'subusers': [], u'bucket_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}}
2015-08-10 14:29:40,823.823 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com']
2015-08-10 14:29:40,823.823 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'user', 'create', '--uid', 'fud', '--display-name', 'Fud', '--email', 'foo@foo.com']
2015-08-10 14:29:40,823.823 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 user create --uid fud --display-name Fud --email foo@foo.com'
2015-08-10 14:29:42,675.675 INFO:tasks.util.rgw:rgwadmin: client.0 : ['user', 'info', '--uid', 'foo']
2015-08-10 14:29:42,676.676 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'user', 'info', '--uid', 'foo']
2015-08-10 14:29:42,676.676 INFO:teuthology.orchestra.run.magna087:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 user info --uid foo'
2015-08-10 14:29:45,108.108 INFO:tasks.util.rgw: json result: {u'auid': 0, u'display_name': u'Foo', u'keys': [{u'access_key': u'9te6NH5mcdcq0Tc5i8i1', u'secret_key': u'Ny4IOauQoL18Gp2zM7lC1vLmoawgqcYP/YGcWfXu', u'user': u'foo'}], u'default_placement': u'', u'temp_url_keys': [], u'caps': [], u'max_buckets': 4, u'swift_keys': [], u'user_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}, u'placement_tags': [], u'suspended': 0, u'op_mask': u'read, write, delete', u'user_id': u'foo', u'email': u'foo@foo.com', u'subusers': [], u'bucket_quota': {u'max_objects': -1, u'enabled': False, u'max_size_kb': -1}}
2015-08-10 14:29:45,108.108 INFO:tasks.util.rgw:sync agent magna087.ceph.redhat.com:8000
2015-08-10 14:29:45,608.608 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:45,601 5377 [radosgw_agent.client][INFO ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10 14:29:45,609.609 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:45,602 5377 [radosgw_agent.client][INFO ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10 14:29:45,614.614 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:29:45,607 5377 [radosgw_agent.sync][DEBUG ] 64 shards to check
2015-08-10 14:30:04,967.967 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:04,960 5377 [radosgw_agent.client][INFO ] creating connection to endpoint: http://magna087.ceph.redhat.com:7280
2015-08-10 14:30:04,967.967 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:04,961 5377 [radosgw_agent.client][INFO ] creating connection to endpoint: http://magna087.ceph.redhat.com:7281
2015-08-10 14:30:04,968.968 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stdout:region map is: {u'region0': [u'r0z0', u'r0z1']}
2015-08-10 14:30:04,969.969 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:04,962 5377 [radosgw_agent.sync][INFO ] Starting sync
2015-08-10 14:30:04,973.973 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:04,966 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 0
2015-08-10 14:30:05,148.148 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,141 5670 [radosgw_agent.worker][INFO ] finished processing shard 0
2015-08-10 14:30:05,149.149 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,141 5377 [radosgw_agent.sync][DEBUG ] synced item (0, []) successfully
2015-08-10 14:30:05,149.149 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,141 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 1
2015-08-10 14:30:05,149.149 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,141 5377 [radosgw_agent.sync][INFO ] 1/64 items processed
2015-08-10 14:30:05,307.307 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,300 5670 [radosgw_agent.worker][INFO ] finished processing shard 1
2015-08-10 14:30:05,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,300 5377 [radosgw_agent.sync][DEBUG ] synced item (1, []) successfully
2015-08-10 14:30:05,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,301 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 2
2015-08-10 14:30:05,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,301 5377 [radosgw_agent.sync][INFO ] 2/64 items processed
2015-08-10 14:30:05,694.694 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,687 5670 [radosgw_agent.worker][INFO ] finished processing shard 2
2015-08-10 14:30:05,694.694 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,687 5377 [radosgw_agent.sync][DEBUG ] synced item (2, []) successfully
2015-08-10 14:30:05,694.694 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,687 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 3
2015-08-10 14:30:05,694.694 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:05,687 5377 [radosgw_agent.sync][INFO ] 3/64 items processed
2015-08-10 14:30:06,054.054 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,047 5670 [radosgw_agent.worker][INFO ] finished processing shard 3
2015-08-10 14:30:06,055.055 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,047 5377 [radosgw_agent.sync][DEBUG ] synced item (3, []) successfully
2015-08-10 14:30:06,055.055 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,047 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 4
2015-08-10 14:30:06,055.055 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,048 5377 [radosgw_agent.sync][INFO ] 4/64 items processed
2015-08-10 14:30:06,206.206 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,199 5670 [radosgw_agent.worker][INFO ] finished processing shard 4
2015-08-10 14:30:06,206.206 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,199 5377 [radosgw_agent.sync][DEBUG ] synced item (4, []) successfully
2015-08-10 14:30:06,206.206 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,199 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 5
2015-08-10 14:30:06,206.206 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:06,199 5377 [radosgw_agent.sync][INFO ] 5/64 items processed
2015-08-10 14:30:09,392.392 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:09,385 5670 [radosgw_agent.worker][INFO ] finished processing shard 5
2015-08-10 14:30:09,393.393 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:09,385 5377 [radosgw_agent.sync][DEBUG ] synced item (5, []) successfully
2015-08-10 14:30:09,393.393 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:09,385 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 6
2015-08-10 14:30:09,393.393 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:09,385 5377 [radosgw_agent.sync][INFO ] 6/64 items processed
2015-08-10 14:30:11,978.978 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:11,970 5670 [radosgw_agent.worker][INFO ] finished processing shard 6
2015-08-10 14:30:11,979.979 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:11,970 5377 [radosgw_agent.sync][DEBUG ] synced item (6, []) successfully
2015-08-10 14:30:11,979.979 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:11,971 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 7
2015-08-10 14:30:11,979.979 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:11,971 5377 [radosgw_agent.sync][INFO ] 7/64 items processed
2015-08-10 14:30:12,122.122 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,114 5670 [radosgw_agent.worker][INFO ] finished processing shard 7
2015-08-10 14:30:12,122.122 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,115 5377 [radosgw_agent.sync][DEBUG ] synced item (7, []) successfully
2015-08-10 14:30:12,123.123 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,115 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 8
2015-08-10 14:30:12,123.123 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,115 5377 [radosgw_agent.sync][INFO ] 8/64 items processed
2015-08-10 14:30:12,348.348 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,341 5670 [radosgw_agent.worker][INFO ] finished processing shard 8
2015-08-10 14:30:12,349.349 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,341 5377 [radosgw_agent.sync][DEBUG ] synced item (8, []) successfully
2015-08-10 14:30:12,349.349 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,341 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 9
2015-08-10 14:30:12,349.349 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,341 5377 [radosgw_agent.sync][INFO ] 9/64 items processed
2015-08-10 14:30:12,492.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,484 5670 [radosgw_agent.worker][INFO ] finished processing shard 9
2015-08-10 14:30:12,492.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,484 5377 [radosgw_agent.sync][DEBUG ] synced item (9, []) successfully
2015-08-10 14:30:12,492.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,484 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 10
2015-08-10 14:30:12,492.492 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,484 5377 [radosgw_agent.sync][INFO ] 10/64 items processed
2015-08-10 14:30:12,676.676 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,668 5670 [radosgw_agent.worker][INFO ] finished processing shard 10
2015-08-10 14:30:12,676.676 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,668 5377 [radosgw_agent.sync][DEBUG ] synced item (10, []) successfully
2015-08-10 14:30:12,677.677 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,669 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 11
2015-08-10 14:30:12,677.677 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,669 5377 [radosgw_agent.sync][INFO ] 11/64 items processed
2015-08-10 14:30:12,836.836 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,828 5670 [radosgw_agent.worker][INFO ] finished processing shard 11
2015-08-10 14:30:12,836.836 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,828 5377 [radosgw_agent.sync][DEBUG ] synced item (11, []) successfully
2015-08-10 14:30:12,836.836 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,828 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 12
2015-08-10 14:30:12,837.837 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,828 5377 [radosgw_agent.sync][INFO ] 12/64 items processed
2015-08-10 14:30:13,003.003 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,996 5670 [radosgw_agent.worker][INFO ] finished processing shard 12
2015-08-10 14:30:13,004.004 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,996 5377 [radosgw_agent.sync][DEBUG ] synced item (12, []) successfully
2015-08-10 14:30:13,004.004 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,996 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 13
2015-08-10 14:30:13,004.004 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:12,996 5377 [radosgw_agent.sync][INFO ] 13/64 items processed
2015-08-10 14:30:13,164.164 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,156 5670 [radosgw_agent.worker][INFO ] finished processing shard 13
2015-08-10 14:30:13,164.164 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,157 5377 [radosgw_agent.sync][DEBUG ] synced item (13, []) successfully
2015-08-10 14:30:13,165.165 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,157 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 14
2015-08-10 14:30:13,165.165 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,157 5377 [radosgw_agent.sync][INFO ] 14/64 items processed
2015-08-10 14:30:13,307.307 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,299 5670 [radosgw_agent.worker][INFO ] finished processing shard 14
2015-08-10 14:30:13,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,300 5377 [radosgw_agent.sync][DEBUG ] synced item (14, []) successfully
2015-08-10 14:30:13,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,300 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 15
2015-08-10 14:30:13,308.308 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,300 5377 [radosgw_agent.sync][INFO ] 15/64 items processed
2015-08-10 14:30:13,500.500 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,492 5670 [radosgw_agent.worker][INFO ] finished processing shard 15
2015-08-10 14:30:13,500.500 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,493 5377 [radosgw_agent.sync][DEBUG ] synced item (15, []) successfully
2015-08-10 14:30:13,501.501 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,493 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 16
2015-08-10 14:30:13,501.501 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,493 5377 [radosgw_agent.sync][INFO ] 16/64 items processed
2015-08-10 14:30:13,668.668 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,661 5670 [radosgw_agent.worker][INFO ] finished processing shard 16
2015-08-10 14:30:13,669.669 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,661 5377 [radosgw_agent.sync][DEBUG ] synced item (16, []) successfully
2015-08-10 14:30:13,669.669 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,661 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 17
2015-08-10 14:30:13,669.669 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,661 5377 [radosgw_agent.sync][INFO ] 17/64 items processed
2015-08-10 14:30:13,795.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,787 5670 [radosgw_agent.worker][INFO ] finished processing shard 17
2015-08-10 14:30:13,795.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,787 5377 [radosgw_agent.sync][DEBUG ] synced item (17, []) successfully
2015-08-10 14:30:13,795.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,787 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 18
2015-08-10 14:30:13,795.795 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:13,787 5377 [radosgw_agent.sync][INFO ] 18/64 items processed
2015-08-10 14:30:18,734.734 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:18,726 5670 [radosgw_agent.worker][INFO ] finished processing shard 18
2015-08-10 14:30:18,735.735 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:18,727 5377 [radosgw_agent.sync][DEBUG ] synced item (18, []) successfully
2015-08-10 14:30:18,735.735 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:18,727 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 19
2015-08-10 14:30:18,735.735 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:18,727 5377 [radosgw_agent.sync][INFO ] 19/64 items processed
2015-08-10 14:30:19,194.194 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,186 5670 [radosgw_agent.worker][INFO ] finished processing shard 19
2015-08-10 14:30:19,195.195 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,187 5377 [radosgw_agent.sync][DEBUG ] synced item (19, []) successfully
2015-08-10 14:30:19,195.195 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,187 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 20
2015-08-10 14:30:19,195.195 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,187 5377 [radosgw_agent.sync][INFO ] 20/64 items processed
2015-08-10 14:30:19,362.362 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,354 5670 [radosgw_agent.worker][INFO ] finished processing shard 20
2015-08-10 14:30:19,362.362 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,355 5377 [radosgw_agent.sync][DEBUG ] synced item (20, []) successfully
2015-08-10 14:30:19,363.363 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,355 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 21
2015-08-10 14:30:19,363.363 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,355 5377 [radosgw_agent.sync][INFO ] 21/64 items processed
2015-08-10 14:30:19,522.522 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,514 5670 [radosgw_agent.worker][INFO ] finished processing shard 21
2015-08-10 14:30:19,522.522 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,514 5377 [radosgw_agent.sync][DEBUG ] synced item (21, []) successfully
2015-08-10 14:30:19,522.522 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,514 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 22
2015-08-10 14:30:19,523.523 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,514 5377 [radosgw_agent.sync][INFO ] 22/64 items processed
2015-08-10 14:30:19,665.665 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,657 5670 [radosgw_agent.worker][INFO ] finished processing shard 22
2015-08-10 14:30:19,665.665 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,657 5377 [radosgw_agent.sync][DEBUG ] synced item (22, []) successfully
2015-08-10 14:30:19,665.665 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,657 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 23
2015-08-10 14:30:19,665.665 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,657 5377 [radosgw_agent.sync][INFO ] 23/64 items processed
2015-08-10 14:30:19,842.842 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,834 5670 [radosgw_agent.worker][INFO ] finished processing shard 23
2015-08-10 14:30:19,842.842 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,834 5377 [radosgw_agent.sync][DEBUG ] synced item (23, []) successfully
2015-08-10 14:30:19,843.843 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,834 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 24
2015-08-10 14:30:19,843.843 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,835 5377 [radosgw_agent.sync][INFO ] 24/64 items processed
2015-08-10 14:30:19,985.985 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,977 5670 [radosgw_agent.worker][INFO ] finished processing shard 24
2015-08-10 14:30:19,986.986 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,977 5377 [radosgw_agent.sync][DEBUG ] synced item (24, []) successfully
2015-08-10 14:30:19,986.986 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,978 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 25
2015-08-10 14:30:19,986.986 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:19,978 5377 [radosgw_agent.sync][INFO ] 25/64 items processed
2015-08-10 14:30:20,162.162 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,153 5670 [radosgw_agent.worker][INFO ] finished processing shard 25
2015-08-10 14:30:20,162.162 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,154 5377 [radosgw_agent.sync][DEBUG ] synced item (25, []) successfully
2015-08-10 14:30:20,162.162 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,154 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 26
2015-08-10 14:30:20,163.163 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,154 5377 [radosgw_agent.sync][INFO ] 26/64 items processed
2015-08-10 14:30:20,329.329 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,321 5670 [radosgw_agent.worker][INFO ] finished processing shard 26
2015-08-10 14:30:20,329.329 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,321 5377 [radosgw_agent.sync][DEBUG ] synced item (26, []) successfully
2015-08-10 14:30:20,330.330 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,321 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 27
2015-08-10 14:30:20,330.330 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,322 5377 [radosgw_agent.sync][INFO ] 27/64 items processed
2015-08-10 14:30:20,488.488 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,481 5670 [radosgw_agent.worker][INFO ] finished processing shard 27
2015-08-10 14:30:20,489.489 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,481 5377 [radosgw_agent.sync][DEBUG ] synced item (27, []) successfully
2015-08-10 14:30:20,489.489 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,481 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 28
2015-08-10 14:30:20,489.489 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,481 5377 [radosgw_agent.sync][INFO ] 28/64 items processed
2015-08-10 14:30:20,580.580 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:20,573 5670 [radosgw_agent.worker][DEBUG ] syncing metadata type user key "foo"
2015-08-10 14:30:24,133.133 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,126 5670 [radosgw_agent.worker][INFO ] finished processing shard 28
2015-08-10 14:30:24,134.134 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,126 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 29
2015-08-10 14:30:24,134.134 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,126 5377 [radosgw_agent.sync][DEBUG ] synced item (28, []) successfully
2015-08-10 14:30:24,225.225 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,218 5377 [radosgw_agent.sync][INFO ] 29/64 items processed
2015-08-10 14:30:24,310.310 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,302 5670 [radosgw_agent.worker][INFO ] finished processing shard 29
2015-08-10 14:30:24,311.311 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,303 5377 [radosgw_agent.sync][DEBUG ] synced item (29, []) successfully
2015-08-10 14:30:24,311.311 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,303 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 30
2015-08-10 14:30:24,311.311 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:24,303 5377 [radosgw_agent.sync][INFO ] 30/64 items processed
2015-08-10 14:30:26,188.188 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,180 5670 [radosgw_agent.worker][INFO ] finished processing shard 30
2015-08-10 14:30:26,188.188 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,180 5377 [radosgw_agent.sync][DEBUG ] synced item (30, []) successfully
2015-08-10 14:30:26,189.189 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,180 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 31
2015-08-10 14:30:26,189.189 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,180 5377 [radosgw_agent.sync][INFO ] 31/64 items processed
2015-08-10 14:30:26,339.339 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,331 5670 [radosgw_agent.worker][INFO ] finished processing shard 31
2015-08-10 14:30:26,339.339 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,332 5377 [radosgw_agent.sync][DEBUG ] synced item (31, []) successfully
2015-08-10 14:30:26,339.339 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,332 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 32
2015-08-10 14:30:26,339.339 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,332 5377 [radosgw_agent.sync][INFO ] 32/64 items processed
2015-08-10 14:30:26,515.515 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,507 5670 [radosgw_agent.worker][INFO ] finished processing shard 32
2015-08-10 14:30:26,516.516 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,508 5377 [radosgw_agent.sync][DEBUG ] synced item (32, []) successfully
2015-08-10 14:30:26,516.516 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,508 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 33
2015-08-10 14:30:26,516.516 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,508 5377 [radosgw_agent.sync][INFO ] 33/64 items processed
2015-08-10 14:30:26,708.708 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,700 5670 [radosgw_agent.worker][INFO ] finished processing shard 33
2015-08-10 14:30:26,709.709 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,701 5377 [radosgw_agent.sync][DEBUG ] synced item (33, []) successfully
2015-08-10 14:30:26,709.709 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,701 5670 [radosgw_agent.worker][INFO ] 5670 is processing shard number 34
2015-08-10 14:30:26,709.709 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,701 5377 [radosgw_agent.sync][INFO ] 34/64 items processed
2015-08-10 14:30:26,868.868 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,860 5670 [radosgw_agent.worker][INFO ] finished processing shard 34
2015-08-10 14:30:26,868.868 INFO:tasks.radosgw_agent.ubuntu@magna087.ceph.redhat.com.8000.syncdaemon.magna087.stderr:2015-08-10 14:30:26,861 5377 [radosgw_agent.sync][DEBUG ] synced item (34, []) successfully
Actions