Actions
Bug #11210
closedradosgw_admin.py fails assert time.time() - timestamp <= (20 * 60)
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2015-03-21T07:20:46.965 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:46.966 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:46.966 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj' 2015-03-21T07:20:48.297 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21: 00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00: 00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]} 2015-03-21T07:20:49.297 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:49.298 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:49.298 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj' 2015-03-21T07:20:51.154 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21: 00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00: 00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]} 2015-03-21T07:20:52.154 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:52.154 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj'] 2015-03-21T07:20:52.155 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj' 2015-03-21T07:20:53.428 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21: 00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00: 00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]} 2015-03-21T07:20:54.429 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task return fn(**kwargs) File "/var/lib/teuthworker/src/ceph-qa-suite_hammer/tasks/radosgw_admin.py", line 827, in task assert time.time() - timestamp <= (20 * 60) AssertionError 2015-03-21T07:20:54.466 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=981aa0a38c0c4e9fa4f16992a510117f AssertionErrorubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814287
ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814280
ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814308
etc.
probabl most of these:
ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi$ teuthology-ls . | grep fail\ 814280 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 4830s 814287 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-profile.yaml} 2621s 814306 fail scheduled_sage@flab rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/ext4.yaml overrides.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_readwrite.yaml} 767s 814308 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 2599s 814329 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/replicated.yaml} 1379s 814350 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec.yaml} 4555s 814357 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/replicated.yaml} 3054s 814378 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec.yaml} 2642s 814392 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 4303s 814399 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-profile.yaml} 2489s 814420 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 2317s
Updated by Alfredo Deza about 9 years ago
- Status changed from New to 12
- Assignee set to Alfredo Deza
Updated by Alfredo Deza about 9 years ago
Just verified that it is a matter of cherry-picking Loic's commit cb3af234e6af6a568b5b0238c747d2d94a132400
Updated by Alfredo Deza about 9 years ago
Actual cherry pick hash is: 97e6d808f086834f2350084d5de36be285ad2bde
Updated by Alfredo Deza about 9 years ago
- Status changed from 12 to 7
cherry-picked the changes to hammer
Updated by Alfredo Deza about 9 years ago
- Status changed from 7 to Resolved
Seems like we are no longer seeing this so marking as resolved for now.
Actions