Project

General

Profile

Actions

Bug #11210

closed

radosgw_admin.py fails assert time.time() - timestamp <= (20 * 60)

Added by Sage Weil about 9 years ago. Updated about 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2015-03-21T07:20:46.965 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:46.966 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:46.966 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj'
2015-03-21T07:20:48.297 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38,
 u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21:
00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00:
00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat
egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]}
2015-03-21T07:20:49.297 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:49.298 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:49.298 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj'
2015-03-21T07:20:51.154 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38,
 u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21:
00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00:
00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat
egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]}
2015-03-21T07:20:52.154 INFO:tasks.util.rgw:rgwadmin: client.0 : ['usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:52.154 INFO:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', 'usage', 'show', '--categories', 'delete_obj']
2015-03-21T07:20:52.155 INFO:teuthology.orchestra.run.plana24:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 usage show --categories delete_obj'
2015-03-21T07:20:53.428 INFO:tasks.util.rgw: json result: {u'summary': [{u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'client0-system-user', u'categories': []}, {u'total': {u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}, u'user': u'foo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38,
 u'ops': 2}]}, {u'total': {u'successful_ops': 0, u'bytes_received': 0, u'bytes_sent': 0, u'ops': 0}, u'user': u'fud', u'categories': []}], u'entries': [{u'owner': u'client0-system-user', u'buckets': [{u'epoch': 1426971600, u'bucket': u'', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'foo', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'categories': [], u'time': u'2015-03-21 21:
00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo', u'categories': [{u'category': u'delete_obj', u'successful_ops': 2, u'bytes_received': 0, u'bytes_sent': 38, u'ops': 2}], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo3', u'categories': [], u'time': u'2015-03-21 21:00:
00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoo4', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}, {u'epoch': 1426971600, u'bucket': u'myfoodata2', u'categories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}, {u'owner': u'fud', u'buckets': [{u'epoch': 1426971600, u'bucket': u'mybar', u'cat
egories': [], u'time': u'2015-03-21 21:00:00.000000Z'}]}]}
2015-03-21T07:20:54.429 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_hammer/tasks/radosgw_admin.py", line 827, in task
    assert time.time() - timestamp <= (20 * 60)
AssertionError
2015-03-21T07:20:54.466 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=981aa0a38c0c4e9fa4f16992a510117f
AssertionError

ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814287
ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814280
ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi/814308
etc.

probabl most of these:

ubuntu@teuthology:/a/sage-2015-03-20_17:42:17-rgw-hammer-distro-basic-multi$ teuthology-ls . | grep fail\ 
814280 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 4830s
814287 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-profile.yaml} 2621s
814306 fail scheduled_sage@flab rgw/multifs/{clusters/fixed-2.yaml frontend/apache.yaml fs/ext4.yaml overrides.yaml rgw_pool_type/ec-cache.yaml tasks/rgw_readwrite.yaml} 767s
814308 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 2599s
814329 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/replicated.yaml} 1379s
814350 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec.yaml} 4555s
814357 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/replicated.yaml} 3054s
814378 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec.yaml} 2642s
814392 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-data-sync.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 4303s
814399 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/apache.yaml overrides.yaml rgw_pool_type/ec-profile.yaml} 2489s
814420 fail scheduled_sage@flab rgw/singleton/{all/radosgw-admin-multi-region.yaml frontend/civetweb.yaml overrides.yaml rgw_pool_type/ec-cache.yaml} 2317s

Actions #1

Updated by Alfredo Deza about 9 years ago

  • Status changed from New to 12
  • Assignee set to Alfredo Deza
Actions #2

Updated by Alfredo Deza about 9 years ago

Just verified that it is a matter of cherry-picking Loic's commit cb3af234e6af6a568b5b0238c747d2d94a132400

Actions #3

Updated by Alfredo Deza about 9 years ago

Actual cherry pick hash is: 97e6d808f086834f2350084d5de36be285ad2bde

Actions #4

Updated by Alfredo Deza about 9 years ago

  • Status changed from 12 to 7

cherry-picked the changes to hammer

Actions #5

Updated by Alfredo Deza about 9 years ago

  • Status changed from 7 to Resolved

Seems like we are no longer seeing this so marking as resolved for now.

Actions

Also available in: Atom PDF