Project

General

Profile

Bug #20617

Exception: timed out waiting for mon to be updated with osd.0: 0 < 47244640268

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-07-12T19:40:39.233 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats'
2017-07-12T19:40:39.300 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.291616 7f493db84700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.310 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.301221 7f493db84700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.340 INFO:teuthology.orchestra.run.smithi172.stdout:47244640268
2017-07-12T19:40:39.341 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats'
2017-07-12T19:40:39.417 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.408817 7fccf4d03700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.435 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.426120 7fccf4d03700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.466 INFO:teuthology.orchestra.run.smithi172.stdout:51539607565
2017-07-12T19:40:39.466 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats'
2017-07-12T19:40:39.540 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.531911 7fe202e7e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.550 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.541517 7fe202e7e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.581 INFO:teuthology.orchestra.run.smithi172.stdout:34359738378
2017-07-12T19:40:39.581 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0'
2017-07-12T19:40:39.653 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.644395 7f42c3178700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.659 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:40:39.650567 7f42c3178700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:40:39.746 INFO:teuthology.orchestra.run.smithi172.stdout:0
2017-07-12T19:40:39.746 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640268
 got 0
 for osd.0
...
2017-07-12T19:41:04.862 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0'
2017-07-12T19:41:04.937 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:41:04.929239 7fab1522e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:04.943 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:41:04.935034 7fab1522e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:05.036 INFO:teuthology.orchestra.run.smithi172.stdout:0
2017-07-12T19:41:05.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640268
 got 0
 for osd.0
2017-07-12T19:41:05.731 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:05.723020 7f6492f27700 -1 mgr handle_mgr_map I was active but no longer am
2017-07-12T19:41:05.745 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:05.737406 7f4bce772300 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:05.745 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:05.737596 7f4bce772300 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:05.747 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:05.739070 7f4bce772300 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:06.037 INFO:teuthology.orchestra.run.smithi172:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0'
2017-07-12T19:41:06.112 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:41:06.104411 7f4249b2e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:06.118 INFO:teuthology.orchestra.run.smithi172.stderr:2017-07-12 19:41:06.109580 7f4249b2e700 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-12T19:41:06.209 INFO:teuthology.orchestra.run.smithi172.stdout:0
2017-07-12T19:41:06.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640268
 got 0
 for osd.0
2017-07-12T19:41:06.386 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.377695 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/server_addr not found
2017-07-12T19:41:06.386 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.377703 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get server_addr not found
2017-07-12T19:41:06.386 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.377712 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/server_port not found
2017-07-12T19:41:06.386 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.377752 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get server_port not found
2017-07-12T19:41:06.387 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378048 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/crt not found
2017-07-12T19:41:06.389 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378053 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get crt not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378058 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/crt_file not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378062 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get crt_file not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378070 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/key not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378073 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get key not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378078 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get x/key_file not found
2017-07-12T19:41:06.390 INFO:tasks.ceph.mgr.x.smithi172.stderr:2017-07-12 19:41:06.378082 7f4bbd5c4700 -1 mgr ceph_config_get ceph_config_get key_file not found
2017-07-12T19:41:07.210 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/divergent_priors2.py", line 38, in task
    manager.flush_pg_stats([0, 1, 2])
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/ceph_manager.py", line 1216, in flush_pg_stats
    format(osd=osd, got=got, need=need))
Exception: timed out waiting for mon to be updated with osd.0: 0
 < 47244640268

/a/sage-2017-07-12_19:30:01-rados-wip-sage-testing-distro-basic-smithi/1392228

ill-timed mgr restart means we blow through the flush stats timeout.

History

#1 Updated by Sage Weil over 6 years ago

  • Status changed from In Progress to Fix Under Review

#2 Updated by Sage Weil over 6 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF