Project

General

Profile

Actions
Copy link

Bug #41513

closed

mgr fails to respawn in 14.2.2->nautilus upgrade

Added by Yuri Weinstein over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Kefu Chai
Category:
ceph-mgr
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.3
ceph-qa-suite:
upgrade/nautilus-p2p
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.front.sepia.ceph.com/yuriw-2019-08-22_16:46:25-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/
Jobs: '4241587', '4241602', '4241597', '4241592'
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2019-08-22_16:46:25-upgrade:nautilus-p2p-nautilus-distro-basic-smithi/4241587/teuthology.log

2019-08-24T16:43:23.634 INFO:teuthology.orchestra.run.smithi180.stderr:s3tests.functional.test_s3.test_object_copy_retaining_metadata ... ok
2019-08-24T16:43:28.339 INFO:teuthology.orchestra.run.smithi180.stderr:s3tests.functional.test_s3.test_object_copy_replacing_metadata ... ok
2019-08-24T16:43:29.701 INFO:teuthology.orchestra.run.smithi180.stderr:s3tests.functional.test_s3.test_object_copy_bucket_not_found ... ok
2019-08-24T16:43:31.093 INFO:teuthology.orchestra.run.smithi180.stderr:s3tests.functional.test_s3.test_object_copy_key_not_found ... ok
2019-08-24T16:43:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 124
2019-08-24T16:43:33.295 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 56, in task
    p.spawn(_run_spawned, ctx, confg, taskname)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/parallel.py", line 64, in _run_spawned
    mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/sequential.py", line 49, in task
    mgr.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph.py", line 1639, in restart
    healthy(ctx=ctx, config=dict(cluster=cluster))
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph.py", line 1477, in healthy
    manager.wait_for_clean()
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph_manager.py", line 2168, in wait_for_clean
    num_active_clean = self.get_num_active_clean()
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph_manager.py", line 2069, in get_num_active_clean
    pgs = self.get_pg_stats()
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph_manager.py", line 1853, in get_pg_stats
    out = self.raw_cluster_cmd('pg', 'dump', '--format=json')
  File "/home/teuthworker/src/github.com_ceph_ceph_nautilus/qa/tasks/ceph_manager.py", line 1157, in raw_cluster_cmd
    stdout=StringIO(),
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi150 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json'
Actions
Copy link

Also available in: Atom PDF