Bug #23007
Updated by Nathan Cutler about 6 years ago
The following tests fail:
* rados/singleton-nomsgr/{all/lfn-upgrade-infernalis.yaml rados.yaml}
* rados/singleton-nomsgr/{all/lfn-upgrade-hammer.yaml rados.yaml}
In both cases we see:
<pre>
2018-02-07T09:41:42.911 INFO:teuthology.orchestra.run.smithi060:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph pg scrub 1.0'
2018-02-07T09:41:43.048 INFO:teuthology.orchestra.run.smithi060.stderr:Error EAGAIN: pg 1.0 primary osd.1 not up
2018-02-07T09:41:43.057 INFO:tasks.create_verify_lfn_objects:ceph_verify_lfn_objects verifying...
</pre>
Then ceph_verify_lfn_objects task runs for awhile, until squawking:
<pre>
2018-02-07T09:35:56.690 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/sequential.py", line 46, in task
mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 2057, in task
fn(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 1485, in do_pg_scrub
self.raw_cluster_cmd('pg', stype, self.get_pgid(pool, pgnum))
File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 881, in raw_cluster_cmd
stdout=StringIO(),
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 423, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 155, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 177, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi016 with status 11: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph pg scrub 1.0'
</pre>
Logs:
* http://pulpito.front.sepia.ceph.com/smithfarm-2018-02-06_21:07:15-rados-wip-jewel-backports-distro-basic-smithi/2160748
* http://pulpito.front.sepia.ceph.com/smithfarm-2018-02-06_21:07:15-rados-wip-jewel-backports-distro-basic-smithi/2160760
After the first run, further attempts to reproduce fail in the same place with a different backtrace:
<pre>
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/sequential.py", line 48, in task
mgr.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 437, in upgrade
upgrade_common(ctx, config, upgrade_old_style)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 382, in upgrade_common
deploy_style(ctx, node, remote, pkgs, system_type)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/__init__.py", line 265, in upgrade_old_style
deb._upgrade_packages(ctx, node, remote, pkgs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/deb.py", line 183, in _upgrade_packages
builder.install_repo()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/packaging.py", line 778, in install_repo
self._install_deb_repo()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/packaging.py", line 976, in _install_deb_repo
repo = self._get_repo()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/packaging.py", line 964, in _get_repo
resp.raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/requests/models.py", line 893, in raise_for_status
raise HTTPError(http_error_msg, response=self)
HTTPError: 502 Server Error: Bad Gateway for url: https://4.chacra.ceph.com/repos/ceph/wip-jewel-backports/a95e2c4958b256d75ea1c732c2f2cce45f024081/ubuntu/trusty/flavors/default/repo
</pre>
Logs:
* http://pulpito.front.sepia.ceph.com/smithfarm-2018-02-15_10:48:02-rados-wip-jewel-backports-distro-basic-smithi/2191706
* http://pulpito.front.sepia.ceph.com/smithfarm-2018-02-15_10:48:02-rados-wip-jewel-backports-distro-basic-smithi/2191707