Project

General

Profile

Actions

Bug #16506

closed

"line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithi

Added by Yuri Weinstein almost 8 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/client-upgrade
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2016-06-26_02:45:02-upgrade:client-upgrade-jewel-distro-basic-smithi/
Jobs: ['278397', '278401']
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-06-26_02:45:02-upgrade:client-upgrade-jewel-distro-basic-smithi/278401/teuthology.log

2016-06-28T01:16:50.551 INFO:teuthology.orchestra.run.smithi012:Running: 'sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring'
2016-06-28T01:16:50.669 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mds/ceph-a/keyring
2016-06-28T01:16:50.672 INFO:tasks.ceph_client:Setting up client nodes...
2016-06-28T01:16:50.672 INFO:teuthology.orchestra.run.smithi009:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'
2016-06-28T01:16:50.750 INFO:teuthology.orchestra.run.smithi009.stderr:/bin/adjust-ulimits: line 16: exec: ceph-coverage: not found
2016-06-28T01:16:50.751 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph.py", line 592, in cluster
    cclient.create_keyring(ctx, cluster_name)
  File "/var/lib/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_client.py", line 40, in create_keyring
    client_keyring,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 394, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 121, in wait
    label=self.label)
CommandFailedError: Command failed on smithi009 with status 127: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'

Related issues 3 (0 open3 closed)

Related to Ceph - Bug #17811: "line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithiResolved11/07/2016

Actions
Has duplicate Ceph - Bug #16718: "/bin/adjust-ulimits: line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithiDuplicate07/18/2016

Actions
Copied to Ceph - Backport #17344: jewel: "line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithiResolvedLoïc DacharyActions
Actions #1

Updated by Yuri Weinstein almost 8 years ago

  • Project changed from rbd to Ceph
Actions #2

Updated by Samuel Just over 7 years ago

  • Status changed from New to Can't reproduce
Actions #3

Updated by Yuri Weinstein over 7 years ago

  • Status changed from Can't reproduce to New

Still seeing this in jewel 10.2.3 point release
Run: http://pulpito.ceph.com/teuthology-2016-08-28_02:45:02-upgrade:client-upgrade-jewel-distro-basic-smithi/
Jobs: 388579, 388580, 388583, 388584
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-08-28_02:45:02-upgrade:client-upgrade-jewel-distro-basic-smithi/388579/teuthology.log

2016-08-28T19:15:50.037 INFO:teuthology.orchestra.run.smithi007:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'
2016-08-28T19:15:50.127 INFO:teuthology.orchestra.run.smithi007.stderr:/bin/adjust-ulimits: line 16: exec: ceph-coverage: not found
2016-08-28T19:15:50.127 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph.py", line 592, in cluster
    cclient.create_keyring(ctx, cluster_name)
  File "/home/teuthworker/src/ceph-qa-suite_jewel/tasks/ceph_client.py", line 40, in create_keyring
    client_keyring,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 192, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 402, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 166, in wait
    label=self.label)
CommandFailedError: Command failed on smithi007 with status 127: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring'
Actions #4

Updated by Samuel Just over 7 years ago

  • Assignee set to Loïc Dachary
Actions #5

Updated by Loïc Dachary over 7 years ago

  • Has duplicate Bug #16718: "/bin/adjust-ulimits: line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithi added
Actions #6

Updated by Loïc Dachary over 7 years ago

  • Status changed from New to In Progress

Re-running twice on VPS

filter="upgrade:client-upgrade/hammer-client-x/basic/{0-cluster/start.yaml 1-install/hammer-client-x.yaml 2-workload/rbd_api_tests.yaml}" 
teuthology-suite --num 2 --verbose --suite-branch jewel --ceph jewel --suite upgrade/client-upgrade --filter "$filter" --machine-type vps --email loic@dachary.org --priority 50
Actions #7

Updated by Loïc Dachary over 7 years ago

The ceph-coverage executable is provided by the ceph-test package but it is excluded from the installation.

Actions #8

Updated by Loïc Dachary over 7 years ago

  • Backport set to jewel
Actions #10

Updated by Loïc Dachary over 7 years ago

  • Status changed from In Progress to Fix Under Review

@Yuri the test passed on VPS (modulo a known bug http://tracker.ceph.com/issues/16717) and a fix is ready for review. I believe this is not a blocker for the upcoming 10.2.3 release.

Actions #11

Updated by Kefu Chai over 7 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions #12

Updated by Nathan Cutler over 7 years ago

  • Copied to Backport #17344: jewel: "line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithi added
Actions #13

Updated by Loïc Dachary over 7 years ago

  • Status changed from Pending Backport to Resolved
Actions #14

Updated by Yuri Weinstein over 7 years ago

  • Related to Bug #17811: "line 16: exec: ceph-coverage: not found" in upgrade:client-upgrade-jewel-distro-basic-smithi added
Actions #15

Updated by Loïc Dachary over 7 years ago

https://github.com/ceph/ceph-qa-suite/pull/1295 fixes it for real in master and the corresponding jewel backport is https://github.com/ceph/ceph-qa-suite/pull/1296

Actions

Also available in: Atom PDF