Actions
Bug #40155
closed"make: *** [hello_world_cpp] Error 127" in rados
% Done:
0%
Source:
Tags:
Backport:
luminous mimic nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/yuriw-2019-06-03_15:13:14-rados-wip-yuri4-testing-2019-05-30-2109-nautilus-distro-basic-smithi/
Job: 4001474
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2019-06-03_15:13:14-rados-wip-yuri4-testing-2019-05-30-2109-nautilus-distro-basic-smithi/4001474/teuthology.log
2019-06-03T15:39:15.522 INFO:tasks.workunit.client.0.smithi037.stderr:+ make all-system
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stdout:g++ -std=c++11 -Wno-unused-parameter -Wall -Wextra -Werror -g -o hello_world_cpp hello_world.cc -lrados -lradosstriper
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:make: g++: Command not found
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:make: *** [hello_world_cpp] Error 127
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:+ cleanup
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_c
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_cpp
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_radosstriper_cpp
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_radosstriper.cc
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_c.c
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world.cc
2019-06-03T15:39:15.528 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.528 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/Makefile
2019-06-03T15:39:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 2
2019-06-03T15:39:15.529 INFO:tasks.workunit:Stopping ['rados/test_librados_build.sh'] on client.0...
2019-06-03T15:39:15.529 INFO:teuthology.orchestra.run.smithi037:Running:
2019-06-03T15:39:15.529 INFO:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2019-06-03T15:39:15.792 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 136, in task
cleanup=cleanup)
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients
timeout=timeout)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 409, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test rados/test_librados_build.sh) on smithi037 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e8040c5146f783fb9c35d955bb1c3484b9370 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h
Updated by Nathan Cutler almost 5 years ago
- Assignee set to David Galloway
A new OS+version combo?
os_type: rhel os_version: '7.6'
The root cause is:
make: g++: Command not found
Assigning to David in the hopes he can add the dependency to the RHEL 7.6 image.
Updated by David Galloway almost 5 years ago
- Category set to User access
- Status changed from New to 4
Is it possible make has a messed up env there? The gcc-c++ package is installed on all RHEL7 testnodes.
https://github.com/ceph/ceph-cm-ansible/blob/master/roles/testnode/vars/redhat_7.yml#L28
Updated by David Galloway almost 5 years ago
Yeah.
[root@smithi009 ~]# yum whatprovides g++ Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager Loading mirror speeds from cached hostfile * epel: ftp.linux.ncsu.edu gcc-c++-4.8.2-16.el7.x86_64 : C++ support for GCC Repo : rhel-7-server-rpms Matched from: Filename : /usr/bin/g++
Note the PATH in the teuthology log
2019-06-03T15:39:13.556 INFO:teuthology.orchestra.run.smithi037:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e8040c5146f783fb9c35d955bb1c3484b9370 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh
Updated by Nathan Cutler almost 5 years ago
The workunit task sets the path to "PATH=$PATH:/usr/sbin" but I don't see how that would lead to "g++: Command not found" if /usr/bin/g++ exists on the system and is executable? Can you elaborate?
Updated by David Galloway almost 5 years ago
Sure, I guess my question is: Did $PATH get munged before the workunit test got ran?
For example, if $PATH is empty, and the task sets the path to PATH=$PATH:/usr/sbin, that would result in PATH = /usr/sbin, right? So /usr/bin/g++ wouldn't be found.
Or am I wayyy off base?
Either way, I looked through the teuthology log for that job and gcc-c++ is installed so I'm not sure this is an infra issue.
Updated by Nathan Cutler almost 5 years ago
David Galloway wrote:
Sure, I guess my question is: Did $PATH get munged before the workunit test got ran?
If it did, that would hopefully be in the log, and it would also cause problems for any other permutation of this test (not just RHEL 7.6).
Anyway, I'll instrument the test to dump the value of PATH to the log.
Updated by Nathan Cutler almost 5 years ago
- Status changed from 4 to In Progress
- Assignee changed from David Galloway to Kefu Chai
moving to Ceph project to get "Pull Request ID" and backports
Updated by Nathan Cutler almost 5 years ago
- Project changed from sepia to Ceph
- Category deleted (
User access) - Pull request ID set to 28484
Updated by Nathan Cutler almost 5 years ago
- Backport set to luminous mimic nautilus
Updated by Kefu Chai almost 5 years ago
- Status changed from In Progress to Pending Backport
Updated by Nathan Cutler almost 5 years ago
- Copied to Backport #40318: luminous: "make: *** [hello_world_cpp] Error 127" in rados added
Updated by Nathan Cutler almost 5 years ago
- Copied to Backport #40319: nautilus: "make: *** [hello_world_cpp] Error 127" in rados added
Updated by Nathan Cutler almost 5 years ago
- Copied to Backport #40320: mimic: "make: *** [hello_world_cpp] Error 127" in rados added
Updated by Nathan Cutler over 4 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions