Project

General

Profile

Bug #40155

"make: *** [hello_world_cpp] Error 127" in rados

Added by Yuri Weinstein about 2 years ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous mimic nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/yuriw-2019-06-03_15:13:14-rados-wip-yuri4-testing-2019-05-30-2109-nautilus-distro-basic-smithi/
Job: 4001474
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2019-06-03_15:13:14-rados-wip-yuri4-testing-2019-05-30-2109-nautilus-distro-basic-smithi/4001474/teuthology.log

2019-06-03T15:39:15.522 INFO:tasks.workunit.client.0.smithi037.stderr:+ make all-system
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stdout:g++ -std=c++11 -Wno-unused-parameter -Wall -Wextra -Werror -g   -o hello_world_cpp hello_world.cc -lrados -lradosstriper
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:make: g++: Command not found
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:make: *** [hello_world_cpp] Error 127
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:+ cleanup
2019-06-03T15:39:15.523 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_c
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_cpp
2019-06-03T15:39:15.524 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_radosstriper_cpp
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.525 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_radosstriper.cc
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world_c.c
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.526 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/hello_world.cc
2019-06-03T15:39:15.528 INFO:tasks.workunit.client.0.smithi037.stderr:+ for f in '$BINARIES$SOURCES'
2019-06-03T15:39:15.528 INFO:tasks.workunit.client.0.smithi037.stderr:+ rm -f /home/ubuntu/cephtest/mnt.0/client.0/tmp/Makefile
2019-06-03T15:39:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 2
2019-06-03T15:39:15.529 INFO:tasks.workunit:Stopping ['rados/test_librados_build.sh'] on client.0...
2019-06-03T15:39:15.529 INFO:teuthology.orchestra.run.smithi037:Running:
2019-06-03T15:39:15.529 INFO:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2019-06-03T15:39:15.792 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 136, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri4-testing-2019-05-30-2109-nautilus/qa/tasks/workunit.py", line 409, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test rados/test_librados_build.sh) on smithi037 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e8040c5146f783fb9c35d955bb1c3484b9370 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h 

Related issues

Copied to Ceph - Backport #40318: luminous: "make: *** [hello_world_cpp] Error 127" in rados Resolved
Copied to Ceph - Backport #40319: nautilus: "make: *** [hello_world_cpp] Error 127" in rados Resolved
Copied to Ceph - Backport #40320: mimic: "make: *** [hello_world_cpp] Error 127" in rados Resolved

History

#1 Updated by Nathan Cutler about 2 years ago

  • Description updated (diff)

#2 Updated by Nathan Cutler about 2 years ago

  • Assignee set to David Galloway

A new OS+version combo?

  os_type: rhel
  os_version: '7.6'

The root cause is:

make: g++: Command not found

Assigning to David in the hopes he can add the dependency to the RHEL 7.6 image.

#3 Updated by David Galloway about 2 years ago

  • Category set to User access
  • Status changed from New to 4

Is it possible make has a messed up env there? The gcc-c++ package is installed on all RHEL7 testnodes.

https://github.com/ceph/ceph-cm-ansible/blob/master/roles/testnode/vars/redhat_7.yml#L28

#4 Updated by David Galloway about 2 years ago

Yeah.

[root@smithi009 ~]# yum whatprovides g++
Loaded plugins: fastestmirror, langpacks, priorities, product-id, search-disabled-repos, subscription-manager
Loading mirror speeds from cached hostfile
 * epel: ftp.linux.ncsu.edu
gcc-c++-4.8.2-16.el7.x86_64 : C++ support for GCC
Repo        : rhel-7-server-rpms
Matched from:
Filename    : /usr/bin/g++

Note the PATH in the teuthology log

2019-06-03T15:39:13.556 INFO:teuthology.orchestra.run.smithi037:> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8b9e8040c5146f783fb9c35d955bb1c3484b9370 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_librados_build.sh

#5 Updated by Nathan Cutler about 2 years ago

The workunit task sets the path to "PATH=$PATH:/usr/sbin" but I don't see how that would lead to "g++: Command not found" if /usr/bin/g++ exists on the system and is executable? Can you elaborate?

#6 Updated by David Galloway about 2 years ago

Sure, I guess my question is: Did $PATH get munged before the workunit test got ran?

For example, if $PATH is empty, and the task sets the path to PATH=$PATH:/usr/sbin, that would result in PATH = /usr/sbin, right? So /usr/bin/g++ wouldn't be found.

Or am I wayyy off base?

Either way, I looked through the teuthology log for that job and gcc-c++ is installed so I'm not sure this is an infra issue.

#7 Updated by Nathan Cutler about 2 years ago

David Galloway wrote:

Sure, I guess my question is: Did $PATH get munged before the workunit test got ran?

If it did, that would hopefully be in the log, and it would also cause problems for any other permutation of this test (not just RHEL 7.6).

Anyway, I'll instrument the test to dump the value of PATH to the log.

#8 Updated by Nathan Cutler about 2 years ago

  • Status changed from 4 to In Progress
  • Assignee changed from David Galloway to Kefu Chai

moving to Ceph project to get "Pull Request ID" and backports

#9 Updated by Nathan Cutler about 2 years ago

  • Project changed from sepia to Ceph
  • Category deleted (User access)
  • Pull request ID set to 28484

#10 Updated by Nathan Cutler about 2 years ago

  • Backport set to luminous mimic nautilus

#11 Updated by Kefu Chai about 2 years ago

  • Status changed from In Progress to Pending Backport

#12 Updated by Nathan Cutler about 2 years ago

  • Copied to Backport #40318: luminous: "make: *** [hello_world_cpp] Error 127" in rados added

#13 Updated by Nathan Cutler about 2 years ago

  • Copied to Backport #40319: nautilus: "make: *** [hello_world_cpp] Error 127" in rados added

#14 Updated by Nathan Cutler about 2 years ago

  • Copied to Backport #40320: mimic: "make: *** [hello_world_cpp] Error 127" in rados added

#15 Updated by Nathan Cutler over 1 year ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF