https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2015-01-21T20:25:43ZCeph teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=467552015-01-21T20:25:43ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Subject</strong> changed from <i>lost needs to be installed on all magna nodes</i> to <i>lsof needs to be installed on all magna nodes</i></li></ul> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=467572015-01-21T20:37:23ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Project</strong> changed from <i>teuthology</i> to <i>sepia</i></li></ul> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=469962015-01-27T19:15:05ZSandon Van Nesssandon@inktank.com
<ul></ul><p>We install lsof on our magna nodes. Its even in /usr/sbin which very commonly on the $PATH. is something stripping environment variables here causing it to not be found?</p>
<pre>
[ubuntu@magna076 ~]$ sudo grep lsof /var/log/yum.log
Jan 09 17:20:48 Installed: lsof-4.87-4.el7.x86_64
</pre>
<pre>
[ubuntu@magna076 ~]$ lsof -v
lsof version information:
revision: 4.87
latest revision: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/
latest FAQ: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
latest man page: ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/lsof_man
constructed: Sun Jan 26 09:55:57 EST 2014
constructed by and on: mockbuild@x86-017.build.eng.bos.redhat.com
compiler: cc
compiler version: 4.8.2 20140120 (Red Hat 4.8.2-12) (GCC)
compiler flags: -DLINUXV=310000 -DGLIBCV=217 -DHASIPv6 -DHASSELINUX -D_FILE_OFFSET_BITS=64 -D_LARGEFILE64_SOURCE -DHAS_STRFTIME -DLSOF_VSTR="3.10.0" -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
loader flags: -L./lib -llsof -lselinux
system info: Linux x86-017.build.eng.bos.redhat.com 2.6.32-431.4.1.el6.x86_64 #1 SMP Thu Dec 19 10:26:41 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
Anyone can list all files.
/dev warnings are disabled.
Kernel ID check is disabled.
</pre> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=469992015-01-27T19:23:56ZGreg Farnumgfarnum@redhat.com
<ul></ul><p>I don't know, I'm just seeing intermittent failures. I somewhere saw from you a reference to paths getting stripped, so I suppose that could just be happening on RHEL machines or something?</p>
<p>We're not doing anything weird with these commands though, so if the PATH is going away it's pretty likely to be a teuthology issue of some kind that will need to be discussed with Zack and resolved globally. Any idea what would be likely to do that?</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=470802015-01-27T22:05:34ZZack Cerza
<ul></ul><p>I just checked, and I don't see a place where teuthology could be affecting $PATH...</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=470872015-01-27T22:33:24ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Project</strong> changed from <i>sepia</i> to <i>teuthology</i></li><li><strong>Subject</strong> changed from <i>lsof needs to be installed on all magna nodes</i> to <i>PATH issues on RHEL7 nodes?</i></li></ul><p>See also <a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: "fsync-tester.sh: line 10: lsof: command not found" (Resolved)" href="https://tracker.ceph.com/issues/10302">#10302</a>. Sandon there suggests adding a "PATH=$PATH:/usr/sbin" to the script, though that seems wonky to me?</p>
<p>Anyway, the command invoking the script is above, and here are the full script contents:</p>
<pre>#!/bin/sh -x
set -e
wget http://ceph.com/qa/fsync-tester.c
gcc fsync-tester.c -o fsync-tester
./fsync-tester
lsof</pre>
<p>This has been running reliably up until the addition of RHEL7 machines. I logged into a RHEL7 box and invoking lsof manually works just fine. :/</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=471352015-01-28T22:04:36ZZack Cerza
<ul></ul><p>I'd personally rather see something like an "<code>echo $PATH; which lsof</code>" to help us figure out if it is indeed a path issue.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=472442015-01-30T19:19:01ZGreg Farnumgfarnum@redhat.com
<ul></ul><p>Ugh, I wasn't getting email notifications on this bug...watched now.</p>
<p>Anyway, that script is one of the ceph.git workunits. I assume that's where you want that info to come from? Or do you mean from in the workunit task?</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=474312015-02-03T22:38:31ZZack Cerza
<ul></ul><p>I meant from the script</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=475492015-02-06T05:26:42ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>Sandon Van Ness</i> to <i>Greg Farnum</i></li></ul><p>I've got a branch to get path info and will run some tests once it's available for install.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=476832015-02-10T06:34:35ZGreg Farnumgfarnum@redhat.com
<ul></ul><p>Well, I tried reproducing this in a VPS setup and was unable to do so. So I've given in and pushed a patch to master and our LTS branches which will echo the $PATH and run "whereis lsof" as part of the script, which will hopefully help us narrow things down a bit.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=480842015-02-14T07:15:15ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Need More Info</i></li></ul> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=482902015-02-18T23:08:00ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>In Progress</i></li><li><strong>Assignee</strong> changed from <i>Greg Farnum</i> to <i>Zack Cerza</i></li></ul><p>gregf@magna002:/a/gregf-2015-02-17_14:18:29-fs-wip-firefly-flock---basic-magna/49584/teuthology.log:<br /><pre>2015-02-18T16:33:07.010 INFO:tasks.workunit.client.0.magna074.stdout:write time: 107.0348s fsync time: 18.4140s
2015-02-18T16:33:07.050 INFO:tasks.workunit.client.0.magna074.stdout:run done 4 fsyncs total, killing random writer
2015-02-18T16:33:07.051 INFO:tasks.workunit.client.0.magna074.stderr:+ echo /usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin
2015-02-18T16:33:07.091 INFO:tasks.workunit.client.0.magna074.stderr:+ whereis lsof
2015-02-18T16:33:07.092 INFO:tasks.workunit.client.0.magna074.stdout:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin
2015-02-18T16:33:07.497 INFO:tasks.workunit.client.0.magna074.stdout:lsof: /usr/sbin/lsof /usr/share/man/man8/lsof.8.gz
2015-02-18T16:33:07.498 INFO:tasks.workunit.client.0.magna074.stderr:+ lsof
2015-02-18T16:33:07.500 INFO:tasks.workunit.client.0.magna074.stderr:/home/ubuntu/cephtest/workunit.client.0/suites/fsync-tester.sh: line 12: lsof: command not found
2015-02-18T16:33:07.501 INFO:tasks.workunit:Stopping suites/fsync-tester.sh on client.0...
2015-02-18T16:33:07.501 INFO:teuthology.orchestra.run.magna074:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2015-02-18T16:33:07.569 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/ceph-qa-suite_firefly/tasks/workunit.py", line 360, in _run_tests
label="workunit test {workunit}".format(workunit=workunit)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 137, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
r.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
label=self.label)
CommandFailedError: Command failed (workunit test suites/fsync-tester.sh) on magna074 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=702dbc0a247c149d53b52d1929f9880bc99d0522 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/suites/fsync-tester.sh'</pre></p>
<p>Okay, so it looks like we're not including /usr/sbin in the PATH that the script is seeing. I don't see that we're including "sudo" anywhere in our execution of this one, which I think means teuthology has to be doing it somehow?</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=483012015-02-19T18:06:15ZGreg Farnumgfarnum@redhat.com
<ul></ul><p>Forgot to update this yesterday: ssh sets up the environment differently depending on whether it's a login shell (or something like that) and RHEL has different PATH values in those two instances!</p>
<p>But in our case we really want the full login experience for all our workings.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=483022015-02-19T18:07:13ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Assignee</strong> changed from <i>Zack Cerza</i> to <i>Greg Farnum</i></li></ul><p>And I guess I'm stuck shepherding this even if I can't solve it on my own right now.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=484122015-02-24T05:07:17ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Fix Under Review</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph-qa-suite/pull/341">https://github.com/ceph/ceph-qa-suite/pull/341</a></p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=484452015-02-24T18:20:15ZZack Cerza
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>Resolved</i></li></ul><p><a class="external" href="https://github.com/ceph/ceph-qa-suite/commit/7e5d8cb61aaa755aa1504cb70ade23b57235a584">https://github.com/ceph/ceph-qa-suite/commit/7e5d8cb61aaa755aa1504cb70ade23b57235a584</a></p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=484492015-02-24T19:15:29ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>Resolved</i> to <i>Pending Backport</i></li></ul><p>This is in master but we'll want it in our other test branches as well. Just waiting to let it get through a few runs and make sure we haven't somehow busted up something else.</p> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=486662015-03-02T06:00:26ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li></ul> teuthology - Bug #10600: PATH issues on RHEL7 nodes?https://tracker.ceph.com/issues/10600?journal_id=487042015-03-02T19:32:28ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul><p>This is on hammer, firefly, dumpling now.</p>