Ceph

Hi Yuri. This test failure is happening in Ubuntu 14.04, right? My commit cited in the description affects RPM distributions only.

(Not that I don't want to look at it, but I will have to do some digging/learning and that will take some time.)

A quick jab at google indicates that the error might be related to a missing LD_PRELOAD=liblttng-ust-fork.so.0 - maybe this is a regression?

Upgrading from v0.87.x Giant
----------------------------

* librbd and librados include lttng tracepoints on distros with
  liblttng 2.4 or later (only Ubuntu Trusty for the ceph.com
  packages). When running a daemon that uses these libraries, i.e. an
  application that calls fork(2) or clone(2) without exec(3), you must
  set LD_PRELOAD=liblttng-ust-fork.so.0 to prevent a crash in the
  lttng atexit handler when the process exits. The only ceph tool that
  requires this is rbd-fuse.

It would seem that this commit has stopped working for some reason:

commit c4939f9aeae73089ab82c1783d7afd9baafd784f
Author: Adam Crume <adamcrume@gmail.com>
Date:   Mon Jul 28 16:32:15 2014 -0700

    lttng: Preload liblttng-ust-fork.so in TESTS_ENVIRONMENT

    This adds LD_PRELOAD=liblttng-ust-fork.so to TESTS_ENVIRONMENT.
    This prevents lttng from complaining when processes are forked.
    The complaints otherwise taint the output and cause tests to fail.

    Signed-off-by: Adam Crume <adamcrume@gmail.com>

http://pulpito.ceph.com/loic-2015-10-09_09:34:50-upgrade:giant-x-hammer---basic-vps/1096482/

http://tracker.ceph.com/issues/13274 seems related

Something happened, somewhere, after http://pulpito.ceph.com/teuthology-2015-10-05_16:05:09-upgrade:hammer-hammer-distro-basic-vps/ : noone of the teuthology.log show the error above.

It does not come from the ceph repository:

git log --merges --format='%H'  79385a85beea9bccd82c99b6bda653f0224c4fcd..ceph/hammer | while read sha1 ; do echo ; git log --format='%aD "* %s":https://github.com/ceph/ceph/commit/%H' ${sha1}^1..${sha1} ; done | cat

Thu, 8 Oct 2015 00:09:24 +0200 "* Merge pull request #6192 from dachary/wip-13401-hammer":https://github.com/ceph/ceph/commit/70e612a395f7f5daf2fa289d8ae522824ef7a38f
Fri, 21 Aug 2015 16:40:34 -0400 "* crush/CrushTester: test fewer inputs when running crushtool":https://github.com/ceph/ceph/commit/2a28114053fbf4ceee09558790ad3cf755270d5f
Sun, 20 Sep 2015 23:42:45 +0200 "* tests: update to match crushmap validation message":https://github.com/ceph/ceph/commit/abc5b5f139b05da0be0fbaa99282be16386980db
Fri, 18 Sep 2015 09:42:47 -0400 "* mon/OSDMonitor: fix crush injection error message":https://github.com/ceph/ceph/commit/25bd2778f8b7f5eb5245efebca56a7348dd064eb
Fri, 18 Sep 2015 09:41:25 -0400 "* mon/OSDMonitor: only test crush ruleset for the newly created pool":https://github.com/ceph/ceph/commit/6635530aa94b1f21c3052e64d4dfe3952f98fe83
Fri, 18 Sep 2015 09:40:13 -0400 "* crush/CrushTester: allow testing by ruleset":https://github.com/ceph/ceph/commit/cc1fedda02fbf3c95d7a00f0b395db5439e96f90

Tue, 6 Oct 2015 11:11:14 -0400 "* Merge pull request #5887 from Abhishekvrshny/wip-13044-hammer":https://github.com/ceph/ceph/commit/065086246290cfa02f67c37c95a2ccdb05872f21
Wed, 5 Aug 2015 15:19:13 +0800 "* crypto: fix unbalanced ceph::crypto::init/ceph::crypto:shutdown":https://github.com/ceph/ceph/commit/3a50b904a2ffd613b695ead1b26c93278044d7df

nor ceph-cm-ansible

ceph/ceph-cm-ansible$ git log --since 2015-10-01 --merges --format='%H'  master | while read sha1 ; do echo ; git log --format='%aD "* %s":https://github.com/ceph/ceph/commit/%H' ${sha1}^1..${sha1} ; done | cat

Tue, 6 Oct 2015 11:23:35 -0500 "* Merge pull request #139 from ceph/wip-dban":https://github.com/ceph/ceph/commit/107f3c46e31f31132622b1f691b0e741e5a61b23
Tue, 22 Sep 2015 13:25:25 -0600 "* Add dummy dban-2.3.0 entry":https://github.com/ceph/ceph/commit/f28b827b3d1412bf70691eb54c7a6e12dd040d86

Tue, 6 Oct 2015 11:23:06 -0500 "* Merge pull request #148 from ceph/wip-teuthology":https://github.com/ceph/ceph/commit/ea87b162c49d45ff16f044369efda7210cb1b295
Fri, 2 Oct 2015 16:35:20 -0600 "* teuthology: Serve logs over http":https://github.com/ceph/ceph/commit/f3d1d93a22e627d1a08dedc1d33aebe0fe8fd1e2

Fri, 2 Oct 2015 11:50:44 -0500 "* Merge pull request #147 from ceph/wip-teuthology":https://github.com/ceph/ceph/commit/d5eae5d035b9a24f5786193b6a9e9e5055e0c59d
Tue, 29 Sep 2015 13:45:03 -0600 "* teuthology: manage teuthology system users":https://github.com/ceph/ceph/commit/6c63105da1d9a3c99e8fd75504ef7464dcbb6fb8

Fri, 2 Oct 2015 09:13:50 -0500 "* Merge pull request #138 from ceph/wip-puddle-updates":https://github.com/ceph/ceph/commit/ec9f2c4ffac6aa4070cc67b7f69020689249193b
Mon, 21 Sep 2015 12:17:05 -0600 "* puddle: new RH-COMMON advisory undergoing testing":https://github.com/ceph/ceph/commit/8a71cf9362d3506c51389f03c80b0699595becab
Fri, 18 Sep 2015 16:43:47 -0600 "* puddle: output RHCEPH 1.3 to separate locations":https://github.com/ceph/ceph/commit/bfc268d7d11ce4757f44a8729830dff9ecb93df9

Thu, 1 Oct 2015 17:46:24 -0500 "* Merge pull request #146 from ceph/wip-paddles":https://github.com/ceph/ceph/commit/5482c8a1d7313f07ac3942378a4984e43a7acfeb
Thu, 1 Oct 2015 10:53:40 -0600 "* pulpito: Restart if the config changes":https://github.com/ceph/ceph/commit/e9732daf99495f6f950ab7d1b081f4db5bdbafa2
Wed, 30 Sep 2015 16:38:49 -0600 "* paddles: Add new paddles role":https://github.com/ceph/ceph/commit/06cf08ddb74dd276790153a0bf7c10e2c5767a2d

Thu, 1 Oct 2015 10:36:58 -0600 "* Merge pull request #145 from ceph/wip-hostname":https://github.com/ceph/ceph/commit/989b9d54dd285ff09a6f64f8f26b03241231fbb5
Wed, 30 Sep 2015 16:27:58 -0400 "* Update documentation for lab_domain var":https://github.com/ceph/ceph/commit/91273540ddc760b8d3e24026e716edd7ffd4c204
Wed, 30 Sep 2015 15:11:26 -0400 "* testnode: set_hostname should be a global task":https://github.com/ceph/ceph/commit/ae09ced8721f635138dad6c27be7b9a53deda5f0

teuthology-suite --verbose --machine-type vps --suite upgrade/hammer --suite-branch hammer --filter="upgrade:hammer/older/{0-cluster/start.yaml 1-install/v0.94.yaml 2-workload/testrados.yaml 3-upgrade-sequence/upgrade-osd-mon-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrados.yaml} distros/ubuntu_14.04.yaml}" --ceph hammer --priority 101 

• pass http://pulpito.ceph.com/loic-2015-10-09_15:40:49-upgrade:hammer-hammer---basic-vps/

That demonstrate this failure does not exist when running tests on the hammer branch.

This warning from #13430 is interesting: 2015-10-08T11:24:58.470 INFO:tasks.workunit.client.0.plana30.stderr:ERROR: ld.so: object 'liblttng-ust-fork.so.0' from LD_PRELOAD cannot be preloaded: ignored.

could it be related to #13305?

See https://github.com/ceph/ceph/pull/6135#issuecomment-146893689

Yuri, are these still happening?

More runs:
http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2015-12-16_17:02:01-upgrade:hammer-x-jewel-distro-basic-openstack/39554/teuthology.log