Ceph » devops

Background:¶

Ceph builds in Jenkins are broken up into three separate jobs: ceph-setup, ceph-build, and ceph-package. These are visible in Jenkins' ceph tab.

In January I duplicated the first job (ceph-setup) so that we'd have a bit more continuous integration to see where things fail. This new job is named ceph-setup-nightly . (Originally I had this job set to pull changes every night and build, but I've actually bumped this up so it watches hourly instead of nightly.) The builds don't go anywhere, and Jenkins is set to discard the output after each build completes. The idea is just to be running the build code more often than "at a release".

Issue:¶

The ceph-setup-nightly has recently been failing. This could be due to a simple difference in the way that Git snapshots are run versus actual tagged Ceph releases, in which case the problem is fairly innocuous. On the other hand, it might point to a more serious problem that we would hit later on when we try to do a real Ceph release.

Investigating the failure in Jenkins:¶

We don't have bisection entirely automated in Jenkins, so the poor-man's bisection is to click on the first failing build and examine the Git changes to see if any are relevant.

Looking at the job here: http://jenkins.ceph.com/job/ceph-setup-nightly/

The last build to succeed was #149, and the first build to fail was #150. So if we click on the "#150" build, we can see what commits triggered Jenkins to build. Those commits are likely to be the ones that caused the failure.

http://jenkins.ceph.com/job/ceph-setup-nightly/150/

This page shows that there was only one change:

  jenkins: show errors and fail fast during job execution (commit: 3f302cf79c9b5a5a3cecc7a50f23560bca8d58ab)

This commit made our error-checking much more strict. In other words, this means that there was probably some action in the process that exited with a non-zero code, and when we switched to more strict error checking, the builds began to fail.

Looking at the console output in both #149 and #150, I see the following

dpkg-source: warning: source directory 'ceph-0.77-489-g6c20728' is not <sourcepackage>-<upstreamversion> 'ceph-0.77'
dpkg-source: info: using source format `1.0'
dpkg-source: info: building ceph in ceph_0.77-1.tar.gz
dpkg-source: info: building ceph in ceph_0.77-1.dsc
mv: cannot stat `release/0.77-489-g6c20728/*.diff.gz': No such file or directory

That looks like our culprit. It looks like there was a problem generating a gzipped-diff file, and then when we try to move it, mv exits with a non-zero exit code, causing the whole build to fail.

Resolution¶

I dug into this a bit more by reading the console output logs on the Jenkins jobs. It looks like this has occurred for a while with the ceph-setup-nightly job; however, the "real" ceph-setup job doesn't have this problem. So it looks like this is just an issue when we're using Git snapshots, and we may have never noticed it during real releases.

Nevertheless it's a good idea to fix this so that the ceph-setup-nightly job works again. That will prevent this issue from masking other potential failures in ceph-setup-nightly that could be more serious.