https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-11-30T12:17:46ZCeph sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=822302016-11-30T12:17:46ZAlfredo Dezaadeza@redhat.com
<ul><li><strong>Project</strong> changed from <i>CI</i> to <i>Infrastructure</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=822412016-11-30T14:53:11ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/82241/diff?detail_id=79342">diff</a>)</li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=822422016-11-30T15:42:29ZDavid Galloway
<ul><li><strong>Project</strong> changed from <i>Infrastructure</i> to <i>sepia</i></li><li><strong>Category</strong> set to <i>Gitbuilder</i></li><li><strong>Assignee</strong> set to <i>David Galloway</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=822432016-11-30T15:42:36ZDavid Galloway
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-4 priority-default closed" href="/issues/18069">Bug #18069</a>: rados suite upgrade tests fail on xenial nodes</i> added</li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=822492016-11-30T16:51:34ZDavid Galloway
<ul></ul><p>Building infernalis on Xenial is stuck due to the distribute library not installing - <a class="external" href="http://tracker.ceph.com/issues/18069#note-6">http://tracker.ceph.com/issues/18069#note-6</a></p>
<p>I'm not quite sure where/how to fix this yet.</p>
<p>I forced a rebuild on v0.80.8. It had been marked PASS from when the gitbuilder was created.</p>
<p>I'm working on getting v0.67.10 on Xenial - <a class="external" href="http://tracker.ceph.com/issues/18069#note-4">http://tracker.ceph.com/issues/18069#note-4</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=836812017-01-03T20:14:53ZDavid Galloway
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Closed</i></li></ul><p>I think this was taken care of by <a class="external" href="https://github.com/ceph/ceph-qa-suite/pull/1292">https://github.com/ceph/ceph-qa-suite/pull/1292</a></p>
<p>Also see <a class="external" href="http://tracker.ceph.com/issues/18069">http://tracker.ceph.com/issues/18069</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849242017-01-27T14:09:30ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>12</i></li></ul><p>This bug is still present in jewel integration testing. We have a branch, wip-jewel-backports, which is based on jewel. This test fails: <a class="external" href="http://pulpito.ceph.com/loic-2017-01-26_22:01:29-rados-wip-jewel-backports-distro-basic-smithi/753358/">http://pulpito.ceph.com/loic-2017-01-26_22:01:29-rados-wip-jewel-backports-distro-basic-smithi/753358/</a></p>
<p>The failure reason is "Failed to fetch package version from <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8</a>"</p>
<p>The test yaml has:</p>
<pre>
os_type: ubuntu
os_version: 14.04
</pre> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849272017-01-27T14:18:41ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Here's another instance of the same failure. This job installs dumpling, then upgrades to firefly, and then to the test branch (wip-jewel-backports in this case): <a class="external" href="http://pulpito.ceph.com/loic-2017-01-26_22:01:29-rados-wip-jewel-backports-distro-basic-smithi/753382/">http://pulpito.ceph.com/loic-2017-01-26_22:01:29-rados-wip-jewel-backports-distro-basic-smithi/753382/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849282017-01-27T14:36:13ZNathan Cutlerncutler@suse.cz
<ul></ul><p><strong>jewel PR</strong>: <a class="external" href="https://github.com/ceph/ceph/pull/13153">https://github.com/ceph/ceph/pull/13153</a></p>
<p>This should fix upgrade:hammer-x/f-h-x-offline in jewel, once the firefly packages for Ubuntu 14.04 are fixed in Shaman that is.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849312017-01-27T15:59:26ZDavid Galloway
<ul></ul><p>I may just try to push an existing build of Firefly to the new CI. Is there a particular version or sha1 needed?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849322017-01-27T16:21:23ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: as far as firefly is concerned, one job specifies v0.80.8 (sha1 69eaad7f8308f21573c604f121956e64679a52a7) and others just want the firefly branch (sha1 8abf95af405e117298c5012aeaa4c60caf86a4fd).</p>
<p>The job that installs dumpling and then upgrades it to firefly uses v0.67.10 (sha1 9d446bd416c52cd785ccf048ca67737ceafcdd7f) for dumpling and then branch firefly (sha1 8abf95af405e117298c5012aeaa4c60caf86a4fd) for the upgrade to firefly.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=849822017-01-27T18:38:44ZDavid Galloway
<ul></ul><p>OK, so you may not need to run those upgrade tests anymore per <a class="external" href="http://dpaste.com/1S7JDSQ">http://dpaste.com/1S7JDSQ</a>#</p>
<p>Nevertheless, I pushed some gitbuilder builds to the CI.</p>
<p><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=9d446bd416c52cd785ccf048ca67737ceafcdd7f">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=9d446bd416c52cd785ccf048ca67737ceafcdd7f</a><br /><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8</a><br /><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=firefly">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=firefly</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=850612017-01-29T03:35:35ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David Thanks for trying. I scheduled a run with just these two tests, but the jobs still fail: <a class="external" href="http://pulpito.ceph.com/smithfarm-2017-01-29_02:56:52-rados-wip-jewel-backports-distro-basic-smithi/">http://pulpito.ceph.com/smithfarm-2017-01-29_02:56:52-rados-wip-jewel-backports-distro-basic-smithi/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=850622017-01-29T03:37:28ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Failure looks like this:</p>
<pre>
2017-01-29T02:59:06.656 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch
2017-01-29T02:59:06.656 INFO:teuthology.packaging:ref: None
2017-01-29T02:59:06.656 INFO:teuthology.packaging:tag: None
2017-01-29T02:59:06.656 INFO:teuthology.packaging:branch: v0.80.8
2017-01-29T02:59:06.656 INFO:teuthology.packaging:sha1: 566dea345152702387becaff00e019bdb088a7fe
2017-01-29T02:59:06.657 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8
2017-01-29T02:59:06.878 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/v0.80.8/69eaad7f8308f21573c604f121956e64679a52a7/ubuntu/trusty/flavors/default/
2017-01-29T02:59:06.878 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 83, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/install/deb.py", line 54, in _update_package_list_and_install
version = builder.version
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/packaging.py", line 538, in version
self._version = self._get_package_version()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/packaging.py", line 946, in _get_package_version
return self._result.json()[0]['extra']['package_manager_version']
KeyError: 'package_manager_version'
</pre> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851492017-01-30T15:43:57ZDavid Galloway
<ul></ul><p>Ah, I see I forgot some additional required JSON. I'll work on that.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851602017-01-30T18:11:00ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David Thanks for trying. I scheduled a run with just these two tests, but the jobs still fail: <a class="external" href="http://pulpito.ceph.com/smithfarm-2017-01-29_02:56:52-rados-wip-jewel-backports-distro-basic-smithi/">http://pulpito.ceph.com/smithfarm-2017-01-29_02:56:52-rados-wip-jewel-backports-distro-basic-smithi/</a></p>
</blockquote>
<p>Fixed and updated my documentation on this process for next time: <a class="external" href="http://wiki.front.sepia.ceph.com/doku.php?id=production:chacra.ceph.com#pushing_a_gitbuilder-built_repo_to_a_chacra_host">http://wiki.front.sepia.ceph.com/doku.php?id=production:chacra.ceph.com#pushing_a_gitbuilder-built_repo_to_a_chacra_host</a></p>
<p>Please retry.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851612017-01-30T18:18:19ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David - thanks, will do. Does your fix cover these two failures, too? <a class="external" href="http://pulpito.ceph.com/smithfarm-2017-01-30_12:05:09-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-01-30_12:05:09-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851622017-01-30T18:22:04ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David - thanks, will do. Does your fix cover these two failures, too? <a class="external" href="http://pulpito.ceph.com/smithfarm-2017-01-30_12:05:09-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-01-30_12:05:09-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/</a></p>
</blockquote>
<p>The package_manager_version error, yes. The second one, no. I'll need to push that to a chacra host. Will let you know when that's done.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851632017-01-30T18:24:46ZNathan Cutlerncutler@suse.cz
<ul></ul><code>filter="rados/singleton-nomsgr/{all/11429.yaml rados.yaml},rados/singleton-nomsgr/{all/13234.yaml rados.yaml}" ./virtualenv/bin/teuthology-suite -k distro --priority 101 --suite rados --ceph jewel --machine-type smithi --ceph-repo http://github.com/ceph/ceph.git --suite-repo http://github.com/ceph/ceph.git --email ncutler@suse.com --filter="$filter"</code>
<p><strong>pass</strong> <a class="external" href="http://pulpito.ceph.com:80/smithfarm-2017-01-30_18:24:07-rados-jewel-distro-basic-smithi/">http://pulpito.ceph.com:80/smithfarm-2017-01-30_18:24:07-rados-jewel-distro-basic-smithi/</a></p>
<p>WOOHOO!</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851662017-01-30T18:35:51ZDavid Galloway
<ul></ul><p>Actually, that second one I'm a bit confused. The tag and sha1 are specified and looks like it defaults to tag.</p>
<pre>
2017-01-30T12:17:35.251 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag
2017-01-30T12:17:35.252 INFO:teuthology.packaging:ref: None
2017-01-30T12:17:35.252 INFO:teuthology.packaging:tag: v0.94.3
2017-01-30T12:17:35.252 INFO:teuthology.packaging:branch: None
2017-01-30T12:17:35.252 INFO:teuthology.packaging:sha1: b671230f7f70b620905eb02c6dbd93d051b53fb7
2017-01-30T12:17:35.303 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v0.94.3^{} -> None
2017-01-30T12:17:35.304 INFO:teuthology.packaging:Tag 'v0.94.3' not found in git://git.ceph.com/ceph-ci; will also look in git://git.ceph.com/ceph
2017-01-30T12:17:35.748 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph v0.94.3^{} -> 95cefea9fd9ab740263bf8bb4796fd864d9afe2b
2017-01-30T12:17:35.749 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b
</pre>
<p>The sha1 requested is built [1] but does not match the version/tag. Which version needs to be installed there?</p>
<p>I see on the gitbuilder host that v0.94.3 -> ../sha1/95cefea9fd9ab740263bf8bb4796fd864d9afe2b</p>
<p>[1] <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=b671230f7f70b620905eb02c6dbd93d051b53fb7">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=b671230f7f70b620905eb02c6dbd93d051b53fb7</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851682017-01-30T19:18:35ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Weird. The test yaml stipulates:</p>
<pre>
- print: "**** Install version lower than v0.94.4"
- install:
tag: v0.94.3
- ceph:
fs: xfs
</pre>
<p>Teuthology somehow converts this into:</p>
<pre>
- print: **** Install version lower than v0.94.4
- install:
tag: v0.94.3
sha1: b671230f7f70b620905eb02c6dbd93d051b53fb7
- ceph:
fs: xfs
</pre>
<p>Where b671230f7f70b620905eb02c6dbd93d051b53fb7 is the SHA1 of wip-jewel-backports (the job was run with <code>--ceph wip-jewel-backports</code>).</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851692017-01-30T19:20:43ZNathan Cutlerncutler@suse.cz
<ul></ul><blockquote>
<p>The sha1 requested is built [1] but does not match the version/tag. Which version needs to be installed there?</p>
</blockquote>
<p>This is trying to install v0.94.3, i.e. 95cefea9fd9ab740263bf8bb4796fd864d9afe2b, so if you push that one to chacra that will do the trick.</p>
<p>The sha1 is not actually requested, though. Teuthology fills that in for some reason, but it is rightfully ignored.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851722017-01-30T20:09:32ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote><blockquote>
<p>The sha1 requested is built [1] but does not match the version/tag. Which version needs to be installed there?</p>
</blockquote>
<p>This is trying to install v0.94.3, i.e. 95cefea9fd9ab740263bf8bb4796fd864d9afe2b, so if you push that one to chacra that will do the trick.</p>
<p>The sha1 is not actually requested, though. Teuthology fills that in for some reason, but it is rightfully ignored.</p>
</blockquote>
<p>Done!</p>
<p><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851802017-01-30T20:38:29ZNathan Cutlerncutler@suse.cz
<ul></ul><p>David, please push this one as well:</p>
<ul>
<li>v10.2.0 3a9fba20ec743699b69bd0181dd6c54dc01c64b9</li>
</ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=851812017-01-30T20:54:58ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>David, please push this one as well:</p>
<ul>
<li>v10.2.0 3a9fba20ec743699b69bd0181dd6c54dc01c64b9</li>
</ul>
</blockquote>
<p>That one's already built: <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=3a9fba20ec743699b69bd0181dd6c54dc01c64b9">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=3a9fba20ec743699b69bd0181dd6c54dc01c64b9</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=853402017-02-01T12:25:42ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: I found a small problem with the v0.94.3 Ubuntu 14.04 packages (sha1 95cefea9fd9ab740263bf8bb4796fd864d9afe2b). They appear to be 0.80.8 instead.</p>
<p><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b</a> returns (note "package_manager_version" field):</p>
<pre>
[{"status": "ready", "sha1": "95cefea9fd9ab740263bf8bb4796fd864d9afe2b", "extra": {"build_url": "", "root_build_cause": "Imported from gitbuilder build", "version": "0.94.3", "node_name": "", "job_name": "", "package_manager_version": "0.80.8-1trusty"}, "url": "https://2.chacra.ceph.com/r/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/flavors/default/", "distro_codename": "trusty", "modified": "2017-01-30 20:08:55.219279", "distro_version": "14.04", "project": "ceph", "flavor": "default", "ref": "v0.94.3", "chacra_url": "https://2.chacra.ceph.com/repos/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/flavors/default/", "archs": ["x86_64"], "distro": "ubuntu"}]
</pre>
<p>and at least one upgrade/hammer-x job goes south:</p>
<pre>
2017-01-31T21:22:34.377 DEBUG:teuthology.misc:System to be installed: Ubuntu
2017-01-31T21:22:34.377 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-mds, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs1, libcephfs-java, libcephfs-jni, librados2, librbd1, rbd-fuse on remote deb x86_64
2017-01-31T21:22:34.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag
2017-01-31T21:22:34.377 INFO:teuthology.packaging:ref: None
2017-01-31T21:22:34.377 INFO:teuthology.packaging:tag: v0.94.3
2017-01-31T21:22:34.377 INFO:teuthology.packaging:branch: None
2017-01-31T21:22:34.377 INFO:teuthology.packaging:sha1: 6d7b95218200aaa6629c6de9bd29b8fcae30760e
2017-01-31T21:22:34.402 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v0.94.3^{} -> None
2017-01-31T21:22:34.403 INFO:teuthology.packaging:Tag 'v0.94.3' not found in git://git.ceph.com/ceph-ci; will also look in git://git.ceph.com/ceph
2017-01-31T21:22:34.859 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph v0.94.3^{} -> 95cefea9fd9ab740263bf8bb4796fd864d9afe2b
2017-01-31T21:22:34.859 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b
2017-01-31T21:22:34.862 INFO:teuthology.orchestra.run.vpm005.stdout:Ubuntu
2017-01-31T21:22:34.863 DEBUG:teuthology.misc:System to be installed: Ubuntu
2017-01-31T21:22:34.863 INFO:teuthology.task.install.deb:Installing packages: ceph, ceph-mds, ceph-common, ceph-fuse, ceph-test, radosgw, python-ceph, libcephfs1, libcephfs-java, libcephfs-jni, librados2, librbd1, rbd-fuse on remote deb x86_64
2017-01-31T21:22:34.863 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag
2017-01-31T21:22:34.863 INFO:teuthology.packaging:ref: None
2017-01-31T21:22:34.863 INFO:teuthology.packaging:tag: v0.94.3
2017-01-31T21:22:34.863 INFO:teuthology.packaging:branch: None
2017-01-31T21:22:34.863 INFO:teuthology.packaging:sha1: 6d7b95218200aaa6629c6de9bd29b8fcae30760e
2017-01-31T21:22:34.883 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph-ci v0.94.3^{} -> None
2017-01-31T21:22:34.883 INFO:teuthology.packaging:Tag 'v0.94.3' not found in git://git.ceph.com/ceph-ci; will also look in git://git.ceph.com/ceph
2017-01-31T21:22:35.336 DEBUG:teuthology.repo_utils:git ls-remote git://git.ceph.com/ceph v0.94.3^{} -> 95cefea9fd9ab740263bf8bb4796fd864d9afe2b
2017-01-31T21:22:35.336 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=95cefea9fd9ab740263bf8bb4796fd864d9afe2b
2017-01-31T21:22:35.534 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/flavors/default/
2017-01-31T21:22:35.534 INFO:teuthology.task.install.deb:Package version is 0.80.8-1trusty
2017-01-31T21:22:35.553 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/flavors/default/
2017-01-31T21:22:35.554 INFO:teuthology.task.install.deb:Package version is 0.80.8-1trusty
</pre>
<p>finally failing with <code>Command failed on vpm005 with status 100: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.80.8-1trusty ceph-mds=0.80.8-1trusty ceph-common=0.80.8-1trusty ceph-fuse=0.80.8-1trusty ceph-test=0.80.8-1trusty radosgw=0.80.8-1trusty python-ceph=0.80.8-1trusty libcephfs1=0.80.8-1trusty libcephfs-java=0.80.8-1trusty libcephfs-jni=0.80.8-1trusty librados2=0.80.8-1trusty librbd1=0.80.8-1trusty rbd-fuse=0.80.8-1trusty'</code></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=853732017-02-01T15:03:31ZDavid Galloway
<ul></ul><p>That was a typo on my part. I re-pushed the extra job info. Should sync with shaman in a bit.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=857782017-02-07T21:43:54ZDavid Galloway
<ul></ul><p>@Nathan, are you satisfied with the outcome of this or do you need anything else?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=857832017-02-08T05:12:17ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>4</i></li><li><strong>Assignee</strong> changed from <i>David Galloway</i> to <i>Nathan Cutler</i></li></ul><p>@David Progress is being made, but I need to do some more runs to verify that there are no more instances of this bug. Assigning to myself and setting status to "Feedback".</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=860412017-02-13T22:38:57ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Assignee</strong> changed from <i>Nathan Cutler</i> to <i>David Galloway</i></li></ul><p>@David: Can you look at this test failure and determine if it's caused by this bug? <a class="external" href="http://pulpito.ceph.com/smithfarm-2017-02-13_20:36:56-rados-wip-jewel-backports-distro-basic-smithi/811008/">http://pulpito.ceph.com/smithfarm-2017-02-13_20:36:56-rados-wip-jewel-backports-distro-basic-smithi/811008/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=862812017-02-17T22:39:51ZDavid Galloway
<ul><li><strong>Assignee</strong> changed from <i>David Galloway</i> to <i>Nathan Cutler</i></li></ul><p>Yep that was my fault. I somehow managed to delete all the packages for that repo.</p>
<p>I re-pushed them and added the extra json needed by teuthology.</p>
<p><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=863642017-02-20T11:32:35ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Assignee</strong> changed from <i>Nathan Cutler</i> to <i>David Galloway</i></li></ul><p>Still seeing this failure (tag: v0.67.10) which had been fixed before: <a class="external" href="http://pulpito.front.sepia.ceph.com/loic-2017-02-20_10:39:39-rados-jewel-distro-basic-smithi/836317/">http://pulpito.front.sepia.ceph.com/loic-2017-02-20_10:39:39-rados-jewel-distro-basic-smithi/836317/</a></p>
<p>Also, firefly builds appear to be missing (again?):</p>
<ul>
<li><a class="external" href="http://pulpito.ceph.com/smithfarm-2017-02-20_18:49:16-upgrade:firefly-x-wip-hammer-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-02-20_18:49:16-upgrade:firefly-x-wip-hammer-backports-distro-basic-vps/</a></li>
<li><a class="external" href="http://pulpito.ceph.com/smithfarm-2017-02-20_18:56:03-upgrade:firefly-x-wip-hammer-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-02-20_18:56:03-upgrade:firefly-x-wip-hammer-backports-distro-basic-vps/</a></li>
</ul>
<p>And the first two (of three failures) in this run appear to be instances of this bug:</p>
<ul>
<li><a class="external" href="http://pulpito.ceph.com/smithfarm-2017-02-20_21:28:11-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-02-20_21:28:11-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/</a></li>
</ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864102017-02-20T21:34:11ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David, ISTR reading somewhere that Shaman keeps builds for only a few days or weeks. Are you safeguarding these builds against that?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864582017-02-21T15:55:04ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David, ISTR reading somewhere that Shaman keeps builds for only a few days or weeks. Are you safeguarding these builds against that?</p>
</blockquote>
<p>Yeah, I thought there wasn't anything implemented to clean up old builds yet but I checked with Andrew and these are getting deleted. Figuring out a fix now.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864612017-02-21T16:19:43ZDavid Galloway
<ul></ul><p>@Nathan, can you guys use <code>use_shaman: False</code> for these runs or are the newer versions of Ceph you're upgrading <strong>to</strong> only built by Jenkins slaves, and thus on chacra nodes, now?</p>
<p>There's no way to flag that a build should be kept on a chacra node.</p>
<p>It sounds like you need to install an upstream release, maybe from download.ceph.com, then upgrade to a CI-built version of Ceph. Is that right?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864712017-02-21T20:08:32ZNathan Cutlerncutler@suse.cz
<ul></ul><blockquote>
<p>@Nathan, can you guys use <code>use_shaman: False</code> for these runs or are the newer versions of Ceph you're upgrading <strong>to</strong> only built by Jenkins slaves, and thus on chacra nodes, now?</p>
</blockquote>
<p>Correct. The way we work is we stage backport PRs in an integration branch, push it to ceph-ci/ceph.git for Shaman to build it, and then run tests on it.</p>
<blockquote>
<p>There's no way to flag that a build should be kept on a chacra node.</p>
</blockquote>
<p>That's a bug. It means that the migration to Shaman breaks all tests that upgrade from legacy releases.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864732017-02-21T20:11:08ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Subject</strong> changed from <i>tests failing with failure to get older package versions</i> to <i>No way to prevent Shaman from deleting builds, breaks upgrade test</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>Urgent</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864742017-02-21T20:11:37ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Subject</strong> changed from <i>No way to prevent Shaman from deleting builds, breaks upgrade test</i> to <i>No way to prevent Shaman from deleting builds, breaks a number of upgrade tests</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=864772017-02-21T20:26:59ZDavid Galloway
<ul><li><strong>Project</strong> changed from <i>sepia</i> to <i>CI</i></li><li><strong>Subject</strong> changed from <i>No way to prevent Shaman from deleting builds, breaks a number of upgrade tests</i> to <i>No way to prevent Chacra from deleting builds, breaks a number of upgrade tests</i></li><li><strong>Category</strong> deleted (<del><i>Gitbuilder</i></del>)</li><li><strong>Assignee</strong> deleted (<del><i>David Galloway</i></del>)</li></ul><p>Nathan Cutler wrote:</p>
<blockquote><blockquote>
<p>@Nathan, can you guys use <code>use_shaman: False</code> for these runs or are the newer versions of Ceph you're upgrading <strong>to</strong> only built by Jenkins slaves, and thus on chacra nodes, now?</p>
</blockquote>
<p>Correct. The way we work is we stage backport PRs in an integration branch, push it to ceph-ci/ceph.git for Shaman to build it, and then run tests on it.</p>
<blockquote>
<p>There's no way to flag that a build should be kept on a chacra node.</p>
</blockquote>
<p>That's a bug. It means that the migration to Shaman breaks all tests that upgrade from legacy releases.</p>
</blockquote>
<p>Well, I should clarify. Chacra nodes can, individually, be set to keep all builds. The 4 chacra nodes we use for test builds, however, are all set to delete builds after a set number of days: <a class="external" href="https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L85-L135">https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L85-L135</a></p>
<p>Shaman's just an aggregator. Chacra's the tool that actually removes repos. I've renamed the bug as such and I'll move this to the CI queue.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=867792017-02-26T03:53:20ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Thanks, David. We really need a way to protect individual builds from being deleted. See <a class="external" href="http://tracker.ceph.com/issues/19080">http://tracker.ceph.com/issues/19080</a> for another example - a test is installing "branch: infernalis" and then upgrading. Shaman gives it the v9.2.1 build, but this is missing a commit that was added to the infernalis branch after the v9.2.1 release. I can fix the test to specify the SHA1, but then the test will fail because Shaman/Chacra will not be able to reliably provide a build for that SHA1.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=867802017-02-26T03:53:57ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>4</i> to <i>12</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=867812017-02-26T03:57:15ZNathan Cutlerncutler@suse.cz
<ul></ul><p>How does Shaman/Chacra determine that a build is a "test build"? Builds specified in test YAML should not fall into this category.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868452017-02-27T22:18:35ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David, it sounds like you could simply put the legacy builds used in the upgrade tests on a Chacra host that is not configured to delete builds after a certain number of days?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868832017-02-28T16:06:27ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David, it sounds like you could simply put the legacy builds used in the upgrade tests on a Chacra host that is not configured to delete builds after a certain number of days?</p>
</blockquote>
<p>The only Chacra host that doesn't delete builds automatically is chacra.ceph.com and according to Andrew, it was intentionally removed from Shaman's config by Alfredo. I'm not sure what the reasoning is or what the ramifications might be of re-adding it to Shaman and pushing permanent builds to it.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868872017-02-28T17:37:08ZAndrew Schoenaschoen@redhat.com
<ul></ul><p>David Galloway wrote:</p>
<blockquote>
<p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David, it sounds like you could simply put the legacy builds used in the upgrade tests on a Chacra host that is not configured to delete builds after a certain number of days?</p>
</blockquote>
<p>The only Chacra host that doesn't delete builds automatically is chacra.ceph.com and according to Andrew, it was intentionally removed from Shaman's config by Alfredo. I'm not sure what the reasoning is or what the ramifications might be of re-adding it to Shaman and pushing permanent builds to it.</p>
</blockquote>
<p>I've updated the config on chacra.ceph.com to update shaman with repo status so that repos stored there can be used. For any newer releases (0.94.10, 12.0.0) you'll have to use chacractl to force a rebuild so shaman is notified. Older repos that doe not exist on chacra.ceph.com will have to be uploaded from download.ceph.com using chacractl.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868882017-02-28T17:48:56ZAndrew Schoenaschoen@redhat.com
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>Thanks, David. We really need a way to protect individual builds from being deleted. See <a class="external" href="http://tracker.ceph.com/issues/19080">http://tracker.ceph.com/issues/19080</a> for another example - a test is installing "branch: infernalis" and then upgrading. Shaman gives it the v9.2.1 build, but this is missing a commit that was added to the infernalis branch after the v9.2.1 release. I can fix the test to specify the SHA1, but then the test will fail because Shaman/Chacra will not be able to reliably provide a build for that SHA1.</p>
</blockquote>
<p>Nathan,</p>
<p>The dev chacra instances are configured to keep a minimum of 5 infernalis repos, repos for the infernalis ref is also configured to be retained for 30 days. <a class="external" href="https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L108">https://github.com/ceph/chacra/blob/master/deploy/playbooks/roles/common/templates/prod.py.j2#L108</a></p>
<p>I just attempted to build infernalis and the problem with infernalis is that any attempt to rebuild fails with errors like: <a class="external" href="https://jenkins.ceph.com/job/ceph-dev-setup/5877/console">https://jenkins.ceph.com/job/ceph-dev-setup/5877/console</a></p>
<p>This would mean that any merge to the infernalis branch also failed to build. IIRC, this problem is related to trying to build infernalis on a xenial node which is the primary node type picked up by ceph-dev-setup.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868912017-02-28T18:45:33ZDavid Galloway
<ul></ul><p>Andrew Schoen wrote:</p>
<blockquote>
<p>David Galloway wrote:</p>
<blockquote>
<p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David, it sounds like you could simply put the legacy builds used in the upgrade tests on a Chacra host that is not configured to delete builds after a certain number of days?</p>
</blockquote>
<p>The only Chacra host that doesn't delete builds automatically is chacra.ceph.com and according to Andrew, it was intentionally removed from Shaman's config by Alfredo. I'm not sure what the reasoning is or what the ramifications might be of re-adding it to Shaman and pushing permanent builds to it.</p>
</blockquote>
<p>I've updated the config on chacra.ceph.com to update shaman with repo status so that repos stored there can be used. For any newer releases (0.94.10, 12.0.0) you'll have to use chacractl to force a rebuild so shaman is notified. Older repos that doe not exist on chacra.ceph.com will have to be uploaded from download.ceph.com using chacractl.</p>
</blockquote>
<p>You sure? I don't see it in <a class="external" href="https://shaman.ceph.com/api/nodes/">https://shaman.ceph.com/api/nodes/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868922017-02-28T18:50:31ZAndrew Schoenaschoen@redhat.com
<ul></ul><p>David Galloway wrote:</p>
<blockquote>
<p>Andrew Schoen wrote:</p>
<blockquote>
<p>David Galloway wrote:</p>
<blockquote>
<p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David, it sounds like you could simply put the legacy builds used in the upgrade tests on a Chacra host that is not configured to delete builds after a certain number of days?</p>
</blockquote>
<p>The only Chacra host that doesn't delete builds automatically is chacra.ceph.com and according to Andrew, it was intentionally removed from Shaman's config by Alfredo. I'm not sure what the reasoning is or what the ramifications might be of re-adding it to Shaman and pushing permanent builds to it.</p>
</blockquote>
<p>I've updated the config on chacra.ceph.com to update shaman with repo status so that repos stored there can be used. For any newer releases (0.94.10, 12.0.0) you'll have to use chacractl to force a rebuild so shaman is notified. Older repos that doe not exist on chacra.ceph.com will have to be uploaded from download.ceph.com using chacractl.</p>
</blockquote>
<p>You sure? I don't see it in <a class="external" href="https://shaman.ceph.com/api/nodes/">https://shaman.ceph.com/api/nodes/</a></p>
</blockquote>
<p>It's purposely out of the shaman rotation, we don't want dev builds being sent there. We just want the upstream releases to register themselves with shaman.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868952017-02-28T20:09:32ZDavid Galloway
<ul></ul><p>Andrew Schoen wrote:</p>
<blockquote>
<p>It's purposely out of the shaman rotation, we don't want dev builds being sent there. We just want the upstream releases to register themselves with shaman.</p>
</blockquote>
<p>Maybe I'm missing a step but shouldn't Shaman know about these repos now then? <a class="external" href="https://chacra.ceph.com/repos/ceph/firefly/">https://chacra.ceph.com/repos/ceph/firefly/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868982017-02-28T20:52:24ZAndrew Schoenaschoen@redhat.com
<ul></ul><p>David Galloway wrote:</p>
<blockquote>
<p>Andrew Schoen wrote:</p>
<blockquote>
<p>It's purposely out of the shaman rotation, we don't want dev builds being sent there. We just want the upstream releases to register themselves with shaman.</p>
</blockquote>
<p>Maybe I'm missing a step but shouldn't Shaman know about these repos now then? <a class="external" href="https://chacra.ceph.com/repos/ceph/firefly/">https://chacra.ceph.com/repos/ceph/firefly/</a></p>
</blockquote>
<p>The repos you linked to have been have been there a long time so they were here before that configuration change I just made today. Only one has a valid sha1 and it's only for centos 6 <a class="external" href="https://chacra.ceph.com/repos/ceph/firefly/8424145d49264624a3b0a204aedb127835161070/centos/6/">https://chacra.ceph.com/repos/ceph/firefly/8424145d49264624a3b0a204aedb127835161070/centos/6/</a></p>
<p>If you're wanting to upload old firefly repos you'll need to import them from download.ceph.com and then use chacractl to to force a rebuild of the repo. When you rebuild the repo it should register with shaman.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=868992017-02-28T21:30:36ZDavid Galloway
<ul><li><strong>Project</strong> changed from <i>CI</i> to <i>sepia</i></li><li><strong>Category</strong> set to <i>Infrastructure Service</i></li><li><strong>Assignee</strong> set to <i>David Galloway</i></li></ul><p>Okay, I can try again to push these old builds to the chacra node that won't delete builds. Can you provide the refs and sha1s that need to be pushed, please? I've gotten confused about exactly what versions/refs/sha1s are needed.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=869382017-03-01T15:51:07ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: OK, I started <a class="external" href="http://pad.ceph.com/p/upgrade-builds">http://pad.ceph.com/p/upgrade-builds</a> and will put the references/SHA1s there. Please update that pad if/when you fix a particular build.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=869582017-03-01T21:27:16ZDavid Galloway
<ul><li><strong>Subject</strong> changed from <i>No way to prevent Chacra from deleting builds, breaks a number of upgrade tests</i> to <i>Pre-hammer official builds missing from CI/Shaman</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=869622017-03-01T22:40:08ZDavid Galloway
<ul></ul><p>8abf95af405e117298c5012aeaa4c60caf86a4fd (tip of "firefly")<br /> - <a class="external" href="https://chacra.ceph.com/repos/ceph/firefly/8abf95af405e117298c5012aeaa4c60caf86a4fd/ubuntu/trusty/flavors/default/">https://chacra.ceph.com/repos/ceph/firefly/8abf95af405e117298c5012aeaa4c60caf86a4fd/ubuntu/trusty/flavors/default/</a><br /> - <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=firefly">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=firefly</a></p>
<p>95cefea9fd9ab740263bf8bb4796fd864d9afe2b (v0.94.3 tag)<br /> - <a class="external" href="https://chacra.ceph.com/repos/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/">https://chacra.ceph.com/repos/ceph/v0.94.3/95cefea9fd9ab740263bf8bb4796fd864d9afe2b/ubuntu/trusty/</a><br /> - <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.94.3">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.94.3</a></p>
<p>baf17c90980f3c61f75775f561ced5b2a1d2141c (tip of "infernalis")<br /> - <a class="external" href="https://chacra.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/ubuntu/trusty/">https://chacra.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/ubuntu/trusty/</a><br /> - <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=baf17c90980f3c61f75775f561ced5b2a1d2141c">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&sha1=baf17c90980f3c61f75775f561ced5b2a1d2141c</a></p>
<p>69eaad7f8308f21573c604f121956e64679a52a7 (v0.80.8 tag)<br /> - <a class="external" href="https://chacra.ceph.com/repos/ceph/v0.80.8/69eaad7f8308f21573c604f121956e64679a52a7/ubuntu/trusty/">https://chacra.ceph.com/repos/ceph/v0.80.8/69eaad7f8308f21573c604f121956e64679a52a7/ubuntu/trusty/</a><br /> - <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=ubuntu%2F14.04%2Fx86_64&ref=v0.80.8</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=874922017-03-10T19:45:47ZDavid Galloway
<ul></ul><p>How's this looking?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=875012017-03-11T09:12:12ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David, please keep the bug open until we do another round of upgrade tests in hammer and jewel. Since that may take awhile, feel free to reassign the bug to me.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=887372017-04-09T19:54:24ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Hi David, could you work your magic on:</p>
<p>tag: v10.2.0<br />sha1: 3a9fba20ec743699b69bd0181dd6c54dc01c64b9</p>
<p>(xenial build is needed - see <a class="external" href="http://pad.ceph.com/p/upgrade-builds">http://pad.ceph.com/p/upgrade-builds</a>)</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=887382017-04-09T19:59:31ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Subject</strong> changed from <i>Pre-hammer official builds missing from CI/Shaman</i> to <i>Various official builds missing from CI/Shaman</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=887492017-04-10T06:36:50ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David, this one is also failing:</p>
<p>Failed to fetch package version from <a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=firefly">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=firefly</a></p>
<p>added to <a class="external" href="http://pad.ceph.com/p/upgrade-builds">http://pad.ceph.com/p/upgrade-builds</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=887832017-04-10T17:41:52ZDavid Galloway
<ul></ul><p>I imported the firefly build for CentOS.</p>
<p>If I import all the 10.2.0-1xenial_amd64 packages from <a class="external" href="http://download.ceph.com/debian-jewel/pool/main/c/ceph/">http://download.ceph.com/debian-jewel/pool/main/c/ceph/</a> would that suffice for "3a9fba20ec743699b69bd0181dd6c54dc01c64b9 (v10.2.0 tag) - xenial" ?</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=887862017-04-10T19:34:07ZDavid Galloway
<ul><li><strong>Assignee</strong> changed from <i>David Galloway</i> to <i>Nathan Cutler</i></li></ul><p>Pushed and updated etherpad. Let me know if you need any others.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=888312017-04-11T13:37:15ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David - please</p>
<p>baf17c90980f3c61f75775f561ced5b2a1d2141c (tip of infernalis) (trusty, xenial, centos)</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=888322017-04-11T13:37:41ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Assignee</strong> changed from <i>Nathan Cutler</i> to <i>David Galloway</i></li></ul> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=888432017-04-11T16:36:21ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David - please</p>
<p>baf17c90980f3c61f75775f561ced5b2a1d2141c (tip of infernalis) (trusty, xenial, centos)</p>
</blockquote>
<p>Will this sha1 eventually be made into an official release? I'm rebuilding it now using the CI but am wondering if I need to push them to the main chacra node so the builds don't get deleted.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=889372017-04-12T12:45:18ZJason Dillamandillaman@redhat.com
<ul></ul><p>@David: Infernalis is an end-of-life release so we won't be doing any official builds. However, we would want to keep the current tip of the infernalis branch around for testing.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=889742017-04-12T14:45:07ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: what @Jason said - the issue here is that when we specify "branch: infernalis" in the test yaml (install task), Shaman/Chacra gives us the "official" build (v9.2.1) which is a different SHA1 than the current tip of infernalis.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=889832017-04-12T15:16:35ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David: what @Jason said - the issue here is that when we specify "branch: infernalis" in the test yaml (install task), Shaman/Chacra gives us the "official" build (v9.2.1) which is a different SHA1 than the current tip of infernalis.</p>
</blockquote>
<p>OK, I rebuilt the tip of infernalis in the CI and pushed copies of trusty, xenial, and centos to the chacra node that won't delete builds.</p>
<p><a class="external" href="https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/24096/">https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/24096/</a><br /><a class="external" href="https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/24064/">https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/24064/</a><br /><a class="external" href="https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/18766/">https://shaman.ceph.com/repos/ceph/infernalis/baf17c90980f3c61f75775f561ced5b2a1d2141c/default/18766/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=894982017-04-17T11:00:57ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David - the "libcephfs-java" package appears to be missing in the Xenial 10.2.0 repo:</p>
<pre>
2017-04-17T10:39:28.867 INFO:teuthology.orchestra.run.smithi117:Running: u'sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=10.2.0-1xenial rbd-fuse=10.2.0-1xenial librbd1=10.2.0-1xenial ceph-fuse=10.2.0-1xenial python-ceph=10.2.0-1xenial ceph-common=10.2.0-1xenial libcephfs-java=10.2.0-1xenial ceph=10.2.0-1xenial libcephfs-jni=10.2.0-1xenial ceph-test=10.2.0-1xenial radosgw=10.2.0-1xenial librados2=10.2.0-1xenial'
2017-04-17T10:39:28.929 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists...
2017-04-17T10:39:29.164 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree...
2017-04-17T10:39:29.164 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information...
2017-04-17T10:39:29.189 INFO:teuthology.orchestra.run.smithi117.stderr:E: Version '10.2.0-1xenial' for 'libcephfs-java' was not found
</pre> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=895102017-04-17T17:47:34ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David - the "libcephfs-java" package appears to be missing in the Xenial 10.2.0 repo:</p>
<p>[...]</p>
</blockquote>
<p>That build was pushed directly from download.ceph.com and the package doesn't appear to have been built/pushed there: <a class="external" href="http://download.ceph.com/debian-jewel/pool/main/c/ceph/">http://download.ceph.com/debian-jewel/pool/main/c/ceph/</a></p>
<p>I was able to find a repo on chacra.ceph.com of the same sha1 using ref 'jewel' instead of 'v10.2.0' and copied the package from there.</p>
<p><a class="external" href="https://chacra.ceph.com/r/ceph/jewel/3a9fba20ec743699b69bd0181dd6c54dc01c64b9/ubuntu/xenial/flavors/default/pool/main/c/ceph/">https://chacra.ceph.com/r/ceph/jewel/3a9fba20ec743699b69bd0181dd6c54dc01c64b9/ubuntu/xenial/flavors/default/pool/main/c/ceph/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=895312017-04-18T06:17:01ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Thanks, David. I realized later that I could have modified the test yaml to exclude that package.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=895912017-04-18T15:10:38ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: Hmm, now it's failing in a slightly different way. Could you have a look?</p>
<p><a class="external" href="http://pulpito.ceph.com/smithfarm-2017-04-18_13:23:55-upgrade:jewel-x-wip-kraken-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-04-18_13:23:55-upgrade:jewel-x-wip-kraken-backports-distro-basic-vps/</a></p>
<p>(I ran the test 5 times to be sure)</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=896072017-04-18T18:40:57ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David: Hmm, now it's failing in a slightly different way. Could you have a look?</p>
<p><a class="external" href="http://pulpito.ceph.com/smithfarm-2017-04-18_13:23:55-upgrade:jewel-x-wip-kraken-backports-distro-basic-vps/">http://pulpito.ceph.com/smithfarm-2017-04-18_13:23:55-upgrade:jewel-x-wip-kraken-backports-distro-basic-vps/</a></p>
<p>(I ran the test 5 times to be sure)</p>
</blockquote>
<p>This worked:</p>
<pre>
sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph-mds=10.2.0-1xenial rbd-fuse=10.2.0-1xenial librbd1=10.2.0-1xenial ceph-fuse=10.2.0-1xenial python-ceph=10.2.0-1xenial ceph-common=10.2.0-1xenial libcephfs-java=10.2.0-1xenial ceph=10.2.0-1xenial libcephfs-jni=10.2.0-1xenial ceph-test=10.2.0-1xenial radosgw=10.2.0-1xenial librados2=10.2.0-1xenial libradosstriper1=10.2.0-1xenial librgw2=10.2.0-1xenial python-rados=10.2.0-1xenial python-cephfs=10.2.0-1xenial python-rbd=10.2.0-1xenial libcephfs1=10.2.0-1xenial
</pre>
<p>Just had to add <code>libradosstriper1=10.2.0-1xenial librgw2=10.2.0-1xenial python-rados=10.2.0-1xenial python-cephfs=10.2.0-1xenial python-rbd=10.2.0-1xenial libcephfs1=10.2.0-1xenial</code></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=906562017-05-03T10:04:26ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David: Please add e832001feaf8c176593e0325c8298e3f16dfb403 (v0.94.6) - trusty build needed for <a class="external" href="https://github.com/ceph/ceph/pull/14930">https://github.com/ceph/ceph/pull/14930</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=907492017-05-04T15:47:03ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David: Please add e832001feaf8c176593e0325c8298e3f16dfb403 (v0.94.6) - trusty build needed for <a class="external" href="https://github.com/ceph/ceph/pull/14930">https://github.com/ceph/ceph/pull/14930</a></p>
</blockquote>
<p><a class="external" href="https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&sha1=e832001feaf8c176593e0325c8298e3f16dfb403">https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&sha1=e832001feaf8c176593e0325c8298e3f16dfb403</a><br /><a class="external" href="https://chacra.ceph.com/r/ceph/v0.94.6/e832001feaf8c176593e0325c8298e3f16dfb403/ubuntu/trusty/flavors/default/">https://chacra.ceph.com/r/ceph/v0.94.6/e832001feaf8c176593e0325c8298e3f16dfb403/ubuntu/trusty/flavors/default/</a></p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=942242017-07-03T22:00:37ZNathan Cutlerncutler@suse.cz
<ul></ul><p>@David Please add the CentOS 7.3, Ubuntu 14.04, and Ubuntu 16.04 builds of v10.2.7 to the Chacra node that doesn't delete builds.</p>
<p>Ideally, this would happen automatically for every point release.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=945672017-07-06T18:12:38ZDavid Galloway
<ul></ul><p>Nathan Cutler wrote:</p>
<blockquote>
<p>@David Please add the CentOS 7.3, Ubuntu 14.04, and Ubuntu 16.04 builds of v10.2.7 to the Chacra node that doesn't delete builds.</p>
<p>Ideally, this would happen automatically for every point release.</p>
</blockquote>
<p>Can you send me the teuthology log where this failed? All point releases are pushed to chacra.ceph.com and this shouldn't have failed because that release /is/ available.</p>
<pre>
git ls-remote git://git.ceph.com/ceph v10.2.7^{}
50e863e0f4bc8f4b9e31156de690d765af245185 refs/tags/v10.2.7^{}
dgalloway@w541 ~ ()$ curl "https://shaman.ceph.com/api/search/?status=ready&project=ceph&sha1=50e863e0f4bc8f4b9e31156de690d765af245185" | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3715 100 3715 0 0 10806 0 --:--:-- --:--:-- --:--:-- 10830
[
{
"status": "ready",
"sha1": "50e863e0f4bc8f4b9e31156de690d765af245185",
"extra": {
"build_url": "https://jenkins.ceph.com/job/ceph-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge/219/",
"root_build_cause": "MANUALTRIGGER",
"version": "10.2.7",
"node_name": "172.21.4.53+omani003",
"job_name": "ceph-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos7,DIST=centos7,MACHINE_SIZE=huge",
"package_manager_version": "10.2.7-0"
},
"url": "https://chacra.ceph.com/r/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/centos/7/flavors/default/",
"distro_codename": null,
"modified": "2017-04-10 17:26:41.629376",
"distro_version": "7",
"project": "ceph",
"flavor": "default",
"ref": "jewel",
"chacra_url": "https://chacra.ceph.com/repos/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/centos/7/flavors/default/",
"archs": [
"source",
"x86_64",
"arm64"
],
"distro": "centos"
},
{
"status": "ready",
"sha1": "50e863e0f4bc8f4b9e31156de690d765af245185",
"extra": {
"build_url": "https://jenkins.ceph.com/job/ceph-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge/219/",
"root_build_cause": "MANUALTRIGGER",
"version": "10.2.7",
"node_name": "172.21.4.52+omani002",
"job_name": "ceph-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=xenial,DIST=xenial,MACHINE_SIZE=huge",
"package_manager_version": "10.2.7-1xenial"
},
"url": "https://chacra.ceph.com/r/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/ubuntu/xenial/flavors/default/",
"distro_codename": "xenial",
"modified": "2017-04-10 16:51:31.647774",
"distro_version": "16.04",
"project": "ceph",
"flavor": "default",
"ref": "jewel",
"chacra_url": "https://chacra.ceph.com/repos/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/ubuntu/xenial/flavors/default/",
"archs": [
"x86_64",
"arm64"
],
"distro": "ubuntu"
},
{
"status": "ready",
"sha1": "50e863e0f4bc8f4b9e31156de690d765af245185",
"extra": {
"build_url": "https://jenkins.ceph.com/job/ceph-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jessie,DIST=jessie,MACHINE_SIZE=huge/218/",
"root_build_cause": "MANUALTRIGGER",
"version": "10.2.7",
"node_name": "172.21.15.124+smithi124",
"job_name": "ceph-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=jessie,DIST=jessie,MACHINE_SIZE=huge",
"package_manager_version": "10.2.7-1~bpo80+1"
},
"url": "https://chacra.ceph.com/r/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/debian/jessie/flavors/default/",
"distro_codename": "jessie",
"modified": "2017-04-10 13:40:13.307632",
"distro_version": "8",
"project": "ceph",
"flavor": "default",
"ref": "jewel",
"chacra_url": "https://chacra.ceph.com/repos/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/debian/jessie/flavors/default/",
"archs": [
"x86_64"
],
"distro": "debian"
},
{
"status": "ready",
"sha1": "50e863e0f4bc8f4b9e31156de690d765af245185",
"extra": {
"build_url": "https://jenkins.ceph.com/job/ceph-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty,MACHINE_SIZE=huge/218/",
"root_build_cause": "MANUALTRIGGER",
"version": "10.2.7",
"node_name": "172.21.1.39+slave-ubuntu03",
"job_name": "ceph-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=trusty,DIST=trusty,MACHINE_SIZE=huge",
"package_manager_version": "10.2.7-1trusty"
},
"url": "https://chacra.ceph.com/r/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/ubuntu/trusty/flavors/default/",
"distro_codename": "trusty",
"modified": "2017-04-10 13:26:44.133870",
"distro_version": "14.04",
"project": "ceph",
"flavor": "default",
"ref": "jewel",
"chacra_url": "https://chacra.ceph.com/repos/ceph/jewel/50e863e0f4bc8f4b9e31156de690d765af245185/ubuntu/trusty/flavors/default/",
"archs": [
"x86_64"
],
"distro": "ubuntu"
}
]
</pre> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=945742017-07-06T19:09:19ZNathan Cutlerncutler@suse.cz
<ul></ul><p>Oh, I didn't realize that the release workflow already has this covered. Thanks, David, and sorry for the trouble.</p>
<p>And I can confirm that the v10.2.7 upgrade test (which I thought might fail) succeeded.</p> sepia - Bug #18089: Various official builds missing from CI/Shamanhttps://tracker.ceph.com/issues/18089?journal_id=961112017-07-28T17:49:26ZDavid Galloway
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Resolved</i></li></ul><p>Think this one's sorted now. Feel free to reopen if you come across builds that need pushing.</p>