Feature #10160
closedAllow teuthology to use alternative gitbuilder URL via config yaml
0%
Description
Per the description. We will expect it to be an identical format we just want to be able to over-ride http://gitbuilder.ceph.com/ with an alternative URL that is expected to mirror that (or atleast be in the same format).
Should be a fairly minor code change.
Updated by Zack Cerza over 9 years ago
Buried in teuthology.task.install
I found:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L257
If we could generalize that (and make it use teuthology.config
instead), that would get us pretty far. Clearly there are several other modules that would need modification though.
Unfortunately, I am currently seeing a ton of this:
2014-12-17 15:46:14,200.200 INFO:teuthology.orchestra.run.magna009:Running: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.89-730-g65be519-1trusty ceph-dbg=0.89-730-g65be519-1trusty ceph-mds=0.89-730-g65be519-1trusty ceph-mds-dbg=0.89-730-g65be519-1trusty ceph-common=0.89-730-g65be519-1trusty ceph-common-dbg=0.89-730-g65be519-1trusty ceph-fuse=0.89-730-g65be519-1trusty ceph-fuse-dbg=0.89-730-g65be519-1trusty ceph-test=0.89-730-g65be519-1trusty ceph-test-dbg=0.89-730-g65be519-1trusty radosgw=0.89-730-g65be519-1trusty radosgw-dbg=0.89-730-g65be519-1trusty python-ceph=0.89-730-g65be519-1trusty libcephfs1=0.89-730-g65be519-1trusty libcephfs1-dbg=0.89-730-g65be519-1trusty libcephfs-java=0.89-730-g65be519-1trusty librados2=0.89-730-g65be519-1trusty librados2-dbg=0.89-730-g65be519-1trusty librbd1=0.89-730-g65be519-1trusty librbd1-dbg=0.89-730-g65be519-1trusty' 2014-12-17 15:46:24,826.826 INFO:teuthology.orchestra.run.magna009.stderr:E: Failed to fetch http://gitbuilder.ceph.redhat.com/ceph-deb-trusty-x86_64-basic/ref/master/pool/main/c/ceph/librbd1-dbg_0.89-730-g65be519-1trusty_amd64.deb 404 Not Found
So... we might want to fix the mirror.
Updated by Sandon Van Ness over 9 years ago
That probably happened due to files still being copied. I think the problem is that the symbolic link in ref/ is being updated even when files are still being copied. I will try to think of a good way around this.
Updated by Zack Cerza over 9 years ago
That seems like a problem that has to have been solved. What do other people running package repo mirrors do?
Updated by Sandon Van Ness over 9 years ago
The complication is there are two parts to this.. There is the actual repos being copied and then there is the symlink that is in /ref that points to the actual folder in the sha1 folder. The problem is there isn't a very nice way to copy the links after based on destination of link.
It should be fixable I think I am just going to have to run a readlink loop on all the links in /ref on the destination gitbuilder and only copy over ones that match the same destination as the sha1 that was just copied (after the sha1 is done syncing). It would be nice if there was an rsync option that would get me this kind of behavior but I seriously doubt that will be the case. A lot of the complication is coming from the fact that we only have one way access to the gitbuilder-archive. Things would be much simpler if the gitbuilders could simply rsync to another mirror but unfortunately for security reasons we can't do that :(
I will work out a solution today.
Updated by Sandon Van Ness over 9 years ago
I believe this should be fixed now. After syncing the sha1 it checks the remote gitbuilder archive server its mirroring from to see if any of the /ref symlinks point to its sha1 and syncs those refs instead of blindly syincing all of the /ref's like it used to. The problem is that often times we have multiple syncs even going from the same gitbuilder due to the network slowness which was causing the /ref to get updated earlier. For example earlier today I saw all these parallell sync's going on:
9036.ceph-rpm-rhel6_5-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:05 PM) (Detached) 9003.ceph-rpm-rhel6_5-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44 (12/18/2014 10:45:05 PM) (Detached) 8968.ceph-rpm-rhel6_5-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19 (12/18/2014 10:45:05 PM) (Detached) 8927.ceph-rpm-rhel6_5-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc (12/18/2014 10:45:05 PM) (Detached) 8896.ceph-rpm-rhel6_5-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4 (12/18/2014 10:45:05 PM) (Detached) 8863.ceph-rpm-rhel6_5-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf (12/18/2014 10:45:05 PM) (Detached) 8825.ceph-rpm-rhel6_4-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9 (12/18/2014 10:45:05 PM) (Detached) 8746.ceph-rpm-rhel6_4-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d (12/18/2014 10:45:05 PM) (Detached) 8725.ceph-rpm-rhel6_4-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806 (12/18/2014 10:45:05 PM) (Detached) 8708.ceph-rpm-rhel6_4-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:05 PM) (Detached) 8668.ceph-rpm-rhel6_4-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44 (12/18/2014 10:45:05 PM) (Detached) 8583.ceph-rpm-rhel6_4-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19 (12/18/2014 10:45:04 PM) (Detached) 8567.ceph-rpm-rhel6_4-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc (12/18/2014 10:45:04 PM) (Detached) 8548.ceph-rpm-rhel6_4-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4 (12/18/2014 10:45:04 PM) (Detached) 8531.ceph-rpm-rhel6_4-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf (12/18/2014 10:45:04 PM) (Detached) 8511.ceph-rpm-rhel6_3-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9 (12/18/2014 10:45:04 PM) (Detached) 8477.ceph-rpm-rhel6_3-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d (12/18/2014 10:45:04 PM) (Detached) 8457.ceph-rpm-rhel6_3-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806 (12/18/2014 10:45:04 PM) (Detached) 8427.ceph-rpm-rhel6_3-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:04 PM) (Detached) 8394.ceph-rpm-rhel6_3-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44 (12/18/2014 10:45:04 PM) (Detached) 8361.ceph-rpm-rhel6_3-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19 (12/18/2014 10:45:04 PM) (Detached) 8330.ceph-rpm-rhel6_3-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc (12/18/2014 10:45:04 PM) (Detached) 8300.ceph-rpm-rhel6_3-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4 (12/18/2014 10:45:04 PM) (Detached) 8270.ceph-rpm-rhel6_3-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf (12/18/2014 10:45:04 PM) (Detached) 8240.ceph-rpm-centos6-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:04 PM) (Detached) 8210.ceph-rpm-centos6-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d (12/18/2014 10:45:04 PM) (Detached) 8180.ceph-rpm-centos6-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9 (12/18/2014 10:45:04 PM) (Detached) 8150.ceph-rpm-centos6-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:04 PM) (Detached) 8120.ceph-rpm-centos6-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44 (12/18/2014 10:45:03 PM) (Detached) 8090.ceph-rpm-centos6-x86_64-basic_a2112fd0721756e580f13299db5bfdca1674c606 (12/18/2014 10:45:03 PM) (Detached) 8059.ceph-rpm-centos6-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19 (12/18/2014 10:45:03 PM) (Detached) 8030.ceph-rpm-centos6-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf (12/18/2014 10:45:03 PM) (Detached) 8000.ceph-rpm-centos6-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4 (12/18/2014 10:45:03 PM) (Detached) 7970.ceph-rpm-centos6_5-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:03 PM) (Detached) 7940.ceph-rpm-centos6_5-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d (12/18/2014 10:45:03 PM) (Detached) 7910.ceph-rpm-centos6_5-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9 (12/18/2014 10:45:03 PM) (Detached) 7880.ceph-rpm-centos6_5-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:03 PM) (Detached) 7850.ceph-rpm-centos6_5-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44 (12/18/2014 10:45:03 PM) (Detached) 7820.ceph-rpm-centos6_5-x86_64-basic_a2112fd0721756e580f13299db5bfdca1674c606 (12/18/2014 10:45:03 PM) (Detached) 7790.ceph-rpm-centos6_5-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19 (12/18/2014 10:45:03 PM) (Detached) 7760.ceph-rpm-centos6_5-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf (12/18/2014 10:45:03 PM) (Detached) 7730.ceph-rpm-centos6_5-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4 (12/18/2014 10:45:03 PM) (Detached) 7700.ceph-rpm-centos6_4-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:03 PM) (Detached) 7670.ceph-rpm-centos6_3-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59 (12/18/2014 10:45:03 PM) (Detached) 5901.ceph-deb-wheezy-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806 (12/18/2014 10:44:02 PM) (Detached) 5871.ceph-deb-trusty-i686-basic_e93039b727f5c5b838abafb532664d8640ae6f2d (12/18/2014 10:44:02 PM) (Detached) 30236.ceph-deb-precise-x86_64-notcmalloc_5439d9ee708f191a8e1aa758b8edd591be98d806 (12/18/2014 10:39:02 PM) (Detached) 28626.ceph-deb-wheezy-x86_64-notcmalloc_e5c02cc778be2b8a0200157b3628913f2b5fd0bf (12/18/2014 10:38:03 PM) (Detached) 28583.ceph-deb-precise-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806 (12/18/2014 10:38:03 PM) (Detached)
Updated by Zack Cerza about 9 years ago
- Status changed from New to Resolved
- Assignee set to Zack Cerza
- Target version set to sprint22
https://github.com/ceph/teuthology/commit/b77af118bfd68194866090a20a8c0f06119d2b96
https://github.com/ceph/teuthology/commit/86c9c68cc821e474f8124604d5ecd096a3cced4b
https://github.com/ceph/teuthology/commit/c0c8441559f4d314db1134a059acd5b46312f6f8
https://github.com/ceph/teuthology/commit/a63a087a3e32450189830c9df4cfb6b22f890be6
https://github.com/ceph/teuthology/commit/e280c0a0879e216678ea31bd173a969925679d23