Feature #10160: Allow teuthology to use alternative gitbuilder URL via config yaml - teuthology - Ceph

Custom queries

Bug queue
Bug triage
Crash queue
Crash triage
Feedback
My issues
Need Review
Pending backports
Product Backlog Scrub

Actions

Copy link

Feature #10160

closed

Allow teuthology to use alternative gitbuilder URL via config yaml

Added by Sandon Van Ness over 9 years ago. Updated over 9 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Zack Cerza

Category:

Target version:

sprint22

% Done:

Source:

other

Tags:

Backport:

Reviewed:

Affected Versions:

Description

Per the description. We will expect it to be an identical format we just want to be able to over-ride http://gitbuilder.ceph.com/ with an alternative URL that is expected to mirror that (or atleast be in the same format).

Should be a fairly minor code change.

History
Notes
Property changes

Actions

Copy link

Updated by Zack Cerza over 9 years ago

Buried in teuthology.task.install I found:
https://github.com/ceph/teuthology/blob/master/teuthology/task/install.py#L257

If we could generalize that (and make it use teuthology.config instead), that would get us pretty far. Clearly there are several other modules that would need modification though.

Unfortunately, I am currently seeing a ton of this:

2014-12-17 15:46:14,200.200 INFO:teuthology.orchestra.run.magna009:Running: 'sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=0.89-730-g65be519-1trusty ceph-dbg=0.89-730-g65be519-1trusty ceph-mds=0.89-730-g65be519-1trusty ceph-mds-dbg=0.89-730-g65be519-1trusty ceph-common=0.89-730-g65be519-1trusty ceph-common-dbg=0.89-730-g65be519-1trusty ceph-fuse=0.89-730-g65be519-1trusty ceph-fuse-dbg=0.89-730-g65be519-1trusty ceph-test=0.89-730-g65be519-1trusty ceph-test-dbg=0.89-730-g65be519-1trusty radosgw=0.89-730-g65be519-1trusty radosgw-dbg=0.89-730-g65be519-1trusty python-ceph=0.89-730-g65be519-1trusty libcephfs1=0.89-730-g65be519-1trusty libcephfs1-dbg=0.89-730-g65be519-1trusty libcephfs-java=0.89-730-g65be519-1trusty librados2=0.89-730-g65be519-1trusty librados2-dbg=0.89-730-g65be519-1trusty librbd1=0.89-730-g65be519-1trusty librbd1-dbg=0.89-730-g65be519-1trusty'
2014-12-17 15:46:24,826.826 INFO:teuthology.orchestra.run.magna009.stderr:E: Failed to fetch http://gitbuilder.ceph.redhat.com/ceph-deb-trusty-x86_64-basic/ref/master/pool/main/c/ceph/librbd1-dbg_0.89-730-g65be519-1trusty_amd64.deb  404  Not Found

So... we might want to fix the mirror.

Actions

Copy link

Updated by Sandon Van Ness over 9 years ago

That probably happened due to files still being copied. I think the problem is that the symbolic link in ref/ is being updated even when files are still being copied. I will try to think of a good way around this.

Actions

Copy link

Updated by Zack Cerza over 9 years ago

That seems like a problem that has to have been solved. What do other people running package repo mirrors do?

Actions

Copy link

Updated by Sandon Van Ness over 9 years ago

The complication is there are two parts to this.. There is the actual repos being copied and then there is the symlink that is in /ref that points to the actual folder in the sha1 folder. The problem is there isn't a very nice way to copy the links after based on destination of link.

It should be fixable I think I am just going to have to run a readlink loop on all the links in /ref on the destination gitbuilder and only copy over ones that match the same destination as the sha1 that was just copied (after the sha1 is done syncing). It would be nice if there was an rsync option that would get me this kind of behavior but I seriously doubt that will be the case. A lot of the complication is coming from the fact that we only have one way access to the gitbuilder-archive. Things would be much simpler if the gitbuilders could simply rsync to another mirror but unfortunately for security reasons we can't do that :(

I will work out a solution today.

Actions

Copy link

Updated by Sandon Van Ness over 9 years ago

I believe this should be fixed now. After syncing the sha1 it checks the remote gitbuilder archive server its mirroring from to see if any of the /ref symlinks point to its sha1 and syncs those refs instead of blindly syincing all of the /ref's like it used to. The problem is that often times we have multiple syncs even going from the same gitbuilder due to the network slowness which was causing the /ref to get updated earlier. For example earlier today I saw all these parallell sync's going on:

        9036.ceph-rpm-rhel6_5-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59     (12/18/2014 10:45:05 PM)        (Detached)
        9003.ceph-rpm-rhel6_5-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44     (12/18/2014 10:45:05 PM)        (Detached)
        8968.ceph-rpm-rhel6_5-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19     (12/18/2014 10:45:05 PM)        (Detached)
        8927.ceph-rpm-rhel6_5-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc     (12/18/2014 10:45:05 PM)        (Detached)
        8896.ceph-rpm-rhel6_5-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4     (12/18/2014 10:45:05 PM)        (Detached)
        8863.ceph-rpm-rhel6_5-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf     (12/18/2014 10:45:05 PM)        (Detached)
        8825.ceph-rpm-rhel6_4-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9     (12/18/2014 10:45:05 PM)        (Detached)
        8746.ceph-rpm-rhel6_4-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d     (12/18/2014 10:45:05 PM)        (Detached)
        8725.ceph-rpm-rhel6_4-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806     (12/18/2014 10:45:05 PM)        (Detached)
        8708.ceph-rpm-rhel6_4-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59     (12/18/2014 10:45:05 PM)        (Detached)
        8668.ceph-rpm-rhel6_4-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44     (12/18/2014 10:45:05 PM)        (Detached)
        8583.ceph-rpm-rhel6_4-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19     (12/18/2014 10:45:04 PM)        (Detached)
        8567.ceph-rpm-rhel6_4-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc     (12/18/2014 10:45:04 PM)        (Detached)
        8548.ceph-rpm-rhel6_4-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4     (12/18/2014 10:45:04 PM)        (Detached)
        8531.ceph-rpm-rhel6_4-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf     (12/18/2014 10:45:04 PM)        (Detached)
        8511.ceph-rpm-rhel6_3-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9     (12/18/2014 10:45:04 PM)        (Detached)
        8477.ceph-rpm-rhel6_3-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d     (12/18/2014 10:45:04 PM)        (Detached)
        8457.ceph-rpm-rhel6_3-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806     (12/18/2014 10:45:04 PM)        (Detached)
        8427.ceph-rpm-rhel6_3-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59     (12/18/2014 10:45:04 PM)        (Detached)
        8394.ceph-rpm-rhel6_3-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44     (12/18/2014 10:45:04 PM)        (Detached)
        8361.ceph-rpm-rhel6_3-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19     (12/18/2014 10:45:04 PM)        (Detached)
        8330.ceph-rpm-rhel6_3-x86_64-basic_2474a7854cd903be35931c08f05fa17e1dfc98cc     (12/18/2014 10:45:04 PM)        (Detached)
        8300.ceph-rpm-rhel6_3-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4     (12/18/2014 10:45:04 PM)        (Detached)
        8270.ceph-rpm-rhel6_3-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf     (12/18/2014 10:45:04 PM)        (Detached)
        8240.ceph-rpm-centos6-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59        (12/18/2014 10:45:04 PM)        (Detached)
        8210.ceph-rpm-centos6-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d     (12/18/2014 10:45:04 PM)        (Detached)
        8180.ceph-rpm-centos6-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9     (12/18/2014 10:45:04 PM)        (Detached)
        8150.ceph-rpm-centos6-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59     (12/18/2014 10:45:04 PM)        (Detached)
        8120.ceph-rpm-centos6-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44     (12/18/2014 10:45:03 PM)        (Detached)
        8090.ceph-rpm-centos6-x86_64-basic_a2112fd0721756e580f13299db5bfdca1674c606     (12/18/2014 10:45:03 PM)        (Detached)
        8059.ceph-rpm-centos6-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19     (12/18/2014 10:45:03 PM)        (Detached)
        8030.ceph-rpm-centos6-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf     (12/18/2014 10:45:03 PM)        (Detached)
        8000.ceph-rpm-centos6-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4     (12/18/2014 10:45:03 PM)        (Detached)
        7970.ceph-rpm-centos6_5-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59      (12/18/2014 10:45:03 PM)        (Detached)
        7940.ceph-rpm-centos6_5-x86_64-basic_e93039b727f5c5b838abafb532664d8640ae6f2d   (12/18/2014 10:45:03 PM)        (Detached)
        7910.ceph-rpm-centos6_5-x86_64-basic_680eb19f06715dfc95bb2521a2be289051d2cbe9   (12/18/2014 10:45:03 PM)        (Detached)
        7880.ceph-rpm-centos6_5-x86_64-basic_94664d6da56f4c50bd2f5adbe35839e21cb9df59   (12/18/2014 10:45:03 PM)        (Detached)
        7850.ceph-rpm-centos6_5-x86_64-basic_cc504694bf0ec8e04938128ad51f10a4bf1f1d44   (12/18/2014 10:45:03 PM)        (Detached)
        7820.ceph-rpm-centos6_5-x86_64-basic_a2112fd0721756e580f13299db5bfdca1674c606   (12/18/2014 10:45:03 PM)        (Detached)
        7790.ceph-rpm-centos6_5-x86_64-basic_04ec8f2a4f79a0a8641d19a9bda7fb871a874f19   (12/18/2014 10:45:03 PM)        (Detached)
        7760.ceph-rpm-centos6_5-x86_64-basic_916a211aaceefc2197922249f0b6b9e580c781cf   (12/18/2014 10:45:03 PM)        (Detached)
        7730.ceph-rpm-centos6_5-x86_64-basic_1a080169f4666436342c300f9a4a4c4854f218f4   (12/18/2014 10:45:03 PM)        (Detached)
        7700.ceph-rpm-centos6_4-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59      (12/18/2014 10:45:03 PM)        (Detached)
        7670.ceph-rpm-centos6_3-x86_64-notcmalloc_94664d6da56f4c50bd2f5adbe35839e21cb9df59      (12/18/2014 10:45:03 PM)        (Detached)
        5901.ceph-deb-wheezy-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806      (12/18/2014 10:44:02 PM)        (Detached)
        5871.ceph-deb-trusty-i686-basic_e93039b727f5c5b838abafb532664d8640ae6f2d        (12/18/2014 10:44:02 PM)        (Detached)
        30236.ceph-deb-precise-x86_64-notcmalloc_5439d9ee708f191a8e1aa758b8edd591be98d806       (12/18/2014 10:39:02 PM)        (Detached)
        28626.ceph-deb-wheezy-x86_64-notcmalloc_e5c02cc778be2b8a0200157b3628913f2b5fd0bf        (12/18/2014 10:38:03 PM)        (Detached)
        28583.ceph-deb-precise-x86_64-basic_5439d9ee708f191a8e1aa758b8edd591be98d806    (12/18/2014 10:38:03 PM)        (Detached)

Actions

Copy link