Feature #5080
closed
distcc for arm gitbuilders
Added by Sage Weil about 11 years ago.
Updated almost 11 years ago.
Description
this would hopefully increase out gitbilder throughput by leveraging 10s of nodes for the builds
maybe we can use up the remaining nodes in our current box?
- Assignee set to Sandon Van Ness
- % Done changed from 0 to 80
Might need to be tweaked some more and fab modifications but I payed around with this. Its actually pretty awesome. I gotta say this is a lot better than the 2hours it was taking =) This was a build of cuttlefish with --without-tcmalloc
CXXLD librbd.la
CXXLD librados-config
CXXLD radosgw-admin
CXXLD radosgw
CXXLD rados
CCLD rbd-fuse
CXXLD rbd
CXXLD ceph-dencoder
make[3]: Leaving directory `/home/ubuntu/ceph-0.61.1/src'
make[2]: Leaving directory `/home/ubuntu/ceph-0.61.1/src'
make[1]: Leaving directory `/home/ubuntu/ceph-0.61.1/src'
Making all in man
make[1]: Entering directory `/home/ubuntu/ceph-0.61.1/man'
make[1]: Nothing to be done for `all'.
make[1]: Leaving directory `/home/ubuntu/ceph-0.61.1/man'
real 6m59.253s
user 10m21.380s
sys 1m31.530s
Seems like a good half of the time was just doing linking which it probably cant distribute all that much. It was pretty cool watching it though =)
http://box.houkouonchi.jp/distcc.png
Also looks like builds were pretty far from the 2 hours gary had mentioned. The gitbuilder seems to be around 70 minutes and my manual build of what took ~7 min with distcc normally took ~45 minutes which is more on with what rossen was saying.
For distcc going on the arm deb gitbuilder with some hackage.
Anyway with building packages it only helps so much. I have gotten builds with packages down to around 40-ish minutes but now a lot of the problem is stuff that is not parallell (a bunch of configures run) and then creating the deb packages takes quite a bit of time by itself so not sure how much more we can speed it up. I would expect the tarball builder to be under 30 min if it gets setup to use distcc as well.
I tested killing random distcc processes and stuff and it seems to handle machines going down pretty well. I think things can be tuned a bit further but how many machines do we want to dedicated to distcc builds? The way things are now it wouldn't be that hard to actually have it just use all of them. The distcc process runs @ 19 nice so it should mostly only use spare CPU usage on the machines.
- Target version set to v0.65
- Translation missing: en.field_story_points set to 1.00
- Status changed from New to Resolved
- % Done changed from 80 to 100
Marking this as resolved. Dedicated machines for distcc now (locked/described as such) and all 3 arm gitbuilders (deb/tarball/kernel) are using distcc.
Also available in: Atom
PDF
