Bug #16944
closed
Migration of teuthology.front to a VM
Added by Zack Cerza over 7 years ago.
Updated over 7 years ago.
Category:
Infrastructure Service
Description
Week before
---
☑ get role updates merged
☑ get inventory updates merged
☑ run role
☑ verify jobs can be scheduled and run from teuthology-new
(using teuthology
's beanstalkd)
☑ sync homes
☑ write up plan
☑ include plan ticket in notice on status.front
☑ send email including plan
Monday morning
---
☑ pause vps queue
☑ add DNS A record to point teuthology-old
at old host (leave teuthology
alone for now)
Monday afternoon
---
☑ shut down miras that need to physically move
☑ no vps jobs should be running
☑ tell workers to stop
☑ kill vps workers if necessary
☑ unpause vps queue
Tuesday morning
---
☑ no workers should be running (and therefore no jobs)
☑ sync homes again
☑ stop beanstalkd on old host
☑ move beanstalkd data to new host
☑ start beanstalkd on new host
☑ move crontabs
☑ build necessary virtualenvs
☑ set timezone to UTC
☑ update DNS (and DHCP names) to point teuthology
to new host
☑ start workers on new host
☑ send email, throw party
How I'm syncing the homedirs:
$ cat ~/bin/Rsyncall.sh
#!/bin/bash
set -ex
for dir in $(ls); do Rsync.sh $dir; done
$ cat ~/bin/Rsync.sh
#!/bin/bash
set -ex
sudo rsync -aAXSHx --info=progress2 --delete --exclude-from=$HOME/rsync_exclude -e "ssh -i $HOME/.ssh/id_rsa" --rsync-path="sudo rsync" --bwlimit=6000 $1 zack@teuthology-new:/home/
$ cat ~/rsync_exclude
archive*/
.cache/
tmp/
virtualenv/
venv/
ubuntu/joao
ubuntu/zhyan
- Description updated (diff)
VPS queue paused.
Marking down miras 005, 012, 016, 019 so they can be physically moved tomorrow.
- Description updated (diff)
mira{003..015,017,018} have been shut down.
Waiting to hear back from dougf if mira016 can be shut down. Will shutdown later tonight either way if I don't hear back.
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
Adding item to update DNS for qa-proxy
- Description updated (diff)
- Status changed from New to In Progress
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
- Status changed from In Progress to Resolved
Also available in: Atom
PDF