Bug #16944: Migration of teuthology.front to a VM - sepia - Ceph

Actions

Copy link

Bug #16944

closed

Migration of teuthology.front to a VM

Added by Zack Cerza over 7 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Immediate

Assignee:

Zack Cerza

Category:

Infrastructure Service

Target version:

% Done:

Source:

other

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

Week before
---
☑ get role updates merged
☑ get inventory updates merged
☑ run role
☑ verify jobs can be scheduled and run from teuthology-new (using teuthology's beanstalkd)
☑ sync homes
☑ write up plan
☑ include plan ticket in notice on status.front
☑ send email including plan

Monday morning
---
☑ pause vps queue
☑ add DNS A record to point teuthology-old at old host (leave teuthology alone for now)

Monday afternoon
---
☑ shut down miras that need to physically move
☑ no vps jobs should be running
☑ tell workers to stop
☑ kill vps workers if necessary
☑ unpause vps queue

Tuesday morning
---
☑ no workers should be running (and therefore no jobs)
☑ sync homes again
☑ stop beanstalkd on old host
☑ move beanstalkd data to new host
☑ start beanstalkd on new host
☑ move crontabs
☑ build necessary virtualenvs
☑ set timezone to UTC
☑ update DNS (and DHCP names) to point teuthology to new host
☑ start workers on new host
☑ send email, throw party

Actions

Copy link

Updated by Zack Cerza over 7 years ago

How I'm syncing the homedirs:

$ cat ~/bin/Rsyncall.sh
#!/bin/bash
set -ex
for dir in $(ls); do Rsync.sh $dir; done

$ cat ~/bin/Rsync.sh
#!/bin/bash
set -ex
sudo rsync -aAXSHx --info=progress2 --delete --exclude-from=$HOME/rsync_exclude -e "ssh -i $HOME/.ssh/id_rsa" --rsync-path="sudo rsync" --bwlimit=6000 $1 zack@teuthology-new:/home/

$ cat ~/rsync_exclude
archive*/
.cache/
tmp/
virtualenv/
venv/
ubuntu/joao
ubuntu/zhyan

Actions

Copy link