implement simple tests for rgw multisite feature
reference tracker ticket: http://tracker.ceph.com/issues/20251
setup rgw multisite cluster with master and secondary zones.
1. On master, create ~20 buckets to generate some metadata sync load.
2. On secondary, use systemctl to restart radosgw a few times, maybe every ~60sec or so.
test result: ensure there is no crash in RGWMetaSyncShardCR::incremental_sync completion.
note: the fix for this ticket was shipped as cutomer hot fix in ceph version 10.2.7-28.0.hotfix.bz1476888.el7cp
we already have test scripts to be run on primary and secondary setup to run, all we need is include the test as part of rgw qa suite.
on master zone - createbuckets.py [run the script in master zone]
from time import sleep
access_key = '7ZYL6IDWBC1C1K3WBI6E'
secret_key = 'loL0logoEpqsKMynNlreeNDCpxkTvQS2ay2sXzw1'
conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = 'magna057.ceph.redhat.com',
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
count = 0
count += 1
bname = 'bzthursbucketnumber%d' % count
bucket = conn.create_bucket(bname)
on secondary zone - rgwstart.sh & [run the script in background, so it restarts rgw every 1 minute]
#! /bin/bash -fv
while true; do
systemctl restart ceph-radosgw@rgw.`hostname -s`
leave the test scripts running for 10 hours and ensure there is no segfault in rgw log [/var/log/ceph/ceph-rgw.<hostname>.log]