Ceph » rgw

On node1, I have master zone and zonegroup configured with ceph version 10.2.2.

On node2, I have secondary zone configured with ceph version 13.2.2.

Two nodes are in same LAN and network speed is good enough.

On node2, checked out the sync status:

$ radosgw-admin sync status
realm 2a30a4d2-f67b-4715-9c49-9e22f6a9e463 (r2)
zonegroup 5792f6b0-5bd1-4851-92d8-58db0943281c (b)
zone 40e29c81-c588-4c05-8230-41cbd634632f (bs3)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 14bf7611-2a60-46fa-b118-1afee5fd8e2f (bs)
init
full sync: 128/128 shards
full sync: 0 buckets to sync
incremental sync: 0/128 shards
data is behind on 128 shards
behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
source: 738d66ff-0a69-40d0-879a-e6198f0fea49 (bs2)
preparing for full sync
full sync: 128/128 shards
full sync: 1 buckets to sync
incremental sync: 0/128 shards
data is behind on 128 shards
behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
$ radosgw-admin data sync run --source-zone bs
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.486 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.486 7f525c634680 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards