Project

General

Profile

Actions
Copy link

Bug #37510

open

luminous cannot sync from jewel

Added by grady chen over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v13.2.2
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

On node1, I have master zone and zonegroup configured with ceph version 10.2.2.

On node2, I have secondary zone configured with ceph version 13.2.2.

Two nodes are in same LAN and network speed is good enough.

On node2, checked out the sync status:

$ radosgw-admin sync status
realm 2a30a4d2-f67b-4715-9c49-9e22f6a9e463 (r2)
zonegroup 5792f6b0-5bd1-4851-92d8-58db0943281c (b)
zone 40e29c81-c588-4c05-8230-41cbd634632f (bs3)
metadata sync syncing
full sync: 0/64 shards
incremental sync: 64/64 shards
metadata is caught up with master
data sync source: 14bf7611-2a60-46fa-b118-1afee5fd8e2f (bs)
init
full sync: 128/128 shards
full sync: 0 buckets to sync
incremental sync: 0/128 shards
data is behind on 128 shards
behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
source: 738d66ff-0a69-40d0-879a-e6198f0fea49 (bs2)
preparing for full sync
full sync: 128/128 shards
full sync: 1 buckets to sync
incremental sync: 0/128 shards
data is behind on 128 shards
behind shards: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127]
$ radosgw-admin data sync run --source-zone bs
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.482 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.484 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.485 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.486 7f52137fe700 0 WARNING: curl operation timed out, network average transfer speed less than 1024 Bytes per second during 30 seconds.
2018-12-04 04:14:52.486 7f525c634680 0 RGW-SYNC:data:sync:init_data_sync_status: ERROR: failed to read remote data log shards

Actions
Copy link
 #1

Updated by Brad Hubbard over 5 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #2

Updated by Abhishek Lekshmanan over 5 years ago

Can you post the radosgw-admin period get output to understand how the configuration looks like

Actions
Copy link
 #3

Updated by grady chen over 5 years ago

Reinstalled the 13.2(the title for this bug may change 'luminous' to 'mimic') and the above issue looks like gone as shown:

$ radosgw-admin sync status
          realm 2a30a4d2-f67b-4715-9c49-9e22f6a9e463 (r2)
      zonegroup 5792f6b0-5bd1-4851-92d8-58db0943281c (b)
           zone 1f4131c2-3270-4b94-bb4f-b165d6f59592 (bs2)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: 14bf7611-2a60-46fa-b118-1afee5fd8e2f (bs)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

but in fact I didn't see metadata in secondary ceph cluster:

$ radosgw-admin metadata list user
[]

and period get is:

$ radosgw-admin period get
{
    "id": "48818e96-49e6-48bb-9349-039d609b0f59",
    "epoch": 23,
    "predecessor_uuid": "dfe78a92-b9bd-420e-ac29-deeb522ac0c5",
    "sync_status": [
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "1_1543558333.699286_5528.1",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "",
        "" 
    ],
    "period_map": {
        "id": "48818e96-49e6-48bb-9349-039d609b0f59",
        "zonegroups": [
            {
                "id": "5792f6b0-5bd1-4851-92d8-58db0943281c",
                "name": "b",
                "api_name": "",
                "is_master": "true",
                "endpoints": [
                    "http://10.210.127.54:7480" 
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "14bf7611-2a60-46fa-b118-1afee5fd8e2f",
                "zones": [
                    {
                        "id": "14bf7611-2a60-46fa-b118-1afee5fd8e2f",
                        "name": "bs",
                        "endpoints": [
                            "http://10.210.127.54:7480" 
                        ],
                        "log_meta": "true",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": "" 
                    },
                    {
                        "id": "1f4131c2-3270-4b94-bb4f-b165d6f59592",
                        "name": "bs2",
                        "endpoints": [
                            "http://10.210.127.53:7480" 
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false",
                        "tier_type": "",
                        "sync_from_all": "true",
                        "sync_from": [],
                        "redirect_zone": "" 
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "2a30a4d2-f67b-4715-9c49-9e22f6a9e463" 
            }
        ],
        "short_zone_ids": [
            {
                "key": "14bf7611-2a60-46fa-b118-1afee5fd8e2f",
                "val": 3434509584
            },
            {
                "key": "1f4131c2-3270-4b94-bb4f-b165d6f59592",
                "val": 184940054
            }
        ]
    },
    "master_zonegroup": "5792f6b0-5bd1-4851-92d8-58db0943281c",
    "master_zone": "14bf7611-2a60-46fa-b118-1afee5fd8e2f",
    "period_config": {
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": 0,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": 0,
            "max_size_kb": 0,
            "max_objects": -1
        }
    },
    "realm_id": "2a30a4d2-f67b-4715-9c49-9e22f6a9e463",
    "realm_name": "r2",
    "realm_epoch": 5
}
Actions
Copy link

Also available in: Atom PDF