Project

General

Profile

Actions
Copy link

Bug #62597

closed

Cloud sync module not working and document not correct

Added by hoan nv 9 months ago. Updated 8 months ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v17.2.6
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hi all.

I was tested s3 cloud sync feature on https://docs.ceph.com/en/quincy/radosgw/cloud-sync-module/ docs

I meet some issues.

1. Parameter on docs not correnct 

"connections": [
      {
        "connection_id": <id>,
        "access_key": <access>,
        "secret": <secret>,
        "endpoint": <endpoint>,
        "host_style" <path | virtual>,  # optional
      } ... ],

Correct is 

"connections": [
      {
        "id": <id>,
        "access_key": <access>,
        "secret": <secret>,
        "endpoint": <endpoint>,
        "host_style" <path | virtual>,  # optional
      } ... ],

2. On code i see config section, but it not include docs.

  AWSSyncConfig_Profile default_profile;
  std::shared_ptr<AWSSyncConfig_Profile> root_profile;

  map<string, std::shared_ptr<AWSSyncConfig_Connection> > connections;
  AWSSyncConfig_ACLProfiles acl_profiles;

  map<string, std::shared_ptr<AWSSyncConfig_Profile> > explicit_profiles;

  AWSSyncConfig_S3 s3;

3. When sync file to bucket rgw-cloud-sync corrupt
0> 2023-08-28T10:16:37.308+0700 7f3770ff9700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f3770ff9700 thread_name:data-sync

 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f387fb97420]
 2: (RGWRESTStreamS3PutObj::send_init(rgw::sal::Object*)+0x54f) [0x7f3880ec1ee9]
 3: (RGWRESTConn::put_obj_send_init(rgw::sal::Object*, rgw_http_param_pair const*, RGWRESTStreamS3PutObj**)+0x632) [0x7f3880ecea0e]
 4: (RGWAWSStreamPutCRF::init()+0xff) [0x7f3880c8fda9]
 5: (RGWStreamSpliceCR::operate(DoutPrefixProvider const*)+0x5af) [0x7f3880d85dd3]
 6: (RGWCoroutine::operate_wrapper(DoutPrefixProvider const*)+0xe) [0x7f38808ed494]
 7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x1e9) [0x7f3880d6b9cf]
 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x1bb) [0x7f3880d6cfe9]
 9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xa3) [0x7f3880d6e5c1]
 10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x6be) [0x7f3880c0b67c]
 11: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x4a) [0x7f3880e7db70]
 12: (RGWRadosThread::Worker::entry()+0x48d) [0x7f3880e3d9f7]
 13: (Thread::entry_wrapper()+0x43) [0x7f387ea80e99]
 14: (Thread::_entry_func(void*)+0xd) [0x7f387ea80eb5]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f387fb8b609]
 16: clone()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

I see rgw Segmentation fault at this line, it


  void RGWRESTStreamS3PutObj::send_init(rgw::sal::Object* obj)
{
  string resource_str;
  string resource;
  string new_url = url;
  string new_host = host;

  const auto& bucket_name = obj->get_bucket()->get_name();

  if (host_style == VirtualStyle) {
    resource_str = obj->get_oid();

    new_url = bucket_name + "."  + new_url;
    new_host = bucket_name + "." + new_host;
  } else {
    resource_str = bucket_name + "/" + obj->get_oid();
  }

obj cannot get bucket-name or oid then it crash.

Thanks.

Related issues 1 (1 open — 0 closed)

Is duplicate of rgw - Bug #57306: rgw: cloud sync crashPending BackportYehuda Sadeh

Actions
Actions
Copy link
 #1

Updated by Casey Bodley 8 months ago

  • Is duplicate of Bug #57306: rgw: cloud sync crash added
Actions
Copy link
 #2

Updated by Casey Bodley 8 months ago

  • Status changed from New to Duplicate

this looks like a duplicate of https://tracker.ceph.com/issues/57306 which was fixed, but never backported to pacific or quincy

Actions
Copy link
 #3

Updated by hoan nv 8 months ago

Yes this is duplicate of https://tracker.ceph.com/issues/57306.

after fix patch on above tracker. I found another bug

Add config for a new bucket then rgw crash and this is log

2023-09-25T15:41:05.675+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3234a0:25RGWAWSHandleRemoteObjCBCR: operate()
2023-09-25T15:41:05.675+0700 7fcc7e7fc700  4 rgw rados thread: AWS: download begin: z=db57cd2b-2213-41ad-8ba8-599edbe6e3ae b=:lab-test-sync-2[db57cd2b-2213-41ad-8ba8-599edbe6e3ae.34129.1]) k=hoannv-test-f
ile-1 size=8 mtime=2023-09-25T15:41:05.464383+0700 etag=d8e2e94a47ddb650ccece3ab451161fb zone_short_id=1662303307 pg_ver=1
2023-09-25T15:41:05.683+0700 7fcc7e7fc700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fcc7e7fc700 thread_name:data-sync

 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fcd997d1420]
 2: /opt/dev/ceph-17/ceph/build/lib/libradosgw.so.2(+0x10db9f9) [0x7fcd9a8c79f9]
 3: /opt/dev/ceph-17/ceph/build/lib/libradosgw.so.2(+0x10dbf9c) [0x7fcd9a8c7f9c]
 4: (AWSSyncConfig::get_path[abi:cxx11](std::shared_ptr<AWSSyncConfig_Profile>&, RGWBucketInfo const&, rgw_obj_key const&)+0x1de) [0x7fcd9a8d3576]
 5: (RGWAWSHandleRemoteObjCBCR::operate(DoutPrefixProvider const*)+0x937) [0x7fcd9a8e9403]
 6: (RGWCoroutine::operate_wrapper(DoutPrefixProvider const*)+0xe) [0x7fcd9a527494]
 7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x1e9) [0x7fcd9a9a3bdb]
 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x1bb) [0x7fcd9a9a51f5]
 9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xa3) [0x7fcd9a9a67cd]
 10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x6be) [0x7fcd9a84567c]
 11: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x4a) [0x7fcd9aab5d7c]
 12: (RGWRadosThread::Worker::entry()+0x48d) [0x7fcd9aa75c03]
 13: (Thread::entry_wrapper()+0x43) [0x7fcd986bae99]
 14: (Thread::_entry_func(void*)+0xd) [0x7fcd986baeb5]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fcd997c5609]
 16: clone()
Actions
Copy link
 #4

Updated by hoan nv 8 months ago

More detail logs

2023-09-25T15:41:05.671+0700 7fcd00ff9700 10 http_client[GET/http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx-st
at=true&rgwx-sync-manifest&rgwx-skip-decrypt]signing_k = 2028562516b9c450df6b375b377e5b9698131f52efe00914092760b2f2cd008c
2023-09-25T15:41:05.671+0700 7fcd00ff9700 10 http_client[GET/http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx-st
at=true&rgwx-sync-manifest&rgwx-skip-decrypt]generated signature = cf70598b9aeb537f454a4fc92c19a919e18e6075b1261fc715797d3fe4c9be3c
2023-09-25T15:41:05.671+0700 7fcd00ff9700 20 http_client[GET/http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx-st
at=true&rgwx-sync-manifest&rgwx-skip-decrypt]sign_request_v4(): sigv4 header: Authorization: AWS4-HMAC-SHA256 Credential=TDR9ABGKINZGEOAU2YUH/20230925/fpt-hn/s3/aws4_request,SignedHeaders=date;host;x-amz-
content-sha256;x-amz-date,Signature=cf70598b9aeb537f454a4fc92c19a919e18e6075b1261fc715797d3fe4c9be3c
2023-09-25T15:41:05.671+0700 7fcd00ff9700 20 http_client[GET/http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx-st
at=true&rgwx-sync-manifest&rgwx-skip-decrypt]sign_request_v4(): sigv4 header: x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-09-25T15:41:05.671+0700 7fcd00ff9700 20 http_client[GET/http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx-st
at=true&rgwx-sync-manifest&rgwx-skip-decrypt]sign_request_v4(): sigv4 header: x-amz-date: 20230925T084105Z
2023-09-25T15:41:05.671+0700 7fcd00ff9700 20 sending request to http://13.0.0.42:8000/lab-test-sync-2/hoannv-test-file-1?rgwx-zonegroup=6d6fed29-dcb5-4f27-b328-01ece0914edc&rgwx-prepend-metadata=true&rgwx
-stat=true&rgwx-sync-manifest&rgwx-skip-decrypt
2023-09-25T15:41:05.671+0700 7fcd00ff9700 20 register_request mgr=0x561c463b5510 req_data->id=3, curl_handle=0x7fcc6c14a010
2023-09-25T15:41:05.671+0700 7fcd8d4e3700 20 link_request req_data=0x7fcce4008ea0 req_data->id=3, curl_handle=0x7fcc6c14a010
2023-09-25T15:41:05.671+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3273a0:18RGWStatRemoteObjCR: operate()
2023-09-25T15:41:05.671+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3273a0:18RGWStatRemoteObjCR: operate()
2023-09-25T15:41:05.671+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3273a0:18RGWStatRemoteObjCR: operate()
2023-09-25T15:41:05.671+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3273a0:18RGWStatRemoteObjCR: operate()
2023-09-25T15:41:05.675+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6e132490:23RGWAWSHandleRemoteObjCR: operate()
2023-09-25T15:41:05.675+0700 7fcc7e7fc700 20 rgw rados thread: stat of remote obj: z=db57cd2b-2213-41ad-8ba8-599edbe6e3ae b=:lab-test-sync-2[db57cd2b-2213-41ad-8ba8-599edbe6e3ae.34129.1]) k=hoannv-test-fi
le-1 size=8 mtime=2023-09-25T15:41:05.464383+0700
2023-09-25T15:41:05.675+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c387d50:op=0x7fcc6c3234a0:25RGWAWSHandleRemoteObjCBCR: operate()
2023-09-25T15:41:05.675+0700 7fcc7e7fc700  4 rgw rados thread: AWS: download begin: z=db57cd2b-2213-41ad-8ba8-599edbe6e3ae b=:lab-test-sync-2[db57cd2b-2213-41ad-8ba8-599edbe6e3ae.34129.1]) k=hoannv-test-f
ile-1 size=8 mtime=2023-09-25T15:41:05.464383+0700 etag=d8e2e94a47ddb650ccece3ab451161fb zone_short_id=1662303307 pg_ver=1
2023-09-25T15:41:05.683+0700 7fcc7e7fc700 -1 *** Caught signal (Segmentation fault) **
 in thread 7fcc7e7fc700 thread_name:data-sync

 ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7fcd997d1420]
 2: /opt/dev/ceph-17/ceph/build/lib/libradosgw.so.2(+0x10db9f9) [0x7fcd9a8c79f9]
 3: /opt/dev/ceph-17/ceph/build/lib/libradosgw.so.2(+0x10dbf9c) [0x7fcd9a8c7f9c]
 4: (AWSSyncConfig::get_path[abi:cxx11](std::shared_ptr<AWSSyncConfig_Profile>&, RGWBucketInfo const&, rgw_obj_key const&)+0x1de) [0x7fcd9a8d3576]
 5: (RGWAWSHandleRemoteObjCBCR::operate(DoutPrefixProvider const*)+0x937) [0x7fcd9a8e9403]
 6: (RGWCoroutine::operate_wrapper(DoutPrefixProvider const*)+0xe) [0x7fcd9a527494]
 7: (RGWCoroutinesStack::operate(DoutPrefixProvider const*, RGWCoroutinesEnv*)+0x1e9) [0x7fcd9a9a3bdb]
 8: (RGWCoroutinesManager::run(DoutPrefixProvider const*, std::__cxx11::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0x1bb) [0x7fcd9a9a51f5]
 9: (RGWCoroutinesManager::run(DoutPrefixProvider const*, RGWCoroutine*)+0xa3) [0x7fcd9a9a67cd]
 10: (RGWRemoteDataLog::run_sync(DoutPrefixProvider const*, int)+0x6be) [0x7fcd9a84567c]
 11: (RGWDataSyncProcessorThread::process(DoutPrefixProvider const*)+0x4a) [0x7fcd9aab5d7c]
 12: (RGWRadosThread::Worker::entry()+0x48d) [0x7fcd9aa75c03]
 13: (Thread::entry_wrapper()+0x43) [0x7fcd986bae99]
 14: (Thread::_entry_func(void*)+0xd) [0x7fcd986baeb5]
 15: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7fcd997c5609]
 16: clone()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
 -9999> 2023-09-25T15:40:45.147+0700 7fcc7d7fa700 20 link_request req_data=0x7fcc6c2ee4a0 req_data->id=25483, curl_handle=0x7fcc6c2a9c60
 -9998> 2023-09-25T15:40:45.151+0700 7fcc7e7fc700 20 rgw rados thread: cr:s=0x7fcc6c066f00:op=0x7fcc6c1054d0:18RGWDataSyncShardCR: operate()
 -9997> 2023-09-25T15:40:45.151+0700 7fcc7e7fc700 20 RGW-SYNC:data:sync:shard[11]: shard_id=11 sync_marker=
Actions
Copy link

Also available in: Atom PDF