Project

General

Profile

Actions
Copy link

Bug #38226

open

rgw: data sync: ERROR: failed to read remote data log info: ret=-2

Added by Iain Buclaw about 5 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
Casey Bodley
Target version:
Ceph - v12.2.11
% Done:

0%

Source:
Tags:
Backport:
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After updating master from 12.2.8 to 12.2.11

2019-02-07 08:45:11.351205 7fbb18e2a000  0 ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable), process radosgw, pid 29241
2019-02-07 08:45:15.169838 7fbb18e2a000  0 starting handler: civetweb

2019-02-07 08:45:15.778179 7fbadf12f700  1 ====== starting new request req=0x7fbadf129140 =====
2019-02-07 08:45:15.784605 7fbadf12f700  1 ====== req done req=0x7fbadf129140 op status=0 http_status=200 ======
2019-02-07 08:45:15.784643 7fbadf12f700  1 civetweb: 0x55b6be09c000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log?type=metadata&id=12&period=0b91d2f9-761f-4281-9aa6-9fdba79fcc2b&max-entries=100&&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 200 0 - -
2019-02-07 08:45:15.966639 7fbade727700  1 ====== starting new request req=0x7fbade721140 =====
2019-02-07 08:45:15.967582 7fbade727700  1 ====== req done req=0x7fbade721140 op status=0 http_status=404 ======
2019-02-07 08:45:15.967602 7fbade727700  1 civetweb: 0x55b6be092000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log/?type=data&id=38&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.040180 7fbadcf24700  1 ====== starting new request req=0x7fbadcf1e140 =====
2019-02-07 08:45:16.041016 7fbadcf24700  1 ====== req done req=0x7fbadcf1e140 op status=0 http_status=404 ======
2019-02-07 08:45:16.041034 7fbadcf24700  1 civetweb: 0x55b6be0a1000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log/?type=data&id=39&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.161224 7fbadc723700  1 ====== starting new request req=0x7fbadc71d140 =====
2019-02-07 08:45:16.162120 7fbadc723700  1 ====== req done req=0x7fbadc71d140 op status=0 http_status=200 ======
2019-02-07 08:45:16.162148 7fbadc723700  1 civetweb: 0x55b6be097000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log?type=metadata&id=2&period=0b91d2f9-761f-4281-9aa6-9fdba79fcc2b&max-entries=100&&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 200 0 - -
2019-02-07 08:45:16.382960 7fbadbf22700  1 ====== starting new request req=0x7fbadbf1c140 =====
2019-02-07 08:45:16.383838 7fbadbf22700  1 ====== req done req=0x7fbadbf1c140 op status=0 http_status=404 ======
2019-02-07 08:45:16.383886 7fbadbf22700  1 civetweb: 0x55b6be0a6000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=42&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.445668 7fbadb721700  1 ====== starting new request req=0x7fbadb71b140 =====
2019-02-07 08:45:16.446916 7fbadb721700  1 ====== req done req=0x7fbadb71b140 op status=0 http_status=404 ======
2019-02-07 08:45:16.446955 7fbadb721700  1 civetweb: 0x55b6be0ab000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=41&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.475568 7fbadaf20700  1 ====== starting new request req=0x7fbadaf1a140 =====
2019-02-07 08:45:16.476729 7fbadaf20700  1 ====== req done req=0x7fbadaf1a140 op status=0 http_status=404 ======
2019-02-07 08:45:16.476778 7fbadaf20700  1 civetweb: 0x55b6be0b0000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=36&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.476824 7fbada71f700  1 ====== starting new request req=0x7fbada719140 =====
2019-02-07 08:45:16.477710 7fbada71f700  1 ====== req done req=0x7fbada719140 op status=0 http_status=404 ======
2019-02-07 08:45:16.477738 7fbada71f700  1 civetweb: 0x55b6be0b5000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=54&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.535398 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535402 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535425 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535427 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535523 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535681 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.535683 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536208 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536213 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536228 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536230 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536401 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536579 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536587 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536603 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536605 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536607 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536664 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536667 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536692 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536695 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536696 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536700 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536714 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536718 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536734 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536735 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536737 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536738 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536739 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536741 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536742 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536757 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536758 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536760 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536782 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536784 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536786 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536787 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536793 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536793 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536796 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.536797 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.545738 7fbade727700  1 ====== starting new request req=0x7fbade721140 =====
2019-02-07 08:45:16.546856 7fbade727700  1 ====== req done req=0x7fbade721140 op status=0 http_status=404 ======
2019-02-07 08:45:16.546873 7fbade727700  1 civetweb: 0x55b6be092000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log/?type=data&id=4&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.582746 7fbadcf24700  1 ====== starting new request req=0x7fbadcf1e140 =====
2019-02-07 08:45:16.583488 7fbadcf24700  1 ====== req done req=0x7fbadcf1e140 op status=0 http_status=404 ======
2019-02-07 08:45:16.583502 7fbadcf24700  1 civetweb: 0x55b6be0a1000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log/?type=data&id=29&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:16.654819 7fbadf12f700  1 ====== starting new request req=0x7fbadf129140 =====
2019-02-07 08:45:16.655935 7fbadf12f700  1 ====== req done req=0x7fbadf129140 op status=0 http_status=200 ======
2019-02-07 08:45:16.655972 7fbadf12f700  1 civetweb: 0x55b6be09c000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log?type=metadata&id=46&period=0b91d2f9-761f-4281-9aa6-9fdba79fcc2b&max-entries=100&&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 200 0 - -
2019-02-07 08:45:16.745756 7fbae9b22700  0 data sync: ERROR: failed to fetch datalog info
2019-02-07 08:45:16.849424 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849428 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849430 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849432 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849447 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849449 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.849450 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851665 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851669 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851670 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851672 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851674 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851675 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851677 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851678 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851679 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851680 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851681 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851682 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851684 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851685 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851686 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851689 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851691 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851692 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851693 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851694 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851713 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851713 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851715 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851717 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851718 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851719 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851721 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851722 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851723 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851724 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851726 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851728 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851729 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851731 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851733 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851734 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851736 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851737 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851738 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851739 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851754 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851761 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851762 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851764 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:16.851764 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.106481 7fbad9f1e700  1 ====== starting new request req=0x7fbad9f18140 =====
2019-02-07 08:45:17.107381 7fbad9f1e700  1 ====== req done req=0x7fbad9f18140 op status=0 http_status=404 ======
2019-02-07 08:45:17.107415 7fbad9f1e700  1 civetweb: 0x55b6be0ba000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=86&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:17.204210 7fbade727700  1 ====== starting new request req=0x7fbade721140 =====
2019-02-07 08:45:17.205234 7fbade727700  1 ====== req done req=0x7fbade721140 op status=0 http_status=404 ======
2019-02-07 08:45:17.205269 7fbade727700  1 civetweb: 0x55b6be092000: 1.2.3.4 - - [07/Feb/2019:08:45:15 +0000] "GET /admin/log/?type=data&id=97&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:17.232342 7fbad971d700  1 ====== starting new request req=0x7fbad9717140 =====
2019-02-07 08:45:17.233701 7fbad971d700  1 ====== req done req=0x7fbad9717140 op status=0 http_status=404 ======
2019-02-07 08:45:17.233752 7fbad971d700  1 civetweb: 0x55b6be0bf000: 1.2.3.4 - - [07/Feb/2019:08:45:16 +0000] "GET /admin/log/?type=data&id=40&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 08:45:17.301422 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301429 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301437 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301441 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301443 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301500 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.301504 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.544305 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.544725 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.544728 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.544731 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.739660 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.739667 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.739671 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740741 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740749 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740752 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740754 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740758 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740763 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740773 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740781 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740786 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740788 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740790 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.740804 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740809 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740815 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740820 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740824 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740833 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740837 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740840 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740853 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740858 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740860 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740863 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740866 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740869 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740871 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740874 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740946 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740953 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.740981 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741012 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741035 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741042 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741046 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741051 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741075 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741080 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741085 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741089 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741153 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741425 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741430 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741434 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741439 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741445 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741447 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741449 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741479 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741482 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741486 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741496 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741502 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741507 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741510 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741544 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741546 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741550 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741557 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741561 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741565 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741583 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741587 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.741590 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.743061 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743065 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743068 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743069 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743071 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743073 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743074 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743076 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743113 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743115 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743117 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743119 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743120 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743122 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743123 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743125 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743127 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743128 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743130 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743132 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743133 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743135 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743136 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743139 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743140 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743141 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743143 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743145 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743145 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743149 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743150 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743151 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743153 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743154 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743160 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743162 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743163 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743165 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743166 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743168 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743169 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743171 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743172 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743173 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743175 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743176 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743178 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743179 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743181 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743183 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743184 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743186 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743187 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743188 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743189 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743191 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743192 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743194 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743195 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743197 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743199 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.743200 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.744051 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.744374 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.746013 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.752428 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.762928 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.860571 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.860581 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.867660 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.867950 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.867981 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868087 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868107 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868132 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868137 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868174 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868180 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868188 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868215 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868253 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868382 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868403 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.868426 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.899683 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.899726 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.899754 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.899987 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.900206 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.900241 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.900250 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.900263 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.900436 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.900501 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.900505 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.900507 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.902962 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.930522 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930784 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930788 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930789 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930790 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930792 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930794 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.930796 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932321 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932326 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932331 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932333 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932335 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932337 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932338 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932340 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932341 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932342 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.932380 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932385 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932443 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932446 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932450 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932452 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932457 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932461 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932464 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.932468 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.942092 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.942099 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.942102 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.942104 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950495 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.950506 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950509 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950571 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950576 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950580 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950583 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950588 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950598 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950602 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950606 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950608 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950613 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950615 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950618 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.950640 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952344 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.952353 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952357 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952360 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952362 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952373 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952378 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952383 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952385 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952388 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952390 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952393 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.952398 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.953854 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.953861 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.956192 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.959209 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.959337 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.962949 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.963983 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.965828 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.967843 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.972164 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.972655 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:17.974314 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.977280 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.977284 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.978310 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.978712 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.978762 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.983969 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.987950 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.997166 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.997190 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.997246 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.997355 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.997363 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:17.999172 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.006790 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014715 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014765 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014863 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014881 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014888 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.014894 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.015223 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.019123 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.019662 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.021852 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.024507 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.039597 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.042182 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.042207 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.042213 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.042217 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.062183 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.062198 7fbae831f700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.062218 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.062255 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.062271 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.071133 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.072433 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.072561 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.072564 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.075875 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.081501 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.081712 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.082327 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.094537 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.112974 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.112996 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.113003 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.121907 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.141827 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.141852 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.075875 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.081501 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.081712 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.082327 7fbae831f700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.094537 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.112974 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.112996 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.113003 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.121907 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.141827 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.141852 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.142171 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.143131 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.144338 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.144387 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.145894 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.145980 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.145983 7fbae6b1c700  0 data sync: ERROR: failed to read remote data log info: ret=-2
2019-02-07 08:45:18.162111 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2
2019-02-07 08:45:18.162119 7fbae6b1c700  0 meta sync: ERROR: RGWBackoffControlCR called coroutine returned -2

This carries on forever...

On the secondary, all I see is:

2019-02-07 16:39:02.250722 7ff9f2751700  1 ====== req done req=0x7ff9f274b140 op status=0 http_status=404 ======
2019-02-07 16:39:02.250766 7ff9f2751700  1 civetweb: 0x555b9b3db000: 5.6.7.8 - - [07/Feb/2019:16:28:44 +0000] "GET /admin/log/?type=data&id=100&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:02.807808 7ff9f2751700  1 ====== req done req=0x7ff9f274b140 op status=0 http_status=404 ======
2019-02-07 16:39:02.807867 7ff9f2751700  1 civetweb: 0x555b9b3db000: 5.6.7.8 - - [07/Feb/2019:16:28:44 +0000] "GET /admin/log/?type=data&id=66&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:03.043726 7ffa237b3700  1 ====== req done req=0x7ffa237ad140 op status=0 http_status=404 ======
2019-02-07 16:39:03.043786 7ffa237b3700  1 civetweb: 0x555b9b1e1000: 5.6.7.8 - - [07/Feb/2019:16:35:52 +0000] "GET /admin/log/?type=data&id=23&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:03.131543 7ff9f2751700  1 ====== req done req=0x7ff9f274b140 op status=0 http_status=404 ======
2019-02-07 16:39:03.131573 7ff9f2751700  1 civetweb: 0x555b9b3db000: 5.6.7.8 - - [07/Feb/2019:16:28:44 +0000] "GET /admin/log/?type=data&id=16&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:03.204995 7ffa207ad700  1 ====== req done req=0x7ffa207a7140 op status=0 http_status=404 ======
2019-02-07 16:39:03.205049 7ffa207ad700  1 civetweb: 0x555b9b1ff000: 5.6.7.8 - - [07/Feb/2019:16:38:56 +0000] "GET /admin/log/?type=data&id=21&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:03.454644 7ffa237b3700  1 ====== req done req=0x7ffa237ad140 op status=0 http_status=404 ======
2019-02-07 16:39:03.454693 7ffa237b3700  1 civetweb: 0x555b9b1e1000: 5.6.7.8 - - [07/Feb/2019:16:35:52 +0000] "GET /admin/log/?type=data&id=51&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
2019-02-07 16:39:03.570363 7ffaab8c3700  1 ====== req done req=0x7ffaab8bd140 op status=0 http_status=404 ======
2019-02-07 16:39:03.570411 7ffaab8c3700  1 civetweb: 0x555b9ac71000: 5.6.7.8 - - [07/Feb/2019:16:34:54 +0000] "GET /admin/log/?type=data&id=5&marker&extra-info=true&rgwx-zonegroup=2489aeaf-d08d-4f4d-b503-3b60e0077321 HTTP/1.1" 404 0 - -
Actions
Copy link
 #1

Updated by Brad Hubbard about 5 years ago

  • Project changed from Ceph to rgw
  • Category deleted (common)
Actions
Copy link
 #2

Updated by Iain Buclaw about 5 years ago

Iain Buclaw wrote:

After updating master from 12.2.8 to 12.2.11

[...]

This carries on forever...

On the secondary, all I see is:

[...]

This may or may not be related.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-November/031350.html

  1. ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l
    7511
  1. ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l
    3509
  1. ./radosgw-gc-bucket-indexes.sh secondary2.rgw.buckets.index | wc -l
    3801

I don't know how data/metadata synchronization is supposed to work if there's orphaned objects/omaps inside the index pool.

I can only assume that radosgw thinks that they are valid and attempts to synchronize them accordingly.

Actions
Copy link
 #3

Updated by Iain Buclaw about 5 years ago

Iain Buclaw wrote:

Iain Buclaw wrote:

After updating master from 12.2.8 to 12.2.11

[...]

This carries on forever...

On the secondary, all I see is:

[...]

This may or may not be related.

http://lists.ceph.com/pipermail/ceph-users-ceph.com/2018-November/031350.html

  1. ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l
    7511
  1. ./radosgw-gc-bucket-indexes.sh secondary1.rgw.buckets.index | wc -l
    3509
  1. ./radosgw-gc-bucket-indexes.sh secondary2.rgw.buckets.index | wc -l
    3801

I don't know how data/metadata synchronization is supposed to work if there's orphaned objects/omaps inside the index pool.

I can only assume that radosgw thinks that they are valid and attempts to synchronize them accordingly.

Reversing the logic to only print the referenced index/shards in the pool.

  1. ./radosgw-gc-bucket-indexes.sh master.rgw.buckets.index | wc -l
    36

This is the same on all secondaries as well.

Actions
Copy link
 #4

Updated by Casey Bodley about 5 years ago

  • Assignee set to Casey Bodley
Actions
Copy link
 #5

Updated by Iain Buclaw about 5 years ago

Upgraded to Mimic and tried out the following:

radosgw-admin reshard stale-instances list
[
"mybucket:0ef1a91a-4aee-427e-bdf8-30589abb2d3e.97248676.1"
]

Only one of the few stale indexes is correctly found on the secondary.

Actions
Copy link
 #6

Updated by qingbo han over 4 years ago

Is it a bug? Has this problem been solved in 12.2.12?

Actions
Copy link

Also available in: Atom PDF