Actions
Bug #17465
closedmultisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors
% Done:
0%
Source:
other
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
An assert that detects deadlocks between coroutines in RGWCoroutinesManager::run() is triggered when a lot of coroutines fail at the same time with an ECANCELED error from cls. All of these failing cls operations originated from a single osd (osd.2).
2016-09-16 22:21:58.394794 28ec5700 0 meta sync: ERROR: can't store key: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272 ret=-125 2016-09-16 22:21:58.399995 2b6ca700 0 meta sync: ERROR: can't store key: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272 ret=-125 2016-09-16 22:21:58.400907 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate() 2016-09-16 22:21:58.401008 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate() 2016-09-16 22:21:58.401059 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate() 2016-09-16 22:21:58.401107 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate() 2016-09-16 22:21:58.401369 372e1700 20 cr:s=0x7690e510:op=0x7690db10:20RGWContinuousLeaseCR: operate() 2016-09-16 22:21:58.401485 372e1700 20 run: stack=0x7690e510 is io blocked 2016-09-16 22:21:58.401731 372e1700 20 cr:s=0x77890f80:op=0x7e3e63b0:19RGWMetaStoreEntryCR: operate() 2016-09-16 22:21:58.401782 372e1700 20 cr:s=0x77890f80:op=0x7e3e63b0:19RGWMetaStoreEntryCR: operate() returned r=-125 2016-09-16 22:21:58.401877 372e1700 20 cr:s=0x77890f80:op=0x77890830:24RGWMetaSyncSingleEntryCR: operate() 2016-09-16 22:21:58.401930 372e1700 20 meta sync: cr:s=0x77890f80:op=0x77890830:24RGWMetaSyncSingleEntryCR: failed to store metadata: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272, got retcode=-125 2016-09-16 22:21:58.402115 372e1700 20 cr:s=0x77ed92d0:op=0x791a8570:19RGWMetaStoreEntryCR: operate() 2016-09-16 22:21:58.402156 372e1700 20 cr:s=0x77ed92d0:op=0x791a8570:19RGWMetaStoreEntryCR: operate() returned r=-125 2016-09-16 22:21:58.402216 372e1700 20 cr:s=0x77890f80:op=0x7a186200:19RGWMetaStoreEntryCR: operate() 2016-09-16 22:21:58.402276 372e1700 20 cr:s=0x77ed92d0:op=0x77ed8c80:24RGWMetaSyncSingleEntryCR: operate() 2016-09-16 22:21:58.402331 372e1700 20 cr:s=0x77ed92d0:op=0x77ed8c80:24RGWMetaSyncSingleEntryCR: operate() returned r=-125 2016-09-16 22:21:58.402407 372e1700 20 stack->operate() returned ret=-125 2016-09-16 22:21:58.402443 372e1700 20 run: stack=0x77ed92d0 is done 2016-09-16 22:21:58.493422 372e1700 -1 /srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/rgw/rgw_coroutine.cc: In function 'int RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*>&)' thread 372e1700 time 2016-09-16 22:21:58.403580 /srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/rgw/rgw_coroutine.cc: 590: FAILED assert(context_stacks.empty() || going_down.read()) ceph version v11.0.0-2288-gceae10b (ceae10b143c9ef54602b51915cd3fb4475b3c3c1) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x5c76e5] 2: (RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0xea0) [0x3729e0] 3: (RGWCoroutinesManager::run(RGWCoroutine*)+0x70) [0x372b80] 4: (RGWRemoteMetaLog::run_sync()+0xf2b) [0x50279b] 5: (RGWMetaSyncProcessorThread::process()+0xd) [0x4070fd] 6: (RGWRadosThread::Worker::entry()+0x133) [0x3a9413] 7: (()+0x7dc5) [0x149addc5] 8: (clone()+0x6d) [0x16185ced]
http://qa-proxy.ceph.com/teuthology/cbodley-2016-09-16_10:00:36-rgw-wip-cbodley-testing---basic-mira/419161/teuthology.log
http://qa-proxy.ceph.com/teuthology/owasserm-2016-09-26_16:09:01-rgw-wip-orit-testing---basic-mira/438706/teuthology.log
Updated by Casey Bodley over 7 years ago
- Subject changed from multisite: ECANCELED errors from cls lead to coroutine assertion failure to multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors
Updated by Casey Bodley over 7 years ago
- Related to Bug #17574: multisite: many duplicate mdlog entries cause race to sync and result in ECANCELED added
Updated by Casey Bodley over 7 years ago
I had added a commit that dumps all of the remaining coroutines when it encounters this deadlock. From the latest run of teuthology (rgw.client.1.log from http://pulpito.ceph.com/teuthology-2016-10-22_17:05:01-rgw-master---basic-smithi/489718/):
-37> 2016-10-22 18:06:17.327596 37b45700 -1 run(): ERROR: deadlock detected, dumping remaining coroutines:
[
{
"stack": "0x76b59a80",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b5ac20"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.620227Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.488970Z"
}
}
]
},
{
"stack": "0x76b5b580",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a17c1f0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.621859Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.743117Z"
}
}
]
},
{
"stack": "0x76b5c300",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a17d4c0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.622234Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.222965Z"
}
}
]
},
{
"stack": "0x76b5d080",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a1814f0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.622610Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.358698Z"
}
}
]
},
{
"stack": "0x76b93230",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x76fbcaf0",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x7752e5e0",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x776d7870",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776da1b0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776db480",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776dc750",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776dda20",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776decf0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776dffc0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e1290",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e2560",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e3830",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e4b00",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e5dd0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e70a0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776e9640",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x776ea910",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x778180f0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7781cc90",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7781df60",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7781f230",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77820500",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x778217d0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77822aa0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77823d70",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77825040",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77826310",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x778275e0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x778288b0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x778292e0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77840860",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77948db0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77a61150",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77840860"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.616761Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.369138Z"
}
}
]
},
{
"stack": "0x77a61e70",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a175ff0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.617163Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.750036Z"
}
}
]
},
{
"stack": "0x77a62bf0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a176d20"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.617533Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.195632Z"
}
}
]
},
{
"stack": "0x77a63970",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a177fa0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.617908Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.350008Z"
}
}
]
},
{
"stack": "0x77a646f0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b50650",
"0x1e66de50",
"0x25f82920",
"0x1b158220",
"0x76a286a0",
"0x76c89a20",
"0x7847a8d0",
"0x77f72880",
"0x76a95190",
"0x77068110",
"0x787d1b50",
"0x787fa170",
"0x78822c70",
"0x775ffb80",
"0x262e2cc0",
"0x7889e370",
"0x788c8cd0",
"0x7acf7c90",
"0x77754370",
"0x7ad34850",
"0x7ad8b530",
"0x7adb42c0",
"0x7addc9d0",
"0x779eab20",
"0x77c60b40",
"0x1e562a00",
"0x7ae7b210",
"0x7aea1a40",
"0x7aec9c70",
"0x7aef2ce0",
"0x7af1b260",
"0x7af43b00",
"0x780bd980",
"0x7af90c30",
"0x7afbc6b0",
"0x7afe8d90",
"0x7a45bbb0",
"0x7b033bd0",
"0x7b0575f0",
"0x7b07cf60",
"0x7b0a3a70",
"0x7b0cbf20",
"0x781e07c0",
"0x781e2880"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.618291Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.036145Z"
}
}
]
},
{
"stack": "0x77a66e20",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e3830"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.606993Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.210518Z"
}
}
]
},
{
"stack": "0x77a67ba0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e4b00"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.607391Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.269976Z"
}
}
]
},
{
"stack": "0x77a690c0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a149ec0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.610145Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.222736Z"
}
}
]
},
{
"stack": "0x77a69de0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a14af20"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.610533Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.883745Z"
}
}
]
},
{
"stack": "0x77a6ab60",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a14c1f0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.610943Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.035938Z"
}
}
]
},
{
"stack": "0x77a6b260",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a14d4c0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.611334Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.368887Z"
}
}
]
},
{
"stack": "0x77a6c700",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a14e790"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.611724Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.774698Z"
}
}
]
},
{
"stack": "0x77a6d480",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a14fa60",
"0x37bbebf0",
"0x76b93230",
"0x76fbcaf0",
"0x7752e5e0",
"0x78419730"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.612070Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.190384Z"
}
}
]
},
{
"stack": "0x77a892d0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776decf0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.605369Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.429607Z"
}
}
]
},
{
"stack": "0x77a8a050",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776dffc0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.605760Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.461269Z"
}
}
]
},
{
"stack": "0x77a8a440",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x778217d0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.600402Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:01.619223Z"
}
}
]
},
{
"stack": "0x77a8b100",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77822aa0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.600739Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.382605Z"
}
}
]
},
{
"stack": "0x77a8be80",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77823d70"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.601085Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.209662Z"
}
}
]
},
{
"stack": "0x77aa6870",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b52da0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.597760Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.517013Z"
}
}
]
},
{
"stack": "0x77aaa6e0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b50ea0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.596480Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.847781Z"
}
}
]
},
{
"stack": "0x77aaa870",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b53f50"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.598119Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.534353Z"
}
}
]
},
{
"stack": "0x77aab5f0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b52020"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.598463Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:58.004535Z"
}
}
]
},
{
"stack": "0x77aee1c0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7781cc90"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.598800Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:58.601418Z"
}
}
]
},
{
"stack": "0x77aeef40",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7781df60",
"0x37b5ab30",
"0x262e10c0",
"0x77ed7e90"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.599147Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.948587Z"
}
}
]
},
{
"stack": "0x77af4660",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e1290"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.606150Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.503615Z"
}
}
]
},
{
"stack": "0x77af53e0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e2560"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.606534Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:58.317154Z"
}
}
]
},
{
"stack": "0x77af6af0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7781f230"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.599591Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.735338Z"
}
}
]
},
{
"stack": "0x77af7810",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77820500"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.599944Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.331426Z"
}
}
]
},
{
"stack": "0x77af9a60",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a150d30"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.612468Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.358016Z"
}
}
]
},
{
"stack": "0x77afa2e0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a155870"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.614041Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.507784Z"
}
}
]
},
{
"stack": "0x77afb000",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a156b40"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.614425Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.310243Z"
}
}
]
},
{
"stack": "0x77afbd80",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a157e10"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.614811Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.257978Z"
}
}
]
},
{
"stack": "0x77afdca0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x778275e0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.602103Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:01.306646Z"
}
}
]
},
{
"stack": "0x77afea20",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x778288b0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.602449Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.461500Z"
}
}
]
},
{
"stack": "0x77b02fb0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e5dd0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.607776Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.033322Z"
}
}
]
},
{
"stack": "0x77b03d30",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e70a0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.608183Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.750169Z"
}
}
]
},
{
"stack": "0x77b069e0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a1590e0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.615217Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.339378Z"
}
}
]
},
{
"stack": "0x77b07760",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a15a3b0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.615597Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.433452Z"
}
}
]
},
{
"stack": "0x77b084e0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a15b680"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.615986Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.270410Z"
}
}
]
},
{
"stack": "0x77b09260",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a15c950"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.616376Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.229532Z"
}
}
]
},
{
"stack": "0x77b0b470",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776e9640"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.608972Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.297641Z"
}
}
]
},
{
"stack": "0x77b0c190",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776ea910"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.609356Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.777364Z"
}
}
]
},
{
"stack": "0x77b0cf10",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a149190"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.609746Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.436384Z"
}
}
]
},
{
"stack": "0x77b10060",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a152000"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.612853Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.074181Z"
}
}
]
},
{
"stack": "0x77b10de0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a1532d0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.613248Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:27:00.195243Z"
}
}
]
},
{
"stack": "0x77b11b60",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x7a1545a0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.613635Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.743327Z"
}
}
]
},
{
"stack": "0x77b1a330",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77825040"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.601421Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.350064Z"
}
}
]
},
{
"stack": "0x77b1b0b0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77826310"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.601758Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:58.343742Z"
}
}
]
},
{
"stack": "0x77b34cf0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x778292e0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.602875Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.233472Z"
}
}
]
},
{
"stack": "0x77b35a70",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776da1b0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.603288Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.729482Z"
}
}
]
},
{
"stack": "0x77b39d00",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x778180f0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.594834Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.970811Z"
}
}
]
},
{
"stack": "0x77b3fca0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77948db0"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.592851Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.872682Z"
}
}
]
},
{
"stack": "0x77b40a20",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x77b48a80"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.594483Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.961428Z"
}
}
]
},
{
"stack": "0x77b41810",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776db480"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.603787Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.372860Z"
}
}
]
},
{
"stack": "0x77b42590",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776dc750"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.604302Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.752181Z"
}
}
]
},
{
"stack": "0x77b42c90",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776dda20"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.604979Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:59.428893Z"
}
}
]
},
{
"stack": "0x77b44da0",
"run_count": 0,
"ops": [
{
"type": "13RGWMetaSyncCR",
"spawned": [
"0x77b4a6d0",
"0x77b3fca0",
"0x77b40a20",
"0x77b39d00",
"0x77aaa6e0",
"0x77aa6870",
"0x77aaa870",
"0x77aab5f0",
"0x77aee1c0",
"0x77aeef40",
"0x77af6af0",
"0x77af7810",
"0x77a8a440",
"0x77a8b100",
"0x77a8be80",
"0x77b1a330",
"0x77b1b0b0",
"0x77afdca0",
"0x77afea20",
"0x77b34cf0",
"0x77b35a70",
"0x77b41810",
"0x77b42590",
"0x77b42c90",
"0x77a892d0",
"0x77a8a050",
"0x77af4660",
"0x77af53e0",
"0x77a66e20",
"0x77a67ba0",
"0x77b02fb0",
"0x77b03d30",
"0x77b0b470",
"0x77b0c190",
"0x77b0cf10",
"0x77a690c0",
"0x77a69de0",
"0x77a6ab60",
"0x77a6b260",
"0x77a6c700",
"0x77a6d480",
"0x77af9a60",
"0x77b10060",
"0x77b10de0",
"0x77b11b60",
"0x77afa2e0",
"0x77afb000",
"0x77afbd80",
"0x77b069e0",
"0x77b07760",
"0x77b084e0",
"0x77b09260",
"0x77a61150",
"0x77a61e70",
"0x77a62bf0",
"0x77a63970",
"0x77a646f0",
"0x76b59a80",
"0x76b5b580",
"0x76b5c300",
"0x76b5d080"
]
}
]
},
{
"stack": "0x77b48a80",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b4a6d0",
"run_count": 0,
"ops": [
{
"type": "25RGWMetaSyncShardControlCR"
},
{
"type": "18RGWMetaSyncShardCR",
"spawned": [
"0x776d7870"
],
"history": [
{
"timestamp": "2016-10-22 17:26:56.589141Z",
"status": "full_sync"
}
],
"status": {
"status": "incremental_sync",
"timestamp": "2016-10-22 17:26:57.974161Z"
}
}
]
},
{
"stack": "0x77b50650",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b50ea0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b52020",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b52da0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b53f50",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x77b5ac20",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x781e07c0",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x78419730",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x7a149190",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a149ec0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a14af20",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a14c1f0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a14d4c0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a14e790",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a14fa60",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a150d30",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a152000",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a1532d0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a1545a0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a155870",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a156b40",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a157e10",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a1590e0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a15a3b0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a15b680",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a15c950",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a175ff0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a176d20",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a177fa0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a17c1f0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a17d4c0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a1814f0",
"run_count": 0,
"ops": [
{
"type": "20RGWContinuousLeaseCR"
}
]
},
{
"stack": "0x7a45bbb0",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x7ae7b210",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
},
{
"stack": "0x7af1b260",
"run_count": 0,
"ops": [
{
"type": "24RGWMetaSyncSingleEntryCR"
},
{
"type": "19RGWMetaStoreEntryCR"
}
]
}
]
Updated by Casey Bodley over 7 years ago
see special handling for ECANCELED in RGWCoroutinesManager::run()
Updated by Casey Bodley over 7 years ago
- Status changed from In Progress to Fix Under Review
Updated by Casey Bodley over 7 years ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to jewel, kraken
Updated by Nathan Cutler over 7 years ago
- Copied to Backport #18286: jewel: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors added
Updated by Nathan Cutler over 7 years ago
- Copied to Backport #18287: kraken: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors added
Updated by Casey Bodley almost 7 years ago
- Status changed from Pending Backport to Resolved
both backports are resolved
Actions