Project

General

Profile

Bug #17465

multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors

Added by Casey Bodley 6 months ago. Updated 3 months ago.

Status:
Pending Backport
Priority:
High
Assignee:
Target version:
-
Start date:
09/30/2016
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

An assert that detects deadlocks between coroutines in RGWCoroutinesManager::run() is triggered when a lot of coroutines fail at the same time with an ECANCELED error from cls. All of these failing cls operations originated from a single osd (osd.2).

2016-09-16 22:21:58.394794 28ec5700  0 meta sync: ERROR: can't store key: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272 ret=-125
2016-09-16 22:21:58.399995 2b6ca700  0 meta sync: ERROR: can't store key: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272 ret=-125
2016-09-16 22:21:58.400907 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate()
2016-09-16 22:21:58.401008 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate()
2016-09-16 22:21:58.401059 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate()
2016-09-16 22:21:58.401107 372e1700 20 cr:s=0x7690e510:op=0x78c40470:20RGWSimpleRadosLockCR: operate()
2016-09-16 22:21:58.401369 372e1700 20 cr:s=0x7690e510:op=0x7690db10:20RGWContinuousLeaseCR: operate()
2016-09-16 22:21:58.401485 372e1700 20 run: stack=0x7690e510 is io blocked
2016-09-16 22:21:58.401731 372e1700 20 cr:s=0x77890f80:op=0x7e3e63b0:19RGWMetaStoreEntryCR: operate()
2016-09-16 22:21:58.401782 372e1700 20 cr:s=0x77890f80:op=0x7e3e63b0:19RGWMetaStoreEntryCR: operate() returned r=-125
2016-09-16 22:21:58.401877 372e1700 20 cr:s=0x77890f80:op=0x77890830:24RGWMetaSyncSingleEntryCR: operate()
2016-09-16 22:21:58.401930 372e1700 20 meta sync: cr:s=0x77890f80:op=0x77890830:24RGWMetaSyncSingleEntryCR: failed to store metadata: bucket.instance:test-client.0-etmo730qle4efwc-271:r0z1.4146.272, got retcode=-125
2016-09-16 22:21:58.402115 372e1700 20 cr:s=0x77ed92d0:op=0x791a8570:19RGWMetaStoreEntryCR: operate()
2016-09-16 22:21:58.402156 372e1700 20 cr:s=0x77ed92d0:op=0x791a8570:19RGWMetaStoreEntryCR: operate() returned r=-125
2016-09-16 22:21:58.402216 372e1700 20 cr:s=0x77890f80:op=0x7a186200:19RGWMetaStoreEntryCR: operate()
2016-09-16 22:21:58.402276 372e1700 20 cr:s=0x77ed92d0:op=0x77ed8c80:24RGWMetaSyncSingleEntryCR: operate()
2016-09-16 22:21:58.402331 372e1700 20 cr:s=0x77ed92d0:op=0x77ed8c80:24RGWMetaSyncSingleEntryCR: operate() returned r=-125
2016-09-16 22:21:58.402407 372e1700 20 stack->operate() returned ret=-125
2016-09-16 22:21:58.402443 372e1700 20 run: stack=0x77ed92d0 is done
2016-09-16 22:21:58.493422 372e1700 -1 /srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/rgw/rgw_coroutine.cc: In function 'int RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*>&)' thread 372e1700 time 2016-09-16 22:21:58.403580
/srv/autobuild-ceph/gitbuilder.git/build/rpmbuild/BUILD/ceph-11.0.0/src/rgw/rgw_coroutine.cc: 590: FAILED assert(context_stacks.empty() || going_down.read())

 ceph version v11.0.0-2288-gceae10b (ceae10b143c9ef54602b51915cd3fb4475b3c3c1)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x5c76e5]
 2: (RGWCoroutinesManager::run(std::list<RGWCoroutinesStack*, std::allocator<RGWCoroutinesStack*> >&)+0xea0) [0x3729e0]
 3: (RGWCoroutinesManager::run(RGWCoroutine*)+0x70) [0x372b80]
 4: (RGWRemoteMetaLog::run_sync()+0xf2b) [0x50279b]
 5: (RGWMetaSyncProcessorThread::process()+0xd) [0x4070fd]
 6: (RGWRadosThread::Worker::entry()+0x133) [0x3a9413]
 7: (()+0x7dc5) [0x149addc5]
 8: (clone()+0x6d) [0x16185ced]

http://qa-proxy.ceph.com/teuthology/cbodley-2016-09-16_10:00:36-rgw-wip-cbodley-testing---basic-mira/419161/teuthology.log
http://qa-proxy.ceph.com/teuthology/owasserm-2016-09-26_16:09:01-rgw-wip-orit-testing---basic-mira/438706/teuthology.log


Related issues

Related to Bug #17574: multisite: many duplicate mdlog entries cause race to sync and result in ECANCELED New 10/13/2016
Copied to Backport #18286: jewel: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors Resolved
Copied to Backport #18287: kraken: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors Resolved

History

#1 Updated by Yehuda Sadeh 5 months ago

  • Assignee set to Casey Bodley

#2 Updated by Casey Bodley 5 months ago

  • Subject changed from multisite: ECANCELED errors from cls lead to coroutine assertion failure to multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors

#3 Updated by Casey Bodley 5 months ago

  • Related to Bug #17574: multisite: many duplicate mdlog entries cause race to sync and result in ECANCELED added

#4 Updated by Casey Bodley 5 months ago

I had added a commit that dumps all of the remaining coroutines when it encounters this deadlock. From the latest run of teuthology (rgw.client.1.log from http://pulpito.ceph.com/teuthology-2016-10-22_17:05:01-rgw-master---basic-smithi/489718/):

   -37> 2016-10-22 18:06:17.327596 37b45700 -1 run(): ERROR: deadlock detected, dumping remaining coroutines:
[
    {
        "stack": "0x76b59a80",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b5ac20" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.620227Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.488970Z" 
                }
            }
        ]
    },
    {
        "stack": "0x76b5b580",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a17c1f0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.621859Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.743117Z" 
                }
            }
        ]
    },
    {
        "stack": "0x76b5c300",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a17d4c0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.622234Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.222965Z" 
                }
            }
        ]
    },
    {
        "stack": "0x76b5d080",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a1814f0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.622610Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.358698Z" 
                }
            }
        ]
    },
    {
        "stack": "0x76b93230",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x76fbcaf0",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x7752e5e0",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x776d7870",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776da1b0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776db480",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776dc750",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776dda20",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776decf0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776dffc0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e1290",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e2560",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e3830",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e4b00",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e5dd0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e70a0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776e9640",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x776ea910",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x778180f0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7781cc90",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7781df60",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7781f230",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77820500",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x778217d0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77822aa0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77823d70",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77825040",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77826310",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x778275e0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x778288b0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x778292e0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77840860",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77948db0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77a61150",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77840860" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.616761Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.369138Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a61e70",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a175ff0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.617163Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.750036Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a62bf0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a176d20" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.617533Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.195632Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a63970",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a177fa0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.617908Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.350008Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a646f0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b50650",
                    "0x1e66de50",
                    "0x25f82920",
                    "0x1b158220",
                    "0x76a286a0",
                    "0x76c89a20",
                    "0x7847a8d0",
                    "0x77f72880",
                    "0x76a95190",
                    "0x77068110",
                    "0x787d1b50",
                    "0x787fa170",
                    "0x78822c70",
                    "0x775ffb80",
                    "0x262e2cc0",
                    "0x7889e370",
                    "0x788c8cd0",
                    "0x7acf7c90",
                    "0x77754370",
                    "0x7ad34850",
                    "0x7ad8b530",
                    "0x7adb42c0",
                    "0x7addc9d0",
                    "0x779eab20",
                    "0x77c60b40",
                    "0x1e562a00",
                    "0x7ae7b210",
                    "0x7aea1a40",
                    "0x7aec9c70",
                    "0x7aef2ce0",
                    "0x7af1b260",
                    "0x7af43b00",
                    "0x780bd980",
                    "0x7af90c30",
                    "0x7afbc6b0",
                    "0x7afe8d90",
                    "0x7a45bbb0",
                    "0x7b033bd0",
                    "0x7b0575f0",
                    "0x7b07cf60",
                    "0x7b0a3a70",
                    "0x7b0cbf20",
                    "0x781e07c0",
                    "0x781e2880" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.618291Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.036145Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a66e20",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e3830" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.606993Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.210518Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a67ba0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e4b00" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.607391Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.269976Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a690c0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a149ec0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.610145Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.222736Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a69de0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a14af20" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.610533Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.883745Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a6ab60",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a14c1f0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.610943Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.035938Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a6b260",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a14d4c0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.611334Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.368887Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a6c700",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a14e790" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.611724Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.774698Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a6d480",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a14fa60",
                    "0x37bbebf0",
                    "0x76b93230",
                    "0x76fbcaf0",
                    "0x7752e5e0",
                    "0x78419730" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.612070Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.190384Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a892d0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776decf0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.605369Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.429607Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a8a050",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776dffc0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.605760Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.461269Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a8a440",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x778217d0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.600402Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:01.619223Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a8b100",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77822aa0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.600739Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.382605Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77a8be80",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77823d70" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.601085Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.209662Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aa6870",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b52da0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.597760Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.517013Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aaa6e0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b50ea0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.596480Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.847781Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aaa870",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b53f50" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.598119Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.534353Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aab5f0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b52020" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.598463Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:58.004535Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aee1c0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7781cc90" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.598800Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:58.601418Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77aeef40",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7781df60",
                    "0x37b5ab30",
                    "0x262e10c0",
                    "0x77ed7e90" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.599147Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.948587Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77af4660",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e1290" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.606150Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.503615Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77af53e0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e2560" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.606534Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:58.317154Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77af6af0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7781f230" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.599591Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.735338Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77af7810",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77820500" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.599944Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.331426Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77af9a60",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a150d30" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.612468Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.358016Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77afa2e0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a155870" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.614041Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.507784Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77afb000",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a156b40" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.614425Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.310243Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77afbd80",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a157e10" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.614811Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.257978Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77afdca0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x778275e0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.602103Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:01.306646Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77afea20",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x778288b0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.602449Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.461500Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b02fb0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e5dd0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.607776Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.033322Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b03d30",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e70a0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.608183Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.750169Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b069e0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a1590e0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.615217Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.339378Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b07760",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a15a3b0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.615597Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.433452Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b084e0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a15b680" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.615986Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.270410Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b09260",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a15c950" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.616376Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.229532Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b0b470",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776e9640" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.608972Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.297641Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b0c190",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776ea910" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.609356Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.777364Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b0cf10",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a149190" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.609746Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.436384Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b10060",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a152000" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.612853Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.074181Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b10de0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a1532d0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.613248Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:27:00.195243Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b11b60",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x7a1545a0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.613635Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.743327Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b1a330",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77825040" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.601421Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.350064Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b1b0b0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77826310" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.601758Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:58.343742Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b34cf0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x778292e0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.602875Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.233472Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b35a70",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776da1b0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.603288Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.729482Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b39d00",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x778180f0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.594834Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.970811Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b3fca0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77948db0" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.592851Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.872682Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b40a20",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x77b48a80" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.594483Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.961428Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b41810",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776db480" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.603787Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.372860Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b42590",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776dc750" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.604302Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.752181Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b42c90",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776dda20" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.604979Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:59.428893Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b44da0",
        "run_count": 0,
        "ops": [
            {
                "type": "13RGWMetaSyncCR",
                "spawned": [
                    "0x77b4a6d0",
                    "0x77b3fca0",
                    "0x77b40a20",
                    "0x77b39d00",
                    "0x77aaa6e0",
                    "0x77aa6870",
                    "0x77aaa870",
                    "0x77aab5f0",
                    "0x77aee1c0",
                    "0x77aeef40",
                    "0x77af6af0",
                    "0x77af7810",
                    "0x77a8a440",
                    "0x77a8b100",
                    "0x77a8be80",
                    "0x77b1a330",
                    "0x77b1b0b0",
                    "0x77afdca0",
                    "0x77afea20",
                    "0x77b34cf0",
                    "0x77b35a70",
                    "0x77b41810",
                    "0x77b42590",
                    "0x77b42c90",
                    "0x77a892d0",
                    "0x77a8a050",
                    "0x77af4660",
                    "0x77af53e0",
                    "0x77a66e20",
                    "0x77a67ba0",
                    "0x77b02fb0",
                    "0x77b03d30",
                    "0x77b0b470",
                    "0x77b0c190",
                    "0x77b0cf10",
                    "0x77a690c0",
                    "0x77a69de0",
                    "0x77a6ab60",
                    "0x77a6b260",
                    "0x77a6c700",
                    "0x77a6d480",
                    "0x77af9a60",
                    "0x77b10060",
                    "0x77b10de0",
                    "0x77b11b60",
                    "0x77afa2e0",
                    "0x77afb000",
                    "0x77afbd80",
                    "0x77b069e0",
                    "0x77b07760",
                    "0x77b084e0",
                    "0x77b09260",
                    "0x77a61150",
                    "0x77a61e70",
                    "0x77a62bf0",
                    "0x77a63970",
                    "0x77a646f0",
                    "0x76b59a80",
                    "0x76b5b580",
                    "0x76b5c300",
                    "0x76b5d080" 
                ]
            }
        ]
    },
    {
        "stack": "0x77b48a80",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b4a6d0",
        "run_count": 0,
        "ops": [
            {
                "type": "25RGWMetaSyncShardControlCR" 
            },
            {
                "type": "18RGWMetaSyncShardCR",
                "spawned": [
                    "0x776d7870" 
                ],
                "history": [
                    {
                        "timestamp": "2016-10-22 17:26:56.589141Z",
                        "status": "full_sync" 
                    }
                ],
                "status": {
                    "status": "incremental_sync",
                    "timestamp": "2016-10-22 17:26:57.974161Z" 
                }
            }
        ]
    },
    {
        "stack": "0x77b50650",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b50ea0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b52020",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b52da0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b53f50",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x77b5ac20",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x781e07c0",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x78419730",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x7a149190",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a149ec0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a14af20",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a14c1f0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a14d4c0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a14e790",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a14fa60",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a150d30",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a152000",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a1532d0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a1545a0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a155870",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a156b40",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a157e10",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a1590e0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a15a3b0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a15b680",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a15c950",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a175ff0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a176d20",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a177fa0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a17c1f0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a17d4c0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a1814f0",
        "run_count": 0,
        "ops": [
            {
                "type": "20RGWContinuousLeaseCR" 
            }
        ]
    },
    {
        "stack": "0x7a45bbb0",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x7ae7b210",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    },
    {
        "stack": "0x7af1b260",
        "run_count": 0,
        "ops": [
            {
                "type": "24RGWMetaSyncSingleEntryCR" 
            },
            {
                "type": "19RGWMetaStoreEntryCR" 
            }
        ]
    }
]

#5 Updated by Casey Bodley 4 months ago

  • Status changed from New to In Progress

#6 Updated by Casey Bodley 3 months ago

see special handling for ECANCELED in RGWCoroutinesManager::run()

#7 Updated by Casey Bodley 3 months ago

  • Status changed from In Progress to Need Review

#8 Updated by Casey Bodley 3 months ago

  • Status changed from Need Review to Pending Backport
  • Backport set to jewel, kraken

#9 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #18286: jewel: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors added

#10 Updated by Nathan Cutler 3 months ago

  • Copied to Backport #18287: kraken: multisite: coroutine deadlock in RGWMetaSyncCR after ECANCELED errors added

Also available in: Atom PDF