Tools » teuthology

This is killing us. This is less than 24 hours since I ran --nuke --stale

dgalloway@teuthology:~$ for owner in $(teuthology-lock --summary | grep -i 'scheduled\|vshankar' | awk '{ print $4 }' | sort -u); do echo $owner; teuthology-nuke --stale --owner $owner --unlock --dry-run; done
scheduled_gsalomon@teuthology
2021-05-27 14:07:23,320.320 INFO:teuthology.nuke:targets:
  smithi081.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjRGF3quvtdo0B6C0wAzJqegMDA9OsGamtJ8J44FneU/cKw8t5DY5PY9RQeTZi2xrYT1pHrwbDu3g32GbmfsaiERI2r9QrdoXmVlef+uh37TL8oSxufkHaO1QmvdgSspbnHkw9Ke2mA9pL08mf9c5ZQxjo0h3TRhTz2d46dz8RsNsLfb5UDFWaQA+TJZR/4Q35b/uf9y2brFE5v0A7z4AdROC3O/ap6YHG8Nlc/CGvWX6FpBp5nz5odN4vTL8f/iKD5DtK70MkDbA0+J0vg2RzC7JdHGedT6wpCkgA0s8Ll6a1t3a+uiuLgO2GjcnWc2oVHWVw7R6qaexyWr+B+M8RoUmR0Il1CKuhXOs07J04lVRkYI00Hdzc5MT+bAce6TmEmkfXL7cM7LTpFrO+PI1WniaDIdZeaDUHxENP6S+EZ/RNXcP+rztIBUuFm4qqkyr/MmnIEgFy7/kJoAcx71ubHbETamjgcAF5AjFORRit2wQi0MHAdIolViAMDruCRxE=
  smithi082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCQ0Xnd72XvraJOaebyb63WJV9jh5kwvxOZTfgDiuYdTBIaINm4J7rZ3kxefTEkpJDW0KjxaLNVjFiouSXzkW3msPCZairy+zK8yJPM7J0QiMRLYZhnWE0Y6IYF1xWrm/d+Q13V7VcKHlE9SfpIHbefVJogddvsgJp7SLuCbXT0S95WsuY2VymItt/cK/eakipFxya6hctzI8Lz2f5+dVL94hYNTvADyVJ+zsfUoDdFvRw8YF2P6SRTvQc/V2V4Qmvfo/NVxJnB2I+xz6Oxv/07/CYVVUdB7WbGNZKrdGc10qp7NKeJRE+xuLL2YLFBeJGStOZtdU+yD1ZRjWImlr/nxg0oFMZi7I4RZEvlBcGkCsGWVd5gS7VOnku4e0J4/0mt6lJ5i+cwU6pUnvClfVA0w3jJsKr3gqoErvkQOMO6G+4nu/OOmsn6RvblsWm39GeYA2SrO8auuIeeqDKf56vLb+mVPFuYvOcCpF7V1wfD2jgyPSjz7YSOymnKFkDa0=
2021-05-27 14:07:23,320.320 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_ideepika@teuthology
2021-05-27 14:07:24,268.268 INFO:teuthology.nuke:targets:
  smithi097.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3nt3vS/XzimPwf86fznbN6VFhYe2P9xaPKmHDXJXV4+/xvOoboGWV6owh3qDaNvdEV9wziVibNjlF0iMNBF0kLJ8OhVuwun2EfDFmDFnRnfupHuiMAlh8yOQl4/NOv5m3TLHNsiGELH8vavsfeFUEHuEV15arjwHLIkzBAaYFWjQNB6/MhnVrPvhEJJNVb5VbqHlvXupv/d0rVylU7azFAuR+49fXqbXsj2y8Cmomtm3NwLY89uglHDkkdSNIACLswKrRtP8lmdJPDiuL+f8WvOYEPb6aw77Qss6wIOeU502uzyzYfic0IqbOV9Zmro1t1AB9cEoF6ymz4A6Nvd8CrYs46DNm8QfoXVpWZ2HwlS0P0HOIlLnEmKNa4q2nScY38c6WcLiodYs1C4yGFKuqzgMoqyT2zE38JKiTVz22o4swYxBrACOBhVXHLRXO8PxVRqHq33kiT9whIWUWBbV3HEOker4XR5pAlKzoxEhEGaTp/YjlKJr7Y4HFfc7HxH8=
2021-05-27 14:07:24,268.268 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_kapandya@teuthology
2021-05-27 14:07:25,274.274 INFO:teuthology.nuke:targets:
  smithi171.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnedXVhGCWtTNU6wefo1v3DLBq2/iNjomkX/bOdEklTEZ07v101ugtijfab+Rg+zKlPAKegcvN5uy0syv/yZnl/App7/seR4j7btHxcbuQ0tKeo9Ew27yHZi5TE1QkxSha3L0zA2u4Yd9Y1/yXwAuaU64P/fEiez9SeurQuYBpIz6vsxXeIUcArd9P9QhZbV2jWCwugeUQTnftJZ6AwA9+FCHE+PpSp8fHvcb9N72rY/uhipTsqP/8ScBLFOUi6e83qcZA6mB7hfTXVxDqrtAZ8r2rFSOXV1h+PXhTOqe3fH5E/teI4UC/fOWMv5KMEsnuS+3rI8eTqR5MGs3WDNfUeHV5WtAFB1JQ66RmwU7PyfVghkkUvNFUsrJ13/cHdOe4x1Yhtqv1JRAwQBDXhVRQbb6di2USIMYim924OtOaOKmIt245/m0Ftmc9YeoyIT2b+Oe4rfEcAOzLt1Q4VLZGIRDVYnfn+CWERc9OkUjIluHfgFX2RzbpixzVvHlSJvU=
2021-05-27 14:07:25,275.275 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_kchai@teuthology
2021-05-27 14:07:26,213.213 INFO:teuthology.nuke:targets:
  {}
2021-05-27 14:07:26,213.213 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_mkogan@teuthology
2021-05-27 14:07:27,225.225 INFO:teuthology.nuke:targets:
  smithi132.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0PfYbB512eLcjCfG+hbgzZgr7VIGrY2mlOGzVi7NdZG8GTOU2A+b7vH9+KVs02WI/bPrMIt0iZz6IaOw8o0t55k5dfwi+y0kP/llqynui29VTV10xlt/T/IHRCfxtY273PWI6t4cQNRvWMqD8G4QkN2Cf03td65oQt40OS3Atydbup+vygmwcFVWsLByQsqyCG1i5A3R3mCt1FPeP/suByyGt22rcFMsK5ok9GInKQGh+kqHNQqefVTH4nD1so8bhivVItiCS39v1n1/+DaDsq8s17TNJ9zDRP2AOb6So0WSR/gWcPU+ksdVjhc1tHvFDj00oD/Ah+Un1GZkmY6a6bIB3ZQ6wSof47l58OlgFgMw8yG+UnJAoOGsxns8ZJK60DkUFnJfLXr8O1tFYvS5HkkZknxFmHdTvsjIA3B6jLivgQzSffsXb0mN7/anOotmGq16pveQcsQY1UzTS1TL4P1KzYQ5w+OjgUTJQGZo/maCf1OUcTwnZvV9pU2Ul2xs=
  smithi185.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmWXnoDKv2IjBEpD5UMWsBGKr+hrAWtLNvTm4o+tSUz+4MVgWsQHfF1AkNrnU+3D4IFOjYFdUowIPw68hJHAuvB3jVf2iIG1YVcZXqISFCAOld7fHLmFCiDl8Dx7t6LU+K+KjQqLH64IxuL90Hqdhl1k7iYrIsoSUI2nx0NdZVlQp9anYbt0bvZbuBhfAGbK0bQqDD6vk1a/PvAJFo0waAAGlhyNE+oUazeR9OT2xNEwIUytZxHxGdAp/yyoYaPzk6sWMvV7bnkG8QR3LQJCjYBxo71l18ULZSWP7yD48tKQ0vb7ADYChVIIyUR0h9jaqqKgF7K+ZUqrEsCrK5EZNYgo1sEDLWCTdseRdzJ1MCt29J6y9o+7vEXjlzOWjmBBv5pzgxhWuTxB9Ls4lAh5752wrMBAjKeaUmexliltQZs2pduhFEafDth92XFZPXL9KoK4yNy/NBcIv1/UqDevri5SdNUx5J2CWYvw4KdSgadea+eR9PyHpB8Qm6y+BUc8s=
2021-05-27 14:07:27,225.225 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_myoungwon@teuthology
2021-05-27 14:07:28,176.176 INFO:teuthology.nuke:targets:
  {}
2021-05-27 14:07:28,176.176 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_ofriedma@teuthology
2021-05-27 14:07:29,177.177 INFO:teuthology.nuke:targets:
  smithi041.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3rODtuCX4tjhzAqkKtLkqR0eVrQvOk+Yfk1U79IXy6Owtf/8jxfq9WLxjtL4KJg7JFrYrEjMihvsjcBU83jOrP6uYDDRKECX6HRhhImasgS1UrJ5f+OtafHxZ4lmvVjcGrUFShVxo3g0jOb0meXlXqNeP3vTb0togt5i1tE8jCVPLEs+iz0mdM9auC/QHVenEuwJH3WEnJWEXGdQBcLKmA7AV9ksFdyCtyxuTtiJG27IztE5lRDXL/mA0ZJBV3cAAb55VN2PiWMGs4MXmGD7Ms4MlRTmU5LOJ8ud0CbBQqZVoQCxleXflSsuEdJmmeWWkQQKIjHgG/cuwhtkpmhUpHFbH3i+Ai4RtjbruHOaH//1GNaTIRJZ0eplQxLBHytdqiWX974KFn+v381O4FxhveBZ9Nrh2hfBOOsxLEngvHXP2f1FpYOZ/OUGDMYVtdmD+msykfrkSRQRbzORtTazYnFfMerFT7MAKK26IJO788mqCHpvIoxb7lWFBcYJEFes=
2021-05-27 14:07:29,178.178 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_pdonnell@teuthology
2021-05-27 14:07:30,122.122 INFO:teuthology.nuke:targets:
  {}
2021-05-27 14:07:30,122.122 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_rfriedma@teuthology
2021-05-27 14:07:31,092.092 INFO:teuthology.nuke:targets:
  gibba030.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCp/pFRHuyyVh8rlXt9/8zP6wNPOUcDTKnQUDtyLPv1xYnb0k7t2q56auS4L+eEjdC2qwKwVj9yL5McI2DiHyJPpIpFpmqUg5ybw4GvV/+XVi6NsqF1a2fpwcPV1AzxHST3X12EmU7KwLZr7LFB8yu2F2wQ3ukij8sX4kCnMO9IcZ3MMAA0TLHRxc7tHW8jT/pclBCnwi72DJPSVTQoOQcwPhMRHP0EkErzlCxGuHcwTQQgAstGS/mV4SKAsyLk2Vjg0fKwqjdMYieoC34NkycFZbhiRfAA4/mndG/BW1dCtbETbm1K2LKc9RSxnflwn++f9tPMB+OuYXHSb10Loww0PJuO6Dg+TaR+c4Z4tVyXjLEdzG5UFH+YQ1F9HT+z9GOtlDrpmZ9id49VA04LabqtBOfq73NfdOnqHSLl+Km7IdsV3AIWgVlb+Ds6B9488i4FZrkzxpeyLXkXNQEii9EZRsWE5tnw8wtNzrcDurGTAY9XrYgwfAbT9iyX8Il/8E0=
2021-05-27 14:07:31,092.092 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_rzarzynski@teuthology
2021-05-27 14:07:32,084.084 INFO:teuthology.nuke:targets:
  smithi110.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD0OFAKPNZbPN3J+RN9xZOAPBio5YmfS0vj8BYlfB/6mLmtXZFxXNZ2recyKJFTemChLkLMfwrkTYyM4wBvJA8J0sAA+cHljtLgNiYWwd4ApwGfhaGqPuRTZjP621RxqxcVLe3POPgcKF3q6DulkoAfawQh1kdPvtEWtbsCGfLKxhBHRklpGN1TaUCeXuTAZqWe8e5uY/PWyfuqIVU06CG85FSGClU+OXqkckXaYJpKEJew/3yQ/szzv2Ivtk7coLI0s9JAtyKfTf/B/h7cV3satV6W9D5Zoh9OPVaU8GDE7kRMnZMOjdHp4zkAiXELTAVkq5X+1TRxfRJ8gVcmrqb45WFt8cj0KYI+vucdZDLoGdYQjrutyQuAZg4ahhGjtxChkjlb4nza/6Dm6ieSdahYHGqFKDtYGbwTDpHSbMhe3AERgwHLrbhdrzzO32cDzyCcpn3ghWVDnVqQMMQs3sA82ILLkfQ403awk3uqlH1U0TBTeHwQ7fjxcQPuqwYKjKU=
2021-05-27 14:07:32,084.084 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_sage@teuthology
2021-05-27 14:07:33,544.544 INFO:teuthology.nuke:targets:
  {}
2021-05-27 14:07:33,544.544 INFO:teuthology.nuke:Not actually nuking anything since --dry-run was passed
scheduled_teuthology@teuthology
2021-05-27 14:07:35,250.250 INFO:teuthology.nuke:targets:
  gibba003.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCld0hdLKMrYfqhXRPeI3Jt4gFSPeVnAzya3a7OBXInfaiYAmRR1kKzpEBsgkMTBLmf9mEYGiYwBPXlzp3jHGSMiHsujAqkJUp9Li0T+Sk0hXdy/14IzZPIaOmR+6hKSB3T9S1h5adUOh7ziaFVtHl3IWes1sg56qjNuV/fensHkgnww59QnH4WuYX5UQg3Fky7F0angBxO9Kgbs7RbEMLoqPayWE1NxvWCnSM9YS3kSOlxNbggtfU2dIfm4IxTwLgCgqfk91FYPqaFPY6mPePZloIFREvQtcMqblFUTJXvinmDmtMgYe0X/GMlnidvYT/mAIqAuyfIxI75Wjv5L6wZd1KWPhf0rE2WZ645btD0kznJ9hLyzRHpayFdTIOrEahAW3GOsNx5gyBpdRAKDAbCKOnWOyKvc34HENl3m2oMqd8u9d3TrbOUpoe5lbNzOFqVJuwBaidSNcg2xuRYYukxtT5Tl5txzTfDSp+sH90SNAP/VQgxK6p66EjgWq3pLok=
  gibba005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDfLj2jT95cDp/IB1dhkTzPsZKihcSkHH+rK/1Iy9BC7W2F85ComBGa1tL/XiBh7fPXSHHbuEBewFnhBdJF2nI990XZRi3onGMBV3obQivtjFb26KzPyW/tkENpbSrYNrQ37cMPC8zKxnu7leKXBO7h/XxetRDQLJW9nAmKZk8BKggnnj58Yt8Ts2LOzdw1pOoR0b/i5IPWgyvT31XNklo44QxWpmX4BDOaTShRDaDCzmGq85TM7X21TaCJjh2BuHIzBvM081NGDYN7vt6ah+sEv/TipIycMQwEJt6irpHKYITEdgcdnSMq7Q7yWsoqRxwxPA6U7CIcIWd9/B8x+8Xr
  gibba011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkmOIsZw76zv+3sukgiuDakHRVt0zRRUaDz1zMIXM3FFXZfse+4g6+uZNX6vbLxguO3m5swvA9sbKIFzWcnkPyfPkGCB7d0fwarpTev92y4wYtqdVZ7eqJM5SaDyyzWEppHZBLPWX2TglUgax0JReXaHcDX5SSIr18EHt4wv33Hk6d/0Mc6mdfo2lbq8ETIYA2b5Sl92s9367TUywY2dIr0VEzu+tR/41xOIsaxbsDMpiw0u23tQkSh9+nzVlQv9G1Wl3rJrdRVFklVvQ6Wg1FLEdtCKUqSj2OnW8BuDUSZBEawn3Lb+xyCX5eiY0oy6ey2Zfpd5J1sq747y1OgRXlSZRqo+kBOsBgqqhWhTonhh53ylXbxKO/a5LsjjitE6dd7JOMIOjATmYCqUu++v0K/zOGVSEsOgP5c/8X94r4QpDp3QWEGerF0crYnRHH5j6HaHO+wmuIfmsUOJPAnSm6a4M3SBVzAzNRLgsP4td5nwCVRpK7SeGn4FPOQkcQ8cU=
  gibba014.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpQAOzuQgjsSxIawFO3gd8th4WBWPaUMnWE3Uy3AwC52PaVCVFqbKIeFxehsgeZiCkYdDwjLHLA9E64yn29zWPf8RLU3uLcnyj6eMz/drZZhMjbLsQm7mf7yTx9aIuWp2svTPIrVRQ+j/5gd3qzi58lGVB/xOoKwMoXsIJf2BNGsoQe1nAltgVCRu/lGLKS/iDZxdqPsIE6WMGxfTHKxb7PIp1UG+mvY3K1/EZvBaElBraymBcrD4WRRSxQ/9p2mu32feAYR12ULHFB7LVbFawqCuwoJFG6mHVGY9TalMcrD+I9qKTxAkT9S1xaSspvAzsOBMkufFoJxoXmGy+rcJXG1fPs18MAjslB2FhAXKHiwltN3xYRYLthNmzRTPGDzglGP53sIkKg8L4UQgVb8FQLo7frINON3fGd1jue3fNNKqTxFJkOpzLwgTtgyx+8gdTsYtfay0OlmS1ZwXvC76mLMEsof6P/S7tYatbY8GT30+tnE7HkR6DvbIJALmeHCU=
  gibba031.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUNyeAm9eVpAQA7q7QDB66UhkKV/MvEFOqMgiJ6pyMC/Nr8Uqi6jGHxaOpAn8nAaYfBQjdg7ATtH5qstYo50aeb75aMl/pNrxa7Naz4BJPTFjsRQVZ8/gLNbmyQW6FKmB5+ajtLfCtXdZ9A0feKFOc53Y4R4YfNfEGud7iU9jNK9lFtUpVYk1QvDp7h+V1+bgacuXC/aXo0hRcdNKrjeBYHpuUWlIhocM2upR5nAojA+qgfV1QCU59YDA2d6Q1akhdPwvEmBJ/hnkACD54vdqgS5X1slwY5SotnYi1LN/yABJWv9BEzOQC5SHyEHw0x75bxah7upluy0uxGG5muCc7lm/zqtS7GMuxq+pJW+ZJkPGzZfsnFM3b+nHvVD1zK3xqML4r2jF5bEUZYCvclLwjfTYvDxL9SAyw7GJHGqWY5kOQW1V6wJUbqUnwQKcSuteByJlWVtr0ts9HyqmrXbLZgxvCIBGzH7PweKWPdINinS1ZPdv4sae+u9v8GQIgyvM=
  gibba034.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDXa+O0TxqAY436A1QUd0g2P1heTgulxdZCy4AQELjZ30NqXsoM/aWEcLGuDcF4n9/8R/obG51en6V405e2Qi8xGFs/WvvqDyu7PLzCQttKB2mmxQ9rUM4hatAx2cPARxt3U+JL+RS9QJQ0CHUMY9GVwNRh7nAgCko1NGppVc04MPubcUZM1bGuZIXD/V01OT76Y0Mgaq58G+1u0OSC9jbs0p0HYZsNw4Wb+i4yMceZVzDCpsJDaO6OpWXTC2b0SxYQjGDRMIhINpMY9n61hln50kS3nTPRk4itF0A+GiEycXvOL4g+o0V4GK827AlaZNr7jkWzil6ScqNKNlWh66pGdvVSE85iSokmCCp/T1R+l67A1J/GNd1QI/3lWYKVJInKIruqD6LkELDCVbLTE884EBwkA41ZSTP0Ce6GsWLFarHfjTuc755kHwkQ+5nfcxQ4ZsQe1s5grHYzMT7VVW330eFWax9JWmCqd57OgOYot+zXYE1mGzyT62gvubT4im8=
  gibba037.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiQMl3x0INlBfZHDX+lQb4u5XaL9CM8wVuFCrSaNifdHN0bfn5GjAQ0io07CGVxshQC7J0D6eRc1Ui66C3bRSfCq6A5oln9a/eVV4gMf44HAWtTvOpseZt8d41ZENV2mRIeDvwS/C7y0hiiiwL7L4XBOu6wX4SBHKXkJWqLPovU9buF2z242fPEnEd6vjhS4XTrAOIFPY/8rA9dOu9ei83WAcK/uZSOqEDv8FknYV0+bhSxWqn7IuuavYL/mYzLp9aDa2dVhuXphW1AtelL6an8StyfOzAfOyIVP3XCTf0DKueDGKZW7kF3xk+a6994ch0tAIa+1VM+KbZtooDZ/Ah
  gibba038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvyN6UzgfY73JsY4CaPF5GHIXEz5hr0SojfNz+fASwE7C3jUirx5JU3ORPTyj4VBm7IlJdaCzs0fd4XxLCCemxgKzPCdVuLGmR+1Er14iCjEKfKAqYHNc/I80m/0GH9BcuomPy5MoU0Bek/lGygktQfV+V+Mubwp9TPqhZmUrodWKtPTIl55A1Vt+bP60YJkYyrV4hIUpx26eFZUbH+uAZOGF1ET6e2LuNaPaFk157CvHsIv92mmXejqNadRRBgfxpyI5fh67YNR84VIWo+yEUacphw5S+fihgFe7q17q3gIfvXyo9wSf0FYgVRzze/ClIY2gOK+jx1jxECN44Qu4WpGfJMAYVfmlNNRA2njk2k+ewSKFzj539YaU36F6hSDq4m0ps2CG/lILKK2UKkgjdXS5T7gDKw+uglwrwqR/CLW9XbUeWnlNh5D8YsKV0esjNW0oUciZiqcMTrJKuP9oj+hQ3iPoj+6Nsff+5xukYpKFT/zfiYLInFwoAhvKRtzk=
  gibba040.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCz4t+r89sadK2l8kHV3+miBfug651GWmWIohj6cH/e3KLgNJBPA7e/9oQATGkX8SrSTJxKvrByLWCLZaBxpjkTTX6JAGql/9/kJTFELUD/43RymeqZIoY50F2TJTSNyMY85+uRyO50GuX8qDxc8t6n837xQL3oRWvJnCYhrhuO9/iYd1R80FMKXpKfzwE/p9T4Mj34l6jkdVPWQC+hxhcknXnG7l7rdv3JdhZV+QoffGw+k2bGbVlAKynmxSYIA/efE3Mf5Bo9TOn0o5l8Mg3DOxjb0PS9m6+qluLlzwgOhck2VOKAbCazN5i1aN2LpDU+9bYL8cSf5hz7yjQwHQznm1iUHMKe6ustaoYMoEN1uJdyljRz8duUr1YxBYgdTl6zEJDoBdPzhxrPBQBxFHTEYlEOCS3mutfNNcYfkJDh0iagn+ypyS7oF3XwagjcXQvAB07vvl7M8cQKHqwycJr0cp2HoQrzrn+Esz/JSPVLjuZ/e6ajYZwioHfNhcra9qM=
  gibba042.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvcvbwNjistr84ZeAYBSe/RuJCEt7WXUQu+msUPlov4Nr0n6nb8WxH2LkKio+QtXIrSeG21xbhdXUxSPownGhxZAQPHOpKt4rEbIL7cQWc6Qt1Z8eHkqKLbNBu4Sa3BGUj19Dj9ORQIhgMI8i9UGZOi3N8H8bko+VMfJXTn3cJiMQf6A+pFQi5XcIfaMQfL3AsiLDOPw+SoTiNiC7r3WseaWP5a/v6RCvMVDR/Isy9H3xWC+C+El1sJm7kk87/rnszSIiz4ozbt6qdy+j8u+ENJ13hUp9t/2HXkS10tXXyVyzfLAEsqx0rKOXoopjHpKIeOdvmmtANT0MHEsOzuES8dGowJ3YaSG4S3mzVYfn/JCL6S0LxhREX/Pj+EXOFw9tGthOl7GPkxOIvWOkLd7IfybQNvhcnv9ye+WeawsPP3PsU/aFiMhcrARc4JnvYDq6YeTjYDP9TXPkUZN5hs2RlAatjupaxB7PU4P34vxpynKzzh2oHRDCQmetqoV/vhTE=
  smithi005.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNc8eiEzpT+1C4Ux1mKGSi2ngie6RibD30k672ucLrakfRDyF6aIbDBFOQPg8pFPy6w+hyZ5Iz120aOzp9yR9qjnnGC+PFkJhQZRrs03WjzUA89gejoSO/GO6Wllx7nGXXY3SJOYz8uSCrp0/hOj/gZM6pl3bAKUo4kX4osd/I2evX0bG3/dwr2m09io3YtWWD3tdfF+iROtECxeD2kMYKS2hA/UoI+fbsqyYA0OQ3AvFFISwuZCXWBKD6XEX5XmoyhoLe3V33n8JIszu6jYHZamxvVss8j8me54rVDDZUyMSijM8wQ/8AYkHzK/GklootrUHR232D2djdu9MFxrVZ
  smithi053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCjjgHiWJpWAcsdAlz+BhdtMF3XE4GIYdArSxyCY9ZIfRzafFAg+R9BWkZoKwWDZphGstqx7GIA2k7FtSq0o6lO5+fP/rSdO6AKwk4MPm5k8zbsDcrHk5NY0obML/QZWNvrRS7+GgrAHizAdGpgh3AI1F0zz7Mlalh4nfsZjarNw5Pz+qrTNpvhNHDQc+WpV7jwtqbJJPNBtJXvvfWaOySITGD+x+p0suU83mTtacruIZmh2EPgukJEXu8+i7qkTYeRs1NVr+LcbHNDNAEZwrIGOzjd9WWf1osort3HUzCMDiAtqaw4eQUwtfJWm2LwWwsH9k+Gc0ewYGkbgWOOclUbGQkfLGoKGVtEuDzCxyH6uw3x92joTO9ZWQyFUJr9HeR2pY67jcgXdSeImG6EdMOMoyZq7hXw2LzHmdd8obV54RxAdZMGPUf4zyWvikB/HDu28QjI3S4thVYjlFbTDe0F2zqndrvbs2myr8IsOavn23xo06l0aKjvyZYKsuseaAk=
  smithi074.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgM7XkVjXwrJeQy50XRpEp2B5qYcZh7+kWl0wQXfhxQcN87DwZNneF8vsqAlx7suaTo+cGBFlUqdY/YtcCaHhfR1M0fasRmyhXLETtBdv69cZ5/Tp7mtNS9zsK02D9iB/HJYZ3kxg+NP8Bpo8JH+s6Rt14vlOmplaVFMhDaQXWdNbdS7HCKPNpcCmZZEe8k1xlAn18lVT4PlnGY0zL86za1rdNRggHHrYGFau3IByDPRABYo8ZGSDwP9UMN4PTIiYy5peK8YXKSWLfeeIb5Ip5sPsRbW5bER8pQ7yej0k1TlJAZkaIKdS+SGkSSEraXAM61lqazYk5GCFo9BA+cgwr7yG9rvksCD/0rNRWNNhPxxfYgsKLO3J2BZMDMSpTpTTKaCNg5ggCj2oXtdfeJI0LnPBYOWt8lkKJstjdXskERytL7g2KcVxWz7BzGObHKXlJo5CdihcNPvG0hL88Rf2L8dOsXb3fVP9mxx/Wu7aL7JiTkErb3yglbzupW1B1+qk=
  smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDo8qJMWHgHQDZEBAgmGwaAzlvLMR/9yGwsd0gR9OyZD2Y/MXhYva/5hHwvwhds3KTb1ylusEX7ZNcxaDQh/K8n85JYhP8M7tZ15WqiQ9M26CLb+5IWlKwckuKAllkskuwu2gpQouk2KDhHAbhpYqA6sYlpP/BEKUcLRhQvUb4E9DU0f93ekd3EgdsNa84O7P5Po2JC/QNm0l1qHEmZK0jF9rBl7Eq3tmMT4sxSH8r43Z4E2Zn6nO70/UVlCmk0W0B9AZgbughLNfnO626ZHMkfrsBydFfxq5g5uPCrpeNx+ZTu12D6fS3i+QI2JzYEEAouub0D7Hpu4EZiBM+awlRvmKe7aeb1v4gNkhu5IMm777VfMhzdresoEUtq15HhVSJSl9hQCgc600HEqOy68290Lag601qHsQa1ZbpbL6eUIrL4oAWItA/btc0YerndgvB4NwdP3r6cBnTYBSvm4FFo8Qqp8sOWupgE/dbTRKnqmaNPRRXdbT34pCknJY8OzmE=
  smithi115.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC/XmjH+08RDSc/AwmmCWRTWIz//I9qR2yZCTsTKMz6XhgxbXFX8ZNvTyemWR1C3bqZH25bwnyMBm2sdf1O3zF8L4ufKshHeO2fTZ42Wg+OA4tjdH8Zgx5F+TDXClxFLLQwgK3yITWx0Pt4ocLDiNQ2lT6gKUArLJ3oHzUMGUlga8K5kVi+/yO+8HDNqD0LEKWBiemLSgcxRbhsL5kcygjwRcibRgo+8VgEOLHVJNpCTHg5RaEVvQ7CqD0L58fhxXUhrVGINp6oLJqUYb9Ff7fR3GG09HVRpaSiJaVE67M7AIjCfdXqpDAL47NkNEX7FsVILIHrRzJM5Qr4k3VxoIbE/viKKX04rpTekgWvnzfsNlxYCE78CrqETK6ZKGPBQYOjbeZY13URvSE7AdbaecfF7JjRHFZuFwGh3625tnfmrSY/BZ6bzPf5sM1WRTfNa1BSopKTOrNlRXwlWAL91E/iAycDrZ8ixai7hAqaqQaJb1aA15QOWkjT0AqOoyjlAwE=

Rishabh asked me:

i spent time with this, went through the ticket and PR but i am not fully sure about whether we want teuthology-nuke --stale --owner $owner --unlock to not to give up unless it has retried a few times or we want it to retry only when it receives error 500 or something else.

along with the ideal behaviour, can you also please leave the recipe, here or on the ticket, for reproducing the issue? it would help me out a lot since i don't ever use these commands or review this codebase.

I personally think -nuke should retry X times if it fails for any reason.

You'd need sudo access on teuthology.front.sepia.ceph.com to reproduce. I can grant that temporarily. But what I run to nuke all stale machines is:

for owner in $(teuthology-lock --summary | grep -i 'scheduled' | awk '{ print $4 }' | sort -u); do echo $owner; teuthology-nuke --stale --owner $owner --unlock ; done

Essentially all of the paddles calls should retry - most of them do already, there are a few used by nuke --stale that aren't wrapped in safe_while() yet. That should be rectified ala https://github.com/ceph/teuthology/pull/1633

Josh Durgin wrote:

Essentially all of the paddles calls should retry - most of them do already, there are a few used by nuke --stale that aren't wrapped in safe_while() yet. That should be rectified ala https://github.com/ceph/teuthology/pull/1633

Not quite yet.

2021-06-16 17:18:29,045.045 ERROR:teuthology.lock.ops:failed to unlock gibba010.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,045.045 ERROR:teuthology.lock.ops:failed to unlock gibba008.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,045.045 ERROR:teuthology.lock.ops:failed to unlock gibba034.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,045.045 ERROR:teuthology.lock.ops:failed to unlock gibba012.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,046.046 ERROR:teuthology.lock.ops:failed to unlock gibba018.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,046.046 ERROR:teuthology.lock.ops:failed to unlock gibba001.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,047.047 ERROR:teuthology.lock.ops:failed to unlock gibba035.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,047.047 INFO:teuthology.orchestra.console:Power off for gibba039 completed
2021-06-16 17:18:29,051.051 ERROR:teuthology.lock.ops:failed to unlock gibba019.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,053.053 ERROR:teuthology.lock.ops:failed to unlock gibba040.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,054.054 ERROR:teuthology.lock.ops:failed to unlock gibba030.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,055.055 ERROR:teuthology.lock.ops:failed to unlock gibba026.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,055.055 ERROR:teuthology.lock.ops:failed to unlock gibba017.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,059.059 ERROR:teuthology.lock.ops:failed to unlock gibba032.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,059.059 ERROR:teuthology.lock.ops:failed to unlock gibba007.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,060.060 ERROR:teuthology.lock.ops:failed to unlock gibba023.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,060.060 ERROR:teuthology.lock.ops:failed to unlock gibba014.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,061.061 INFO:teuthology.lock.ops:unlocked gibba013.front.sepia.ceph.com
2021-06-16 17:18:29,062.062 ERROR:teuthology.lock.ops:failed to unlock gibba022.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,064.064 ERROR:teuthology.lock.ops:failed to unlock gibba025.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,071.071 INFO:teuthology.orchestra.console:Power off for gibba043 completed
2021-06-16 17:18:29,078.078 INFO:teuthology.orchestra.console:Power off for gibba036 completed
2021-06-16 17:18:29,080.080 ERROR:teuthology.lock.ops:failed to unlock gibba011.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,080.080 ERROR:teuthology.lock.ops:failed to unlock gibba037.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,084.084 ERROR:teuthology.lock.ops:failed to unlock gibba005.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,084.084 ERROR:teuthology.lock.ops:failed to unlock gibba044.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,090.090 INFO:teuthology.lock.ops:unlocked gibba002.front.sepia.ceph.com
2021-06-16 17:18:29,093.093 INFO:teuthology.lock.ops:unlocked gibba029.front.sepia.ceph.com
2021-06-16 17:18:29,093.093 INFO:teuthology.lock.ops:unlocked gibba041.front.sepia.ceph.com
2021-06-16 17:18:29,093.093 INFO:teuthology.lock.ops:unlocked gibba020.front.sepia.ceph.com
2021-06-16 17:18:29,171.171 INFO:teuthology.lock.ops:unlocked gibba015.front.sepia.ceph.com
2021-06-16 17:18:29,172.172 INFO:teuthology.lock.ops:unlocked gibba009.front.sepia.ceph.com
2021-06-16 17:18:29,201.201 ERROR:teuthology.lock.ops:failed to unlock gibba043.front.sepia.ceph.com. reason: 500
2021-06-16 17:18:29,203.203 INFO:teuthology.lock.ops:unlocked gibba027.front.sepia.ceph.com
2021-06-16 17:18:29,210.210 INFO:teuthology.lock.ops:unlocked gibba039.front.sepia.ceph.com
2021-06-16 17:18:30,144.144 INFO:teuthology.lock.ops:unlocked gibba038.front.sepia.ceph.com
2021-06-16 17:18:30,177.177 INFO:teuthology.lock.ops:unlocked gibba028.front.sepia.ceph.com
2021-06-16 17:18:31,525.525 INFO:teuthology.lock.ops:unlocked gibba036.front.sepia.ceph.com

dgalloway@teuthology:~$ teuthology --version
1.1.0-f359b10d

Looks like there is a problem with the retry mechanism of this particular unlock request to paddles. I have a fix out for review: https://github.com/ceph/teuthology/pull/1675

I just used https://github.com/ceph/teuthology/pull/1675 to run --nuke --stale and did not get any 500 errors!

We need this fix applied to unlocking as well apparently.

http://qa-proxy.ceph.com/teuthology/teuthology-2021-09-05_12:15:02-powercycle-octopus-distro-basic-gibba/6376171/supervisor.6376171.log

2021-09-20T17:55:03.486 ERROR:teuthology.lock.ops:failed to unlock gibba015.front.sepia.ceph.com. reason: 500

The run where David saw this was using an older version of teuthology without this fix - since he cleared out older jobs from the queue last week we shouldn't see this anymore.