Project

General

Profile

Actions

Bug #42012

closed

mon osd_snap keys grow unbounded

Added by Sage Weil over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

ceph-monstore-tool /var/lib/ceph/mon/ceph-ufm03 dump-keys |grep osd_snap| cut
-c-29 |uniq -c
      2 osd_snap / purged_snap_10_000
      1 osd_snap / purged_snap_12_000
     75 osd_snap / purged_snap_13_000
      4 osd_snap / purged_snap_14_000
 778106 osd_snap / purged_snap_15_000
    861 osd_snap / purged_snap_1_0000
    323 osd_snap / purged_snap_4_0000
     88 osd_snap / purged_snap_5_0000
      2 osd_snap / purged_snap_7_0000
      2 osd_snap / purged_snap_8_0000
      2 osd_snap / removed_epoch_10_0
      1 osd_snap / removed_epoch_12_0
     75 osd_snap / removed_epoch_13_0
      4 osd_snap / removed_epoch_14_0
2316417 osd_snap / removed_epoch_15_0
    970 osd_snap / removed_epoch_1_00
    324 osd_snap / removed_epoch_4_00
     89 osd_snap / removed_epoch_5_00
      2 osd_snap / removed_epoch_7_00
      2 osd_snap / removed_epoch_8_00
      2 osd_snap / removed_snap_10_00
      1 osd_snap / removed_snap_12_00
     75 osd_snap / removed_snap_13_00
      4 osd_snap / removed_snap_14_00
2720849 osd_snap / removed_snap_15_00
   1161 osd_snap / removed_snap_1_000
    379 osd_snap / removed_snap_4_000
     89 osd_snap / removed_snap_5_000
      2 osd_snap / removed_snap_7_000
      2 osd_snap / removed_snap_8_000

osdmap pool 15 has a lot of deleted snaps,

    removed_snaps [1~2274,2276~3,227a~6,2281~3,2285~3,2289~2,228c~4,2291~1,2293~3,2297~30948,32be0~2,32be3~3,32be7~2,32bea~4,32bef~4,32bf4~4,32bf9~3,32bfd~1,32bff~1,32c01~1,32c03~4,32c08~2,32c0b~5,32c11~1,32c13~1,32c15~3,32c19~4,32c1e~3,32c23~3,32c27~3,32c2b~4,32c30~8,32c39~6,32c40~2,32c43~1,32c45~3,32c49~1,32c4b~3,32c50~4,32c55~3,32c59~3,32c5d~1,32c5f~9,32c69~3,32c6d~1,32c6f~7,32c77~2,32c7a~1,32c7c~2,32c7f~2,32c82~3,32c86~2,32c89~2,32c8d~5,32c93~7,32c9b~5,32ca1~4,32ca6~4,32cab~2,32cae~7,32cb6~5,32cbc~6a0d4,9cd91~4,9cd96~4,9cd9c~7,9cda4~3,9cda8~5,9cdae~9,9cdb8~2,9cdbc~2,9cdbf~1,9cdc1~1,9cdc3~c,9cdd0~5,9cdd7~5,9cddd~7,9cde5~4,9cdea~2,9cded~3,9cdf1~1,9cdf4~3,9cdf8~2,9cdfb~4,9ce00~4,9ce05~3,9ce09~1,9ce0b~2,9ce0e~2,9ce12~1,9ce14~3,9ce18~4,9ce1d~1,9ce1f~2,9ce22~4,9ce27~5,9ce2d~7,9ce35~6,9ce3c~2,9ce40~1,9ce43~4,9ce48~3,9ce4c~3,9ce50~6,9ce57~6,9ce5e~3,9ce62~2,9ce65~5,9ce6b~4,9ce70~5,9ce77~4,9ce7c~3,9ce80~4,9ce85~7a20f,117095~9,11709f~9,1170a9~2,1170ac~2,1170af~8,1170b8~5,1170bf~1,1170c2~4,1170c7~2,1170ca~1,1170cc~5,1170d2~8,1170db~9,1170e6~2,1170e9~3,1170ee~2,1170f1~1,1170f3~2,1170f7~4,1170fc~2,1170ff~2,117102~3,117106~2,117109~b,117115~7,11711d~3,117121~1,117123~1,117125~3,117129~1,11712b~5,117131~2,117135~3,117139~2,11713d~5,117143~5,117149~4,11714e~8,117157~4,11715c~2,11715f~4,117164~1,117166~1,117168~2,11716b~13,11717f~5,117185~2,117188~2,11718b~1,11718e~1,117190~2,117193~3,117197~4,11719c~1,11719e~8,1171a7~8,1171b0~375fd,14e7ae~3f4b2,18dc61~1,18dc63~5,18dc69~2,18dc6c~16,18dc83~3,18dc88~2,18dc8b~2,18dc8e~3,18dc92~3,18dc96~5,18dc9c~3,18dca0~2,18dca3~2,18dca6~4,18dcab~3,18dcaf~5,18dcb5~1,18dcb7~5,18dcbd~9,18dcc7~5,18dccd~1,18dccf~3,18dcd3~2,18dcd6~1,18dcd8~1,18dcda~3,18dcde~2,18dce2~3,18dce6~1,18dce8~3,18dcec~5,18dcf2~5,18dcf8~6,18dcff~3,18dd03~2,18dd06~3,18dd0a~5,18dd10~3,18dd14~2,18dd17~1,18dd19~3,18dd1e~1,18dd20~2,18dd23~1,18dd25~c,18dd32~1,18dd34~2,18dd37~2,18dd3a~5,18dd40~2,18dd43~a,18dd4e~3,18dd52~3,18dd57~5,18dd5d~1,18dd5f~2,18dd62~2,18dd65~5,18dd6b~2,18dd6e~2,18dd71~4,18dd76~4b3ff,1d919f~273ce,20056e~3,200572~3,200576~3,20057a~2,20057d~3,200581~2,200584~2,200587~2,20058a~3,20058e~5,200594~3,200598~2,20059b~1,20059d~2,2005a0~5,2005a6~1,2005a8~3,2005ac~2,2005af~1,2005b1~2,2005b4~3,2005b8~1,2005ba~1,2005bc~2,2005bf~3,2005c3~3,2005c7~2,2005ca~3,2005ce~3,2005d2~3,2005d7~5,2005dd~2,2005e0~2,2005e3~4,2005e8~5,2005ee~3,2005f2~1,2005f4~2,2005f7~4,2005fc~4,200601~2,200604~3,200608~2,20060b~2,20060e~3,200612~1,200614~6,20061b~3,20061f~3,200623~1,200625~4,20062a~4,20062f~4,200634~3,200638~1,20063a~5,200640~2,200643~1,200645~2,200648~5,20064e~5,200655~3,200659~4,20065e~3,200662~3,200666~872e0,287947~2,28794a~4,28794f~3,287953~5,287959~5,28795f~4,287964~6,28796b~3,28796f~2,287972~4,287977~3,28797b~3,28797f~5,287986~4,28798b~6,287992~4,287997~4,28799c~4,2879a1~1,2879a3~5,2879a9~1,2879ab~2,2879ae~3,2879b2~3,2879b6~4,2879bb~2,2879be~7,2879c6~3,2879ca~3,2879ce~3,2879d2~1,2879d4~3,2879d8~3,2879dc~4,2879e2~6,2879e9~2,2879ec~5,2879f2~4,2879f7~2,2879fa~4,2879ff~5,287a05~4,287a0b~2,287a0e~2,287a11~1,287a13~5,287a19~2,287a1c~2,287a1f~9,287a29~5,287a2f~2,287a32~3,287a36~2,287a39~2,287a3c~3,287a40~2,287a43~4,287a48~3,287a4c~1,287a4e~2,287a51~2,287a54~5,287a5a~5,287a60~4,287a65~4,287a6a~4,287a6f~5,287a75~4,287a7a~3,287a7e~2,287a81~2,287a84~4,287a89~8622b,30dcb5~5,30dcbb~3,30dcbf~4,30dcc4~2,30dcc7~4,30dccc~1,30dcce~3,30dcd2~1,30dcd4~6,30dcdb~2,30dcde~3,30dce2~6,30dcea~4,30dcef~2,30dcf2~4,30dcf7~3,30dcfb~2,30dcfe~4,30dd03~2,30dd06~4,30dd0c~4,30dd11~5,30dd17~1,30dd19~5,30dd1f~2,30dd22~2,30dd26~1,30dd28~2,30dd2c~3,30dd30~3,30dd35~3,30dd39~3,30dd3d~3,30dd41~3,30dd46~3,30dd4a~6,30dd51~2,30dd54~3,30dd58~2,30dd5b~5,30dd61~3,30dd66~2,30dd69~1,30dd6b~3,30dd70~2,30dd74~3,30dd78~3,30dd7c~2,30dd7f~1,30dd81~4,30dd86~4,30dd8b~3,30dd8f~2,30dd92~1,30dd94~6,30dd9b~3,30dd9f~1,30dda1~3,30dda5~3,30dda9~3,30ddad~3,30ddb1~3,30ddb5~1,30ddb7~5,30ddbd~3,30ddc1~3,30ddc6~8f15c,39cf24~7,39cf2c~2,39cf2f~1,39cf31~3,39cf35~4,39cf3a~2,39cf3d~4,39cf42~2,39cf45~4,39cf4a~4,39cf4f~1,39cf51~4,39cf56~2,39cf59~4,39cf5e~2,39cf61~4,39cf66~2,39cf69~4,39cf6e~3,39cf72~3,39cf77~4,39cf7c~5,39cf83~6,39cf8a~3,39cf8e~1,39cf90~3,39cf94~3,39cf98~3,39cf9c~6,39cfa4~5,39cfaa~1,39cfac~4,39cfb1~5,39cfb7~1,39cfb9~3,39cfbd~4,39cfc2~4,39cfc7~3,39cfcb~1,39cfcd~4,39cfd2~3,39cfd6~1,39cfd8~3,39cfdc~2,39cfdf~2,39cfe2~2,39cfe5~4,39cfeb~3,39cfef~4,39cff4~1,39cff6~3,39cffa~6,39d001~2,39d004~2,39d007~2,39d00a~2,39d00d~3,39d011~3,39d016~3,39d01a~4,39d01f~3,39d023~3,39d027~1,39d029~2,39d02c~3,39d030~3,39d034~4,39d039~3,39d03e~3,39d042~3,39d046~4,39d04b~4,39d050~3,39d054~5,39d05a~8ebae,42bc09~4,42bc0e~2,42bc11~5,42bc17~3,42bc1b~3,42bc1f~3,42bc23~3,42bc27~6,42bc2e~3,42bc32~5,42bc38~2,42bc3b~4,42bc40~5,42bc46~3,42bc4a~2,42bc4d~4,42bc52~3,42bc56~3,42bc5a~1,42bc5c~5,42bc62~2,42bc65~5,42bc6b~3,42bc6f~3,42bc73~4,42bc78~2,42bc7b~4,42bc80~5,42bc86~1,42bc88~4,42bc8d~4,42bc92~3,42bc96~5,42bc9c~3,42bca0~2,42bca3~2,42bca6~2,42bca9~3,42bcad~5,42bcb3~4,42bcb8~1,42bcba~2,42bcbd~2,42bcc0~3,42bcc4~2,42bcc7~1,42bcc9~3,42bccd~2,42bcd0~3,42bcd4~3,42bcd8~1,42bcda~3,42bcde~2,42bce1~3,42bce5~2,42bce8~3,42bcec~3,42bcf0~2,42bcf3~4,42bcf8~3,42bcfc~4,42bd01~2,42bd04~3,42bd08~1,42bd0a~5,42bd11~5,42bd17~3,42bd1b~1,42bd1d~3,42bd21~3,42bd25~1,42bd27~4,42bd2c~4,42bd31~4,42bd36~5,42bd3d~3,42bd41~3,42bd45~4,42bd4a~4,42bd4f~3,42bd53~3,42bd57~4f22,430c7a~1,430c7c~6,430c83~3,430c87~6,430c8e~3,430c92~1,430c94~4,430c99~4,430c9e~5,430ca4~1,430ca6~3,430caa~2,430cad~6,430cb4~2,430cb7~4,430cbc~3,430cc0~1,430cc2~5,430cc9~4,430cce~6,430cd5~3,430cd9~3,430cdd~2,430ce0~4,430ce5~1,430ce7~4,430cec~3,430cf0~4,430cf5~5,430cfc~5,430d02~2,430d05~5,430d0b~2,430d0e~4,430d13~1,430d15~5,430d1b~1,430d1d~5,430d23~3,430d27~1,430d29~3,430d2d~3,430d31~3,430d35~3,430d39~2,430d3c~3,430d41~3,430d45~1,430d47~5,430d4d~6,430d54~5,430d5b~3,430d5f~4,430d64~2,430d67~5,430d6d~2,430d70~1,430d72~5,430d78~2,430d7b~1,430d7d~5,430d83~2,430d86~3,430d8a~2,430d8d~4,430d92~2,430d95~2,430d98~3,430d9c~2,430d9f~5,430da5~2,430da8~2,430dac~3,430db0~3,430db4~3,430db8~3,430dbc~2,430dbf~2,430dc2~d497,43e25b~10941,44eb9d~6,44eba4~2,44eba7~2,44ebaa~1,44ebac~6,44ebb4~6,44ebbb~3,44ebbf~3,44ebc4~4,44ebc9~2,44ebcc~4,44ebd1~3,44ebd5~1,44ebd7~5,44ebdd~1,44ebdf~5,44ebe5~2,44ebe8~3,44ebec~4,44ebf1~3,44ebf5~1,44ebf7~4,44ebfc~1,44ebfe~5,44ec04~1,44ec06~4,44ec0b~2,44ec0e~4,44ec14~4,44ec19~3,44ec1d~1,44ec1f~5,44ec25~3,44ec29~5,44ec30~4,44ec35~5,44ec3b~2,44ec3e~4,44ec43~2,44ec46~4,44ec4b~4,44ec50~1,44ec52~3,44ec57~4,44ec5c~2,44ec5f~4,44ec64~3,44ec68~3,44ec6c~1,44ec6e~4,44ec73~4,44ec78~3,44ec7c~6,44ec83~1,44ec85~2,44ec88~3,44ec8c~4,44ec91~1,44ec93~4,44ec98~1,44ec9a~6,44eca1~1,44eca3~2,44eca6~5,44ecac~3,44ecb0~4,44ecb5~4,44ecba~4,44ecbf~1,44ecc1~2,44ecc4~4,44ecc9~3,44eccd~4,44ecd2~2,44ecd5~4,44ecda~2,44ecdd~4,44ece2~1,44ece4~22de3,471ac8~9,471ad2~2,471ad5~3,471ad9~4,471adf~3,471ae3~4,471ae8~2,471aeb~3,471aef~2,471af2~6,471af9~2,471afd~4,471b02~2,471b05~6,471b0c~2,471b0f~3,471b13~1,471b15~6,471b1c~1,471b1e~2,471b21~4,471b26~1,471b28~2,471b2b~4,471b30~2,471b33~1,471b35~7,471b3d~4,471b42~3,471b46~2,471b49~4,471b4e~1,471b50~2,471b53~5,471b59~4,471b5e~2,471b61~4,471b66~3,471b6a~2,471b6d~4,471b73~4,471b78~3,471b7c~2,471b7f~3,471b83~3,471b87~2,471b8a~5,471b90~5,471b96~1,471b98~3,471b9c~2,471ba0~3,471ba4~3,471ba8~3,471bac~2,471baf~6,471bb6~3,471bba~1,471bbc~2,471bbf~1,471bc1~5,471bc7~3,471bcb~3,471bcf~2,471bd2~2,471bd5~5,471bdb~2,471bde~2,471be1~6,471be8~3,471bec~3,471bf0~1,471bf2~2,471bf5~5,471bfb~5,471c01~2,471c04~1,471c06~1,471c08~3,471c0c~232e7,494ef4~5,494efa~2,494efd~4,494f02~3,494f06~6,494f0d~4,494f12~3,494f16~3,494f1a~4,494f1f~6,494f26~3,494f2a~3,494f2e~2,494f31~3,494f35~4,494f3b~5,494f41~1,494f43~4,494f48~2,494f4b~5,494f51~3,494f55~4,494f5a~2,494f5d~6,494f64~3,494f68~4,494f6d~4,494f73~5,494f79~3,494f7d~4,494f82~5,494f88~2,494f8b~4,494f90~4,494f95~6,494f9c~2,494f9f~3,494fa3~4,494fa8~4,494fad~1,494faf~5,494fb5~4,494fba~7,494fc2~2,494fc5~5,494fcb~4,494fd0~4,494fd5~6,494fdc~5,494fe2~1,494fe4~5,494fea~2,494fed~3,494ff1~2,494ff4~3,494ff8~4,494ffd~4,495002~4,495007~2,49500a~4,49500f~7,495017~2,49501a~5,495020~3,495024~2,495027~4,49502c~3,495030~3,495035~6,49503d~1,49503f~3,495043~1,495045~3,495049~2,49504c~3,495050~d,49505e~5,495064~2,495067~a,495072~3,495076~2360c,4b8683~7,4b868b~7,4b8693~2,4b8696~1,4b8698~4,4b869d~4,4b86a2~2,4b86a5~5,4b86ab~1,4b86ad~3,4b86b1~2,4b86b4~1,4b86b6~6,4b86bd~2,4b86c0~3,4b86c4~5,4b86ca~3,4b86ce~4,4b86d3~5,4b86d9~2,4b86dc~4,4b86e1~4,4b86e6~1,4b86e8~6,4b86ef~4,4b86f4~1,4b86f6~3,4b86fa~1,4b86fc~3,4b8700~5,4b8707~4,4b870c~3,4b8710~1,4b8712~2,4b8715~3,4b8719~2,4b871c~3,4b8720~1,4b8722~4,4b8727~4,4b872c~2,4b872f~1,4b8731~3,4b8735~2,4b8738~4,4b873d~2,4b8740~2,4b8743~3,4b8747~1,4b8749~2,4b874c~1,4b874e~4,4b8753~1,4b8755~3,4b8759~4,4b875f~5,4b8765~1,4b8767~3,4b876b~2,4b876e~3,4b8772~2,4b8775~3,4b8779~2,4b877c~4,4b8781~2,4b8784~4,4b8789~5,4b878f~3,4b8794~3,4b8799~4,4b879e~3,4b87a2~3,4b87a7~4,4b87ac~2,4b87af~6,4b87b6~5,4b87bc~4,4b87c1~2,4b87c4~4,4b87c9~3,4b87cd~3b1a,4bc2e8~5f,4bc348~b1,4bc3ff~6ec,4bcaec~5ba5,4c2692~2519,4c4bac~1,4c4bae~2,4c4bb2~1,4c4bb5~2,4c4bb8~2,4c4bbc~2,4c4bc0~2,4c4bc4~1,4c4bc7~2,4c4bca~2,4c4bce~2,4c4bd1~1,4c4bd4~2,4c4bd7~1,4c4bd9~1,4c4bdc~1,4c4bde~2,4c4be1~1,4c4be3~1,4c4be5~1,4c4be7~2,4c4beb~2,4c4bef~2,4c4bf3~1,4c4bf5~1,4c4bf7~2,4c4bfa~1,4c4bfd~1,4c4bff~2,4c4c03~2,4c4c06~1,4c4c08~2,4c4c0c~2,4c4c0f~1,4c4c12~1,4c4c14~2,4c4c18~1,4c4c1a~2,4c4c1d~1,4c4c1f~1,4c4c21~1,4c4c24~1,4c4c26~1,4c4c28~2,4c4c2b~1,4c4c2e~1,4c4c30~1,4c4c32~1,4c4c35~1,4c4c37~2,4c4c3a~1,4c4c3c~2,4c4c3f~1,4c4c42~1,4c4c44~1,4c4c47~1,4c4c49~1,4c4c4b~1,4c4c4d~2,4c4c51~2,4c4c55~1,4c4c57~1b7,4c4e10~2,4c4e14~1,4c4e16~1,4c4e19~2,4c4e1c~2,4c4e20~1,4c4e22~2,4c4e26~1,4c4e28~1,4c4e2a~1,4c4e2c~2,4c4e30~2,4c4e34~1,4c4e36~1,4c4e38~1,4c4e3a~2,4c4e3e~2,4c4e42~2,4c4e46~1,4c4e48~1,4c4e4a~1,4c4e4c~1,4c4e4e~2,4c4e52~1,4c4e54~2,4c4e58~1,4c4e5b~2,4c4e5f~1,4c4e61~1,4c4e63~2,4c4e66~1,4c4e69~1,4c4e6b~1,4c4e6d~2,4c4e71~2,4c4e74~1,4c4e76~2,4c4e7a~1,4c4e7c~1,4c4e7e~1,4c4e80~2,4c4e84~1,4c4e86~2,4c4e8a~2,4c4e8e~1,4c4e90~1,4c4e92~2,4c4e96~1,4c4e98~1,4c4e9a~1,4c4e9d~2,4c4ea0~1,4c4ea2~1,4c4ea4~1,4c4ea6~1,4c4ea8~2,4c4eac~1,4c4eae~2,4c4eb1~1,4c4eb4~1,4c4eb6~1,4c4eb8~22e,4c50e7~27a,4c5362~1,4c5364~1,4c5367~2,4c536a~1,4c536c~2,4c536f~1,4c5372~2,4c5375~1,4c5378~1,4c537a~1,4c537c~1,4c537e~2,4c5381~1,4c5383~1,4c5385~1,4c5387~1,4c538a~1,4c538c~2,4c5390~2,4c5394~2,4c5398~1,4c539a~1,4c539d~2,4c53a1~2,4c53a4~1,4c53a6~1,4c53a9~1,4c53ab~3,4c53af~1,4c53b2~2,4c53b5~1,4c53b8~1,4c53bb~2,4c53bf~1,4c53c1~2,4c53c4~1,4c53c7~1,4c53c9~1,4c53cb~1,4c53cd~1,4c53d0~2,4c53d4~2,4c53d7~1,4c53da~2,4c53dd~1,4c53df~1,4c53e2~2,4c53e6~2,4c53ea~1,4c53ec~1,4c53ef~1,4c53f1~1,4c53f3~2,4c53f7~2,4c53fa~1,4c53fd~2,4c5401~2,4c5404~1,4c5406~1,4c5409~1,4c540b~4ae,4c58ba~1,4c58bc~2,4c58bf~1,4c58c2~1,4c58c5~2,4c58c9~2,4c58cc~1,4c58ce~1,4c58d0~1,4c58d2~1,4c58d5~1,4c58d7~1,4c58d9~2,4c58dd~2,4c58e0~1,4c58e2~2,4c58e5~2,4c58e8~1,4c58eb~2,4c58ef~1,4c58f1~1,4c58f3~1,4c58f6~1,4c58f8~1,4c58fb~1,4c58fd~2,4c5900~1,4c5902~1,4c5905~2,4c5908~1,4c590a~1,4c590c~1,4c590e~2,4c5912~1,4c5914~1,4c5917~1,4c5919~1,4c591b~2,4c591f~2,4c5923~1,4c5925~2,4c5928~1,4c592a~1,4c592d~1,4c592f~2,4c5932~2,4c5936~1,4c5939~2,4c593d~1,4c593f~2,4c5943~1,4c5945~1,4c5947~1,4c594a~2,4c594d~1,4c5950~1,4c5952~1,4c5954~1,4c5956~2,4c5959~1,4c595b~1,4c595d~1,4c5960~1,4c5962~2,4c5965~6da9,4cc70f~2,4cc712~1,4cc714~1,4cc76b~2d64,4cf4d0~1,4cf4d2~1,4cf4d4~1,4cf530~2ed5,4d2406~1,4d2408~1,4d240a~1,4d240c~1,4d240e~1,4d2469~2fda,4d544d~1,4d544f~2,4d5453~1,4d5455~1,4d5457~1,4d54a8~30f6,4d85a0~1,4d85a2~1,4d85a4~1,4d85a7~1,4d85a9~1,4d85ab~1,4d85ae~2,4d8606~3286,4db88d~1,4db890~1,4db892~1,4db894~1,4db896~1,4db899~1,4db89c~1,4db89e~2,4db8f3~2,4db8fb~3380,4dec84~1,4dec86~1,4dec88~1,4dec8a~1,4dec8d~1,4dec8f~1,4dec91~1,4dec93~1,4dec95~1,4dec97~1,4dec99~1,4dec9b~1,4deceb~33ee,4e20da~1,4e20dc~1,4e20de~1,4e20e0~1,4e20e3~1,4e20e6~1,4e20e8~2,4e20ec~2,4e20f0~1,4e2149~3420,4e556a~1,4e556c~1,4e556e~1,4e5570~1,4e5573~1,4e5575~2,4e5578~1,4e557a~1,4e557c~1,4e557e~1,4e55d9~342f,4e8a0a~1,4e8a0c~1,4e8a0e~1,4e8a11~2,4e8a15~1,4e8a18~1,4e8a1a~1,4e8a1c~2,4e8a1f~1,4e8a78~33d6,4ebe58~1,4ebe5a~1,4ebe5c~1,4ebe5e~1,4ebe60~1,4ebe62~1,4ebe64~1,4ebe67~2,4ebe6a~1,4ebe6c~1,4ebe6e~1,4ebebf~342b,4ef2eb~1,4ef2ed~2,4ef2f0~1,4ef2f3~1,4ef2f5~2,4ef2f8~1,4ef2fa~1,4ef2fc~1,4ef2fe~1,4ef30c~2,4ef35c~3431,4f278f~1,4f2791~1,4f2793~1,4f2797~1,4f2799~1,4f279c~1,4f279f~1,4f27a1~1,4f27a4~1,4f27a6~1,4f27a9~1,4f27fd~342d,4f5c2b~2,4f5c2f~1,4f5c31~2,4f5c34~2,4f5c37~1,4f5c3a~1,4f5c3c~1,4f5c3f~1,4f5c9a~342f,4f90ca~1,4f90cc~1,4f90ce~1,4f90d0~2,4f90d4~1,4f90d7~2,4f90da~1,4f90dc~1,4f90de~1,4f9139~342d,4fc567~1,4fc569~1,4fc56b~1,4fc56d~1,4fc56f~1,4fc572~1,4fc574~1,4fc576~1,4fc578~2,4fc57b~1,4fc5d6~342b,4ffa03~1,4ffa06~2,4ffa0a~1,4ffa0c~1,4ffa0f~2,4ffa12~2,4ffa15~1,4ffa17~1,4ffa71~342c,502e9f~1,502ea1~1,502ea3~1,502ea5~1,502ea7~2,502eaa~2,502ead~2,502eb0~1,502f0d~342c,50633a~1,50633d~2,506340~1,506343~2,506346~1,506348~1,50634a~1,50634c~1,50634e~1,5063a9~342c,5097d6~1,5097d8~1,5097db~1,5097dd~2,5097e1~2,5097e4~1,5097e6~2,5097e9~1,5097f0~2,509847~3430,50cc79~1,50cc7b~2,50cc7e~2,50cc81~1,50cc83~1,50cc85~1,50cc87~1,50cc89~2,50cce7~3430,510118~2,51011c~1,51011f~1,510122~1,510124~1,510126~2,51012a~2,51012d~1,510187~3430,5135b8~2,5135bc~1,5135be~1,5135c0~1,5135c2~2,5135c5~1,5135c7~1,5135ca~2,513627~342e,516a56~2,516a5a~1,516a5c~1,516a5e~1,516a60~1,516a62~2,516a66~2,516a69~1,516ac5~342c,519ef3~1,519ef5~1,519ef7~1,519ef9~1,519efc~1,519efe~1,519f00~2,519f03~1,519f05~2,519f61~3475,51d3d7~1,51d3da~1,51d3dc~2,51d3df~1,51d3e2~2,51d3e5~1,51d3e7~1,51d3e9~2,51d447~349f,5208e7~1,5208e9~1,5208eb~1,5208ed~1,5208f0~1,5208f2~1,5208f4~1,5208f6~1,5208f8~1,5208fb~1,5208fd~1,520957~34a8,523e01~2,523e04~2,523e08~2,523e0b~1,523e0d~2,523e10~1,523e13~1,523e70~34c4,527335~1,527338~1,52733a~1,52733c~1,52733e~2,527341~1,527344~2,527348~1,52734a~1,5273a6~3514,52a8bb~1,52a8be~2,52a8c2~1,52a8c4~1,52a8c6~2,52a8c9~1,52a8cb~1,52a8cd~1,52a8d0~1,52a92c~3523,52de5c~2,52de5f~1,52de62~1,52de64~1,52de66~1,52dec2~ed3,52ed96~1,52ed99~1,52ed9b~1,52ed9d~2,52eda0~1,52eda2~1,52eda4~1,52eda6~1,52eda8~1,52edab~1,52ee07~2f6,52f0fe~1,52f101~1,52f103~2,52f107~2,52f10b~2,52f10e~1,52f110~1,52f113~1,52f16f~2f6,52f467~2,52f46b~1,52f46d~1,52f46f~1,52f471~2,52f474~1,52f476~1,52f478~1,52f47a~1,52f4d7~2f6,52f7cf~1,52f7d1~1,52f7d3~1,52f7d5~1,52f7d7~2,52f7da~1,52f7dd~1,52f7df~2,52f7e3~1,52f83f~2f6,52fb36~1,52fb39~1,52fb3b~1,52fb3d~2,52fb40~1,52fb42~1,52fb44~1,52fb46~1,52fb49~2,52fba7~2f6,52fe9f~1,52fea1~2,52fea5~1,52fea7~2,52feaa~1,52feac~1,52feaf~1,52feb1~2,52ff0f~2f6,530206~1,530209~1,53020b~1,53020d~1,53020f~2,530212~1,530214~1,530217~1,530219~2,530277~2f6,53056e~1,530570~1,530572~1,530574~1,530577~2,53057a~1,53057c~1,53057f~2,530582~1,5305df~2f6,5308d6~1,5308d8~1,5308da~2,5308dd~1,5308e0~2,5308e4~2,5308e8~1,5308ea~1,530947~2f6,530c3e~1,530c40~1,530c42~1,530c44~1,530c46~1,530c48~2,530c4c~2,530c4f~1,530c51~1,530caf~2f6,530fa6~1,530fa8~1,530fab~2,530fae~1,530fb0~1,530fb2~1,530fb4~1,530fb6~1,530fb8~1,530fbb~1,531017~68,531081~1,531083~1,531085~1,531087~1,531089~2,53108d~2,531090~1,531093~2,5310f1~68,53115a~1,53115d~1,53115f~1,531161~1,531163~2,531166~1,531169~1,53116b~2,53116e~1,5311cb~68,531234~2,531238~1,53123a~2,53123e~2,531241~1,531243~1,531245~1,531247~1,5312a5~68,53130f~1,531311~2,531315~1,531317~2,53131a~1,53131c~1,53131e~1,531320~1,531322~1,53137f~67,5313f2~1,5313f4~1,5313f6~1,5313f8~1,5313fb~1,5313fd~1,531459~59,5314b3~1,5314b5~1,5314b7~1,5314b9~1,5314bb~1,5314bd~2,5314c0~1,5314c2~1,5314c4~1,5314c6~1,531524~68,53158d~1,531590~1,531592~1,531594~2,531597~1,531599~1,53159b~1,53159d~1,5315a0~1,5315a2~1,5315fe~68,531667~2,53166a~1,53166d~1,53166f~2,531673~1,531675~2,531678~1,53167a~1,5316d8~68,531741~2,531745~1,531747~2,53174b~2,53174f~2,531753~1,531755~1,5317b2~68,53181b~1,53181e~2,531822~1,531824~2,531828~2,53182b~1,53182e~1,531830~1,53188c~67,5318ff~1,531901~1,531903~1,531905~1,531908~1,53190a~1,531966~2f5,531c67~1,531c69~1,531c6c~2,531c70~1,531c72~1,531cce~2f5,531fd0~1,531fd2~2,531fd5~1,531fd7~1,531fd9~1,532036~21b,53225e~2,532262~2,532266~1,532268~1,5322c4~67,532337~1,532339~1,53233b~1,53233d~1,53233f~1,532341~1,53239e~67,532410~1,532412~1,532414~1,532416~1,532419~1,53241b~1,532478~67,5324eb~1,5324ed~1,5324ef~1,5324f1~1,5324f4~1,5324f6~1,532552~67,5325c6~2,5325c9~1,5325cb~1,5325ce~2,53262c~67]
Actions #1

Updated by Brian Koebbe over 4 years ago

Link to the full "dump-keys | grep osd_snap"
https://wustl.box.com/s/3r7bgv32hs5hw4jmgmywbo9qvqrqsmwn

Actions #2

Updated by Sage Weil over 4 years ago

This is (mostly) fixed in master by https://github.com/ceph/ceph/pull/30518. There is still one set of per-epoch keys for purged_snaps, but it's less than before. Once the upgrade hits octopus, the old keys are removed.

But... for nautilus and mimic, there isn't a quick/easy fix for the problem.

Actions #3

Updated by Josh Durgin over 4 years ago

Sage says he wants to confirm we're not leaving behind more keys than we need in master

Actions #4

Updated by Sage Weil over 4 years ago

  • Assignee set to Sage Weil
Actions #5

Updated by Sage Weil over 4 years ago

  • Status changed from 12 to In Progress

Okay, in octopus, there are now 2 sets of keys

- purged_snap_*: map intervals of snaps that are purged. adjacent records are merged into a single key, so this is pretty space efficient.

- purged_epoch_*: map epoch -> interval_set of snaps purged that epoch. This is recorded based on time, so it will grow as the cluster ages. It is used to ensure that the purged_snaps are propagated to OSDs. We could periodically coalesce these into fewer keys (with a larger value), if we decide it is worth the effort.

On the osd, we have

- PSE_ purged_snap_epoch_*: mirror of the purged_epoch_* keys. these are written but never used, so we can remove them.
- PSN_ purged_snap_* equivalent that we use for lookups.

Actions #6

Updated by Sage Weil over 4 years ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 31866
Actions #7

Updated by Sage Weil over 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF