telegeo02:~ # ceph --cluster geoceph status cluster: id: c27607d1-9852-4aa2-b953-b5e3fa3845ea health: HEALTH_WARN 12410/2950041 objects misplaced (0.421%) Degraded data redundancy: 4353/2950041 objects degraded (0.148%), 33 pgs degraded, 33 pgs undersized services: mon: 3 daemons, quorum telegeo02,geo09,sen2agriprod mgr: sen2agriprod(active), standbys: geo09, telegeo02 mds: cephfs-2/2/2 up {0=geo09=up:active,1=sen2agriprod=up:active}, 1 up:standby osd: 80 osds: 77 up, 77 in; 95 remapped pgs data: pools: 2 pools, 384 pgs objects: 134k objects, 1973 GB usage: 4733 GB used, 253 TB / 258 TB avail pgs: 4353/2950041 objects degraded (0.148%) 12410/2950041 objects misplaced (0.421%) 256 active+clean 95 active+clean+remapped 33 active+undersized+degraded io: client: 10836 kB/s wr, 0 op/s rd, 17 op/s wr (rem: see ticket #24402 for the mismatch btw the global/pool RAW) telegeo02:~ # ceph --cluster geoceph df detail GLOBAL: SIZE AVAIL RAW USED %RAW USED OBJECTS 258T 253T 4646G 1.76 131k POOLS: NAME ID QUOTA OBJECTS QUOTA BYTES USED %USED MAX AVAIL OBJECTS DIRTY READ WRITE RAW USED cephfs_data_21p3 5 N/A N/A 1936G 0.87 215T 118298 115k 48377 6965k 2213G cephfs_metadata 6 N/A N/A 413M 0 84198G 16687 16687 55252 642k 1133M telegeo02:~ # ceph --cluster geoceph osd df tree ID CLASS WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS TYPE NAME -1 268.34656 - 194T 3417G 191T 0 0 - root default -2 201.94464 - 194T 3417G 191T 1.71 0.92 - host telegeo02 55 hdd 7.27730 1.00000 7451G 132G 7319G 1.77 0.95 176 osd.55 56 hdd 7.27730 1.00000 7451G 124G 7327G 1.67 0.89 165 osd.56 57 hdd 7.27730 1.00000 7451G 122G 7329G 1.65 0.88 162 osd.57 58 hdd 7.27730 1.00000 7451G 122G 7329G 1.64 0.88 163 osd.58 59 hdd 7.27730 1.00000 7451G 114G 7337G 1.53 0.82 162 osd.59 60 hdd 7.27730 0 0 0 0 0 0 0 osd.60 61 hdd 7.27730 1.00000 7451G 125G 7326G 1.68 0.90 162 osd.61 62 hdd 7.27730 1.00000 7451G 125G 7326G 1.68 0.90 167 osd.62 63 hdd 7.27730 1.00000 7451G 121G 7330G 1.63 0.87 164 osd.63 64 hdd 7.27730 1.00000 7451G 112G 7339G 1.51 0.81 151 osd.64 65 hdd 7.27730 1.00000 7451G 118G 7333G 1.59 0.85 155 osd.65 66 hdd 7.27730 1.00000 7451G 127G 7324G 1.71 0.91 164 osd.66 67 hdd 7.27730 1.00000 7451G 120G 7331G 1.61 0.86 157 osd.67 68 hdd 7.27730 1.00000 7451G 116G 7335G 1.57 0.84 153 osd.68 69 hdd 7.27730 1.00000 7451G 123G 7328G 1.66 0.89 163 osd.69 70 hdd 7.27730 1.00000 7451G 119G 7332G 1.60 0.86 158 osd.70 71 hdd 7.27730 1.00000 7451G 111G 7340G 1.49 0.80 142 osd.71 72 hdd 7.27730 1.00000 7451G 131G 7320G 1.77 0.95 165 osd.72 73 hdd 7.27730 1.00000 7451G 127G 7324G 1.71 0.91 166 osd.73 74 hdd 7.27730 1.00000 7451G 125G 7326G 1.69 0.90 162 osd.74 75 hdd 7.27730 1.00000 7451G 126G 7325G 1.70 0.91 164 osd.75 76 hdd 7.27730 1.00000 7451G 120G 7331G 1.61 0.86 157 osd.76 77 hdd 7.27730 1.00000 7451G 123G 7328G 1.66 0.89 159 osd.77 78 hdd 7.27730 1.00000 7451G 123G 7328G 1.66 0.89 158 osd.78 79 hdd 7.27730 1.00000 7451G 123G 7328G 1.66 0.89 158 osd.79 43 oldhdd 1.81929 1.00000 1862G 47423M 1816G 2.49 1.33 60 osd.43 44 oldhdd 1.81929 1.00000 1862G 44705M 1819G 2.34 1.25 57 osd.44 46 oldhdd 1.81929 1.00000 1862G 61883M 1802G 3.24 1.74 78 osd.46 47 oldhdd 1.81929 1.00000 1862G 41171M 1822G 2.16 1.16 53 osd.47 48 oldhdd 1.81929 1.00000 1862G 44379M 1819G 2.33 1.25 57 osd.48 49 oldhdd 1.81929 1.00000 1862G 47812M 1816G 2.51 1.34 58 osd.49 50 oldhdd 1.81929 1.00000 1862G 35170M 1828G 1.84 0.99 43 osd.50 51 oldhdd 1.81929 1.00000 1862G 41311M 1822G 2.17 1.16 54 osd.51 52 oldhdd 1.81929 1.00000 1862G 47636M 1816G 2.50 1.34 62 osd.52 53 oldhdd 1.81929 1.00000 1862G 36816M 1826G 1.93 1.03 44 osd.53 54 oldhdd 1.81929 1.00000 1862G 41027M 1822G 2.15 1.15 49 osd.54 -3 66.40192 - 65201G 1524G 63677G 2.34 1.25 - host telegeo03 0 oldhdd 1.36440 1.00000 1397G 32789M 1365G 2.29 1.23 41 osd.0 1 oldhdd 1.36440 1.00000 1397G 28847M 1368G 2.02 1.08 38 osd.1 2 oldhdd 1.36440 1.00000 1397G 36884M 1361G 2.58 1.38 44 osd.2 3 oldhdd 1.36440 1.00000 1397G 36814M 1361G 2.57 1.38 43 osd.3 4 oldhdd 1.36440 0 0 0 0 0 0 0 osd.4 5 oldhdd 1.36440 1.00000 1397G 21216M 1376G 1.48 0.79 26 osd.5 6 oldhdd 1.36440 1.00000 1397G 41085M 1357G 2.87 1.54 54 osd.6 7 oldhdd 1.36440 1.00000 1397G 40496M 1357G 2.83 1.52 51 osd.7 8 oldhdd 1.36440 1.00000 1397G 27107M 1370G 1.89 1.01 37 osd.8 9 oldhdd 1.36440 1.00000 1397G 34754M 1363G 2.43 1.30 46 osd.9 10 oldhdd 1.36440 1.00000 1397G 42603M 1355G 2.98 1.59 56 osd.10 11 oldhdd 1.36440 1.00000 1397G 32769M 1365G 2.29 1.23 42 osd.11 12 oldhdd 1.36440 1.00000 1397G 28907M 1368G 2.02 1.08 38 osd.12 13 oldhdd 1.36440 1.00000 1397G 35245M 1362G 2.46 1.32 46 osd.13 14 oldhdd 1.36440 1.00000 1397G 28041M 1369G 1.96 1.05 36 osd.14 15 oldhdd 1.36440 1.00000 1397G 23276M 1374G 1.63 0.87 32 osd.15 16 oldhdd 1.36440 1.00000 1397G 39504M 1358G 2.76 1.48 50 osd.16 17 oldhdd 1.36440 1.00000 1397G 31720M 1366G 2.22 1.19 43 osd.17 18 oldhdd 1.36440 1.00000 1397G 31156M 1366G 2.18 1.17 39 osd.18 19 oldhdd 1.36440 1.00000 1397G 42570M 1355G 2.98 1.59 53 osd.19 20 oldhdd 1.36440 1.00000 1397G 34290M 1363G 2.40 1.28 44 osd.20 21 oldhdd 1.36440 0 0 0 0 0 0 0 osd.21 22 oldhdd 1.36440 1.00000 1397G 33695M 1364G 2.36 1.26 46 osd.22 23 oldhdd 1.36440 1.00000 1397G 37282M 1360G 2.61 1.39 51 osd.23 24 oldhdd 1.36440 1.00000 1397G 29578M 1368G 2.07 1.11 41 osd.24 25 oldhdd 1.36440 1.00000 1397G 36321M 1361G 2.54 1.36 46 osd.25 26 oldhdd 1.36440 1.00000 1397G 35300M 1362G 2.47 1.32 43 osd.26 27 oldhdd 1.36440 1.00000 1397G 31795M 1366G 2.22 1.19 45 osd.27 28 oldhdd 1.36440 1.00000 1397G 35398M 1362G 2.47 1.32 45 osd.28 29 oldhdd 1.36440 1.00000 1397G 37415M 1360G 2.62 1.40 45 osd.29 30 oldhdd 1.81929 1.00000 1862G 44643M 1819G 2.34 1.25 63 osd.30 31 oldhdd 1.81929 1.00000 1862G 41820M 1822G 2.19 1.17 56 osd.31 32 oldhdd 1.81929 1.00000 1862G 48453M 1815G 2.54 1.36 64 osd.32 33 oldhdd 1.81929 1.00000 1862G 48022M 1816G 2.52 1.35 61 osd.33 34 oldhdd 1.81929 1.00000 1862G 33951M 1829G 1.78 0.95 42 osd.34 35 oldhdd 1.81929 1.00000 1862G 42846M 1821G 2.25 1.20 52 osd.35 36 oldhdd 1.81929 1.00000 1862G 42323M 1821G 2.22 1.19 55 osd.36 37 oldhdd 1.81929 1.00000 1862G 36076M 1827G 1.89 1.01 48 osd.37 38 oldhdd 1.81929 1.00000 1862G 45854M 1818G 2.40 1.29 61 osd.38 39 oldhdd 1.81929 1.00000 1862G 45556M 1818G 2.39 1.28 55 osd.39 40 oldhdd 1.81929 1.00000 1862G 45675M 1818G 2.39 1.28 58 osd.40 41 oldhdd 1.81929 1.00000 1862G 54602M 1809G 2.86 1.53 73 osd.41 42 oldhdd 1.81929 1.00000 1862G 42497M 1821G 2.23 1.19 61 osd.42 45 oldhdd 1.81929 1.00000 1862G 41895M 1822G 2.20 1.18 57 osd.45 -5 0 - 0 0 0 0 0 - host telegeo04 TOTAL 258T 4942G 253T 1.87 MIN/MAX VAR: 0.79/1.74 STDDEV: 0.50 (rem: see ticket #24403 mentionning this strange output) telegeo02:~ # ceph --cluster geoceph mds metadata [ { "name": "sen2agriprod" }, { "name": "geo09" }, { "name": "telegeo02", "addr": "10.36.2.2:6800/737495544", "arch": "x86_64", "ceph_version": "ceph version 12.2.5-407-g5e7ea8cf03 (5e7ea8cf03603e1dc8937665b599f6a8fcb0213e) luminous (stable)", "cpu": "Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz", "distro": "opensuse", "distro_description": "openSUSE Leap 42.3", "distro_version": "42.3", "hostname": "telegeo02", "kernel_description": "#1 SMP Sat Apr 7 05:22:50 UTC 2018 (f24992c)", "kernel_version": "4.4.126-48-default", "mem_swap_kb": "2104316", "mem_total_kb": "131933332", "os": "Linux" } ]