https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2017-11-15T16:02:13Z
Ceph
RADOS - Support #22132: OSDs stuck in "booting" state after catastrophic data loss
https://tracker.ceph.com/issues/22132?journal_id=102282
2017-11-15T16:02:13Z
Maxim Manuylov
<ul></ul><pre>
core@mm-ceph-mon-0 ~ $ ceph -s
cluster:
id: ecf1b1ee-d10f-741d-4e01-5124fb84ec4b
health: HEALTH_OK
services:
mon: 3 daemons, quorum mm-ceph-mon-2,mm-ceph-mon-0,mm-ceph-mon-1
mgr: mm-ceph-mon-1(active), standbys: mm-ceph-mon-2, mm-ceph-mon-0
osd: 5 osds: 0 up, 0 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 bytes
usage: 0 kB used, 0 kB / 0 kB avail
pgs:
core@mm-ceph-mon-0 ~ $ ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 0.30945 root default
-4 0.06189 host mm-ceph-osd-0
0 0.06189 osd.0 down 0 1.00000
-3 0.06189 host mm-ceph-osd-1
4 0.06189 osd.4 down 0 1.00000
-5 0.06189 host mm-ceph-osd-2
3 0.06189 osd.3 down 0 1.00000
-6 0.06189 host mm-ceph-osd-3
2 0.06189 osd.2 down 0 1.00000
-2 0.06189 host mm-ceph-osd-4
1 0.06189 osd.1 down 0 1.00000
core@mm-ceph-mon-0 ~ $ ceph osd dump
epoch 6
fsid ecf1b1ee-d10f-741d-4e01-5124fb84ec4b
created 2017-11-15 15:56:37.832653
modified 2017-11-15 15:56:45.402958
flags sortbitwise,recovery_deletes,purged_snapdirs
crush_version 5
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
max_osd 5
osd.0 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) - - - - exists,new fc9f64c3-5301-4981-9668-96fbb3d2b606
osd.1 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) - - - - exists,new 2252cf36-ccda-469e-9836-6dcb55891517
osd.2 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) - - - - exists,new cdc1a1c6-7016-4470-ab1a-0cce2809092e
osd.3 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) - - - - exists,new 1dbb0120-850b-4c59-bbce-c43bef2161d8
osd.4 down out weight 0 up_from 0 up_thru 0 down_at 0 last_clean_interval [0,0) - - - - exists,new 1780ee97-2f46-441b-80bb-714dc5cd2f1b
core@mm-ceph-mon-0 ~ $
</pre>
RADOS - Support #22132: OSDs stuck in "booting" state after catastrophic data loss
https://tracker.ceph.com/issues/22132?journal_id=102283
2017-11-15T16:07:20Z
Maxim Manuylov
<ul><li><strong>File</strong> <a href="/attachments/download/3089/osd.log">osd.log</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/3089/osd.log">View</a> added</li></ul><p>Attaching OSD log (one of).</p>
RADOS - Support #22132: OSDs stuck in "booting" state after catastrophic data loss
https://tracker.ceph.com/issues/22132?journal_id=103218
2017-12-05T00:23:29Z
Greg Farnum
gfarnum@redhat.com
<ul><li><strong>Tracker</strong> changed from <i>Bug</i> to <i>Support</i></li><li><strong>Project</strong> changed from <i>Ceph</i> to <i>RADOS</i></li><li><strong>Subject</strong> changed from <i>OSDs stuck in "booting" state after entire cluster redeploy</i> to <i>OSDs stuck in "booting" state after catastrophic data loss</i></li><li><strong>Category</strong> deleted (<del><i>OSD</i></del>)</li><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>This isn't impossible but I believe you've gone about it the wrong way. See <a class="external" href="http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-mon/#monitor-store-failures">http://docs.ceph.com/docs/master/rados/troubleshooting/troubleshooting-mon/#monitor-store-failures</a>, and I recommend discussing on the mailing list if you have questions. :)</p>