Project

General

Profile

Actions
Copy link

Bug #5920

closed

mon daemon crashes

Added by Dmitry Panov over 10 years ago. Updated over 10 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Joao Eduardo Luis
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello!

I'd like to report about the problem. May be it will help you to improve Ceph.

I've created Ceph cluster on a test farm to evaluate its ability to recover from the serious failures.
3 identical servers were used in my cluster. Each server is equipped by 2 SATA hard drives. Each hard drive has 3 partitions.
/dev/sda1 and /dev/sdb1 partitions are used in RAID 1 array for swap area. /dev/sda2 and /dev/sdb2 partitions are used in RAID1 array for the root filesystem.
I was going to simulate the failure of the hard drive. I pulled out hard drive /dev/sdb on node sn2. Nothing serious was happened. As soon as the cluster returned back to the working condition, I repeated the test by pulling out hard drive /dev/sda on node sn2. The root filesystem was immediately remounted in read only mode and the system became poorly responsive and unstable. After further experiments, I was unable to make the OS working if the primary hard drive (/dev/sda) is lost despite of the fact that RAID 1 was used. Ceph cluster was unable to sync with the node sn2 and I decided to reformat partitions /dev/sda3 and /dev/sdb3 on node sn2. My idea was to clean the diskspace and re-sync the node as it was a new node. Of course, I was wrong. Then I followed by this tutorial [[http://ceph.com/w/index.php?title=Replacing_a_failed_disk/OSD&redirect=no]]. I didn't follow by it exactly as I used xfs filesystem instead of brtfs. Finally, osd daemons were able to run and have been successfully synchronized with the other nodes, sn1 and sn3.
The only problem is that mon daemon crashes during the start.
Please see the attached files.

Best regards,
Dmitry

Files

Download all files
fstab (851 Bytes) fstab Dmitry Panov, 08/09/2013 05:42 AM
ceph-mon.b.log (8.25 KB) ceph-mon.b.log Dmitry Panov, 08/09/2013 05:42 AM
ceph-osd.2.log (219 KB) ceph-osd.2.log Dmitry Panov, 08/09/2013 05:42 AM
ceph-osd.3.log (128 KB) ceph-osd.3.log Dmitry Panov, 08/09/2013 05:42 AM
core (41.2 MB) core Dmitry Panov, 08/09/2013 05:43 AM
ceph.conf (1.06 KB) ceph.conf Dmitry Panov, 08/09/2013 05:43 AM
ceph.txt (3.8 KB) ceph.txt Dmitry Panov, 08/09/2013 05:43 AM
ceph.conf (1.06 KB) ceph.conf Dmitry Panov, 08/14/2013 01:27 AM
ceph.log (77.7 KB) ceph.log Dmitry Panov, 08/14/2013 01:27 AM
ceph-mon.b.log (31.9 KB) ceph-mon.b.log Dmitry Panov, 08/14/2013 01:27 AM
ceph-osd.2.log (1.82 MB) ceph-osd.2.log Dmitry Panov, 08/14/2013 01:27 AM
ceph-osd.3.log (11.9 KB) ceph-osd.3.log Dmitry Panov, 08/14/2013 01:27 AM
ceph-status.txt (302 Bytes) ceph-status.txt Dmitry Panov, 08/14/2013 01:27 AM
core.bz2 (856 KB) core.bz2 Dmitry Panov, 08/14/2013 01:27 AM
Actions
Copy link

Also available in: Atom PDF