Project

General

Profile

Actions

Bug #10225

closed

keyvaluestore: OSDs do not start after few weeks of downtime (osd init failed / unable to read osd superblock)

Added by Dmitry Smirnov over 9 years ago. Updated about 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

On "Giant" I've created seven KV OSDs (on 4 or 5 different hosts) before cluster went down due to cascade of OSD crashes (see #9978).
All KV OSDs were stopped as it was suggested that they may be causing the problem (unfortunately it did not relieve situation as filestore-based OSD are crashing as well...).
Some weeks passed. Now I'm trying to start KV OSD (to capture yet another crash log) but none of them are starting:

2014-12-03 17:40:26.739455 7f6a23e88880  0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578), process ceph-osd, pid 28587
2014-12-03 17:40:26.741767 7f6a23e88880  5 basedir /var/lib/ceph/osd/ceph-7
2014-12-03 17:40:26.741782 7f6a23e88880 10 mount fsid is ce1d4ff1-6ea7-48a0-b07e-9c9268acf892
2014-12-03 17:40:30.682296 7f6a23e88880 20 (init)genericobjectmap: seq is 1976429
2014-12-03 17:40:30.682420 7f6a23e88880  5 umount /var/lib/ceph/osd/ceph-7
2014-12-03 17:40:30.953776 7f6a23e88880  5 test_mount basedir /var/lib/ceph/osd/ceph-7
2014-12-03 17:40:30.953921 7f6a23e88880  5 basedir /var/lib/ceph/osd/ceph-7
2014-12-03 17:40:30.953932 7f6a23e88880 10 mount fsid is ce1d4ff1-6ea7-48a0-b07e-9c9268acf892
2014-12-03 17:40:34.745314 7f6a23e88880 20 (init)genericobjectmap: seq is 1976429
2014-12-03 17:40:34.745427 7f6a23e88880 15 read meta/23c2fcde/osd_superblock/0//-1 0~0
2014-12-03 17:40:34.745518 7f6a23e88880 20 lookup_strip_header failed to get strip_header  cid meta oid 23c2fcde/osd_superblock/0//-1
2014-12-03 17:40:34.745525 7f6a23e88880 10 read meta/23c2fcde/osd_superblock/0//-1 0~0 header isn't exist: r = -2
2014-12-03 17:40:34.745529 7f6a23e88880 -1 osd.7 0 OSD::init() : unable to read osd superblock
2014-12-03 17:40:34.745531 7f6a23e88880  5 umount /var/lib/ceph/osd/ceph-7
2014-12-03 17:40:35.012015 7f6a23e88880 -1 ESC[0;31m ** ERROR: osd init failed: (22) Invalid argumentESC[0m
Actions

Also available in: Atom PDF