Bug #3676
closedosd keeps crashing at ReplicatedPG::scan_range()
0%
Description
This specific osd (osd.17) keeps crashing at the same location, as I tried to bring it back. It would start peering and recovery, but sooner or later, it would crash again, and it never finished the recovery.
Our other larger cluster, with 76 osds, crashed and we could not bring it up. Well, at the rate the recovery is running, it probably would take months to complete. We shut it down and access the data directly, since our data are immutable. And built a smaller one.
This is a small cluster with only 30 osds. The log file is attached. Here's the version information:
ceph version 0.55.1-1-g7c7469a (7c7469a19b0d563a448486adce9f326c6e5bd66d)
Running on Debian Wheezy. This is a build from Samuel Just, with a backport that fixed the constant osd crash.
Some more information:
root@s1:/tmp# ceph -s
health HEALTH_WARN 251 pgs backfill; 38 pgs backfilling; 289 pgs degraded; 1 pgs recovering; 6 pgs recovery_wait; 34 pgs stale; 34 pgs stuck stale; 296 pgs stuck unclean; recovery 76564/2415766 degraded (3.169%)
monmap e1: 3 mons at {a=10.1.3.1:6789/0,b=10.1.3.2:6789/0,c=10.1.3.3:6789/0}, election epoch 14, quorum 0,1,2 a,b,c
osdmap e540: 30 osds: 29 up, 29 in
pgmap v8905: 11452 pgs: 11122 active+clean, 98 active+degraded+wait_backfill, 6 active+recovery_wait, 34 stale+active+clean, 37 active+degraded+backfilling, 153 active+degraded+remapped+wait_backfill, 1 active+degraded+remapped+backfilling, 1 active+recovering; 2745 GB data, 7581 GB used, 46419 GB / 54001 GB avail; 76564/2415766 degraded (3.169%)
mdsmap e1: 0/0/1 up
I tried over 10 times to bring it back, but everytime it just crashed without finishing recovery. Please let me know if I can provide more information.
Files
Updated by Sage Weil over 11 years ago
- Status changed from New to Need More Info
int r = osd->store->getattr(coll, *p, OI_ATTR, bl);
assert(r >= 0);
an file is missing the user.ceph._ xattr.
you can identify which object it is by adding 'debug osd = 0/20' and reproducing the crash. you can repair the object by copying the xattr value from a replica on another node.
what fs are you using? what are the mount options?
Updated by Xiaopong Tran over 11 years ago
I'm using xfs, with no specific mount options, just the default.
I added the debug settings, and got a large log file. How do I look for the object ID in that file? Specific log format I should be scanning for?
Thanks
Updated by Sage Weil over 11 years ago
Xiaopong Tran wrote:
I'm using xfs, with no specific mount options, just the default.
I added the debug settings, and got a large log file. How do I look for the object ID in that file? Specific log format I should be scanning for?
Thanks
Sigh.. I take it back. The block of code is
void ReplicatedPG::scan_range(hobject_t begin, int min, int max, BackfillInterval *bi)
{
assert(is_locked());
dout(10) << "scan_range from " << begin << dendl;
bi->begin = begin;
bi->objects.clear(); // for good measure
vector<hobject_t> ls;
ls.reserve(max);
int r = osd->store->collection_list_partial(coll, begin, min, max,
0, &ls, &bi->end);
assert(r >= 0);
dout(10) << " got " << ls.size() << " items, next " << bi->end << dendl;
dout(20) << ls << dendl;
for (vector<hobject_t>::iterator p = ls.begin(); p != ls.end(); ++p) {
ObjectContext *obc = NULL;
if (is_primary())
obc = _lookup_object_context(*p);
if (obc) {
bi->objects[*p] = obc->obs.oi.version;
dout(20) << " " << *p << " " << obc->obs.oi.version << dendl;
} else {
bufferlist bl;
int r = osd->store->getattr(coll, *p, OI_ATTR, bl);
assert(r >= 0);
object_info_t oi(bl);
bi->objects[*p] = oi.version;
dout(20) << " " << *p << " " << oi.version << dendl;
}
}
}
and the object isn't dumped prior to the failed assertion. You can modify the code to print it, e.g.
int r = osd->store->getattr(coll, *p, OI_ATTR, bl);
if (r < 0)
dout(0) << "error getting attr on " << *p << ": " << cpp_strerror(r) << dendl;
assert(r >= 0);
and then you can see it. Unfortunately the STL iterator p isn't easy to inspect via gdb.
Updated by Sage Weil almost 11 years ago
- Status changed from Need More Info to Can't reproduce
Updated by Bernd Eckenfels over 6 years ago
This is just a FYI in my older ceph version 0.94.10-1trusty is see the same crash. The code is a bit different, it will skip on -ENOENT. I chnaged this now to log and skip on all error codes. The loggng of the object ID should be added in all cases I think (not sure about latest source, however).
I see the following logs now:
/var/log/ceph/ceph-osd.39.log:2017-08-17 19:16:47.775537 7f3389e85700 0 osd.39 pg_epoch: 28294 pg[11.1d8( v 28289'32157569 (21329'32154561,28289'32157569] lb -1/0//0 local-les=28207 n=0 ec=2440 les/c 28207/28166 28206/28206/22659) [39,65]/[65,25] r=-1 lpr=28206 pi=21552-28205/377 luod=0'0 crt=28289'32157569 lcod 28256'32157568 active+remapped] error getting attr on -13/1d8/temp_11.1d8_0_139460080_8/head: (61) No data available
So this is -ENODATA (61) and I am not sure where the file should be, but the directory
/var/lib/ceph/osd/ceph-39/current/11.1d8_TEMP
is empty.
Linux: 4.4.0-92-generic #115~14.04.1-Ubuntu
Mounted with: /dev/nvme2n1p1 on /var/lib/ceph/osd/ceph-26 type xfs (rw,noatime,inode64)