Ceph

I'm using xfs, with no specific mount options, just the default.

I added the debug settings, and got a large log file. How do I look for the object ID in that file? Specific log format I should be scanning for?

Thanks

Xiaopong Tran wrote:

I'm using xfs, with no specific mount options, just the default.

I added the debug settings, and got a large log file. How do I look for the object ID in that file? Specific log format I should be scanning for?

Thanks

Sigh.. I take it back. The block of code is

void ReplicatedPG::scan_range(hobject_t begin, int min, int max, BackfillInterval *bi)
{
  assert(is_locked());
  dout(10) << "scan_range from " << begin << dendl;
  bi->begin = begin;
  bi->objects.clear();  // for good measure

  vector<hobject_t> ls;
  ls.reserve(max);
  int r = osd->store->collection_list_partial(coll, begin, min, max,
                          0, &ls, &bi->end);
  assert(r >= 0);
  dout(10) << " got " << ls.size() << " items, next " << bi->end << dendl;
  dout(20) << ls << dendl;

  for (vector<hobject_t>::iterator p = ls.begin(); p != ls.end(); ++p) {
    ObjectContext *obc = NULL;
    if (is_primary())
      obc = _lookup_object_context(*p);
    if (obc) {
      bi->objects[*p] = obc->obs.oi.version;
      dout(20) << "  " << *p << " " << obc->obs.oi.version << dendl;
    } else {
      bufferlist bl;
      int r = osd->store->getattr(coll, *p, OI_ATTR, bl);
      assert(r >= 0);
      object_info_t oi(bl);
      bi->objects[*p] = oi.version;
      dout(20) << "  " << *p << " " << oi.version << dendl;
    }
  }
}

and the object isn't dumped prior to the failed assertion. You can modify the code to print it, e.g.

      int r = osd->store->getattr(coll, *p, OI_ATTR, bl);
      if (r < 0)
         dout(0) << "error getting attr on " << *p << ": " << cpp_strerror(r) << dendl;
      assert(r >= 0);

and then you can see it. Unfortunately the STL iterator p isn't easy to inspect via gdb.

This is just a FYI in my older ceph version 0.94.10-1trusty is see the same crash. The code is a bit different, it will skip on -ENOENT. I chnaged this now to log and skip on all error codes. The loggng of the object ID should be added in all cases I think (not sure about latest source, however).

I see the following logs now:

/var/log/ceph/ceph-osd.39.log:2017-08-17 19:16:47.775537 7f3389e85700 0 osd.39 pg_epoch: 28294 pg[11.1d8( v 28289'32157569 (21329'32154561,28289'32157569] lb -1/0//0 local-les=28207 n=0 ec=2440 les/c 28207/28166 28206/28206/22659) [39,65]/[65,25] r=-1 lpr=28206 pi=21552-28205/377 luod=0'0 crt=28289'32157569 lcod 28256'32157568 active+remapped] error getting attr on -13/1d8/temp_11.1d8_0_139460080_8/head: (61) No data available

So this is -ENODATA (61) and I am not sure where the file should be, but the directory

/var/lib/ceph/osd/ceph-39/current/11.1d8_TEMP

is empty.

Linux: 4.4.0-92-generic #115~14.04.1-Ubuntu
Mounted with: /dev/nvme2n1p1 on /var/lib/ceph/osd/ceph-26 type xfs (rw,noatime,inode64)