Ceph » rbd

I do have a snapshot of the root drive from the failure on 12/31, and looked at it briefly to see if I could spot some kind of corruption based on what fsck said, but had no success. I could conceivably copy that 40GB image somewhere if it were useful in some way.

Note: there is a known cache corruption bug [1] that is scheduled to be fixed in 0.94.10, but that wouldn't explain the assertion failure.

[1] http://tracker.ceph.com/issues/16546

It's possible that somehow a cache read request failed (which shouldn't be possible), which resulted in the bh being marked in the error state [1]. With enough cache pressure, the `trim()` would have attempted to evict a block in the error state and would have failed the assertion.

I still need to verify this assumption but even fixing this assertion failure doesn't get to the root cause of why the read request failed (which would still result in IO errors within your VM).

@Brian: do you have any non-default settings in your Ceph configuration (either client or server side)? OSD timeouts?

[1] https://github.com/ceph/ceph/blob/v0.94.9/src/osdc/ObjectCacher.cc#L807

Jason Dillaman wrote:

@Brian: any chance you had your nodes configured to capture core dumps? That would make solving this relatively simple.

Provided via private channels due to sensitive information contained within.

@Brian: do you have any non-default settings in your Ceph configuration (either client or server side)? OSD timeouts?

[1] https://github.com/ceph/ceph/blob/v0.94.9/src/osdc/ObjectCacher.cc#L807

[client]
rados mon op timeout = 180
osd op timeout = 180

I realize default is 0, and could explain the bulk of our problems. I'll try to determine when/why this was set in our environment.

@Brian: Thanks! That "osd op timeout = 180" setting is the problem -- it would have caused "ETIMEDOUT" errors to bubble up to your VMs and it set the stage for the cache to have bhs marked as error state. I would recommend yanking that setting post haste from any librbd client configs.

The provided core dump confirms that the bh state was error with an error code of ETIMEDOUT (-110).

(gdb) print (*('ObjectCacher::BufferHead'*)0x00007f4748ff3380)
$17 = {
  <LRUObject> = {
    lru_next = 0x0, 
    lru_prev = 0x0, 
    lru_pinned = false, 
    lru = 0x0, 
    lru_list = 0x0
  }, 
  members of ObjectCacher::BufferHead: 
  static STATE_MISSING = 0, 
  static STATE_CLEAN = 1, 
  static STATE_ZERO = 2, 
  static STATE_DIRTY = 3, 
  static STATE_RX = 4, 
  static STATE_TX = 5, 
  static STATE_ERROR = 6, 
  state = 6, 
  ref = 0, 
  ex = {
    start = 184320, 
    length = 8192
  }, 
  dontneed = false, 
  ob = 0x7f4748377250, 
  bl = {
    _buffers = empty std::list, 
    _len = 0, 
    _memcopy_count = 0, 
    append_buffer = {
      _raw = 0x0, 
      _off = 0, 
      _len = 0
    }, 
    last_p = {
      bl = 0x7f4748ff33d0, 
      ls = 0x7f4748ff33d0, 
      off = 0, 
      p = {
        _raw = , 
        _off = 0, 
        _len = 0
      }, 
      p_off = 0
    }, 
    static CLAIM_DEFAULT = 0, 
    static CLAIM_ALLOW_NONSHAREABLE = 1
  }, 
  last_write_tid = 0, 
  last_read_tid = 44752963, 
  last_write = {
    tv = {
      tv_sec = 0, 
      tv_nsec = 0
    }
  }, 
  snapc = {
    seq = {
      val = 0
    }, 
    snaps = std::vector of length 0, capacity 0
  }, 
  error = -110, 
  waitfor_read = std::map with 0 elements
}