Ceph

Hit this bug yesterday. The gdb output:

#3  0x00007fd772bfd6c5 in raise () from /lib64/libc.so.6
#4  0x00007fd772bfeea5 in abort () from /lib64/libc.so.6
#5  0x00007fd7734a37b5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#6  0x00007fd7734a1886 in ?? () from /usr/lib64/libstdc++.so.6
#7  0x00007fd7734a18b3 in std::terminate() () from /usr/lib64/libstdc++.so.6
#8  0x00007fd7734a19ae in __cxa_throw () from /usr/lib64/libstdc++.so.6
#9  0x00000000005db978 in ceph::__ceph_assert_fail (assertion=0x608171 "!missing.is_missing(soid)", file=<value optimized out>, line=2579, func=<value optimized out>)
    at common/assert.cc:30
#10 0x0000000000491c09 in ReplicatedPG::sub_op_modify (this=0x1968000, op=0x2a1c600) at osd/ReplicatedPG.cc:2579
#11 0x00000000004d5684 in OSD::dequeue_op (this=0x1567000, pg=0x1968000) at osd/OSD.cc:5146
#12 0x00000000005dc5b3 in ThreadPool::worker (this=0x15673e8) at common/WorkQueue.cc:44
#13 0x00000000005065ad in ThreadPool::WorkThread::entry() ()
#14 0x000000000047a42a in Thread::_entry_func (arg=<value optimized out>) at ./common/Thread.h:41
#15 0x00007fd77408ca3a in start_thread () from /lib64/libpthread.so.0
#16 0x00007fd772ca977d in clone () from /lib64/libc.so.6
#17 0x0000000000000000 in ?? ()
(gdb) frame 10
#10 0x0000000000491c09 in ReplicatedPG::sub_op_modify (this=0x1968000, op=0x2a1c600) at osd/ReplicatedPG.cc:2579
2579    osd/ReplicatedPG.cc: No such file or directory.
        in osd/ReplicatedPG.cc
(gdb) print soid
$1 = (const sobject_t &) @0x2a1c798: {oid = {name = "10000004f47.000001b0"}, snap = {val = 18446744073709551614}}
(gdb) print acting
$2 = std::vector of length 2, capacity 2 = {0, 3}
(gdb) print info
$3 = {pgid = {v = {preferred = {v = 65535}, ps = {v = 99}, pool = {v = 0}}}, last_update = {version = 216, epoch = 611, __pad = 0}, last_complete = {version = 137,
    epoch = 569, __pad = 0}, log_tail = {version = 167, epoch = 600, __pad = 0}, log_backlog = true, purged_snaps = {_size = 0, m = std::map with 0 elements},
  stats = {version = {version = 215, epoch = 611, __pad = 0}, reported = {version = 934, epoch = 297, __pad = 0}, state = 0, log_start = {version = 0, epoch = 0,
      __pad = 0}, ondisk_log_start = {version = 0, epoch = 0, __pad = 0}, created = 2, parent = {v = {preferred = {v = 0}, ps = {v = 0}, pool = {v = 0}}},
    parent_split_bits = 0, last_scrub = {version = 96, epoch = 506, __pad = 0}, last_scrub_stamp = {tv = {tv_sec = 1298959219, tv_nsec = 605245000}},
    num_bytes = 75497472, num_kb = 73728, num_objects = 18, num_object_clones = 0, num_object_copies = 0, num_objects_missing_on_primary = 0,
    num_objects_degraded = 0, log_size = 0, ondisk_log_size = 0, num_rd = 843, num_rd_kb = 195984, num_wr = 216, num_wr_kb = 313336, num_objects_unfound = 0,
    up = std::vector of length 0, capacity 0, acting = std::vector of length 0, capacity 0}, history = {epoch_created = 2, last_epoch_started = 611,
    last_epoch_clean = 3, last_epoch_split = 609, same_up_since = 601, same_acting_since = 607, same_primary_since = 297, last_scrub = {version = 96, epoch = 506,
      __pad = 0}, last_scrub_stamp = {tv = {tv_sec = 1298959219, tv_nsec = 605245000}}}}
(gdb) print history
No symbol "history" in current context.
(gdb) print missing
$4 = {missing = std::map with 1 elements = {[{oid = {name = "10000004f47.000001b0"}, snap = {val = 18446744073709551614}}] = {need = {version = 117, epoch = 569,
        __pad = 0}, have = {version = 0, epoch = 0, __pad = 0}}}, rmissing = std::map with 1 elements = {[{version = 117, epoch = 569, __pad = 0}] = {oid = {name =
    "10000004f47.000001b0"}, snap = {val = 18446744073709551614}}}}

I put the log files, cosd binary and core dump on http://www.megaupload.com/?d=GRORERXY