Starting program: /usr/bin/ceph-osd -d --debug_ms 20 -i 17 -c /etc/ceph/ceph.conf [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [New Thread 0x7ffff55c9700 (LWP 9545)] 2012-12-13 17:02:32.362368 7ffff7fea780 0 ceph version 0.55 (690f8175606edf37a3177c27a3949c78fd37099f), process ceph-osd, pid 9542 starting osd.17 at :/0 osd_data /var/lib/ceph/osd/ceph-17 /var/lib/ceph/osd/ceph-17/journal 2012-12-13 17:02:32.362907 7ffff7fea780 10 -- :/0 rank.bind :/0 2012-12-13 17:02:32.362922 7ffff7fea780 10 accepter.accepter.bind 2012-12-13 17:02:32.362945 7ffff7fea780 10 accepter.accepter.bind bound on random port 0.0.0.0:6800/0 2012-12-13 17:02:32.362960 7ffff7fea780 10 accepter.accepter.bind bound to 0.0.0.0:6800/0 2012-12-13 17:02:32.362985 7ffff7fea780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6800/9542 need_addr=1 2012-12-13 17:02:32.362993 7ffff7fea780 10 -- :/0 rank.bind :/0 2012-12-13 17:02:32.362996 7ffff7fea780 10 accepter.accepter.bind 2012-12-13 17:02:32.363004 7ffff7fea780 10 accepter.accepter.bind bound on random port 0.0.0.0:6801/0 2012-12-13 17:02:32.363018 7ffff7fea780 10 accepter.accepter.bind bound to 0.0.0.0:6801/0 2012-12-13 17:02:32.363027 7ffff7fea780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6801/9542 need_addr=1 2012-12-13 17:02:32.363032 7ffff7fea780 10 -- :/0 rank.bind :/0 2012-12-13 17:02:32.363035 7ffff7fea780 10 accepter.accepter.bind 2012-12-13 17:02:32.363044 7ffff7fea780 10 accepter.accepter.bind bound on random port 0.0.0.0:6802/0 2012-12-13 17:02:32.363048 7ffff7fea780 10 accepter.accepter.bind bound to 0.0.0.0:6802/0 2012-12-13 17:02:32.363062 7ffff7fea780 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6802/9542 need_addr=1 [New Thread 0x7ffff4687700 (LWP 9546)] [New Thread 0x7ffff3e86700 (LWP 9547)] 2012-12-13 17:02:32.433333 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount FIEMAP ioctl is supported and appears to work 2012-12-13 17:02:32.433358 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option 2012-12-13 17:02:32.433861 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount did NOT detect btrfs 2012-12-13 17:02:32.499932 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount syncfs(2) syscall fully supported (by glibc and kernel) 2012-12-13 17:02:32.533354 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount syscall(SYS_syncfs, fd) fully supported 2012-12-13 17:02:32.566591 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount syscall(__NR_syncfs, fd) fully supported 2012-12-13 17:02:32.566730 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount found snaps <> 2012-12-13 17:02:32.601013 7ffff7fea780 0 filestore(/var/lib/ceph/osd/ceph-17) mount: enabling WRITEAHEAD journal mode: btrfs not detected [New Thread 0x7ffff3685700 (LWP 9548)] 2012-12-13 17:02:32.629874 7ffff7fea780 1 journal _open /var/lib/ceph/osd/ceph-17/journal fd 25: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 2012-12-13 17:02:32.635563 7ffff7fea780 1 journal _open /var/lib/ceph/osd/ceph-17/journal fd 25: 1048576000 bytes, block size 4096 bytes, directio = 1, aio = 0 [New Thread 0x7ffff2e84700 (LWP 9549)] [New Thread 0x7ffff2683700 (LWP 9550)] [New Thread 0x7ffff1e82700 (LWP 9551)] [New Thread 0x7ffff1681700 (LWP 9552)] [New Thread 0x7ffff0e80700 (LWP 9553)] [New Thread 0x7ffff067f700 (LWP 9554)] [New Thread 0x7fffefe7e700 (LWP 9555)] [New Thread 0x7fffef67d700 (LWP 9556)] [New Thread 0x7fffeee7c700 (LWP 9557)] [Thread 0x7ffff3685700 (LWP 9548) exited] [Thread 0x7ffff067f700 (LWP 9554) exited] [Thread 0x7ffff0e80700 (LWP 9553) exited] [Thread 0x7ffff1681700 (LWP 9552) exited] [Thread 0x7ffff1e82700 (LWP 9551) exited] 2012-12-13 17:02:32.654056 7ffff7fea780 1 journal close /var/lib/ceph/osd/ceph-17/journal [Thread 0x7ffff2e84700 (LWP 9549) exited] [Thread 0x7ffff2683700 (LWP 9550) exited] [Thread 0x7fffefe7e700 (LWP 9555) exited] [Thread 0x7fffef67d700 (LWP 9556) exited] [Thread 0x7fffeee7c700 (LWP 9557) exited] [New Thread 0x7fffeee7c700 (LWP 9558)] [New Thread 0x7fffef67d700 (LWP 9559)] [New Thread 0x7fffefe7e700 (LWP 9560)] [New Thread 0x7ffff2683700 (LWP 9561)] [New Thread 0x7ffff3685700 (LWP 9562)] [New Thread 0x7ffff2e84700 (LWP 9563)] [New Thread 0x7ffff1e82700 (LWP 9564)] [New Thread 0x7ffff1681700 (LWP 9565)] [New Thread 0x7ffff0e80700 (LWP 9566)] [New Thread 0x7ffff067f700 (LWP 9567)] [New Thread 0x7fffee67b700 (LWP 9568)] [New Thread 0x7fffede7a700 (LWP 9569)] [New Thread 0x7fffed679700 (LWP 9570)] [New Thread 0x7fffece78700 (LWP 9571)] [New Thread 0x7fffec677700 (LWP 9572)] [New Thread 0x7fffebe76700 (LWP 9573)] [New Thread 0x7fffeb675700 (LWP 9574)] Program received signal SIGABRT, Aborted. 0x00007ffff5e45425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff5e45425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff5e48b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff679769d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #3 0x00007ffff6795846 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #4 0x00007ffff6795873 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #5 0x00007ffff679596e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 #6 0x000000000081c6e7 in copy_out (dest=, l=, o=, this=) at ./include/buffer.h:194 #7 ceph::buffer::list::iterator::copy (this=0x7fffffffc8e0, len=4, dest=0x7fffffffc64c "") at common/buffer.cc:526 #8 0x000000000064456e in decode_raw (t=@0x7fffffffc64c: 0, p=...) at ./include/encoding.h:57 #9 decode (p=..., v=) at ./include/encoding.h:102 #10 decode (m=..., p=...) at ./include/encoding.h:555 #11 0x000000000069a158 in PG::read_state (this=0x7660000, store=0x1472000) at osd/PG.cc:2641 #12 0x0000000000621d6a in OSD::load_pgs (this=0x14b5000) at osd/OSD.cc:1429 #13 0x0000000000622e40 in OSD::init (this=0x14b5000) at osd/OSD.cc:877 #14 0x0000000000561ee9 in main (argc=, argv=) at ceph_osd.cc:440 (gdb) bt full #0 0x00007ffff5e45425 in raise () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #1 0x00007ffff5e48b8b in abort () from /lib/x86_64-linux-gnu/libc.so.6 No symbol table info available. #2 0x00007ffff679769d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #3 0x00007ffff6795846 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #4 0x00007ffff6795873 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #5 0x00007ffff679596e in __cxa_throw () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6 No symbol table info available. #6 0x000000000081c6e7 in copy_out (dest=, l=, o=, this=) at ./include/buffer.h:194 No locals. #7 ceph::buffer::list::iterator::copy (this=0x7fffffffc8e0, len=4, dest=0x7fffffffc64c "") at common/buffer.cc:526 howmuch = __PRETTY_FUNCTION__ = "void ceph::buffer::list::iterator::copy(unsigned int, char*)" #8 0x000000000064456e in decode_raw (t=@0x7fffffffc64c: 0, p=...) at ./include/encoding.h:57 No locals. #9 decode (p=..., v=) at ./include/encoding.h:102 e = 0 #10 decode (m=..., p=...) at ./include/encoding.h:555 n = #11 0x000000000069a158 in PG::read_state (this=0x7660000, store=0x1472000) at osd/PG.cc:2641 bl = {_buffers = { >> = { _M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, _M_node = {_M_next = 0x7fffffffc860, _M_prev = 0x7fffffffc860}}}, }, _len = 0, append_buffer = {_raw = 0x0, _off = 0, _len = 0}, last_p = {bl = 0x7fffffffc860, ls = 0x7fffffffc860, off = 0, p = {_M_node = 0x7fffffffc860}, p_off = 0}} p = {bl = 0x7fffffffc860, ls = 0x7fffffffc860, off = 0, p = {_M_node = 0x7fffffffc860}, p_off = 0} struct_v = 4 '\004' #12 0x0000000000621d6a in OSD::load_pgs (this=0x14b5000) at osd/OSD.cc:1429 pgid = {m_pool = 6, m_seed = 7111, m_preferred = -1} snap = {val = 18446744073709551614} role = pg = 0x7660000 rctx = {start_time = {tv = {tv_sec = 1355418153, tv_nsec = 903588000}}, query_map = 0x0, info_map = 0x0, notify_list = 0x0, on_applied = 0x0, on_safe = 0x0, transaction = 0x0} it = {_M_current = 0x14ca2a0} __PRETTY_FUNCTION__ = "void OSD::load_pgs()" ---Type to continue, or q to quit--- ls = { >> = { _M_impl = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x14c7000, _M_finish = 0x14ca918, _M_end_of_storage = 0x14cb000}}, } r = __func__ = "load_pgs" #13 0x0000000000622e40 in OSD::init (this=0x14b5000) at osd/OSD.cc:877 lock = {mutex = @0x14b5010} __PRETTY_FUNCTION__ = "int OSD::init()" r = admin_socket = #14 0x0000000000561ee9 in main (argc=, argv=) at ceph_osd.cc:440 mkkey = flushjournal = convertfilestore = false supported = 8389123 args = { >> = { _M_impl = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x143f000, _M_finish = 0x143f000, _M_end_of_storage = 0x143f040}}, } get_osd_fsid = false id = cluster_fsid = {uuid = "\362B\004\201\v$JԷh\244\032m\260=\016"} osd_fsid = {uuid = "] FݷmD\034\262\274\344\234\000\204\036\260"} r = cluster_messenger = 0x1457a00 mkfs = mkjournal = dump_journal = false get_cluster_fsid = false messenger_hbserver = 0x1459a00 hb_addr = {type = 0, nonce = 0, {addr = {ss_family = 0, __ss_align = 0, __ss_padding = '\000' }, addr4 = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, addr6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = { __in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}} mc = { = {_vptr.Dispatcher = 0xbfd0d0, cct = 0x1441000}, monmap = {epoch = 0, fsid = {uuid = '\000' }, mon_addr = {_M_t = { _M_impl = {, std::allocator > const, entity_addr_t> > >> = {<__gnu_cxx::new_allocator, std::allocator > const, entity_addr_t> > >> = {}, }, _M_key_compare = {, std::allocator >, std::basic_string, std::allocator >, bool>> = {}, }, _M_header = {_M_color = std::_S_red, _M_parent = 0x14702c0, _M_left = 0x1470420, _M_right = 0x1470160}, _M_node_count = 3}}}, last_changed = {tv = {tv_sec = 0, tv_nsec = 0}}, created = {tv = { ---Type to continue, or q to quit--- tv_sec = 0, tv_nsec = 0}}, addr_name = {_M_t = { _M_impl = {, std::allocator > > > >> = {<__gnu_cxx::new_allocator, std::allocator > > > >> = {}, }, _M_key_compare = {> = {}, }, _M_header = {_M_color = std::_S_red, _M_parent = 0x1470210, _M_left = 0x1470370, _M_right = 0x14700b0}, _M_node_count = 3}}}, rank_name = {, std::allocator >, std::allocator, std::allocator > > >> = { _M_impl = {, std::allocator > >> = {<__gnu_cxx::new_allocator, std::allocator > >> = {}, }, _M_start = 0x1469200, _M_finish = 0x1469218, _M_end_of_storage = 0x1469220}}, }, rank_addr = { >> = { _M_impl = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x1440480, _M_finish = 0x1440618, _M_end_of_storage = 0x14406a0}}, }}, state = MC_STATE_NONE, messenger = 0x1457000, cur_mon = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0xc06958 ""}}, cur_con = 0x0, rng = {m_z = 9542, m_w = 123}, entity_name = {type = 0, id = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0xc06958 ""}}, type_id = { static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0xc06958 ""}}}, my_addr = {type = 0, nonce = 0, {addr = {ss_family = 0, __ss_align = 0, __ss_padding = '\000' }, addr4 = {sin_family = 0, sin_port = 0, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"}, addr6 = {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {__in6_u = {__u6_addr8 = '\000' , __u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}, monc_lock = {name = 0x93beac "MonClient::monc_lock", id = -1, recursive = false, lockdep = true, backtrace = false, _m = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\002", '\000' , __align = 0}, nlock = 0, locked_by = 0, cct = 0x0, logger = 0x0}, timer = {cct = 0x1441000, lock = @0x7fffffffe010, cond = {_vptr.Cond = 0x8e31b0, _c = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}, waiter_mutex = 0x0}, safe_callbacks = true, thread = 0x0, schedule = {_M_t = { _M_impl = { > >> = {<__gnu_cxx::new_allocator > >> = {}, }, _M_key_compare = {> = {}, }, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe0d0, _M_right = 0x7fffffffe0d0}, _M_node_count = 0}}}, events = {_M_t = { _M_impl = { > > > >> = {<__gnu_cxx::new_allocator > > > >> = {}, }, _M_key_compare = {> = {}, }, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe100, _M_right = 0x7fffffffe100}, _M_node_count = 0}}}, stopping = false}, finisher = {cct = 0x1441000, finisher_lock = {name = 0x8e676b "Finisher::finisher_lock", id = -1, recursive = false, lockdep = true, backtrace = false, _m = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\002", '\000' , __align = 0}, nlock = 0, locked_by = 0, cct = 0x0, logger = 0x0}, finisher_cond = {_vptr.Cond = 0x8e31b0, _c = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}, waiter_mutex = 0x0}, finisher_empty_cond = { ---Type to continue, or q to quit--- _vptr.Cond = 0x8e31b0, _c = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}, waiter_mutex = 0x0}, finisher_stop = false, finisher_running = false, finisher_queue = { >> = { _M_impl = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}}, }, finisher_queue_rval = {, std::allocator > >> = { _M_impl = { > >> = {<__gnu_cxx::new_allocator > >> = {}, }, _M_node = {_M_next = 0x7fffffffe230, _M_prev = 0x7fffffffe230}}}, }, logger = 0x0, finisher_thread = { = {_vptr.Thread = 0x8ebd90, thread_id = 0}, fin = 0x7fffffffe130}}, authorize_handler_registry = 0x0, initialized = false, log_client = 0x0, more_log_pending = false, auth_supported = 0x0, hunting = true, auth_cond = {_vptr.Cond = 0x8e31b0, _c = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}, waiter_mutex = 0x0}, want_monmap = true, want_keys = 0, global_id = 0, map_cond = {_vptr.Cond = 0x8e31b0, _c = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 0, __broadcast_seq = 0}, __size = '\000' , __align = 0}, waiter_mutex = 0x0}, authenticate_err = 0, waiting_for_session = { >> = { _M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, _M_node = {_M_next = 0x7fffffffe328, _M_prev = 0x7fffffffe328}}}, }, sub_have = {_M_t = { _M_impl = {, std::allocator > const, ceph_mon_subscribe_item> > >> = {<__gnu_cxx::new_allocator, std::allocator > const, ceph_mon_subscribe_item> > >> = {}, }, _M_key_compare = {, std::allocator >, std::basic_string, std::allocator >, bool>> = {}, }, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe340, _M_right = 0x7fffffffe340}, _M_node_count = 0}}}, sub_renew_sent = {tv = {tv_sec = 0, tv_nsec = 0}}, sub_renew_after = {tv = {tv_sec = 0, tv_nsec = 0}}, auth = 0x0, keyring = 0x0, rotating_secrets = 0x0, version_requests = {_M_t = { _M_impl = { > >> = {<__gnu_cxx::new_allocator > >> = {}, }, _M_key_compare = {> = {}, }, _M_header = { _M_color = std::_S_red, _M_parent = 0x0, _M_left = 0x7fffffffe398, _M_right = 0x7fffffffe398}, _M_node_count = 0}}}, version_req_id = 0} s = "\000pC\001\000\000\000\000[\000\000\000\000\000\000\000\\\000\000" dump_pg_log = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0xc06958 ""}} client_messenger = 0x1457000 client_throttler = {cct = 0x1441000, name = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x1438378 "osd_client_bytes"}}, logger = 0x144a1b0, count = {val = 0}, max = {val = 524288000}, lock = {name = 0x93e7c9 "Throttle::lock", id = -1, recursive = false, lockdep = true, backtrace = false, _m = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , "\002", '\000' , __align = 0}, nlock = 0, locked_by = 0, cct = 0x0, logger = 0x0}, cond = { >> = { _M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, ---Type to continue, or q to quit--- _M_node = {_M_next = 0x7fffffffe450, _M_prev = 0x7fffffffe450}}}, }} val = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0xc06958 ""}} end = 0x143525a "" magic = {static npos = , _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x1438138 "ceph osd volume v026"}} get_journal_fsid = false whoami = 17 w = 17 messenger_hbclient = 0x1457500 err = 0 (gdb)