Project

General

Profile

Actions
Copy link

Bug #13778

closed

OSD crash (giant)

Added by 忆 秋 over 8 years ago. Updated about 8 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
0.86
ceph-qa-suite:
fs
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

-1> 2015-11-12 11:05:10.798295 7fe9364fc880 5 osd.0 pg_epoch: 657 pg[2.aa(unlocked)] enter Initial
0> 2015-11-12 11:05:10.812489 7fe9364fc880 -1 ** Caught signal (Aborted) *
in thread 7fe9364fc880

ceph version 0.87.2 (87a7cec9ab11c677de2ab23a7668a77d2f5b955e)
1: /usr/bin/ceph-osd() [0xa86922]
2: (()+0xf130) [0x7fe934e96130]
3: (gsignal()+0x37) [0x7fe9338b05d7]
4: (abort()+0x148) [0x7fe9338b1cc8]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fe9341b49b5]
6: (()+0x5e926) [0x7fe9341b2926]
7: (()+0x5e953) [0x7fe9341b2953]
8: (()+0x5eb73) [0x7fe9341b2b73]
9: (pg_log_entry_t::decode_with_checksum(ceph::buffer::list::iterator&)+0x21a) [0x76106a]
10: (PGLog::read_log(ObjectStore*, coll_t, hobject_t, pg_info_t const&, std::map<eversion_t, hobject_t, std::less<eversion_t>, std::allocator<std::pair<eversion_t const, hobject_t> > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream<char, std::char_traits<char>, std::allocator<char> >&, std::set<std::string, std::less<std::string>, std::allocator<std::string> >)+0xf18) [0x748268]
11: (PG::read_state(ObjectStore, ceph::buffer::list&)+0x2d6) [0x7c2a96]
12: (OSD::load_pgs()+0x966) [0x695956]
13: (OSD::init()+0x730) [0x697a60]
14: (main()+0x24c7) [0x622757]
15: (__libc_start_main()+0xf5) [0x7fe93389caf5]
16: /usr/bin/ceph-osd() [0x63aa79]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.0.log
-- end dump of recent events ---

Actions
Copy link
 #1

Updated by Loïc Dachary over 8 years ago

  • Subject changed from Why OSD can not start? to OSD crash (giant)
  • Status changed from New to Won't Fix

Could you please upgrade to hammer ? If the problem still happens, please re-open this issue. The giant release (0.87) is no longer supported.

Actions
Copy link
 #2

Updated by Johannes Erdfelt about 8 years ago

I appear to have run into this same problem, but I am running hammer:

ceph-0.94.5-0.el6.x86_64

I have one OSD that won't stay running now:

2016-02-02 17:40:50.191900 7f9a3f56e800 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5642
2016-02-02 17:40:50.630201 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) backend generic (magic 0xef53)
2016-02-02 17:40:51.393738 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is supported and appears to work
2016-02-02 17:40:51.395129 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-02-02 17:40:51.470464 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: syscall(SYS_syncfs, fd) fully supported
2016-02-02 17:40:51.601887 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) limited size xattrs
2016-02-02 17:40:51.825407 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-02-02 17:40:52.182720 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
2016-02-02 17:40:52.250622 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
2016-02-02 17:40:52.313684 7f9a3f56e800 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2016-02-02 17:40:52.337571 7f9a3f56e800 0 osd.8 50551 crush map has features 69793480704, adjusting msgr requires for clients
2016-02-02 17:40:52.339033 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648 was 8705, adjusting msgr requires for mons
2016-02-02 17:40:52.339115 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648, adjusting msgr requires for osds
2016-02-02 17:40:52.339183 7f9a3f56e800 0 osd.8 50551 load_pgs
2016-02-02 17:40:58.200696 7f9a3f56e800 -1 ** Caught signal (Aborted) *
in thread 7f9a3f56e800

ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xa4c1c5]
 2: (()+0xf790) [0x7f9a3e2ca790]
 3: (gsignal()+0x35) [0x7f9a3cf94625]
 4: (abort()+0x175) [0x7f9a3cf95e05]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x7f9a3d84ea7d]
 6: (()+0xbcbd6) [0x7f9a3d84cbd6]
 7: (()+0xbcc03) [0x7f9a3d84cc03]
 8: (()+0xbcd22) [0x7f9a3d84cd22]
 9: (pg_log_entry_t::decode_with_checksum(ceph::buffer::list::iterator&)+0x12d) [0x7b074d]
 10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map&lt;eversion_t, hobject_t, std::less&lt;eversion_t&gt;, std::allocator&lt;std::pair&lt;eversion_t const, hobject_t&gt; > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >&, std::set&lt;std::string, std::less&lt;std::string&gt;, std::allocator&lt;std::string&gt; >)+0xb74) [0x7ccef4]
 11: (PG::read_state(ObjectStore, ceph::buffer::list&)+0x34c) [0x82962c]
 12: (OSD::load_pgs()+0x1646) [0x688b96]
 13: (OSD::init()+0x176e) [0x68caee]
 14: (main()+0x384f) [0x62eb8f]
 15: (__libc_start_main()+0xfd) [0x7f9a3cf80d5d]
 16: /usr/bin/ceph-osd() [0x62a299]
 NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- begin dump of recent events ---
130> 2016-02-02 17:40:50.171206 7f9a3f56e800 5 asok(0x5118000) register_command perfcounters_dump hook 0x5088050
-129> 2016-02-02 17:40:50.171281 7f9a3f56e800 5 asok(0x5118000) register_command 1 hook 0x5088050
-128> 2016-02-02 17:40:50.171292 7f9a3f56e800 5 asok(0x5118000) register_command perf dump hook 0x5088050
-127> 2016-02-02 17:40:50.171307 7f9a3f56e800 5 asok(0x5118000) register_command perfcounters_schema hook 0x5088050
-126> 2016-02-02 17:40:50.171315 7f9a3f56e800 5 asok(0x5118000) register_command 2 hook 0x5088050
-125> 2016-02-02 17:40:50.171321 7f9a3f56e800 5 asok(0x5118000) register_command perf schema hook 0x5088050
-124> 2016-02-02 17:40:50.171328 7f9a3f56e800 5 asok(0x5118000) register_command perf reset hook 0x5088050
-123> 2016-02-02 17:40:50.171333 7f9a3f56e800 5 asok(0x5118000) register_command config show hook 0x5088050
-122> 2016-02-02 17:40:50.171339 7f9a3f56e800 5 asok(0x5118000) register_command config set hook 0x5088050
-121> 2016-02-02 17:40:50.171346 7f9a3f56e800 5 asok(0x5118000) register_command config get hook 0x5088050
-120> 2016-02-02 17:40:50.171351 7f9a3f56e800 5 asok(0x5118000) register_command config diff hook 0x5088050
-119> 2016-02-02 17:40:50.171357 7f9a3f56e800 5 asok(0x5118000) register_command log flush hook 0x5088050
-118> 2016-02-02 17:40:50.171363 7f9a3f56e800 5 asok(0x5118000) register_command log dump hook 0x5088050
-117> 2016-02-02 17:40:50.171368 7f9a3f56e800 5 asok(0x5118000) register_command log reopen hook 0x5088050
-116> 2016-02-02 17:40:50.191900 7f9a3f56e800 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5642
-115> 2016-02-02 17:40:50.237948 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6804/5642 need_addr=1
-114> 2016-02-02 17:40:50.241932 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6806/5642 need_addr=1
-113> 2016-02-02 17:40:50.248035 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6807/5642 need_addr=1
-112> 2016-02-02 17:40:50.253372 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6808/5642 need_addr=1
-111> 2016-02-02 17:40:50.262560 7f9a3f56e800 1 finished global_init_daemonize
-110> 2016-02-02 17:40:50.490710 7f9a3f56e800 5 asok(0x5118000) init /var/run/ceph/ceph-osd.8.asok
-109> 2016-02-02 17:40:50.492565 7f9a3f56e800 5 asok(0x5118000) bind_and_listen /var/run/ceph/ceph-osd.8.asok
-108> 2016-02-02 17:40:50.496863 7f9a3f56e800 5 asok(0x5118000) register_command 0 hook 0x50800b0
-107> 2016-02-02 17:40:50.497582 7f9a3f56e800 5 asok(0x5118000) register_command version hook 0x50800b0
-106> 2016-02-02 17:40:50.497611 7f9a3f56e800 5 asok(0x5118000) register_command git_version hook 0x50800b0
-105> 2016-02-02 17:40:50.497624 7f9a3f56e800 5 asok(0x5118000) register_command help hook 0x5088140
-104> 2016-02-02 17:40:50.497652 7f9a3f56e800 5 asok(0x5118000) register_command get_command_descriptions hook 0x5088130
-103> 2016-02-02 17:40:50.499699 7f9a3f56e800 10 monclient(hunting): build_initial_monmap
-102> 2016-02-02 17:40:50.500254 7f9a3a8e5700 5 asok(0x5118000) entry start
-101> 2016-02-02 17:40:50.598615 7f9a3f56e800 5 adding auth protocol: cephx
-100> 2016-02-02 17:40:50.600147 7f9a3f56e800 5 adding auth protocol: cephx
-99> 2016-02-02 17:40:50.600590 7f9a3f56e800 5 asok(0x5118000) register_command objecter_requests hook 0x5088170
-98> 2016-02-02 17:40:50.605820 7f9a3f56e800 1 - 0.0.0.0:6804/5642 messenger.start
97> 2016-02-02 17:40:50.607473 7f9a3f56e800 1 - :/0 messenger.start
96> 2016-02-02 17:40:50.609772 7f9a3f56e800 1 - 0.0.0.0:6808/5642 messenger.start
95> 2016-02-02 17:40:50.612225 7f9a3f56e800 1 - 0.0.0.0:6807/5642 messenger.start
94> 2016-02-02 17:40:50.614750 7f9a3f56e800 1 - 0.0.0.0:6806/5642 messenger.start
93> 2016-02-02 17:40:50.618250 7f9a3f56e800 1 - :/0 messenger.start
-92> 2016-02-02 17:40:50.625307 7f9a3f56e800 2 osd.8 0 mounting /data/osd8/ceph/data /var/lib/ceph/osd/osd.8/journal
-91> 2016-02-02 17:40:50.630201 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) backend generic (magic 0xef53)
-90> 2016-02-02 17:40:51.393738 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is supported and appears to work
-89> 2016-02-02 17:40:51.395129 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-88> 2016-02-02 17:40:51.470464 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: syscall(SYS_syncfs, fd) fully supported
-87> 2016-02-02 17:40:51.601887 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) limited size xattrs
-86> 2016-02-02 17:40:51.825407 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-85> 2016-02-02 17:40:52.110985 7f9a3f56e800 2 journal open /var/lib/ceph/osd/osd.8/journal fsid 2988ec18-fcc1-4aa6-85de-cb8adc56a007 fs_op_seq 4126731
-84> 2016-02-02 17:40:52.182720 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
-83> 2016-02-02 17:40:52.184238 7f9a3f56e800 2 journal open advancing committed_seq 4126730 to fs op_seq 4126731
-82> 2016-02-02 17:40:52.186354 7f9a3f56e800 2 journal read_entry 12788428800 : seq 4126731 960 bytes
-81> 2016-02-02 17:40:52.188012 7f9a3f56e800 2 journal No further valid entries found, journal is most likely valid
-80> 2016-02-02 17:40:52.190625 7f9a3f56e800 2 journal No further valid entries found, journal is most likely valid
-79> 2016-02-02 17:40:52.191045 7f9a3f56e800 3 journal journal_replay: end of journal, done.
-78> 2016-02-02 17:40:52.250622 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
-77> 2016-02-02 17:40:52.262409 7f9a3f56e800 2 osd.8 0 boot
-76> 2016-02-02 17:40:52.271472 7f9a3f56e800 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-75> 2016-02-02 17:40:52.276610 7f9a3f56e800 1 <cls> cls/log/cls_log.cc:312: Loaded log class!
-74> 2016-02-02 17:40:52.280916 7f9a3f56e800 1 <cls> cls/user/cls_user.cc:367: Loaded user class!
-73> 2016-02-02 17:40:52.291492 7f9a3f56e800 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-72> 2016-02-02 17:40:52.301374 7f9a3f56e800 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
-71> 2016-02-02 17:40:52.309660 7f9a3f56e800 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
-70> 2016-02-02 17:40:52.313684 7f9a3f56e800 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
-69> 2016-02-02 17:40:52.330932 7f9a3f56e800 1 <cls> cls/rgw/cls_rgw.cc:3047: Loaded rgw class!
-68> 2016-02-02 17:40:52.337571 7f9a3f56e800 0 osd.8 50551 crush map has features 69793480704, adjusting msgr requires for clients
-67> 2016-02-02 17:40:52.339033 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648 was 8705, adjusting msgr requires for mons
-66> 2016-02-02 17:40:52.339115 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648, adjusting msgr requires for osds
-65> 2016-02-02 17:40:52.339183 7f9a3f56e800 0 osd.8 50551 load_pgs
-64> 2016-02-02 17:40:54.981646 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2(unlocked)] enter Initial
-63> 2016-02-02 17:40:55.140037 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2( v 50545'13223 (50235'10149,50545'13223] local-les=50551 n=3321 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13223 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.158392 0 0.000000
-62> 2016-02-02 17:40:55.140902 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2( v 50545'13223 (50235'10149,50545'13223] local-les=50551 n=3321 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13223 lcod 0'0 mlcod 0'0 inactive] enter Reset
-61> 2016-02-02 17:40:55.144239 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3(unlocked)] enter Initial
-60> 2016-02-02 17:40:55.247112 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3( v 50541'13961 (50243'10920,50541'13961] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13961 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.102861 0 0.000000
-59> 2016-02-02 17:40:55.248870 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3( v 50541'13961 (50243'10920,50541'13961] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13961 lcod 0'0 mlcod 0'0 inactive] enter Reset
-58> 2016-02-02 17:40:55.251434 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5(unlocked)] enter Initial
-57> 2016-02-02 17:40:55.456313 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5( v 50500'13626 (50243'10567,50500'13626] local-les=50551 n=3269 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50500'13624 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.204879 0 0.000000
-56> 2016-02-02 17:40:55.457458 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5( v 50500'13626 (50243'10567,50500'13626] local-les=50551 n=3269 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50500'13624 lcod 0'0 mlcod 0'0 inactive] enter Reset
-55> 2016-02-02 17:40:55.466063 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9(unlocked)] enter Initial
-54> 2016-02-02 17:40:55.618691 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9( v 50537'13519 (50235'10469,50537'13519] local-les=50551 n=3280 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50537'13519 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.152628 0 0.000000
-53> 2016-02-02 17:40:55.619974 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9( v 50537'13519 (50235'10469,50537'13519] local-les=50551 n=3280 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50537'13519 lcod 0'0 mlcod 0'0 inactive] enter Reset
-52> 2016-02-02 17:40:55.625042 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e(unlocked)] enter Initial
-51> 2016-02-02 17:40:55.755310 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e( v 50537'13216 (50243'10179,50537'13216] local-les=50551 n=3311 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13216 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.130268 0 0.000000
-50> 2016-02-02 17:40:55.756944 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e( v 50537'13216 (50243'10179,50537'13216] local-les=50551 n=3311 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13216 lcod 0'0 mlcod 0'0 inactive] enter Reset
-49> 2016-02-02 17:40:55.763638 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10(unlocked)] enter Initial
-48> 2016-02-02 17:40:55.895574 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10( v 50545'13931 (50243'10880,50545'13931] local-les=50551 n=3389 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13931 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.131937 0 0.000000
-47> 2016-02-02 17:40:55.897028 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10( v 50545'13931 (50243'10880,50545'13931] local-les=50551 n=3389 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13931 lcod 0'0 mlcod 0'0 inactive] enter Reset
-46> 2016-02-02 17:40:55.900392 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12(unlocked)] enter Initial
-45> 2016-02-02 17:40:56.078102 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12( v 50541'13782 (50235'10737,50541'13782] local-les=50551 n=3334 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50541'13782 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.177710 0 0.000000
-44> 2016-02-02 17:40:56.079598 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12( v 50541'13782 (50235'10737,50541'13782] local-les=50551 n=3334 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50541'13782 lcod 0'0 mlcod 0'0 inactive] enter Reset
-43> 2016-02-02 17:40:56.081403 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13(unlocked)] enter Initial
-42> 2016-02-02 17:40:56.216696 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13( v 50500'13547 (50235'10507,50500'13547] local-les=50551 n=3372 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13545 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.135292 0 0.000000
-41> 2016-02-02 17:40:56.218669 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13( v 50500'13547 (50235'10507,50500'13547] local-les=50551 n=3372 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13545 lcod 0'0 mlcod 0'0 inactive] enter Reset
-40> 2016-02-02 17:40:56.226157 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14(unlocked)] enter Initial
-39> 2016-02-02 17:40:56.363115 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14( v 50539'13379 (50243'10324,50539'13379] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13376 lcod 0'0 inactive NOTIFY] exit Initial 0.136958 0 0.000000
-38> 2016-02-02 17:40:56.365358 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14( v 50539'13379 (50243'10324,50539'13379] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13376 lcod 0'0 inactive NOTIFY] enter Reset
-37> 2016-02-02 17:40:56.368174 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16(unlocked)] enter Initial
-36> 2016-02-02 17:40:56.521665 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16( v 50500'13405 (50243'10374,50500'13405] local-les=50551 n=3219 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13403 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.153492 0 0.000000
-35> 2016-02-02 17:40:56.522516 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16( v 50500'13405 (50243'10374,50500'13405] local-les=50551 n=3219 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13403 lcod 0'0 mlcod 0'0 inactive] enter Reset
-34> 2016-02-02 17:40:56.530583 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19(unlocked)] enter Initial
-33> 2016-02-02 17:40:56.654350 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19( v 50500'13898 (50243'10881,50500'13898] local-les=50551 n=3380 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'13896 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.123767 0 0.000000
-32> 2016-02-02 17:40:56.656036 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19( v 50500'13898 (50243'10881,50500'13898] local-les=50551 n=3380 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'13896 lcod 0'0 mlcod 0'0 inactive] enter Reset
-31> 2016-02-02 17:40:56.661445 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a(unlocked)] enter Initial
-30> 2016-02-02 17:40:56.781691 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a( v 50543'13168 (50235'10120,50543'13168] local-les=50551 n=3228 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13165 lcod 0'0 inactive NOTIFY] exit Initial 0.120247 0 0.000000
-29> 2016-02-02 17:40:56.782016 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a( v 50543'13168 (50235'10120,50543'13168] local-les=50551 n=3228 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13165 lcod 0'0 inactive NOTIFY] enter Reset
-28> 2016-02-02 17:40:56.787180 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b(unlocked)] enter Initial
-27> 2016-02-02 17:40:56.960516 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b( v 50500'13498 (50235'10441,50500'13498] local-les=50551 n=3287 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13496 lcod 0'0 inactive NOTIFY] exit Initial 0.173337 0 0.000000
-26> 2016-02-02 17:40:56.962869 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b( v 50500'13498 (50235'10441,50500'13498] local-les=50551 n=3287 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13496 lcod 0'0 inactive NOTIFY] enter Reset
-25> 2016-02-02 17:40:56.970848 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f(unlocked)] enter Initial
-24> 2016-02-02 17:40:57.086203 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f( v 50500'13670 (50243'10660,50500'13670] local-les=50551 n=3291 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'13668 lcod 0'0 inactive NOTIFY] exit Initial 0.115356 0 0.000000
-23> 2016-02-02 17:40:57.087869 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f( v 50500'13670 (50243'10660,50500'13670] local-les=50551 n=3291 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'13668 lcod 0'0 inactive NOTIFY] enter Reset
-22> 2016-02-02 17:40:57.095066 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21(unlocked)] enter Initial
-21> 2016-02-02 17:40:57.229652 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21( v 50500'14107 (50243'11099,50500'14107] local-les=50551 n=3431 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'14105 lcod 0'0 inactive NOTIFY] exit Initial 0.134585 0 0.000000
-20> 2016-02-02 17:40:57.231151 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21( v 50500'14107 (50243'11099,50500'14107] local-les=50551 n=3431 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'14105 lcod 0'0 inactive NOTIFY] enter Reset
-19> 2016-02-02 17:40:57.236505 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24(unlocked)] enter Initial
-18> 2016-02-02 17:40:57.366887 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24( v 50543'13657 (50235'10603,50543'13657] local-les=50551 n=3394 ec=1 les/c 50551/50551 50550/50550/50530) [7,8] r=1 lpr=0 pi=50526-50549/9 crt=50500'13654 lcod 0'0 inactive NOTIFY] exit Initial 0.130383 0 0.000000
-17> 2016-02-02 17:40:57.368625 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24( v 50543'13657 (50235'10603,50543'13657] local-les=50551 n=3394 ec=1 les/c 50551/50551 50550/50550/50530) [7,8] r=1 lpr=0 pi=50526-50549/9 crt=50500'13654 lcod 0'0 inactive NOTIFY] enter Reset
-16> 2016-02-02 17:40:57.372953 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27(unlocked)] enter Initial
-15> 2016-02-02 17:40:57.540487 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27( v 50500'13527 (50235'10511,50500'13527] local-les=50551 n=3349 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13525 lcod 0'0 inactive NOTIFY] exit Initial 0.167534 0 0.000000
-14> 2016-02-02 17:40:57.542029 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27( v 50500'13527 (50235'10511,50500'13527] local-les=50551 n=3349 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13525 lcod 0'0 inactive NOTIFY] enter Reset
-13> 2016-02-02 17:40:57.550242 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29(unlocked)] enter Initial
-12> 2016-02-02 17:40:57.681730 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29( v 50500'14108 (50243'11080,50500'14108] local-les=50551 n=3381 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'14106 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.131488 0 0.000000
-11> 2016-02-02 17:40:57.683480 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29( v 50500'14108 (50243'11080,50500'14108] local-les=50551 n=3381 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'14106 lcod 0'0 mlcod 0'0 inactive] enter Reset
-10> 2016-02-02 17:40:57.691238 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b(unlocked)] enter Initial
-9> 2016-02-02 17:40:57.827298 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b( v 50508'13549 (50243'10496,50508'13549] local-les=50551 n=3377 ec=1 les/c 50551/50551 50550/50550/50550) [8,7] r=0 lpr=0 crt=50508'13549 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.136060 0 0.000000
-8> 2016-02-02 17:40:57.828249 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b( v 50508'13549 (50243'10496,50508'13549] local-les=50551 n=3377 ec=1 les/c 50551/50551 50550/50550/50550) [8,7] r=0 lpr=0 crt=50508'13549 lcod 0'0 mlcod 0'0 inactive] enter Reset
-7> 2016-02-02 17:40:57.832675 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d(unlocked)] enter Initial
-6> 2016-02-02 17:40:57.960588 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d( v 50541'13861 (50243'10807,50541'13861] local-les=50551 n=3353 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13861 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.127913 0 0.000000
-5> 2016-02-02 17:40:57.962296 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d( v 50541'13861 (50243'10807,50541'13861] local-les=50551 n=3353 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13861 lcod 0'0 mlcod 0'0 inactive] enter Reset
-4> 2016-02-02 17:40:57.968125 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e(unlocked)] enter Initial
-3> 2016-02-02 17:40:58.127859 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e( v 50537'13473 (50243'10454,50537'13473] local-les=50551 n=3301 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13473 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.159735 0 0.000000
-2> 2016-02-02 17:40:58.129330 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e( v 50537'13473 (50243'10454,50537'13473] local-les=50551 n=3301 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13473 lcod 0'0 mlcod 0'0 inactive] enter Reset
-1> 2016-02-02 17:40:58.130892 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.32(unlocked)] enter Initial
0> 2016-02-02 17:40:58.200696 7f9a3f56e800 -1 ** Caught signal (Aborted) *
in thread 7f9a3f56e800

ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xa4c1c5]
 2: (()+0xf790) [0x7f9a3e2ca790]
 3: (gsignal()+0x35) [0x7f9a3cf94625]
 4: (abort()+0x175) [0x7f9a3cf95e05]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x7f9a3d84ea7d]
 6: (()+0xbcbd6) [0x7f9a3d84cbd6]
 7: (()+0xbcc03) [0x7f9a3d84cc03]
 8: (()+0xbcd22) [0x7f9a3d84cd22]
 9: (pg_log_entry_t::decode_with_checksum(ceph::buffer::list::iterator&)+0x12d) [0x7b074d]
 10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map&lt;eversion_t, hobject_t, std::less&lt;eversion_t&gt;, std::allocator&lt;std::pair&lt;eversion_t const, hobject_t&gt; > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >&, std::set&lt;std::string, std::less&lt;std::string&gt;, std::allocator&lt;std::string&gt; >)+0xb74) [0x7ccef4]
 11: (PG::read_state(ObjectStore, ceph::buffer::list&)+0x34c) [0x82962c]
 12: (OSD::load_pgs()+0x1646) [0x688b96]
 13: (OSD::init()+0x176e) [0x68caee]
 14: (main()+0x384f) [0x62eb8f]
 15: (__libc_start_main()+0xfd) [0x7f9a3cf80d5d]
 16: /usr/bin/ceph-osd() [0x62a299]
 NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
-- end dump of recent events ---

Actions
Copy link
 #3

Updated by Johannes Erdfelt about 8 years ago

I appear to have run into this same problem, but I am running hammer:

ceph-0.94.5-0.el6.x86_64

I have one OSD that won't stay running now:

{{{
2016-02-02 17:40:50.191900 7f9a3f56e800 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5642
2016-02-02 17:40:50.630201 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) backend generic (magic 0xef53)
2016-02-02 17:40:51.393738 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is supported and appears to work
2016-02-02 17:40:51.395129 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
2016-02-02 17:40:51.470464 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: syscall(SYS_syncfs, fd) fully supported
2016-02-02 17:40:51.601887 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) limited size xattrs
2016-02-02 17:40:51.825407 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
2016-02-02 17:40:52.182720 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
2016-02-02 17:40:52.250622 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
2016-02-02 17:40:52.313684 7f9a3f56e800 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
2016-02-02 17:40:52.337571 7f9a3f56e800 0 osd.8 50551 crush map has features 69793480704, adjusting msgr requires for clients
2016-02-02 17:40:52.339033 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648 was 8705, adjusting msgr requires for mons
2016-02-02 17:40:52.339115 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648, adjusting msgr requires for osds
2016-02-02 17:40:52.339183 7f9a3f56e800 0 osd.8 50551 load_pgs
2016-02-02 17:40:58.200696 7f9a3f56e800 -1 ** Caught signal (Aborted) *
in thread 7f9a3f56e800

ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xa4c1c5]
 2: (()+0xf790) [0x7f9a3e2ca790]
 3: (gsignal()+0x35) [0x7f9a3cf94625]
 4: (abort()+0x175) [0x7f9a3cf95e05]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x7f9a3d84ea7d]
 6: (()+0xbcbd6) [0x7f9a3d84cbd6]
 7: (()+0xbcc03) [0x7f9a3d84cc03]
 8: (()+0xbcd22) [0x7f9a3d84cd22]
 9: (pg_log_entry_t::decode_with_checksum(ceph::buffer::list::iterator&)+0x12d) [0x7b074d]
 10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map&lt;eversion_t, hobject_t, std::less&lt;eversion_t&gt;, std::allocator&lt;std::pair&lt;eversion_t const, hobject_t&gt; > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >&, std::set&lt;std::string, std::less&lt;std::string&gt;, std::allocator&lt;std::string&gt; >)+0xb74) [0x7ccef4]
 11: (PG::read_state(ObjectStore, ceph::buffer::list&)+0x34c) [0x82962c]
 12: (OSD::load_pgs()+0x1646) [0x688b96]
 13: (OSD::init()+0x176e) [0x68caee]
 14: (main()+0x384f) [0x62eb8f]
 15: (__libc_start_main()+0xfd) [0x7f9a3cf80d5d]
 16: /usr/bin/ceph-osd() [0x62a299]
 NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- begin dump of recent events ---
130> 2016-02-02 17:40:50.171206 7f9a3f56e800 5 asok(0x5118000) register_command perfcounters_dump hook 0x5088050
-129> 2016-02-02 17:40:50.171281 7f9a3f56e800 5 asok(0x5118000) register_command 1 hook 0x5088050
-128> 2016-02-02 17:40:50.171292 7f9a3f56e800 5 asok(0x5118000) register_command perf dump hook 0x5088050
-127> 2016-02-02 17:40:50.171307 7f9a3f56e800 5 asok(0x5118000) register_command perfcounters_schema hook 0x5088050
-126> 2016-02-02 17:40:50.171315 7f9a3f56e800 5 asok(0x5118000) register_command 2 hook 0x5088050
-125> 2016-02-02 17:40:50.171321 7f9a3f56e800 5 asok(0x5118000) register_command perf schema hook 0x5088050
-124> 2016-02-02 17:40:50.171328 7f9a3f56e800 5 asok(0x5118000) register_command perf reset hook 0x5088050
-123> 2016-02-02 17:40:50.171333 7f9a3f56e800 5 asok(0x5118000) register_command config show hook 0x5088050
-122> 2016-02-02 17:40:50.171339 7f9a3f56e800 5 asok(0x5118000) register_command config set hook 0x5088050
-121> 2016-02-02 17:40:50.171346 7f9a3f56e800 5 asok(0x5118000) register_command config get hook 0x5088050
-120> 2016-02-02 17:40:50.171351 7f9a3f56e800 5 asok(0x5118000) register_command config diff hook 0x5088050
-119> 2016-02-02 17:40:50.171357 7f9a3f56e800 5 asok(0x5118000) register_command log flush hook 0x5088050
-118> 2016-02-02 17:40:50.171363 7f9a3f56e800 5 asok(0x5118000) register_command log dump hook 0x5088050
-117> 2016-02-02 17:40:50.171368 7f9a3f56e800 5 asok(0x5118000) register_command log reopen hook 0x5088050
-116> 2016-02-02 17:40:50.191900 7f9a3f56e800 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process ceph-osd, pid 5642
-115> 2016-02-02 17:40:50.237948 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6804/5642 need_addr=1
-114> 2016-02-02 17:40:50.241932 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6806/5642 need_addr=1
-113> 2016-02-02 17:40:50.248035 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6807/5642 need_addr=1
-112> 2016-02-02 17:40:50.253372 7f9a3f56e800 1 accepter.accepter.bind my_inst.addr is 0.0.0.0:6808/5642 need_addr=1
-111> 2016-02-02 17:40:50.262560 7f9a3f56e800 1 finished global_init_daemonize
-110> 2016-02-02 17:40:50.490710 7f9a3f56e800 5 asok(0x5118000) init /var/run/ceph/ceph-osd.8.asok
-109> 2016-02-02 17:40:50.492565 7f9a3f56e800 5 asok(0x5118000) bind_and_listen /var/run/ceph/ceph-osd.8.asok
-108> 2016-02-02 17:40:50.496863 7f9a3f56e800 5 asok(0x5118000) register_command 0 hook 0x50800b0
-107> 2016-02-02 17:40:50.497582 7f9a3f56e800 5 asok(0x5118000) register_command version hook 0x50800b0
-106> 2016-02-02 17:40:50.497611 7f9a3f56e800 5 asok(0x5118000) register_command git_version hook 0x50800b0
-105> 2016-02-02 17:40:50.497624 7f9a3f56e800 5 asok(0x5118000) register_command help hook 0x5088140
-104> 2016-02-02 17:40:50.497652 7f9a3f56e800 5 asok(0x5118000) register_command get_command_descriptions hook 0x5088130
-103> 2016-02-02 17:40:50.499699 7f9a3f56e800 10 monclient(hunting): build_initial_monmap
-102> 2016-02-02 17:40:50.500254 7f9a3a8e5700 5 asok(0x5118000) entry start
-101> 2016-02-02 17:40:50.598615 7f9a3f56e800 5 adding auth protocol: cephx
-100> 2016-02-02 17:40:50.600147 7f9a3f56e800 5 adding auth protocol: cephx
-99> 2016-02-02 17:40:50.600590 7f9a3f56e800 5 asok(0x5118000) register_command objecter_requests hook 0x5088170
-98> 2016-02-02 17:40:50.605820 7f9a3f56e800 1 - 0.0.0.0:6804/5642 messenger.start
97> 2016-02-02 17:40:50.607473 7f9a3f56e800 1 - :/0 messenger.start
96> 2016-02-02 17:40:50.609772 7f9a3f56e800 1 - 0.0.0.0:6808/5642 messenger.start
95> 2016-02-02 17:40:50.612225 7f9a3f56e800 1 - 0.0.0.0:6807/5642 messenger.start
94> 2016-02-02 17:40:50.614750 7f9a3f56e800 1 - 0.0.0.0:6806/5642 messenger.start
93> 2016-02-02 17:40:50.618250 7f9a3f56e800 1 - :/0 messenger.start
-92> 2016-02-02 17:40:50.625307 7f9a3f56e800 2 osd.8 0 mounting /data/osd8/ceph/data /var/lib/ceph/osd/osd.8/journal
-91> 2016-02-02 17:40:50.630201 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) backend generic (magic 0xef53)
-90> 2016-02-02 17:40:51.393738 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is supported and appears to work
-89> 2016-02-02 17:40:51.395129 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option
-88> 2016-02-02 17:40:51.470464 7f9a3f56e800 0 genericfilestorebackend(/data/osd8/ceph/data) detect_features: syscall(SYS_syncfs, fd) fully supported
-87> 2016-02-02 17:40:51.601887 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) limited size xattrs
-86> 2016-02-02 17:40:51.825407 7f9a3f56e800 0 filestore(/data/osd8/ceph/data) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled
-85> 2016-02-02 17:40:52.110985 7f9a3f56e800 2 journal open /var/lib/ceph/osd/osd.8/journal fsid 2988ec18-fcc1-4aa6-85de-cb8adc56a007 fs_op_seq 4126731
-84> 2016-02-02 17:40:52.182720 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
-83> 2016-02-02 17:40:52.184238 7f9a3f56e800 2 journal open advancing committed_seq 4126730 to fs op_seq 4126731
-82> 2016-02-02 17:40:52.186354 7f9a3f56e800 2 journal read_entry 12788428800 : seq 4126731 960 bytes
-81> 2016-02-02 17:40:52.188012 7f9a3f56e800 2 journal No further valid entries found, journal is most likely valid
-80> 2016-02-02 17:40:52.190625 7f9a3f56e800 2 journal No further valid entries found, journal is most likely valid
-79> 2016-02-02 17:40:52.191045 7f9a3f56e800 3 journal journal_replay: end of journal, done.
-78> 2016-02-02 17:40:52.250622 7f9a3f56e800 1 journal _open /var/lib/ceph/osd/osd.8/journal fd 19: 12884901888 bytes, block size 4096 bytes, directio = 1, aio = 1
-77> 2016-02-02 17:40:52.262409 7f9a3f56e800 2 osd.8 0 boot
-76> 2016-02-02 17:40:52.271472 7f9a3f56e800 1 <cls> cls/replica_log/cls_replica_log.cc:141: Loaded replica log class!
-75> 2016-02-02 17:40:52.276610 7f9a3f56e800 1 <cls> cls/log/cls_log.cc:312: Loaded log class!
-74> 2016-02-02 17:40:52.280916 7f9a3f56e800 1 <cls> cls/user/cls_user.cc:367: Loaded user class!
-73> 2016-02-02 17:40:52.291492 7f9a3f56e800 1 <cls> cls/statelog/cls_statelog.cc:306: Loaded log class!
-72> 2016-02-02 17:40:52.301374 7f9a3f56e800 1 <cls> cls/version/cls_version.cc:227: Loaded version class!
-71> 2016-02-02 17:40:52.309660 7f9a3f56e800 1 <cls> cls/refcount/cls_refcount.cc:231: Loaded refcount class!
-70> 2016-02-02 17:40:52.313684 7f9a3f56e800 0 <cls> cls/hello/cls_hello.cc:271: loading cls_hello
-69> 2016-02-02 17:40:52.330932 7f9a3f56e800 1 <cls> cls/rgw/cls_rgw.cc:3047: Loaded rgw class!
-68> 2016-02-02 17:40:52.337571 7f9a3f56e800 0 osd.8 50551 crush map has features 69793480704, adjusting msgr requires for clients
-67> 2016-02-02 17:40:52.339033 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648 was 8705, adjusting msgr requires for mons
-66> 2016-02-02 17:40:52.339115 7f9a3f56e800 0 osd.8 50551 crush map has features 344671387648, adjusting msgr requires for osds
-65> 2016-02-02 17:40:52.339183 7f9a3f56e800 0 osd.8 50551 load_pgs
-64> 2016-02-02 17:40:54.981646 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2(unlocked)] enter Initial
-63> 2016-02-02 17:40:55.140037 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2( v 50545'13223 (50235'10149,50545'13223] local-les=50551 n=3321 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13223 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.158392 0 0.000000
-62> 2016-02-02 17:40:55.140902 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2( v 50545'13223 (50235'10149,50545'13223] local-les=50551 n=3321 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13223 lcod 0'0 mlcod 0'0 inactive] enter Reset
-61> 2016-02-02 17:40:55.144239 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3(unlocked)] enter Initial
-60> 2016-02-02 17:40:55.247112 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3( v 50541'13961 (50243'10920,50541'13961] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13961 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.102861 0 0.000000
-59> 2016-02-02 17:40:55.248870 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.3( v 50541'13961 (50243'10920,50541'13961] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13961 lcod 0'0 mlcod 0'0 inactive] enter Reset
-58> 2016-02-02 17:40:55.251434 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5(unlocked)] enter Initial
-57> 2016-02-02 17:40:55.456313 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5( v 50500'13626 (50243'10567,50500'13626] local-les=50551 n=3269 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50500'13624 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.204879 0 0.000000
-56> 2016-02-02 17:40:55.457458 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.5( v 50500'13626 (50243'10567,50500'13626] local-les=50551 n=3269 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50500'13624 lcod 0'0 mlcod 0'0 inactive] enter Reset
-55> 2016-02-02 17:40:55.466063 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9(unlocked)] enter Initial
-54> 2016-02-02 17:40:55.618691 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9( v 50537'13519 (50235'10469,50537'13519] local-les=50551 n=3280 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50537'13519 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.152628 0 0.000000
-53> 2016-02-02 17:40:55.619974 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.9( v 50537'13519 (50235'10469,50537'13519] local-les=50551 n=3280 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50537'13519 lcod 0'0 mlcod 0'0 inactive] enter Reset
-52> 2016-02-02 17:40:55.625042 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e(unlocked)] enter Initial
-51> 2016-02-02 17:40:55.755310 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e( v 50537'13216 (50243'10179,50537'13216] local-les=50551 n=3311 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13216 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.130268 0 0.000000
-50> 2016-02-02 17:40:55.756944 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.e( v 50537'13216 (50243'10179,50537'13216] local-les=50551 n=3311 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13216 lcod 0'0 mlcod 0'0 inactive] enter Reset
-49> 2016-02-02 17:40:55.763638 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10(unlocked)] enter Initial
-48> 2016-02-02 17:40:55.895574 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10( v 50545'13931 (50243'10880,50545'13931] local-les=50551 n=3389 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13931 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.131937 0 0.000000
-47> 2016-02-02 17:40:55.897028 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.10( v 50545'13931 (50243'10880,50545'13931] local-les=50551 n=3389 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50545'13931 lcod 0'0 mlcod 0'0 inactive] enter Reset
-46> 2016-02-02 17:40:55.900392 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12(unlocked)] enter Initial
-45> 2016-02-02 17:40:56.078102 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12( v 50541'13782 (50235'10737,50541'13782] local-les=50551 n=3334 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50541'13782 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.177710 0 0.000000
-44> 2016-02-02 17:40:56.079598 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.12( v 50541'13782 (50235'10737,50541'13782] local-les=50551 n=3334 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50541'13782 lcod 0'0 mlcod 0'0 inactive] enter Reset
-43> 2016-02-02 17:40:56.081403 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13(unlocked)] enter Initial
-42> 2016-02-02 17:40:56.216696 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13( v 50500'13547 (50235'10507,50500'13547] local-les=50551 n=3372 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13545 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.135292 0 0.000000
-41> 2016-02-02 17:40:56.218669 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.13( v 50500'13547 (50235'10507,50500'13547] local-les=50551 n=3372 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13545 lcod 0'0 mlcod 0'0 inactive] enter Reset
-40> 2016-02-02 17:40:56.226157 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14(unlocked)] enter Initial
-39> 2016-02-02 17:40:56.363115 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14( v 50539'13379 (50243'10324,50539'13379] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13376 lcod 0'0 inactive NOTIFY] exit Initial 0.136958 0 0.000000
-38> 2016-02-02 17:40:56.365358 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.14( v 50539'13379 (50243'10324,50539'13379] local-les=50551 n=3344 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13376 lcod 0'0 inactive NOTIFY] enter Reset
-37> 2016-02-02 17:40:56.368174 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16(unlocked)] enter Initial
-36> 2016-02-02 17:40:56.521665 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16( v 50500'13405 (50243'10374,50500'13405] local-les=50551 n=3219 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13403 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.153492 0 0.000000
-35> 2016-02-02 17:40:56.522516 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.16( v 50500'13405 (50243'10374,50500'13405] local-les=50551 n=3219 ec=1 les/c 50551/50551 50550/50550/50550) [8,5] r=0 lpr=0 crt=50500'13403 lcod 0'0 mlcod 0'0 inactive] enter Reset
-34> 2016-02-02 17:40:56.530583 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19(unlocked)] enter Initial
-33> 2016-02-02 17:40:56.654350 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19( v 50500'13898 (50243'10881,50500'13898] local-les=50551 n=3380 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'13896 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.123767 0 0.000000
-32> 2016-02-02 17:40:56.656036 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.19( v 50500'13898 (50243'10881,50500'13898] local-les=50551 n=3380 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'13896 lcod 0'0 mlcod 0'0 inactive] enter Reset
-31> 2016-02-02 17:40:56.661445 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a(unlocked)] enter Initial
-30> 2016-02-02 17:40:56.781691 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a( v 50543'13168 (50235'10120,50543'13168] local-les=50551 n=3228 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13165 lcod 0'0 inactive NOTIFY] exit Initial 0.120247 0 0.000000
-29> 2016-02-02 17:40:56.782016 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1a( v 50543'13168 (50235'10120,50543'13168] local-les=50551 n=3228 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13165 lcod 0'0 inactive NOTIFY] enter Reset
-28> 2016-02-02 17:40:56.787180 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b(unlocked)] enter Initial
-27> 2016-02-02 17:40:56.960516 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b( v 50500'13498 (50235'10441,50500'13498] local-les=50551 n=3287 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13496 lcod 0'0 inactive NOTIFY] exit Initial 0.173337 0 0.000000
-26> 2016-02-02 17:40:56.962869 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1b( v 50500'13498 (50235'10441,50500'13498] local-les=50551 n=3287 ec=1 les/c 50551/50551 50550/50550/50528) [5,8] r=1 lpr=0 pi=50524-50549/9 crt=50500'13496 lcod 0'0 inactive NOTIFY] enter Reset
-25> 2016-02-02 17:40:56.970848 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f(unlocked)] enter Initial
-24> 2016-02-02 17:40:57.086203 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f( v 50500'13670 (50243'10660,50500'13670] local-les=50551 n=3291 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'13668 lcod 0'0 inactive NOTIFY] exit Initial 0.115356 0 0.000000
-23> 2016-02-02 17:40:57.087869 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.1f( v 50500'13670 (50243'10660,50500'13670] local-les=50551 n=3291 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'13668 lcod 0'0 inactive NOTIFY] enter Reset
-22> 2016-02-02 17:40:57.095066 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21(unlocked)] enter Initial
-21> 2016-02-02 17:40:57.229652 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21( v 50500'14107 (50243'11099,50500'14107] local-les=50551 n=3431 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'14105 lcod 0'0 inactive NOTIFY] exit Initial 0.134585 0 0.000000
-20> 2016-02-02 17:40:57.231151 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.21( v 50500'14107 (50243'11099,50500'14107] local-les=50551 n=3431 ec=1 les/c 50551/50551 50550/50550/50509) [0,8] r=1 lpr=0 pi=50501-50549/11 crt=50500'14105 lcod 0'0 inactive NOTIFY] enter Reset
-19> 2016-02-02 17:40:57.236505 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24(unlocked)] enter Initial
-18> 2016-02-02 17:40:57.366887 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24( v 50543'13657 (50235'10603,50543'13657] local-les=50551 n=3394 ec=1 les/c 50551/50551 50550/50550/50530) [7,8] r=1 lpr=0 pi=50526-50549/9 crt=50500'13654 lcod 0'0 inactive NOTIFY] exit Initial 0.130383 0 0.000000
-17> 2016-02-02 17:40:57.368625 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.24( v 50543'13657 (50235'10603,50543'13657] local-les=50551 n=3394 ec=1 les/c 50551/50551 50550/50550/50530) [7,8] r=1 lpr=0 pi=50526-50549/9 crt=50500'13654 lcod 0'0 inactive NOTIFY] enter Reset
-16> 2016-02-02 17:40:57.372953 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27(unlocked)] enter Initial
-15> 2016-02-02 17:40:57.540487 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27( v 50500'13527 (50235'10511,50500'13527] local-les=50551 n=3349 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13525 lcod 0'0 inactive NOTIFY] exit Initial 0.167534 0 0.000000
-14> 2016-02-02 17:40:57.542029 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.27( v 50500'13527 (50235'10511,50500'13527] local-les=50551 n=3349 ec=1 les/c 50551/50551 50550/50550/50516) [1,8] r=1 lpr=0 pi=50503-50549/11 crt=50500'13525 lcod 0'0 inactive NOTIFY] enter Reset
-13> 2016-02-02 17:40:57.550242 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29(unlocked)] enter Initial
-12> 2016-02-02 17:40:57.681730 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29( v 50500'14108 (50243'11080,50500'14108] local-les=50551 n=3381 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'14106 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.131488 0 0.000000
-11> 2016-02-02 17:40:57.683480 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.29( v 50500'14108 (50243'11080,50500'14108] local-les=50551 n=3381 ec=1 les/c 50551/50551 50550/50550/50550) [8,1] r=0 lpr=0 crt=50500'14106 lcod 0'0 mlcod 0'0 inactive] enter Reset
-10> 2016-02-02 17:40:57.691238 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b(unlocked)] enter Initial
-9> 2016-02-02 17:40:57.827298 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b( v 50508'13549 (50243'10496,50508'13549] local-les=50551 n=3377 ec=1 les/c 50551/50551 50550/50550/50550) [8,7] r=0 lpr=0 crt=50508'13549 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.136060 0 0.000000
-8> 2016-02-02 17:40:57.828249 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2b( v 50508'13549 (50243'10496,50508'13549] local-les=50551 n=3377 ec=1 les/c 50551/50551 50550/50550/50550) [8,7] r=0 lpr=0 crt=50508'13549 lcod 0'0 mlcod 0'0 inactive] enter Reset
-7> 2016-02-02 17:40:57.832675 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d(unlocked)] enter Initial
-6> 2016-02-02 17:40:57.960588 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d( v 50541'13861 (50243'10807,50541'13861] local-les=50551 n=3353 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13861 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.127913 0 0.000000
-5> 2016-02-02 17:40:57.962296 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2d( v 50541'13861 (50243'10807,50541'13861] local-les=50551 n=3353 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50541'13861 lcod 0'0 mlcod 0'0 inactive] enter Reset
-4> 2016-02-02 17:40:57.968125 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e(unlocked)] enter Initial
-3> 2016-02-02 17:40:58.127859 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e( v 50537'13473 (50243'10454,50537'13473] local-les=50551 n=3301 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13473 lcod 0'0 mlcod 0'0 inactive] exit Initial 0.159735 0 0.000000
-2> 2016-02-02 17:40:58.129330 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.2e( v 50537'13473 (50243'10454,50537'13473] local-les=50551 n=3301 ec=1 les/c 50551/50551 50550/50550/50550) [8,0] r=0 lpr=0 crt=50537'13473 lcod 0'0 mlcod 0'0 inactive] enter Reset
-1> 2016-02-02 17:40:58.130892 7f9a3f56e800 5 osd.8 pg_epoch: 50551 pg[0.32(unlocked)] enter Initial
0> 2016-02-02 17:40:58.200696 7f9a3f56e800 -1 ** Caught signal (Aborted) *
in thread 7f9a3f56e800

ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
 1: /usr/bin/ceph-osd() [0xa4c1c5]
 2: (()+0xf790) [0x7f9a3e2ca790]
 3: (gsignal()+0x35) [0x7f9a3cf94625]
 4: (abort()+0x175) [0x7f9a3cf95e05]
 5: (_gnu_cxx::_verbose_terminate_handler()+0x12d) [0x7f9a3d84ea7d]
 6: (()+0xbcbd6) [0x7f9a3d84cbd6]
 7: (()+0xbcc03) [0x7f9a3d84cc03]
 8: (()+0xbcd22) [0x7f9a3d84cd22]
 9: (pg_log_entry_t::decode_with_checksum(ceph::buffer::list::iterator&)+0x12d) [0x7b074d]
 10: (PGLog::read_log(ObjectStore*, coll_t, coll_t, ghobject_t, pg_info_t const&, std::map&lt;eversion_t, hobject_t, std::less&lt;eversion_t&gt;, std::allocator&lt;std::pair&lt;eversion_t const, hobject_t&gt; > >&, PGLog::IndexedLog&, pg_missing_t&, std::basic_ostringstream&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; >&, std::set&lt;std::string, std::less&lt;std::string&gt;, std::allocator&lt;std::string&gt; >)+0xb74) [0x7ccef4]
 11: (PG::read_state(ObjectStore, ceph::buffer::list&)+0x34c) [0x82962c]
 12: (OSD::load_pgs()+0x1646) [0x688b96]
 13: (OSD::init()+0x176e) [0x68caee]
 14: (main()+0x384f) [0x62eb8f]
 15: (__libc_start_main()+0xfd) [0x7f9a3cf80d5d]
 16: /usr/bin/ceph-osd() [0x62a299]
 NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.

--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 keyvaluestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/ceph-osd.8.log
-- end dump of recent events ---
}}}

Actions
Copy link
 #4

Updated by Johannes Erdfelt about 8 years ago

Ugh, sorry for the duplicate comment. I was trying to edit my previous comment to fix the formatting and not only did it post a new comment, but it also the attempt to fix the formatting failed too.

I should add that this host had a power failure recently. The file system the OSD is on had some file system errors and that resulted in quite a few inconsistent PGs (~100). I repaired all of the inconsistent PGs (which also ended up with a few corrupted objects as well). Since then, this OSD has been troublesome. Clients have been stalling but restarting this OSD usually gets the client "unstuck". This worked for a few days but now the OSD won't even start :(

Actions
Copy link

Also available in: Atom PDF