Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2016-06-07T02:05:06Z
Ceph
Redmine
rbd - Bug #16176 (Resolved): objectmap does not show object existence correctly
https://tracker.ceph.com/issues/16176
2016-06-07T02:05:06Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>in the latest master, i create 10GB rbd and use rbd bench-write to fill objects, and i used "ctrl + c" to terminate bench-write process, then i check objects,it shows there are 26 objects</p>
<p>[root@ctrl src]# ./rados -c /mnt/data/devs/ceph.conf -p rbd ls | grep rbd_data.113a74b0dc51 | wc -l<br />2016-06-07 10:02:45.681910 7f438e50ba40 -1 WARNING: the following dangerous and experimental features are enabled: *<br />2016-06-07 10:02:45.682067 7f438e50ba40 -1 WARNING: the following dangerous and experimental features are enabled: *<br />2016-06-07 10:02:45.713863 7f438e50ba40 -1 WARNING: the following dangerous and experimental features are enabled: *<br />26<br />[root@ctrl src]#</p>
<p>but when i use 'rbd du' to disk usage, it only shows that only 6 objects exist, and the objectmap flag is valid<br />NAME PROVISIONED USED<br />test 10240M 24576k</p>
<p>[root@ctrl src]# ./rbd -c /mnt/data/devs/ceph.conf -p rbd info test<br />2016-06-07 10:03:55.902227 7f6963a84d80 -1 WARNING: the following dangerous and experimental features are enabled: *<br />2016-06-07 10:03:55.902758 7f6963a84d80 -1 WARNING: the following dangerous and experimental features are enabled: *<br />2016-06-07 10:03:55.935078 7f6963a84d80 -1 WARNING: the following dangerous and experimental features are enabled: *<br />rbd image 'test':<br /> size 10240 MB in 2560 objects<br /> order 22 (4096 kB objects)<br /> block_name_prefix: rbd_data.113a74b0dc51<br /> format: 2<br /> features: layering, exclusive-lock, object-map, fast-diff, deep-flatten<br /> flags:</p>
Ceph - Feature #15346 (Resolved): add error handle for leveldb or rocksdb in bluestore or filestore
https://tracker.ceph.com/issues/15346
2016-04-01T02:31:28Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>recently i tested bluestore, i met error that rocksdb corruptted(submit_transaction always return -1)but ceph does not handle this error.</p>
Ceph - Bug #15298 (Can't reproduce): rocksdb corrupt with bluefs
https://tracker.ceph.com/issues/15298
2016-03-29T07:22:05Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>2016-03-29 09:09:18.236160 7f821c7ff700 4 rocksdb: (Original Log Time 2016/03/29-09:09:18.236050) EVENT_LOG_v1 {"time_micros": 1459213758236023, "job": 11158, "event": "compaction_finished", "compaction_time_micros": 3262213, "output_level": 1, "num_output_files": 8, "total_output_size": 12833518, "num_input_records": 79130, "num_output_records": 68472, "num_subcompactions": 1, "lsm_state": [0, 8, 82, 520, 367, 0, 0]}<br />2016-03-29 09:09:18.236439 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236435, "job": 11158, "event": "table_file_deletion", "file_number": 72280}<br />2016-03-29 09:09:18.236464 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236462, "job": 11158, "event": "table_file_deletion", "file_number": 72025}<br />2016-03-29 09:09:18.236489 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236487, "job": 11158, "event": "table_file_deletion", "file_number": 72008}<br />2016-03-29 09:09:18.236506 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236504, "job": 11158, "event": "table_file_deletion", "file_number": 72006}<br />2016-03-29 09:09:18.236521 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236519, "job": 11158, "event": "table_file_deletion", "file_number": 72005}<br />2016-03-29 09:09:18.236536 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236535, "job": 11158, "event": "table_file_deletion", "file_number": 72004}<br />2016-03-29 09:09:18.236568 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236566, "job": 11158, "event": "table_file_deletion", "file_number": 72002}<br />2016-03-29 09:09:18.236584 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236583, "job": 11158, "event": "table_file_deletion", "file_number": 72001}<br />2016-03-29 09:09:18.236600 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236599, "job": 11158, "event": "table_file_deletion", "file_number": 71998}<br />2016-03-29 09:09:18.236617 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236615, "job": 11158, "event": "table_file_deletion", "file_number": 71996}<br />2016-03-29 09:09:18.236631 7f821c7ff700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1459213758236629, "job": 11158, "event": "table_file_deletion", "file_number": 71995}<br />2016-03-29 09:09:18.236716 7f821c7ff700 2 rocksdb: Waiting after background compaction error: NotFound: , Accumulated background error counts: 1</p>
rbd - Bug #13558 (Resolved): rbd : enable feature objectmap
https://tracker.ceph.com/issues/13558
2015-10-21T11:19:58Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>how to produce this bug:</p>
<p>rbd -p rbd create --new-format --size 1024 test1<br />rbd feature enable test1 exclusive-lock<br />rbd feature enable test1 object-map<br />2015-10-21 18:52:48.261482 7f3a83a477c0 -1 librbd::ObjectMap: error refreshing object map: (2) No such file or directory</p>
rbd - Bug #13553 (Resolved): rbd clone issue
https://tracker.ceph.com/issues/13553
2015-10-21T07:19:11Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>rbd CLI : rbd clone --image-feature layering rbd/test@snap rbd/clone</p>
<p>output : rbd: image format can only be set when creating or importing an image</p>
Ceph - Bug #12637 (Resolved): bug of rados ls with keyvaluestore
https://tracker.ceph.com/issues/12637
2015-08-06T16:08:16Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>if leveldb is usedas keyvaluestore backend , run 'rados -p rbd ls', then cluster crash , log shows as follow : </p>
<pre><code><del>5> 2015-08-06 15:49:53.176711 7f38a4038700 10 osd.0 pg_epoch: 12 pg[0.0( empty local-les=12 n=0 ec=1 les/c 12/12 11/11/9) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean] handle_message: 0x3b13d00<br /> -4> 2015-08-06 15:49:53.176728 7f38a4038700 10 osd.0 pg_epoch: 12 pg[0.0( empty local-les=12 n=0 ec=1 les/c 12/12 11/11/9) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean] do_pg_op osd_op(client.14108.0:1 [pgls start_epoch 0] 0.0 ack+read+known_if_redirected e12) v5<br /> -3> 2015-08-06 15:49:53.176744 7f38a4038700 5 -</del> op tracker -- seq: 63, time: 2015-08-06 15:49:53.176744, event: started, op: osd_op(client.14108.0:1 [pgls start_epoch 0] 0.0 ack+read+known_if_redirected e12)<br /> -2> 2015-08-06 15:49:53.176760 7f38a4038700 10 osd.0 pg_epoch: 12 pg[0.0( empty local-les=12 n=0 ec=1 les/c 12/12 11/11/9) [0,1] r=0 lpr=11 crt=0'0 mlcod 0'0 active+clean] pgls pg=0.0 count 1024<br /> -1> 2015-08-06 15:49:53.176781 7f38a4038700 10 collection_list_partial 0.0_head start:MIN/0 is_max:0<br /> 0> 2015-08-06 15:49:53.193495 7f38a4038700 -1 os/GenericObjectMap.cc: In function 'static std::string GenericObjectMap::header_key(const coll_t&, const ghobject_t&)' thread 7f38a4038700 time 2015-08-06 15:49:53.176820<br />os/GenericObjectMap.cc: 160: FAILED assert(oid.shard_id != shard_id_t::NO_SHARD)</code></pre>
<pre><code>ceph version 9.0.1-1988-gb63f2af (b63f2afadbc3b4436c936c5a9b62dcd416ea3afe)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xb1443b]<br /> 2: (GenericObjectMap::header_key(coll_t const&, ghobject_t const&)+0x3e9) [0x9fe4c9]<br /> 3: (GenericObjectMap::list_objects(coll_t const&, ghobject_t, int, std::vector&lt;ghobject_t, std::allocator&lt;ghobject_t&gt; ><strong>, ghobject_t</strong>)+0x129) [0xa06fd9]<br /> 4: (KeyValueStore::collection_list_partial(coll_t, ghobject_t, int, int, snapid_t, std::vector&lt;ghobject_t, std::allocator&lt;ghobject_t&gt; ><strong>, ghobject_t</strong>)+0x1ff) [0x91b06f]<br /> 5: (PGBackend::objects_list_partial(hobject_t const&, int, int, snapid_t, std::vector&lt;hobject_t, std::allocator&lt;hobject_t&gt; ><strong>, hobject_t</strong>)+0x227) [0x8b05a7]<br /> 6: (ReplicatedPG::do_pg_op(std::tr1::shared_ptr&lt;OpRequest&gt;)+0x102a) [0x85265a]<br /> 7: (ReplicatedPG::do_op(std::tr1::shared_ptr&lt;OpRequest&gt;&)+0x1eb) [0x877a2b]<br /> 8: (ReplicatedPG::do_request(std::tr1::shared_ptr&lt;OpRequest&gt;&, ThreadPool::TPHandle&)+0x58c) [0x8155dc]<br /> 9: (OSD::dequeue_op(boost::intrusive_ptr&lt;PG&gt;, std::tr1::shared_ptr&lt;OpRequest&gt;, ThreadPool::TPHandle&)+0x3bd) [0x6719cd]<br /> 10: (PGQueueable::RunVis::operator()(std::tr1::shared_ptr&lt;OpRequest&gt;&)+0x5d) [0x671c1d]<br /> 11: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x8de) [0x69625e]<br /> 12: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x875) [0xb03e05]<br /> 13: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xb05f20]<br /> 14: (()+0x8182) [0x7f38d4344182]<br /> 15: (clone()+0x6d) [0x7f38d250447d]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
Ceph - Bug #11695 (Won't Fix): keyvaluestore: can not overwrite object after snapshot is created
https://tracker.ceph.com/issues/11695
2015-05-21T13:28:32Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>recently i build a setup with key-value db back, but after snapshot is created, i overwrite one object, then OSD daemon is corrupt, I track down this issue , in KeyValueStore.cc</p>
<pre><code>if (exist_clone) {<br /> dout(0) << "BUG: clone failed will lead to paritial transaction applied" << dendl;<br /> }</code></pre>
<p>It seem that KeyValueStore::_clone do not implement this semantic as filestore does, you can reproduce this error with following commands</p>
<p>1. rbd -p rbd --new-format --size 10240 create test <br />2. write data to rbd <br />3. rbd snap create rbd/test@snap <br />4. overwrite previous object</p>
Ceph - Feature #11028 (In Progress): LMDB key/value backend for Ceph
https://tracker.ceph.com/issues/11028
2015-03-05T08:56:14Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p><a class="external" href="https://wiki.ceph.com/Planning/Blueprints/Infernalis/LMDB_key%2F%2Fvalue_backend_for_Ceph">https://wiki.ceph.com/Planning/Blueprints/Infernalis/LMDB_key%2F%2Fvalue_backend_for_Ceph</a></p>
Ceph - Bug #9535 (Duplicate): monitor crashed after restarting
https://tracker.ceph.com/issues/9535
2014-09-19T01:41:42Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>recently when i restarted my ceph cluster , the monitor crashed , below is the output of monitor log</p>
<p>2014-09-19 11:29:30.568856 7f89c94077a0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mon, pid 39903<br />2014-09-19 11:29:30.743139 7f89c94077a0 0 starting mon.a rank 0 at 192.101.12.51:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid beb286e8-9c6f-4687-babe-1b8924479760<br />2014-09-19 11:29:30.743496 7f89c94077a0 1 mon.a@-1(probing) e1 preinit fsid beb286e8-9c6f-4687-babe-1b8924479760<br />2014-09-19 11:29:30.744615 7f89c94077a0 1 mon.a@-1(probing).paxosservice(pgmap 1229030..1229550) refresh upgraded, format 0 -> 1<br />2014-09-19 11:29:30.744637 7f89c94077a0 1 mon.a@-1(probing).pg v0 on_upgrade discarding in-core PGMap<br />2014-09-19 11:29:30.834870 7f89c94077a0 0 mon.a@-1(probing).mds e1 print_map<br />epoch 1<br />flags 0<br />created 2014-07-15 15:55:02.229706<br />modified 2014-07-15 15:55:02.229738<br />tableserver 0<br />root 0<br />session_timeout 60<br />session_autoclose 300<br />max_file_size 1099511627776<br />last_failure 0<br />last_failure_osd_epoch 0<br />compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap}<br />max_mds 1<br />in <br />up {}<br />failed <br />stopped <br />data_pools 0<br />metadata_pool 1<br />inline_data disabled</p>
<p>2014-09-19 11:29:30.835568 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br />2014-09-19 11:29:30.835579 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br />2014-09-19 11:29:30.835581 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br />2014-09-19 11:29:30.835582 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br />2014-09-19 11:29:30.836276 7f89c94077a0 1 mon.a@-1(probing).paxosservice(auth 1501..1695) refresh upgraded, format 0 -> 1<br />2014-09-19 11:29:30.837200 7f89c94077a0 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 7f89c94077a0 time 2014-09-19 11:29:30.836402<br />mon/AuthMonitor.cc: 155: FAILED assert(ret == 0)</p>
<pre><code>ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)<br /> 1: (AuthMonitor::update_from_paxos(bool*)+0x2611) [0x5e8041]<br /> 2: (PaxosService::refresh(bool*)+0x193) [0x530c43]<br /> 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x4ced07]<br /> 4: (Monitor::init_paxos()+0xd5) [0x4cee95]<br /> 5: (Monitor::preinit()+0x981) [0x4f3d51]<br /> 6: (main()+0x3caa) [0x4b620a]<br /> 7: (__libc_start_main()+0xfd) [0x376f41ecdd]<br /> 8: /usr/bin/ceph-mon() [0x4b0af9]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
<p>--- begin dump of recent events ---<br /> <del>36> 2014-09-19 11:29:30.563118 7f89c94077a0 5 asok(0x3b80e00) register_command perfcounters_dump hook 0x3b38010<br /> -35> 2014-09-19 11:29:30.563159 7f89c94077a0 5 asok(0x3b80e00) register_command 1 hook 0x3b38010<br /> -34> 2014-09-19 11:29:30.563191 7f89c94077a0 5 asok(0x3b80e00) register_command perf dump hook 0x3b38010<br /> -33> 2014-09-19 11:29:30.563198 7f89c94077a0 5 asok(0x3b80e00) register_command perfcounters_schema hook 0x3b38010<br /> -32> 2014-09-19 11:29:30.563204 7f89c94077a0 5 asok(0x3b80e00) register_command 2 hook 0x3b38010<br /> -31> 2014-09-19 11:29:30.563210 7f89c94077a0 5 asok(0x3b80e00) register_command perf schema hook 0x3b38010<br /> -30> 2014-09-19 11:29:30.563227 7f89c94077a0 5 asok(0x3b80e00) register_command config show hook 0x3b38010<br /> -29> 2014-09-19 11:29:30.563234 7f89c94077a0 5 asok(0x3b80e00) register_command config set hook 0x3b38010<br /> -28> 2014-09-19 11:29:30.563243 7f89c94077a0 5 asok(0x3b80e00) register_command config get hook 0x3b38010<br /> -27> 2014-09-19 11:29:30.563257 7f89c94077a0 5 asok(0x3b80e00) register_command log flush hook 0x3b38010<br /> -26> 2014-09-19 11:29:30.563264 7f89c94077a0 5 asok(0x3b80e00) register_command log dump hook 0x3b38010<br /> -25> 2014-09-19 11:29:30.563270 7f89c94077a0 5 asok(0x3b80e00) register_command log reopen hook 0x3b38010<br /> -24> 2014-09-19 11:29:30.568856 7f89c94077a0 0 ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74), process ceph-mon, pid 39903<br /> -23> 2014-09-19 11:29:30.571827 7f89c94077a0 5 asok(0x3b80e00) init /var/run/ceph/ceph-mon.a.asok<br /> -22> 2014-09-19 11:29:30.571868 7f89c94077a0 5 asok(0x3b80e00) bind_and_listen /var/run/ceph/ceph-mon.a.asok<br /> -21> 2014-09-19 11:29:30.572135 7f89c94077a0 5 asok(0x3b80e00) register_command 0 hook 0x3b30030<br /> -20> 2014-09-19 11:29:30.572159 7f89c94077a0 5 asok(0x3b80e00) register_command version hook 0x3b30030<br /> -19> 2014-09-19 11:29:30.572172 7f89c94077a0 5 asok(0x3b80e00) register_command git_version hook 0x3b30030<br /> -18> 2014-09-19 11:29:30.572180 7f89c94077a0 5 asok(0x3b80e00) register_command help hook 0x3b38030<br /> -17> 2014-09-19 11:29:30.572188 7f89c94077a0 5 asok(0x3b80e00) register_command get_command_descriptions hook 0x3b38090<br /> -16> 2014-09-19 11:29:30.572274 7f89c8404700 5 asok(0x3b80e00) entry start<br /> -15> 2014-09-19 11:29:30.743139 7f89c94077a0 0 starting mon.a rank 0 at 192.101.12.51:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid beb286e8-9c6f-4687-babe-1b8924479760<br /> -14> 2014-09-19 11:29:30.743237 7f89c94077a0 1 -</del> 192.101.12.51:6789/0 learned my addr 192.101.12.51:6789/0<br /> -13> 2014-09-19 11:29:30.743255 7f89c94077a0 1 accepter.accepter.bind my_inst.addr is 192.101.12.51:6789/0 need_addr=0<br /> -12> 2014-09-19 11:29:30.743282 7f89c94077a0 5 adding auth protocol: none<br /> -11> 2014-09-19 11:29:30.743289 7f89c94077a0 5 adding auth protocol: none<br /> -10> 2014-09-19 11:29:30.743496 7f89c94077a0 1 mon.a@-1(probing) e1 preinit fsid beb286e8-9c6f-4687-babe-1b8924479760<br /> -9> 2014-09-19 11:29:30.744615 7f89c94077a0 1 mon.a@-1(probing).paxosservice(pgmap 1229030..1229550) refresh upgraded, format 0 -> 1<br /> -8> 2014-09-19 11:29:30.744637 7f89c94077a0 1 mon.a@-1(probing).pg v0 on_upgrade discarding in-core PGMap<br /> -7> 2014-09-19 11:29:30.834858 7f89c94077a0 4 mon.a@-1(probing).mds e1 new map<br /> -6> 2014-09-19 11:29:30.834870 7f89c94077a0 0 mon.a@-1(probing).mds e1 print_map<br />epoch 1<br />flags 0<br />created 2014-07-15 15:55:02.229706<br />modified 2014-07-15 15:55:02.229738<br />tableserver 0<br />root 0<br />session_timeout 60<br />session_autoclose 300<br />max_file_size 1099511627776<br />last_failure 0<br />last_failure_osd_epoch 0<br />compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap}<br />max_mds 1<br />in <br />up {}<br />failed <br />stopped <br />data_pools 0<br />metadata_pool 1<br />inline_data disabled</p>
<pre><code>-5> 2014-09-19 11:29:30.835568 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br /> -4> 2014-09-19 11:29:30.835579 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br /> -3> 2014-09-19 11:29:30.835581 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br /> -2> 2014-09-19 11:29:30.835582 7f89c94077a0 0 mon.a@-1(probing).osd e2668 crush map has features 2200130813952, adjusting msgr requires<br /> -1> 2014-09-19 11:29:30.836276 7f89c94077a0 1 mon.a@-1(probing).paxosservice(auth 1501..1695) refresh upgraded, format 0 -> 1<br /> 0> 2014-09-19 11:29:30.837200 7f89c94077a0 -1 mon/AuthMonitor.cc: In function 'virtual void AuthMonitor::update_from_paxos(bool*)' thread 7f89c94077a0 time 2014-09-19 11:29:30.836402<br />mon/AuthMonitor.cc: 155: FAILED assert(ret == 0)</code></pre>
<pre><code>ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)<br /> 1: (AuthMonitor::update_from_paxos(bool*)+0x2611) [0x5e8041]<br /> 2: (PaxosService::refresh(bool*)+0x193) [0x530c43]<br /> 3: (Monitor::refresh_from_paxos(bool*)+0x57) [0x4ced07]<br /> 4: (Monitor::init_paxos()+0xd5) [0x4cee95]<br /> 5: (Monitor::preinit()+0x981) [0x4f3d51]<br /> 6: (main()+0x3caa) [0x4b620a]<br /> 7: (__libc_start_main()+0xfd) [0x376f41ecdd]<br /> 8: /usr/bin/ceph-mon() [0x4b0af9]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
<p>--- logging levels ---<br /> 0/ 5 none<br /> 0/ 1 lockdep<br /> 0/ 1 context<br /> 1/ 1 crush<br /> 1/ 5 mds<br /> 1/ 5 mds_balancer<br /> 1/ 5 mds_locker<br /> 1/ 5 mds_log<br /> 1/ 5 mds_log_expire<br /> 1/ 5 mds_migrator<br /> 0/ 1 buffer<br /> 0/ 1 timer<br /> 0/ 1 filer<br /> 0/ 1 striper<br /> 0/ 1 objecter<br /> 0/ 5 rados<br /> 0/ 5 rbd<br /> 0/ 5 journaler<br /> 0/ 5 objectcacher<br /> 0/ 5 client<br /> 0/ 5 osd<br /> 0/ 5 optracker<br /> 0/ 5 objclass<br /> 1/ 3 filestore<br /> 1/ 3 keyvaluestore<br /> 1/ 3 journal<br /> 0/ 5 ms<br /> 1/ 5 mon<br /> 0/10 monc<br /> 1/ 5 paxos<br /> 0/ 5 tp<br /> 1/ 5 auth<br /> 1/ 5 crypto<br /> 1/ 1 finisher<br /> 1/ 5 heartbeatmap<br /> 1/ 5 perfcounter<br /> 1/ 5 rgw<br /> 1/ 5 javaclient<br /> 1/ 5 asok<br /> 1/ 1 throttle<br /> <del>2/-2 (syslog threshold)<br /> -1/-1 (stderr threshold)<br /> max_recent 10000<br /> max_new 1000<br /> log_file /var/log/ceph/mon.a.log<br />--</del> end dump of recent events ---<br />2014-09-19 11:29:30.838931 7f89c94077a0 -1 *<strong>* Caught signal (Aborted) *</strong><br /> in thread 7f89c94077a0</p>
<pre><code>ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)<br /> 1: /usr/bin/ceph-mon() [0x7edb41]<br /> 2: /lib64/libpthread.so.0() [0x376fc0f500]<br /> 3: (gsignal()+0x35) [0x376f4328e5]<br /> 4: (abort()+0x175) [0x376f4340c5]<br /> 5: (_<em>gnu_cxx::</em>_verbose_terminate_handler()+0x12d) [0x3c6cabea5d]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
<p>--- begin dump of recent events ---<br /> 0> 2014-09-19 11:29:30.838931 7f89c94077a0 -1 *<strong>* Caught signal (Aborted) *</strong><br /> in thread 7f89c94077a0</p>
<pre><code>ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)<br /> 1: /usr/bin/ceph-mon() [0x7edb41]<br /> 2: /lib64/libpthread.so.0() [0x376fc0f500]<br /> 3: (gsignal()+0x35) [0x376f4328e5]<br /> 4: (abort()+0x175) [0x376f4340c5]<br /> 5: (_<em>gnu_cxx::</em>_verbose_terminate_handler()+0x12d) [0x3c6cabea5d]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
<p>--- logging levels ---<br /> 0/ 5 none<br /> 0/ 1 lockdep<br /> 0/ 1 context<br /> 1/ 1 crush<br /> 1/ 5 mds<br /> 1/ 5 mds_balancer<br /> 1/ 5 mds_locker<br /> 1/ 5 mds_log<br /> 1/ 5 mds_log_expire<br /> 1/ 5 mds_migrator<br /> 0/ 1 buffer<br /> 0/ 1 timer<br /> 0/ 1 filer<br /> 0/ 1 striper<br /> 0/ 1 objecter<br /> 0/ 5 rados<br /> 0/ 5 rbd<br /> 0/ 5 journaler<br /> 0/ 5 objectcacher<br /> 0/ 5 client<br /> 0/ 5 osd<br /> 0/ 5 optracker<br /> 0/ 5 objclass<br /> 1/ 3 filestore<br /> 1/ 3 keyvaluestore<br /> 1/ 3 journal<br /> 0/ 5 ms<br /> 1/ 5 mon<br /> 0/10 monc<br /> 1/ 5 paxos<br /> 0/ 5 tp<br /> 1/ 5 auth<br /> 1/ 5 crypto<br /> 1/ 1 finisher<br /> 1/ 5 heartbeatmap<br /> 1/ 5 perfcounter<br /> 1/ 5 rgw<br /> 1/ 5 javaclient<br /> 1/ 5 asok<br /> 1/ 1 throttle<br /> <del>2/-2 (syslog threshold)<br /> -1/-1 (stderr threshold)<br /> max_recent 10000<br /> max_new 1000<br /> log_file /var/log/ceph/mon.a.log<br />--</del> end dump of recent events ---</p>
Ceph - Bug #9217 (Resolved): wrong value of osd::op_w_latency
https://tracker.ceph.com/issues/9217
2014-08-25T02:40:53Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>when i use fio to test rbd performance (4kb random write) , i found that osd::op_w_latency which i got from perf counter is much larger than the latency reported by fio, after i digged into it and found that, with XFS , after data is written to journal , osd sends a reply to client to imply that the write request has completed , it updates the osd::op_w_latency perf counter in ReplicatedPG::eval_repop function , when data is written to filestore, the osd::op_w_latency perf counter updates again, so op_w_latency perf counter records the apply latency , not the commit latency , is this the defined behavior or a bug?</p>
Ceph - Bug #8589 (Resolved): scrub error with leveldb backend
https://tracker.ceph.com/issues/8589
2014-06-12T01:41:24Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>when ceph cluster began to scrub , ceph cluster became unhealthy , several pgs is active+clean+inconsistent, </p>
<pre><code>cluster 7ba3148c-6998-42df-af67-20a2f068fdf9<br /> health HEALTH_ERR 31 pgs inconsistent; 31 scrub errors<br /> monmap e1: 1 mons at {a=172.16.96.201:6789/0}, election epoch 1, quorum 0 a<br /> osdmap e173: 20 osds: 20 up, 20 in<br /> pgmap v23655: 7808 pgs, 4 pools, 2400 GB data, 600 kobjects<br /> 432 GB used, 18188 GB / 18621 GB avail<br /> 7776 active+clean<br /> 31 active+clean+inconsistent<br /> 1 active+clean+scrubbing</code></pre>
<p>the attach file is detail log.</p>
rbd - Bug #8564 (Resolved): osd cannot be restarted when leveldb is used as backend
https://tracker.ceph.com/issues/8564
2014-06-09T02:47:50Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>hi all , recently i enabled leveldb as filestore backend, after restarting my cluster , an osd is crashed, from the log , i get an error like this :</p>
<p>2014-06-09 16:22:12.250078 7f86a5610700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7f86a5610700 time 2014-06-09 16:22:12.248673<br />os/KeyValueStore.cc: 1524: FAILED assert(0 == "unexpected error")</p>
<pre><code>ceph version 0.80-820-g5d606cd (5d606cd0d00698699c91a378a1bd9f71cc8a77c9)<br /> 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x750) [0x9e6f20]<br /> 2: (KeyValueStore::_do_transactions(std::list&lt;ObjectStore::Transaction*, std::allocator&lt;ObjectStore::Transaction*&gt; >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9e8d1e]<br /> 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x9a) [0x9e8e2a]<br /> 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb62a3a]<br /> 5: (ThreadPool::WorkThread::entry()+0x10) [0xb63c90]<br /> 6: (()+0x7e9a) [0x7f86aed2de9a]<br /> 7: (clone()+0x6d) [0x7f86ad2d8ccd]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
rbd - Bug #8529 (Resolved): vm get killed when leveldb backend enable
https://tracker.ceph.com/issues/8529
2014-06-03T23:00:48Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>hi all , i enabled leveldb backend , but i use 'attach-device' to attach rbd to vm , the vm get killed , i get a segment fault from dmesg</p>
<p>[1185387.398884] virbr0: port 1(vnet0) entered forwarding state<br />[1185387.398890] virbr0: port 1(vnet0) entered forwarding state<br />[1185402.342142] kvm<sup><a href="#fn211895">211895</a></sup>: segfault at 7f3900000018 ip 00007f3a358bd009 sp 00007f399a7fb7f0 error 6 in librados.so.2.0.0[7f3a35570000+66b000]</p>
<p>the qemu log for this vm :</p>
<p>common/buffer.cc: In function 'ceph::buffer::ptr::ptr(const ceph::buffer::ptr&, unsigned int, unsigned int)' thread 7fc7beffd700 time 2014-06-04 05:29:20.835467<br />common/buffer.cc: 574: FAILED assert(_raw)<br /> ceph version 0.80-716-g884a6b3 (884a6b374af5c08c4c8d3c5f6488f415b120a6ab)<br /> 1: (()+0x34d052) [0x7fc85a695052]<br /> 2: (ceph::buffer::list::append(ceph::buffer::ptr const&, unsigned int, unsigned int)+0x2f) [0x7fc85a6960ef]<br /> 3: (ceph::buffer::list::splice(unsigned int, unsigned int, ceph::buffer::list*)+0xba) [0x7fc85a6964ca]<br /> 4: (Striper::StripedReadResult::add_partial_sparse_result(CephContext*, ceph::buffer::list&, std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > > const&, unsigned long, std::vector<std::pair<unsigned long, unsigned long>, std::allocator<std::pair<unsigned long, unsigned long> > > const&)+0xa49) [0x7fc85b42c689]<br /> 5: (librbd::C_AioRead::finish(int)+0xfa) [0x7fc85b3bb76a]<br /> 6: (Context::complete(int)+0x9) [0x7fc85b3bba19]<br /> 7: (librbd::AioRequest::complete(int)+0x3d) [0x7fc85b3bbaed]<br /> 8: (librados::C_AioComplete::finish(int)+0x1d) [0x7fc85a5ec7cd]<br /> 9: (Context::complete(int)+0x9) [0x7fc85b3bba19]<br /> 10: (Finisher::finisher_thread_entry()+0x1c8) [0x7fc85a67ac48]<br /> 11: (()+0x7e9a) [0x7fc8578f7e9a]<br /> 12: (clone()+0x6d) [0x7fc8576243fd]<br /> NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.<br />terminate called after throwing an instance of 'ceph::FailedAssertion'<br />2014-06-03 21:29:20.868+0000: shutting down</p>
Ceph - Bug #8381 (Resolved): osd crash when osd use leveldb as filestore
https://tracker.ceph.com/issues/8381
2014-05-18T20:05:48Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>when osd use leveldb as filestore(add configur options osd_objectstore = keyvaluestore-dev in ceph.conf), i used qemu rbd , after i attached rbd to one vm , everything seems ok , i can get disk label by 'fdisk -l' , however , when i used 'dd if=/dev/zero of=/dev/vdb bs=1M &', the ceph cluster crashed, i saw "bad op" error, the attached file is detail log.</p>
<pre><code>-8> 2014-05-19 10:47:44.828717 7f55579aa700 10 osd.0 pg_epoch: 245 pg[3.3f( v 245'1 (0'0,245'1] local-les=222 n=1 ec=44 les/c 222/222 221/221/173) [0,44] r=0 lpr=221 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] append_log adding 1 keys<br /> -7> 2014-05-19 10:47:44.828753 7f55579aa700 10 write_log with: dirty_to: 0'0, dirty_from: 4294967295'18446744073709551615, dirty_divergent_priors: 0, writeout_from: 245'1, trimmed:<br /> -6> 2014-05-19 10:47:44.828821 7f5564dff700 12 KeyValueStore::op_tp worker wq KeyValueStore::OpWQ start processing 0x8b7dc20 (1 active)<br /> -5> 2014-05-19 10:47:44.828818 7f55579aa700 10 osd.0 pg_epoch: 245 pg[3.3f( v 245'1 (0'0,245'1] local-les=222 n=1 ec=44 les/c 222/222 221/221/173) [0,44] r=0 lpr=221 luod=0'0 crt=0'0 lcod 0'0 mlcod 0'0 active+clean] eval_repop repgather(0x76a0840 245'1 rep_tid=1 committed?=0 applied?=0 lock=0 op=osd_op(client.4358.0:104 rbd_data.10426b8b4567.0000000000000004 [set-alloc-hint object_size 4194304 write_size 4194304,write 0~4194304] 3.bf28a03f ack+ondisk+write e245) v4) wants=ad<br /> -4> 2014-05-19 10:47:44.828839 7f55579aa700 10 osd.0 245 dequeue_op 0xa57c780 finish<br /> -3> 2014-05-19 10:47:44.828843 7f55579aa700 15 OSD::op_tp worker wq OSD::OpWQ done processing 0x1 (0 active)<br /> -2> 2014-05-19 10:47:44.828846 7f55579aa700 20 OSD::op_tp worker waiting<br /> -1> 2014-05-19 10:47:44.829013 7f5564dff700 -1 bad op 2307<br /> 0> 2014-05-19 10:47:44.830448 7f5564dff700 -1 os/KeyValueStore.cc: In function 'unsigned int KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)' thread 7f5564dff700 time 2014-05-19 10:47:44.829023<br />os/KeyValueStore.cc: 1457: FAILED assert(0)</code></pre>
<pre><code>ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)<br /> 1: (KeyValueStore::_do_transaction(ObjectStore::Transaction&, KeyValueStore::BufferTransaction&, SequencerPosition&, ThreadPool::TPHandle*)+0x1d0) [0x9d8720]<br /> 2: (KeyValueStore::_do_transactions(std::list&lt;ObjectStore::Transaction*, std::allocator&lt;ObjectStore::Transaction*&gt; >&, unsigned long, ThreadPool::TPHandle*)+0x8e) [0x9daa0e]<br /> 3: (KeyValueStore::_do_op(KeyValueStore::OpSequencer*, ThreadPool::TPHandle&)+0x97) [0x9dab17]<br /> 4: (ThreadPool::worker(ThreadPool::WorkThread*)+0x68a) [0xb5101a]<br /> 5: (ThreadPool::WorkThread::entry()+0x10) [0xb52270]<br /> 6: (()+0x7e9a) [0x7f556d321e9a]<br /> 7: (clone()+0x6d) [0x7f556b8ccccd]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
Ceph - Bug #7756 (Rejected): cannot get through configuration when configure.ac check xfs.h
https://tracker.ceph.com/issues/7756
2014-03-18T01:36:16Z
Xinxin Shu
xinxin.shu5040@gmail.com
<p>when i used configure.ac to config ceph, i get an error "configure: error: xfs/xfs.h not found (--without-libxfs to disable)", i checked configure.ac and found that, in order to support compiling without xfs, checks for xfs header file were added to configure.ac.</p>
<p>my system os is ubuntu-12.10, kernel version is 3.6.6-030606-generic, i used "apt-get install xfsprogs" to install xfs, we cannot find xfs head file(xfs.h), but had a kernel module xfs.</p>