Ceph : Issueshttps://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-03-16T02:22:27ZCeph
Redmine Ceph - Bug #15153 (Closed): when tacking an OpRequest, the time stamp of "dispatched" event is 0 https://tracker.ceph.com/issues/151532016-03-16T02:22:27Zchuanhong wangchuanhong_wang@163.com
<p>software: ceph-0.94.5+centos7<br />bug: the time stamp of "dispatched" event is 0, and the log listed as below.</p>
<p>2016-03-15 18:08:47.770223 7f5c99db1700 5 -- op tracker -- seq: 101899, time: 0.000000, event: dispatched, op: osd_repop(client.13938058.0:13708 1.319 85ebbb19/rbd_data.164a402ae8944a.000000000000f5f0/head//1 v 12638'75259)<br />2016-03-15 18:08:47.770552 7f5cbebb8700 5 -- op tracker -- seq: 101885, time: 2016-03-15 18:08:47.770551, event: journaled_completion_queued, op: osd_op(client.13938058.0:13655 rbd_data.164a402ae8944a.000000000000d7a0 [set-alloc-hint object_size 4194304 write_size 4194304,write 438272~4096] 1.4cf9af6d ack+ondisk+write+known_if_redirected e12638)<br />2016-03-15 18:08:47.770566 7f5ca80ec700 5 -- op tracker -- seq: 101897, time: 2016-03-15 18:08:47.770565, event: started, op: osd_repop(client.13938037.0:16406 1.2fd 5429befd/rbd_data.164a402ae8944a.000000000000db0d/head//1 v 12638'80535)<br />2016-03-15 18:08:47.770787 7f5ca60e8700 5 -- op tracker -- seq: 101893, time: 2016-03-15 18:08:47.770787, event: started, op: osd_repop(client.13938034.0:15792 1.379 65960f79/rbd_data.164a402ae8944a.0000000000002f61/head//1 v 12638'132245)<br />2016-03-15 18:08:47.770975 7f5cb93ad700 5 -- op tracker -- seq: 101885, time: 2016-03-15 18:08:47.770975, event: op_commit, op: osd_op(client.13938058.0:13655 rbd_data.164a402ae8944a.000000000000d7a0 [set-alloc-hint object_size 4194304 write_size 4194304,write 438272~4096] 1.4cf9af6d ack+ondisk+write+known_if_redirected e12638)<br />2016-03-15 18:08:47.771058 7f5c9a9bd700 5 -- op tracker -- seq: 101900, time: 2016-03-15 18:08:47.770943, event: header_read, op: osd_repop_reply(client.13938058.0:13655 1.36d ondisk, result = 0)<br />2016-03-15 18:08:47.771068 7f5c9a9bd700 5 -- op tracker -- seq: 101900, time: 2016-03-15 18:08:47.770945, event: throttled, op: osd_repop_reply(client.13938058.0:13655 1.36d ondisk, result = 0)<br />2016-03-15 18:08:47.771076 7f5c9a9bd700 5 -- op tracker -- seq: 101900, time: 2016-03-15 18:08:47.771029, event: all_read, op: osd_repop_reply(client.13938058.0:13655 1.36d ondisk, result = 0)<br />2016-03-15 18:08:47.771158 7f5c9a9bd700 5 -- op tracker -- seq: 101900, time: 0.000000, event: dispatched, op: osd_repop_reply(client.13938058.0:13655 1.36d ondisk, result = 0)</p> Ceph - Bug #12251 (Resolved): the output is wrong when runing "ceph osd reweight"https://tracker.ceph.com/issues/122512015-07-09T03:14:52Zchuanhong wangchuanhong_wang@163.com
<p>environment: centos7 + ceph v0.94<br />problem: when doing "ceph osd reweight", there is something wrong in the output. For example, the internal value of weight is 58982, but the command output in the bracket is 8589822.<br />[root@c8 ~]# ceph osd reweight 0 0.9<br />reweighted osd.0 to 0.9 (8589822)<br />At line 5625 of OSDMonitor.cc, should ios::hex be replaced by std::hex?</p> Ceph - Tasks #12118 (Rejected): ceph-mon crashed when starting uphttps://tracker.ceph.com/issues/121182015-06-23T07:22:09Zchuanhong wangchuanhong_wang@163.com
<p>environment: ceph v0.87 + centos7<br />ceph-mon crashed when start up. Device 9 is not exist but the mon still check the health of it. Threre is the stack info below, and the crush is attatched.</p>
<p>Program received signal SIGABRT, Aborted.<br />[Switching to Thread 0x7f5b91227700 (LWP 26238)]<br />0x00007f5b98dbf989 in raise () from /lib64/libc.so.6<br />(gdb) bt<br />#0 0x00007f5b98dbf989 in raise () from /lib64/libc.so.6<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a> 0x00007f5b98dc1098 in abort () from /lib64/libc.so.6<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: BUG at fs/ceph/caps.c:2178 (Closed)" href="https://tracker.ceph.com/issues/2">#2</a> 0x00007f5b996c39d5 in <i>gnu_cxx::__verbose_terminate_handler() () from /lib64/libstdc++.so.6<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: leaked dentry ref on umount (Closed)" href="https://tracker.ceph.com/issues/3">#3</a> 0x00007f5b996c1946 in ?? () from /lib64/libstdc++.so.6<br /><a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: lockdep warning in socket code (Closed)" href="https://tracker.ceph.com/issues/4">#4</a> 0x00007f5b996c1973 in std::terminate() () from /lib64/libstdc++.so.6<br /><a class="issue tracker-1 status-5 priority-3 priority-lowest closed" title="Bug: ./rados lspools sometimes hangs after listing all pools? (Closed)" href="https://tracker.ceph.com/issues/5">#5</a> 0x00007f5b996c1b9f in _<em>cxa_throw () from /lib64/libstdc++.so.6<br /><a class="issue tracker-2 status-6 priority-3 priority-lowest closed" title="Feature: libceph could use a backward-compatible-to function (Rejected)" href="https://tracker.ceph.com/issues/6">#6</a> 0x00000000007d77ca in ceph::</em>_ceph_assert_fail (assertion=assertion@entry=0x9867a7 "exists(osd)", file=file@entry=0x98678c "./osd/OSDMap.h", line=line@entry=442, <br /> func=func@entry=0x98cfa0 <OSDMap::get_addr(int) const::__PRETTY_FUNCTION</i>> "const entity_addr_t& OSDMap::get_addr(int) const") at common/assert.cc:77<br /><a class="issue tracker-6 status-3 priority-3 priority-lowest closed" title="Documentation: Document Monitor Commands (Resolved)" href="https://tracker.ceph.com/issues/7">#7</a> 0x000000000055e662 in OSDMap::get_addr (osd=9, this=0x4080088) at osd/OSDMap.h:442<br /><a class="issue tracker-3 status-5 priority-4 priority-default closed" title="Support: Document differences from S3 (Closed)" href="https://tracker.ceph.com/issues/8">#8</a> 0x0000000000604a11 in get_addr (osd=9, this=0x4080088) at osd/OSDMap.h:406<br />Python Exception <type 'exceptions.IndexError'> list index out of range: <br /><a class="issue tracker-2 status-8 priority-3 priority-lowest closed" title="Feature: Access unimported data (Won't Fix)" href="https://tracker.ceph.com/issues/9">#9</a> OSDMonitor::get_health (this=0x4080000, summary=std::list, detail=0x7f5b912252e0) at mon/OSDMonitor.cc:2119<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: osd: Replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/10">#10</a> 0x00000000005a9e81 in Monitor::get_health (this=0x4070000, status="", detailbl=0x7f5b91225bd0, f=0x0) at mon/Monitor.cc:2099<br /><a class="issue tracker-4 status-3 priority-3 priority-lowest closed" title="Cleanup: mds: replace ALLOW_MESSAGES_FROM macro (Resolved)" href="https://tracker.ceph.com/issues/11">#11</a> 0x00000000005bec14 in Monitor::handle_command (this=0x4070000, m=0x3e72400) at mon/Monitor.cc:2588<br /><a class="issue tracker-2 status-3 priority-3 priority-lowest closed" title="Feature: uclient: Make cap handling smarter (Resolved)" href="https://tracker.ceph.com/issues/12">#12</a> 0x00000000005c0cbb in Monitor::dispatch (this=0x4070000, s=0x3e69a00, m=0x3e72400, src_is_mon=<optimized out>) at mon/Monitor.cc:3240<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: uclient: Make readdir use the cache (Resolved)" href="https://tracker.ceph.com/issues/13">#13</a> 0x00000000005c173d in Monitor::_ms_dispatch (this=this@entry=0x4070000, m=m@entry=0x3e72400) at mon/Monitor.cc:3156<br /><a class="issue tracker-1 status-10 priority-4 priority-default closed" title="Bug: osd: pg split breaks if not all osds are up (Duplicate)" href="https://tracker.ceph.com/issues/14">#14</a> 0x00000000005e2343 in Monitor::ms_dispatch (this=0x4070000, m=0x3e72400) at mon/Monitor.h:785<br /><a class="issue tracker-1 status-3 priority-4 priority-default closed" title="Bug: mds rejoin: invented dirfrags (MDCache.cc:3469) (Resolved)" href="https://tracker.ceph.com/issues/15">#15</a> 0x0000000000918d2a in ms_deliver_dispatch (m=0x3e72400, this=0x4060000) at msg/Messenger.h:532<br /><a class="issue tracker-1 status-3 priority-5 priority-high3 closed" title="Bug: mds restart vs dbench (Resolved)" href="https://tracker.ceph.com/issues/16">#16</a> DispatchQueue::entry (this=0x40601b8) at msg/DispatchQueue.cc:185<br /><a class="issue tracker-1 status-6 priority-4 priority-default closed" title="Bug: rm -r failure (Rejected)" href="https://tracker.ceph.com/issues/17">#17</a> 0x00000000007bbe7d in DispatchQueue::DispatchThread::entry (this=<optimized out>) at msg/DispatchQueue.h:104<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: reconnect fixups (Resolved)" href="https://tracker.ceph.com/issues/18">#18</a> 0x00007f5b9a39ddf3 in start_thread () from /lib64/libpthread.so.0<br /><a class="issue tracker-2 status-3 priority-4 priority-default closed" title="Feature: rbd (Resolved)" href="https://tracker.ceph.com/issues/19">#19</a> 0x00007f5b98e803dd in clone () from /lib64/libc.so.6<br />(gdb) f 6<br /><a class="issue tracker-2 status-6 priority-3 priority-lowest closed" title="Feature: libceph could use a backward-compatible-to function (Rejected)" href="https://tracker.ceph.com/issues/6">#6</a> 0x00000000007d77ca in ceph::__ceph_assert_fail (assertion=assertion@entry=0x9867a7 "exists(osd)", file=file@entry=0x98678c "./osd/OSDMap.h", line=line@entry=442, <br /> func=func@entry=0x98cfa0 <OSDMap::get_addr(int) const::__PRETTY_FUNCTION__> "const entity_addr_t& OSDMap::get_addr(int) const") at common/assert.cc:77<br />77 throw FailedAssertion(bt);<br />(gdb) f 7<br /><a class="issue tracker-6 status-3 priority-3 priority-lowest closed" title="Documentation: Document Monitor Commands (Resolved)" href="https://tracker.ceph.com/issues/7">#7</a> 0x000000000055e662 in OSDMap::get_addr (osd=9, this=0x4080088) at osd/OSDMap.h:442<br />442 assert(exists(osd));<br />(gdb) p osd<br />$1 = 9<br />(gdb)</p> Ceph - Support #12085 (New): data is no well-distributed among osds of host with staw algorithmhttps://tracker.ceph.com/issues/120852015-06-19T08:13:26Zchuanhong wangchuanhong_wang@163.com
<p>environment: ceph0.87+centos7<br />problem: there are three host in my ceph cluster, and the crush algorithm of the host is straw. It seemds that data is not well-distributes in osds. For example, in host ceph2, osd.1 and osd.4 are full, but just 26% of osd.6 is used, %35 of osd.8 is used. my question is, what kind of algorithm is prefered by a host? Is four items too few in a 'straw' buckt? if so, how many items would be added at least in a 'straw' bucket?<br />There is the crush of my cluster in the attachment.</p>
[root@ceph2 ~]# ceph -s<br /> cluster cbc79ef9-fbc3-41ad-a726-47359f8d84b3<br /> health HEALTH_ERR6 pgs backfill_toofull; 3 pgs inconsistent; 6 pgs stuck unclean; recovery 366/1516021 objects degraded (0.024%); 9296/1516021 objects misplaced (0.613%); 1 full osd(s); 3 scrub errors<br /> monmap e7: 3 mons at {ceph1=192.168.200.246:6789/0,ceph2=192.168.200.247:6789/0,ceph3=192.168.200.242:6789/0}, election epoch 1312, quorum 0,1,2 ceph3,ceph1,ceph2<br /> osdmap e14700: 12 osds: 12 up, 12 in<br /> flags full<br /> pgmap v3244880: 4608 pgs, 12 pools, 550 GB data, 491 kobjects<br /> 1703 GB used, 8000 GB / 9704 GB avail<br /> 366/1516021 objects degraded (0.024%); 9296/1516021 objects misplaced (0.613%)<br /> 4599 active+clean<br /> 3 active+clean+inconsistent<br /> 6 active+remapped+backfill_toofull<br /> client io 15594 B/s rd, 21 op/s<br />[root@ceph2 ~]# ceph osd tree
<ol>
<li>id weight type name up/down reweight<br />-1 9.47 root default<br />-3 1.1 host ceph2<br />4 0.27 osd.4 up 1<br />6 0.27 osd.6 up 1<br />8 0.27 osd.8 up 1<br />1 0.29 osd.1 up 1<br />-4 1.1 host ceph3<br />0 0.29 osd.0 up 1<br />2 0.27 osd.2 up 1<br />3 0.27 osd.3 up 1<br />5 0.27 osd.5 up 1<br />-2 7.27 host ceph1<br />13 1.82 osd.13 up 1<br />14 1.82 osd.14 up 1<br />15 1.81 osd.15 up 1<br />16 1.82 osd.16 up 1<br />[root@ceph2 ceph]# df -h<br />Filesystem Size Used Avail Use% Mounted on<br />/dev/mapper/vg_sys-lv_root 97G 8.1G 84G 9% /<br />devtmpfs 32G 0 32G 0% /dev<br />tmpfs 32G 0 32G 0% /dev/shm<br />tmpfs 32G 675M 31G 3% /run<br />tmpfs 32G 0 32G 0% /sys/fs/cgroup<br />/dev/sda1 380M 98M 258M 28% /boot<br />/dev/sdc1 280G 280G 20K 100% /var/lib/ceph/osd/ceph-4<br />/dev/sde1 280G 98G 182G 35% /var/lib/ceph/osd/ceph-8<br />/dev/sdb1 293G 274G 20G 94% /var/lib/ceph/osd/ceph-1<br />/dev/sdd1 280G 71G 209G 26% /var/lib/ceph/osd/ceph-6</li>
</ol> rbd - Bug #12018 (Resolved): rbd and pool quota do not go well togetherhttps://tracker.ceph.com/issues/120182015-06-15T09:17:18Zchuanhong wangchuanhong_wang@163.com
<p>The pool "rbd" is already full, but I still flatten a clone successfully. After flatten, "rbd" has more objects, but the "USED" space has no change, thar's weird.</p>
<p>[root@c8 ~]# ceph -s<br /> cluster 927e8bfb-a47a-4538-aaab-b74a8c6163b9<br /> health HEALTH_WARN<br /> pool 'rbd' is full<br /> monmap e1: 1 mons at {c7=10.118.202.97:6789/0}<br /> election epoch 1, quorum 0 c7<br /> osdmap e46: 2 osds: 2 up, 2 in<br /> pgmap v605: 128 pgs, 2 pools, 2048 MB data, 518 objects<br /> 4180 MB used, 36759 MB / 40940 MB avail<br /> 128 active+clean<br />[root@c8 ~]# rbd list<br />im1<br />im1_clone1<br />[root@c8 ~]# ceph df<br />GLOBAL:<br /> SIZE AVAIL RAW USED <span>RAW USED <br /> 40940M 36759M 4180M 10.21 <br />POOLS:<br /> NAME ID USED %USED MAX AVAIL OBJECTS <br /> rbd 0 2048M 5.00 18379M 518 <br /> volums 1 0 0 18379M 0 <br />[root@c8 ~]# rbd flatten im1_clone1<br />Image flatten: 100</span> complete...done.<br />[root@c8 ~]# ceph -s<br /> cluster 927e8bfb-a47a-4538-aaab-b74a8c6163b9<br /> health HEALTH_WARN<br /> pool 'rbd' is full<br /> monmap e1: 1 mons at {c7=10.118.202.97:6789/0}<br /> election epoch 1, quorum 0 c7<br /> osdmap e46: 2 osds: 2 up, 2 in<br /> pgmap v625: 128 pgs, 2 pools, 2048 MB data, 3078 objects<br /> 4216 MB used, 36723 MB / 40940 MB avail<br /> 128 active+clean<br /> client io 614 B/s rd, 0 B/s wr, 145 op/s<br />[root@c8 ~]# ceph df<br />GLOBAL:<br /> SIZE AVAIL RAW USED %RAW USED <br /> 40940M 36723M 4216M 10.30 <br />POOLS:<br /> NAME ID USED %USED MAX AVAIL OBJECTS <br /> rbd 0 2048M 5.00 18360M 3078 <br /> volums 1 0 0 18360M 0</p> Ceph - Bug #11840 (Resolved): avoid complicated cache tier, like cache loop and three level cachehttps://tracker.ceph.com/issues/118402015-06-02T02:05:37Zchuanhong wangchuanhong_wang@163.com
<blockquote>
<p>there are three pool in ceph cluster, e.g, t1, t2 t3, and t2 is a cache of t1.<br />1, we can set t1 as a cache of t2 too, then t1 is the cache of t2, t2 also is the cache of t1,a loop of cache generated;<br />2, we can also set t3 as a cache of t2 ort t1, then a three level cache tier generated, t3 is the cache of t2, t2 is the cache of t1.<br />cache loop, and three level cache are weird, also maybe reduce the IO performance, so it'd be better to avoid complicated tier like them.</p>
</blockquote>
<p>we don't support multiple tiering, so we'd better fail a "ceph osd tier add" command which intent to add a tier for another tier.</p>