Ceph : Issues
https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2022-07-06T10:25:03Z
Ceph
Redmine
Ceph - Bug #56479 (New): Cannot automatically recover from slow ops warning on ceph-mon
https://tracker.ceph.com/issues/56479
2022-07-06T10:25:03Z
Yao Ning
zay11022@163.com
<p>cluster:<br /> id: b8e99618-7d99-4b50-b4cc-b633292c0fe3<br /> health: HEALTH_WARN<br /> 9 slow ops, oldest one blocked for 196 sec, mon.ceph-test-2 has slow ops</p>
<p>mon.ceph-test-2 always show 9 slow ops until reported failure osd restarted, showed in ceph-mon log</p>
<p>2022-07-06 18:09:32.068 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:09:37.069 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:09:42.235 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:09:47.235 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:09:52.236 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:09:57.237 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)<br />2022-07-06 18:10:02.237 7f9ce2212700 -1 mon.ceph-test-2@0(leader) e1 get_health_metrics reporting 9 slow ops, oldest is osd_failure(failed timeout osd.1 [v2:192.168.0.104:6800/815752,v1:192.168.0.104:6801/815752] for 15sec e1863 v1863)</p>
<p>How to reproduce?</p>
<p>1) 3 host, 3 osd on each host; <br />hostA: 192.168.0.104<br />hostB: 192.168.0.22<br />hostC: 192.168.0.41</p>
<p>2) ceph.conf settings to prevent osd being marked down<br />[global]<br />mon_osd_reporter_subtree_level = host<br />mon_osd_min_down_reporters = 3</p>
<p>3) On hostA, use iptables to drop network packets from hostC<br />iptables -I INPUT -s 192.168.0.41/32 -j DROP</p>
<p>4) wait enough times until slowops appeared in ceph -s</p>
<p>5) stop all osds on hostC</p>
<p>6) remove the iptables rules on hostA<br />iptables -D INPUT -s 192.168.0.41/32 -j DROP</p>
<p>7) start all osds on hostC</p>
<p>8)and then you can find slowops warn always appeared on ceph -s</p>
<p>I think the main reason causes this problem is, in OSDMonitor.cc, failure_info logged when some osds report others' failure, but failure osds are not marked down immediatelly because of not enough osd report it down. Finally, the reporter(osds) may restart or reboot so that they lost the failure's peer osds and when they are started, the heartbeat with failure osds become normal. So no more event will be sent to ceph mon to tell this, and failure_info always on ceph monitor.</p>
RADOS - Bug #51168 (New): ceph-osd state machine crash during peering process
https://tracker.ceph.com/issues/51168
2021-06-10T15:58:50Z
Yao Ning
zay11022@163.com
<pre>
[root@ceph-128 ~]# ceph crash info 2021-06-10_15:30:01.935970Z_f5d8ab5b-8613-408b-ac22-75425c5cf672
{
"os_version_id": "7",
"assert_condition": "abort",
"utsname_release": "3.10.0-1127.el7.x86_64",
"os_name": "CentOS Linux",
"entity_name": "osd.43",
"assert_file": "/builddir/build/BUILD/ceph-14.2.18/src/osd/PG.cc",
"timestamp": "2021-06-10 15:30:01.935970Z",
"process_name": "ceph-osd",
"utsname_machine": "x86_64",
"assert_line": 7274,
"utsname_sysname": "Linux",
"os_version": "7 (Core)",
"os_id": "centos",
"assert_thread_name": "tp_osd_tp",
"utsname_version": "#1 SMP Tue Mar 31 23:36:51 UTC 2020",
"backtrace": [
"(()+0xf630) [0x7f0101be1630]",
"(gsignal()+0x37) [0x7f01009d4387]",
"(abort()+0x148) [0x7f01009d5a78]",
"(ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0x1a5) [0x55859f824b98]",
"(PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::my_context)+0xc3) [0x55859f9d5643]",
"(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::deep_construct(boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>* const&, boost::statechart::state_machine<PG::RecoveryState::RecoveryMachine, PG::RecoveryState::Initial, std::allocator<boost::statechart::none>, boost::statechart::null_exception_translator>&)+0x36) [0x55859fa2b4e6]",
"(boost::statechart::simple_state<PG::RecoveryState::ReplicaActive, PG::RecoveryState::Started, PG::RecoveryState::RepNotRecovering, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0x1d6) [0x55859fa2be26]",
"(boost::statechart::simple_state<PG::RecoveryState::RepNotRecovering, PG::RecoveryState::ReplicaActive, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::react_impl(boost::statechart::event_base const&, void const*)+0xd3) [0x55859fa2b483]",
"(PG::do_peering_event(std::shared_ptr<PGPeeringEvent>, PG::RecoveryCtx*)+0x2dd) [0x55859f9efe0d]",
"(OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b4) [0x55859f92c2c4]",
"(PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x51) [0x55859fb94951]",
"(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x914) [0x55859f920d84]",
"(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x55859fedcfe6]",
"(ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55859fedfb00]",
"(()+0x7ea5) [0x7f0101bd9ea5]",
"(clone()+0x6d) [0x7f0100a9c96d]"
],
"utsname_hostname": "ceph-138",
"assert_msg": "/builddir/build/BUILD/ceph-14.2.18/src/osd/PG.cc: In function 'PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)' thread 7f00daa4b700 time 2021-06-10 23:30:01.927379\n/builddir/build/BUILD/ceph-14.2.18/src/osd/PG.cc: 7274: ceph_abort_msg(\"we got a bad state machine event\")\n",
"crash_id": "2021-06-10_15:30:01.935970Z_f5d8ab5b-8613-408b-ac22-75425c5cf672",
"assert_func": "PG::RecoveryState::Crashed::Crashed(boost::statechart::state<PG::RecoveryState::Crashed, PG::RecoveryState::RecoveryMachine>::my_context)",
"ceph_version": "14.2.18"
}
</pre>
RADOS - Bug #48028 (Won't Fix - EOL): ceph-mon always suffer lots of slow ops from v14.2.9
https://tracker.ceph.com/issues/48028
2020-10-28T15:21:54Z
Yao Ning
zay11022@163.com
<p>root@worker-2:~# docker exec ceph-mon-worker-2 ceph -s<br /> cluster:<br /> id: 299a04ba-dd3e-43a7-af17-628190cf742f<br /> health: HEALTH_WARN<br /> 41 slow ops, oldest one blocked for 38274 sec, mon.worker-2 has slow ops</p>
<pre><code>services:<br /> mon: 3 daemons, quorum worker-2,worker-12,worker-22 (age 17m)<br /> mgr: worker-22(active, since 11d), standbys: worker-12, worker-2<br /> mds: cephfs:1 {0=worker-22=up:active} 2 up:standby<br /> osd: 300 osds: 300 up (since 56m), 300 in (since 11d)</code></pre>
<pre><code>task status:<br /> scrub status:<br /> mds.worker-22: idle</code></pre>
<pre><code>data:<br /> pools: 2 pools, 16384 pgs<br /> objects: 78.68M objects, 300 TiB<br /> usage: 645 TiB used, 1.5 PiB / 2.2 PiB avail<br /> pgs: 16384 active+clean</code></pre>
<pre><code>io:<br /> client: 5.1 KiB/s rd, 315 MiB/s wr, 1 op/s rd, 188 op/s wr</code></pre>
mgr - Bug #47789 (New): ceph-mgr: failed assert in Thread.cc
https://tracker.ceph.com/issues/47789
2020-10-08T05:53:13Z
Yao Ning
zay11022@163.com
<p>2020-10-08 06:05:36.986 7f548b370e40 1 mgr[py] Loading python module 'rbd_support'<br />2020-10-08 06:05:37.161 7f548b370e40 1 mgr[py] Loading python module 'restful'<br />2020-10-08 06:05:38.156 7f548b370e40 1 mgr[py] Loading python module 'selftest'<br />2020-10-08 06:05:38.473 7f548b370e40 1 mgr[py] Loading python module 'status'<br />2020-10-08 06:05:38.613 7f548b370e40 1 mgr[py] Loading python module 'telegraf'<br />2020-10-08 06:05:38.782 7f548b370e40 1 mgr[py] Loading python module 'telemetry'<br />2020-10-08 06:05:39.265 7f548b370e40 1 mgr[py] Loading python module 'test_orchestrator'<br />2020-10-08 06:05:39.466 7f548b370e40 1 mgr[py] Loading python module 'volumes'<br />2020-10-08 06:05:39.684 7f548b370e40 1 mgr[py] Loading python module 'zabbix'<br />2020-10-08 06:05:39.788 7f5477366700 0 ms_deliver_dispatch: unhandled message 0x55bac19e0a00 mon_map magic: 0 v1 from mon.2 v2:10.0.6.33:3300/0<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn e: '/usr/bin/ceph-mgr'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 0: '/usr/bin/ceph-mgr'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 1: '-f'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 2: '--cluster'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 3: 'ceph'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 4: '--id'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 5: 'control3'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 6: '--setuser'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 7: 'ceph'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 8: '--setgroup'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn 9: 'ceph'<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn respawning with exe /usr/bin/ceph-mgr<br />2020-10-08 06:05:39.789 7f5477366700 1 mgr respawn exe_path /proc/self/exe<br />2020-10-08 06:05:39.791 7f5477b67700 1 mgr load Constructed class from module: prometheus<br />2020-10-08 06:05:39.795 7f5477b67700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/common/Thread.cc: In function 'void Thread::create(const char*, size_t)' thread 7f5477b67700 time 2020-10-08 06:05:39.793849<br />/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/gigantic/release/14.2.11/rpm/el7/BUILD/ceph-14.2.11/src/common/Thread.cc: 157: FAILED ceph_assert(ret == 0)</p>
<pre><code>ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)<br /> 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x7f54825ce025]<br /> 2: (()+0x25c1ed) [0x7f54825ce1ed]<br /> 3: (()+0x324992) [0x7f5482696992]<br /> 4: (()+0x208f28) [0x55babec96f28]<br /> 5: (FunctionContext::finish(int)+0x2c) [0x55babebf383c]<br /> 6: (Context::complete(int)+0x9) [0x55babebef319]<br /> 7: (Finisher::finisher_thread_entry()+0x16f) [0x7f548265738f]<br /> 8: (()+0x7e65) [0x7f547feb5e65]<br /> 9: (clone()+0x6d) [0x7f547eb6388d]</code></pre>
<p>2020-10-08 06:05:39.800 7f5477b67700 -1 *<strong>* Caught signal (Aborted) *</strong><br /> in thread 7f5477b67700 thread_name:mgrsb-fin</p>
<pre><code>ceph version 14.2.11 (f7fdb2f52131f54b891a2ec99d8205561242cdaf) nautilus (stable)<br /> 1: (()+0xf5f0) [0x7f547febd5f0]<br /> 2: (gsignal()+0x37) [0x7f547ea9b337]<br /> 3: (abort()+0x148) [0x7f547ea9ca28]<br /> 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x7f54825ce074]<br /> 5: (()+0x25c1ed) [0x7f54825ce1ed]<br /> 6: (()+0x324992) [0x7f5482696992]<br /> 7: (()+0x208f28) [0x55babec96f28]<br /> 8: (FunctionContext::finish(int)+0x2c) [0x55babebf383c]<br /> 9: (Context::complete(int)+0x9) [0x55babebef319]<br /> 10: (Finisher::finisher_thread_entry()+0x16f) [0x7f548265738f]<br /> 11: (()+0x7e65) [0x7f547feb5e65]<br /> 12: (clone()+0x6d) [0x7f547eb6388d]<br /> NOTE: a copy of the executable, or `objdump -rdS &lt;executable&gt;` is needed to interpret this.</code></pre>
CephFS - Bug #46844 (Won't Fix): ceph-fuse: writing data can exceed quota when mount with ceph-fuse
https://tracker.ceph.com/issues/46844
2020-08-06T07:51:58Z
Yao Ning
zay11022@163.com
<p>setfattr -n ceph.quota.max_bytes -v 1073741824 /opt/cephfs/test1/</p>
<p>use ceph-fuse mount the dir /cephfs/test1/ to /mnt/</p>
<p>[root@tempest-testshareshrinkcephfs-server-1583097679 ~]# sh -c "dd bs=64M count=20 if=/dev/urandom of=/mnt/t1 conv=fsync iflag=fullblock" <br />20+0 records in<br />20+0 records out<br />1342177280 bytes (1.3 GB) copied, 47.4257 s, 28.3 MB/s<br />[root@tempest-testshareshrinkcephfs-server-1583097679 ~]# df -h<br />Filesystem Size Used Avail Use% Mounted on<br />devtmpfs 220M 0 220M 0% /dev<br />tmpfs 244M 0 244M 0% /dev/shm<br />tmpfs 244M 8.6M 235M 4% /run<br />tmpfs 244M 0 244M 0% /sys/fs/cgroup<br />/dev/vda1 20G 1.3G 19G 7% /<br />tmpfs 49M 0 49M 0% /run/user/0<br />ceph-fuse 1.0G 1.0G 0 100% /mnt<br />[root@tempest-testshareshrinkcephfs-server-1583097679 ~]# du -sh /mnt/*<br />1.3G /mnt/t1</p>
<p>we expect error return here like below:</p>
<p>[root@tempest-testshareshrinkcephfs-server-1583097679 ~]# sh -c "dd bs=64M count=1 if=/dev/urandom of=/mnt/t2 conv=fsync iflag=fullblock" <br />dd: error writing ‘/mnt/t2’: Disk quota exceeded<br />1+0 records in<br />0+0 records out<br />0 bytes (0 B) copied, 0.51842 s, 0.0 kB/s</p>
RADOS - Support #23455 (Resolved): osd: large number of inconsistent objects after recover or bac...
https://tracker.ceph.com/issues/23455
2018-03-24T07:20:24Z
Yao Ning
zay11022@163.com
<p>large number of inconsistent objects after recover or backfilling.</p>
<p>reproduce method:</p>
<p>1) create rbd volume and, map to a client. use fio to write data to the volume, as below</p>
<p>[job]<br />ioengine=libaio<br />direct=1<br />filename=/dev/rbd0<br />rw=randwrite<br />norandommap<br />bs=8k<br />iodepth=2<br />random_distribution=zipf:1.2<br />time_based<br />runtime=360000</p>
<p>2) stop serveral osds<br />3) create snapshot for this volume<br />4) and then start osds again<br />5) finally, do deep-scrub and the inconsistent occurs</p>
<p>like below:<br />host=$(( $i % 2 + 1))</p>
<p>ssh jewel-ceph-${host} "systemctl stop ceph-osd.target" <br />echo "stop osd on jewel-ceph-${host}" <br />sleep 60</p>
<p>rbd snap create nyao@${i}<br />sleep 60<br />rbd snap create nyao@s${i}<br />sleep 60<br />rbd snap create nyao@ss${i}<br />sleep 60<br />rbd snap create nyao@sss${i}<br />echo "create snap nyao@${i}"</p>
<p>sleep 120<br />ssh jewel-ceph-${host} "systemctl start ceph-osd.target" <br />echo "start osd on jewel-ceph-${host}"</p>
<p>for j in {0..5};<br />do<br /> ceph osd deep-scrub osd.${j}<br />done<br />echo "scrub all osd" <br />sleep 3600</p>
<p>it seems if I disable fiemap features, then no inconsistent object any more.</p>
bluestore - Bug #20557 (Closed): segmentation fault with rocksdb|BlueStore and jemalloc
https://tracker.ceph.com/issues/20557
2017-07-10T12:47:24Z
Yao Ning
zay11022@163.com
<p>Jul 10 12:00:43 server-69 ceph-osd: ceph version 12.0.3 (f2337d1b42fa49dbb0a93e4048a42762e3dffbbf)<br />Jul 10 12:00:43 server-69 ceph-osd: 1: (()+0x9aa2bf) [0x7fe303c9b2bf]<br />Jul 10 12:00:43 server-69 ceph-osd: 2: (()+0xf100) [0x7fe3006fb100]<br />Jul 10 12:00:43 server-69 ceph-osd: 3: (()+0x1cdff) [0x7fe302eb5dff]<br />Jul 10 12:00:43 server-69 ceph-osd: 4: (rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, rocksdb::Block*, unsigned int, rocksdb::Slice const&, unsigned long, bool, rocksdb::Cache::Priority)+0xd6) [0x7fe303f6d916]<br />Jul 10 12:00:43 server-69 ceph-osd: 5: (rocksdb::BlockBasedTable::MaybeLoadDataBlockToCache(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::Slice, rocksdb::BlockBasedTable::CachableEntry<rocksdb::Block>*, bool)+0x3dc) [0x7fe303f6e9ac]<br />Jul 10 12:00:43 server-69 ceph-osd: 6: (rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::BlockIter*, bool, rocksdb::Status)+0x127) [0x7fe303f6ec07]<br />Jul 10 12:00:43 server-69 ceph-osd: 7: (rocksdb::BlockBasedTable::BlockEntryIteratorState::NewSecondaryIterator(rocksdb::Slice const&)+0x89) [0x7fe303f77229]<br />Jul 10 12:00:43 server-69 ceph-osd: 8: (()+0xca78f6) [0x7fe303f988f6]<br />Jul 10 12:00:43 server-69 ceph-osd: 9: (()+0xca7ebd) [0x7fe303f98ebd]<br />Jul 10 12:00:43 server-69 ceph-osd: 10: (()+0xca7ecf) [0x7fe303f98ecf]<br />Jul 10 12:00:43 server-69 ceph-osd: 11: (rocksdb::MergingIterator::Seek(rocksdb::Slice const&)+0xce) [0x7fe303f80fbe]<br />Jul 10 12:00:43 server-69 ceph-osd: 12: (rocksdb::DBIter::Seek(rocksdb::Slice const&)+0x174) [0x7fe304000444]<br />Jul 10 12:00:43 server-69 ceph-osd: 13: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::lower_bound(std::string const&, std::string const&)+0xa2) [0x7fe303bf4e42]<br />Jul 10 12:00:43 server-69 ceph-osd: 14: (RocksDBStore::RocksDBWholeSpaceIteratorImpl::upper_bound(std::string const&, std::string const&)+0x30) [0x7fe303bf6730]<br />Jul 10 12:00:43 server-69 ceph-osd: 15: (BlueStore::_collection_list(BlueStore::Collection*, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> ><strong>, ghobject_t</strong>)+0xbc2) [0x7fe303b60c42]<br />Jul 10 12:00:43 server-69 ceph-osd: 16: (BlueStore::collection_list(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> ><strong>, ghobject_t</strong>)+0xe8) [0x7fe303b624c8]<br />Jul 10 12:00:43 server-69 ceph-osd: 17: (BlueStore::collection_list(coll_t const&, ghobject_t const&, ghobject_t const&, int, std::vector<ghobject_t, std::allocator<ghobject_t> ><strong>, ghobject_t</strong>)+0x72) [0x7fe303b80ce2]<br />Jul 10 12:00:43 server-69 ceph-osd: 18: (OSD::clear_temp_objects()+0x537) [0x7fe303780357]<br />Jul 10 12:00:43 server-69 ceph-osd: 19: (OSD::init()+0x1f7d) [0x7fe3037b838d]<br />Jul 10 12:00:43 server-69 ceph-osd: 20: (main()+0x2ea7) [0x7fe3036c1547]<br />Jul 10 12:00:43 server-69 ceph-osd: 21: (__libc_start_main()+0xf5) [0x7fe2ff50cb15]<br />Jul 10 12:00:43 server-69 ceph-osd: 22: (()+0x469906) [0x7fe30375a906]<br />Jul 10 12:00:43 server-69 ceph-osd: 2017-07-10 12:00:43.440569 7fe3032cebc0 -1 *<strong>* Caught signal (Segmentation fault) *</strong><br />Jul 10 12:00:43 server-69 ceph-osd: in thread 7fe3032cebc0 thread_name:ceph-osd</p>
Ceph - Bug #19996 (Resolved): infinit loops in filestore::fiemap()
https://tracker.ceph.com/issues/19996
2017-05-20T14:36:39Z
Yao Ning
zay11022@163.com
<p>since fiemap can get extents based on offset --> len<br />but we should consider last extents is retrieved when len == 0<br />even though it is not last fiemap extents</p>
<p>Fixed in master by <a class="external" href="https://github.com/ceph/ceph/pull/14367">https://github.com/ceph/ceph/pull/14367</a></p>
Ceph - Backport #19394 (Resolved): OSD blocks all readonly ops when OSD reaches full
https://tracker.ceph.com/issues/19394
2017-03-28T07:21:18Z
Yao Ning
zay11022@163.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/14181">https://github.com/ceph/ceph/pull/14181</a></p>
Ceph - Backport #19311 (Resolved): segmentation fault when call do_fiemap() in filestore
https://tracker.ceph.com/issues/19311
2017-03-20T07:09:11Z
Yao Ning
zay11022@163.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/14044">https://github.com/ceph/ceph/pull/14044</a></p>
Ceph - Backport #19083 (Resolved): jewel: osd: preserve allocation hint attribute during recovery
https://tracker.ceph.com/issues/19083
2017-02-25T02:40:27Z
Yao Ning
zay11022@163.com
<p><a class="external" href="https://github.com/ceph/ceph/pull/13647">https://github.com/ceph/ceph/pull/13647</a></p>
Ceph - Bug #18446 (Resolved): osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KE...
https://tracker.ceph.com/issues/18446
2017-01-08T06:54:37Z
Yao Ning
zay11022@163.com
<p>_zero() in filestore always return -EOPNOTSUPP if there is no flags FALLOC_FL_KEEP_SIZE.</p>
Ceph - Bug #17610 (Resolved): FileStore: fiemap cannot be totally retrieved in xfs when the numbe...
https://tracker.ceph.com/issues/17610
2016-10-19T09:30:07Z
Yao Ning
zay11022@163.com
<p>Retrieve Fiemap from XFS filesystem:</p>
<pre><code>0: [0..7]: 22108440..22108447<br /> 1: [8..15]: hole<br /> 2: [16..23]: 22108456..22108463<br /> 3: [24..31]: hole<br /> 4: [32..39]: 22108472..22108479<br /> 5: [40..47]: hole<br /> 6: [48..55]: 22108488..22108495<br /> 7: [56..63]: hole<br /> ...<br /> ...<br /> 3991: [31928..31935]: hole<br /> 3992: [31936..31943]: 22057896..22057903<br /> 3993: [31944..31951]: hole<br /> 3994: [31952..31959]: 22057960..22057967<br /> 3995: [31960..31967]: hole<br /> 3996: [31968..31975]: 22058008..22058015<br /> 3997: [31976..31983]: hole<br /> 3998: [31984..31991]: 22058040..22058047</code></pre>
<p>Actually, 2000 extents should be retrived here.</p>
<p>However, the result is:<br />total extents retrieved: 1364</p>
Ceph - Bug #11375 (Duplicate): Proxy_Read BUG in Cachetier when using ecpool as cold-pool
https://tracker.ceph.com/issues/11375
2015-04-13T05:17:00Z
Yao Ning
zay11022@163.com
<pre>
-6> 2015-04-13 13:02:47.497843 7fa41a74e700 5 -- op tracker -- seq: 11715258, time: 2015-04
-13 13:02:47.497843, event: op_applied, op: osd_op(client.15088.1:5081260
rb.0.56cc.6b8b4567.0000000004c2 [set-alloc-hint object_size 4194304 write_size 4194304,write
3760128~8192] 2.20cc17d2 ondisk+write e76)
-5> 2015-04-13 13:02:47.497993 7fa41cf53700 5 -- op tracker -- seq: 11715266, time: 2015-04
-13 13:02:47.497992, event: write_thread_in_journal_buffer, op: osd_repop(osd.3.0:2424900 2.187
f5bf6387/rb.0.5511.6b8b4567.00000000071c/head//2 v 76'13933)
-4> 2015-04-13 13:02:47.498258 7fa41a74e700 5 -- op tracker -- seq: 11715264, time: 2015-04
-13 13:02:47.498258, event: op_applied, op: osd_op(client.15088.1:5081275
rb.0.5511.6b8b4567.00000000057b [set-alloc-hint object_size 4194304 write_size 4194304,write
3256320~16384] 2.a695533e ondisk+write e76)
-3> 2015-04-13 13:02:47.498676 7fa41a74e700 5 -- op tracker -- seq: 11715263, time: 2015-04
-13 13:02:47.498676, event: op_applied, op: osd_op(client.15088.1:5081271
rb.0.5511.6b8b4567.0000000004c6 [set-alloc-hint object_size 4194304 write_size 4194304,write
3207168~20480] 2.8850ad85 ondisk+write e76)
-2> 2015-04-13 13:02:47.505922 7fa3fb5cc700 1 -- 192.168.1.71:0/14876 <== osd.0
192.168.1.70:6800/18480 128275 ==== osd_op_reply(1211124 rb.0.56bb.6b8b4567.000000005089 [list-
snaps] v0'0 uv13943 ondisk = 0) v6 ==== 198+0+48 (3855621112 0 4193251864) 0xcda8580 con
0x804b5a0
-1> 2015-04-13 13:02:47.506278 7fa3fb5cc700 1 -- 192.168.1.71:0/14876 <== osd.0
192.168.1.70:6800/18480 128276 ==== osd_op_reply(1211122 rb.0.56bb.6b8b4567.000000005089 [read
3764224~12288] v0'0 uv13943 ondisk = 0) v6 ==== 198+0+12288 (589373748 0 0) 0xcda8580 con
0x804b5a0
0> 2015-04-13 13:02:47.506290 7fa405cee700 -1 osd/ReplicatedPG.cc: In function 'void
ReplicatedPG::finish_proxy_read(hobject_t, ceph_tid_t, int)' thread 7fa405cee700 time 2015-04-13
13:02:47.481190
osd/ReplicatedPG.cc: 2075: FAILED assert(op == prdop->op)
ceph version 0.94 (e61c4f093f88e44961d157f65091733580cea79a)
1: (ReplicatedPG::finish_proxy_read(hobject_t, unsigned long, int)+0xb78) [0x8957a8]
2: (C_ProxyRead::finish(int)+0xe7) [0x8c98d7]
3: (Context::complete(int)+0x9) [0x6987b9]
4: (Finisher::finisher_thread_entry()+0x188) [0xa514c8]
5: /lib64/libpthread.so.0() [0x31792079d1]
6: (clone()+0x6d) [0x3178ee8b6d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
</pre>
Ceph - Bug #10428 (Closed): duplicated PGLog transaction inserted
https://tracker.ceph.com/issues/10428
2014-12-23T17:50:01Z
Yao Ning
zay11022@163.com
<p>Since append_log will call write_if_dirty any way, it always call the write_log() in PGLog. While writeout_from will always updateto the first modified eversion of the pg_log, p->version >= writeout_from must be true so that it will be inserted all non-submitted pg_log to transaction in _write_log (where I mark in the code). Thus, there is no need to submit two exactly same transaction to the FileStore. it seems to be reasonable to remove the first one to improve the performance<br />In the dump of transaction information below,
{ "offset": 1215090688,<br />"seq": 2040475,<br />"transactions": [
{ "trans_num": 0,<br />"ops": [
{ "op_num": 0,<br />"op_name": "omap_setkeys",<br />"collection": "meta",<br />"oid": "5172fe8b\/pglog_0.76\/0\/\/-1",<br />"attr_lens": { "0000000198.00000000000000017999": 171}},
{ "op_num": 1,<br />"op_name": "omap_setkeys",<br />"collection": "meta",<br />"oid": "16ef7597\/infos\/head\/\/-1",<br />"attr_lens": { "0.76_epoch": 4,<br />"0.76_info": 729}},
{ "op_num": 2,<br />"op_name": "omap_rmkeys",<br />"collection": "meta",<br />"oid": "5172fe8b\/pglog_0.76\/0\/\/-1"},
{ "op_num": 3,<br />"op_name": "omap_setkeys",<br />"collection": "meta",<br />"oid": "5172fe8b\/pglog_0.76\/0\/\/-1",<br />"attr_lens": { "0000000198.00000000000000017999": 171,<br />"can_rollback_to": 12}},
{ "op_num": 4,<br />"op_name": "write",<br />"collection": "0.76_head",<br />"oid": "bd96de76\/rb.0.105c.6b8b4567.0000000013fa\/head\/\/0",<br />"length": 4096,<br />"offset": 4100096,<br />"bufferlist length": 4096},
{ "op_num": 5,<br />"op_name": "setattr",<br />"collection": "0.76_head",<br />"oid": "bd96de76\/rb.0.105c.6b8b4567.0000000013fa\/head\/\/0",<br />"name": "",<br />"length": 258},
{ "op_num": 6,<br />"op_name": "setattr",<br />"collection": "0.76_head",<br />"oid": "bd96de76\/rb.0.105c.6b8b4567.0000000013fa\/head\/\/0",<br />"name": "snapset",<br />"length": 31}]}]},<br />Here is one of the write transaction, it also says that op_num: 0 is actually the same with op_num: 3.</p>
<p>so is it safe to do that? Maybe I will miss something.</p>