Project

General

Profile

Actions

Bug #62173

open

"rados get" command can't get complete data when bluestore uses LruBufferCacheShard

Added by taki zhao 10 months ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I set the buffer cache of bluestore to LRU.

When I put an object whose size is about 100MB to rados using "./bin/rados -p cache_pool put tar /root/Boost.tar.bz2" command, I cannot get all the data using "./bin/rados -p cache_pool get tar /root/tmp1" command (I get the /root/tmp1 whose size is about 12MB), and the command does not seem to be terminated.

It is worth mentioning that I am using the vstart environment, and the vstart startup command is "../src/vstart.sh --debug --new -x --localhost --bluestore".

backtrace

0> 2023-07-26T15:48:58.726+0800 fff80e5caf90 -1 ** Caught signal (Aborted) *
in thread fff80e5caf90 thread_name:tp_osd_tp

```cpp
1: _kernel_rt_sigreturn()
2: gsignal()
3: abort()
4: /usr/lib64/libc.so.6(+0x2fa0c) [0xfffc2dbcfa0c]
5: /usr/lib64/libc.so.6(+0x2fa8c) [0xfffc2dbcfa8c]
6: (LruBufferCacheShard::_rm(BlueStore::Buffer*)+0xf4) [0xaaac355770bc]
7: (BlueStore::BufferSpace::_rm_buffer(BlueStore::BufferCacheShard*, std::_Rb_tree_iterator<std::pair<unsigned int const, std::unique_ptr<BlueStore::Buffer, std::default_delete<BlueStore::Buffer> > > >)+0x40) [0xaaac35582a5c]
8: (LruBufferCacheShard::_trim_to(unsigned long)+0xd8) [0xaaac355aab6c]
9: (BlueStore::BufferSpace::did_read(BlueStore::BufferCacheShard*, unsigned int, ceph::buffer::v15_2_0::list&)+0x210) [0xaaac355a4114]
10: (BlueStore::_generate_read_result_bl(boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, std::map<unsigned long, ceph::buffer::v15_2_0::list, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, ceph::buffer::v15_2_0::list> > >&, std::vector<ceph::buffer::v15_2_0::list, std::allocator<ceph::buffer::v15_2_0::list> >&, std::map<boost::intrusive_ptr<BlueStore::Blob>, std::
_cxx11::list<BlueStore::read_req_t, std::allocator<BlueStore::read_req_t> >, std::less<boost::intrusive_ptr<BlueStore::Blob> >, std::allocator<std::pair<boost::intrusive_ptr<BlueStore::Blob> const, std::__cxx11::list<BlueStore::read_req_t, std::allocator<BlueStore::read_req_t> > > > >&, bool, bool*, ceph::buffer::v15_2_0::list&)+0x5f8) [0xaaac355140d4]
11: (BlueStore::_do_read(BlueStore::Collection*, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int, unsigned long)+0x81c) [0xaaac35520770]
12: (BlueStore::read(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ghobject_t const&, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&, unsigned int)+0x3a8) [0xaaac355581d4]
13: (ReplicatedBackend::objects_read_sync(hobject_t const&, unsigned long, unsigned long, unsigned int, ceph::buffer::v15_2_0::list*)+0x94) [0xaaac35385030]
14: (PrimaryLogPG::do_read(PrimaryLogPG::OpContext*, OSDOp&)+0x738) [0xaaac350a2218]
15: (PrimaryLogPG::do_osd_ops(PrimaryLogPG::OpContext*, std::vector<OSDOp, std::allocator<OSDOp> >&)+0xa08) [0xaaac350dbfc4]
16: (PrimaryLogPG::prepare_transaction(PrimaryLogPG::OpContext*)+0x158) [0xaaac350ebfe8]
17: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x474) [0xaaac350ec83c]
18: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x2f68) [0xaaac350f0a50]
19: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xad4) [0xaaac350f701c]
20: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x470) [0xaaac34f5dcfc]
21: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x80) [0xaaac35260fb0]
22: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x26ec) [0xaaac34f8b0c4]
23: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x830) [0xaaac356e0fe0]
24: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0xaaac356e3c8c]
25: (Thread::entry_wrapper()+0x4c) [0xaaac356ccfdc]
26: (Thread::_entry_func(void*)+0xc) [0xaaac356ccffc]
27: /usr/lib64/libpthread.so.0(+0x88cc) [0xfffc2e1388cc]
28: /usr/lib64/libc.so.6(+0xda12c) [0xfffc2dc7a12c]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.


Files

QQ截图20230726155931.jpg (26.8 KB) QQ截图20230726155931.jpg taki zhao, 07/26/2023 08:07 AM
Actions #1

Updated by taki zhao 10 months ago

I am new to the community so I didn't notice that there is a bluestore subproject to create an issue. I have recreate [an issue](https://tracker.ceph.com/issues/62175). Sorry for that.

Actions

Also available in: Atom PDF