Project

General

Profile

osd.17.log.txt

Alex Litvak, 10/01/2019 07:09 AM

Download (18.5 KB)

 
1
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block) aio_submit retries 16
2
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.867 7f093d71e700 -1 bdev(0x55b72c156000 /var/lib/ceph/osd/ceph-17/block)  aio submit got (11) Resource temporarily unavailable
3
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: In fun
4
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: 757: F
5
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
6
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55b71f668cf4]
7
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55b71f668ec2]
8
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
9
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
10
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
11
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::T
12
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55b71f9b1b84]
13
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, s
14
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
15
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
16
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3965) [0x55b71f96de15]
17
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
18
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
19
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
20
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) [0x55b71f7c6ef4]
21
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) [0x55b71fdc5ce3]
22
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
23
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (()+0x7dd5) [0x7f0971da9dd5]
24
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (clone()+0x6d) [0x7f0970c7002d]
25
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.879 7f093d71e700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/
26
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/14.2.2/rpm/el7/BUILD/ceph-14.2.2/src/os/bluestore/KernelDevice.cc: 757: F
27
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
28
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14a) [0x55b71f668cf4]
29
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55b71f668ec2]
30
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
31
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
32
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
33
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::T
34
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55b71f9b1b84]
35
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, s
36
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
37
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
38
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3965) [0x55b71f96de15]
39
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
40
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
41
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
42
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) [0x55b71f7c6ef4]
43
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) [0x55b71fdc5ce3]
44
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
45
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (()+0x7dd5) [0x7f0971da9dd5]
46
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (clone()+0x6d) [0x7f0970c7002d]
47
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: *** Caught signal (Aborted) **
48
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  in thread 7f093d71e700 thread_name:tp_osd_tp
49
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
50
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: (()+0xf5d0) [0x7f0971db15d0]
51
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: (gsignal()+0x37) [0x7f0970ba82c7]
52
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: (abort()+0x148) [0x7f0970ba99b8]
53
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55b71f668d43]
54
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55b71f668ec2]
55
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
56
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
57
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
58
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::T
59
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55b71f9b1b84]
60
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&,
61
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
62
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
63
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3965) [0x55b71f96de15]
64
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
65
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
66
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
67
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) [0x55b71f7c6ef4]
68
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) [0x55b71fdc5ce3]
69
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  20: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
70
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  21: (()+0x7dd5) [0x7f0971da9dd5]
71
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  22: (clone()+0x6d) [0x7f0970c7002d]
72
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: 2019-09-30 22:52:58.883 7f093d71e700 -1 *** Caught signal (Aborted) **
73
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  in thread 7f093d71e700 thread_name:tp_osd_tp
74
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  ceph version 14.2.2 (4f8fa0a0024755aae7d95567c63f11d6862d55be) nautilus (stable)
75
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  1: (()+0xf5d0) [0x7f0971db15d0]
76
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  2: (gsignal()+0x37) [0x7f0970ba82c7]
77
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  3: (abort()+0x148) [0x7f0970ba99b8]
78
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x199) [0x55b71f668d43]
79
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  5: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55b71f668ec2]
80
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  6: (KernelDevice::aio_submit(IOContext*)+0x701) [0x55b71fd61ca1]
81
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  7: (BlueStore::_txc_aio_submit(BlueStore::TransContext*)+0x42) [0x55b71fc29892]
82
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  8: (BlueStore::_txc_state_proc(BlueStore::TransContext*)+0x42b) [0x55b71fc496ab]
83
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  9: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::T
84
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  10: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54) [0x55b71f9b1b84]
85
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  11: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&,
86
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  12: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xf12) [0x55b71f90e322]
87
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  13: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0xfae) [0x55b71f969b7e]
88
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  14: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x3965) [0x55b71f96de15]
89
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  15: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xbd4) [0x55b71f96f8a4]
90
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  16: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1a9) [0x55b71f7a9ea9]
91
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  17: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0x55b71fa475d2]
92
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  18: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x9f4) [0x55b71f7c6ef4]
93
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  19: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x433) [0x55b71fdc5ce3]
94
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  20: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x55b71fdc8d80]
95
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  21: (()+0x7dd5) [0x7f0971da9dd5]
96
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  22: (clone()+0x6d) [0x7f0970c7002d]
97
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
98
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]: --- begin dump of recent events ---
99
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9999> 2019-09-30 20:46:02.076 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038
100
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9998> 2019-09-30 20:46:03.776 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038
101
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9997> 2019-09-30 20:46:04.277 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038
102
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9996> 2019-09-30 20:46:04.777 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038
103
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9995> 2019-09-30 20:46:04.905 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _tune_cache_size target: 8485076992 heap: 398680064 unmapped: 9035776 mapped: 389644288 old cache_size: 5064831794 new cache siz
104
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9994> 2019-09-30 20:46:04.905 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _trim_shards cache_size: 5064831794 kv_alloc: 1979711488 kv_used: 120635251 meta_alloc: 1979711488 meta_used: 117486636 data_all
105
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9993> 2019-09-30 20:46:05.813 7f094ccd2700 10 monclient: tick
106
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9992> 2019-09-30 20:46:05.813 7f094ccd2700 10 monclient: _check_auth_rotating have uptodate secrets (they expire after 2019-09-30 20:45:35.819641)
107
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9991> 2019-09-30 20:46:09.913 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _tune_cache_size target: 8485076992 heap: 398680064 unmapped: 9035776 mapped: 389644288 old cache_size: 5064831794 new cache siz
108
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9990> 2019-09-30 20:46:09.913 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _trim_shards cache_size: 5064831794 kv_alloc: 1979711488 kv_used: 120635251 meta_alloc: 1979711488 meta_used: 117486836 data_all
109
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9989> 2019-09-30 20:46:10.681 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038
110
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9988> 2019-09-30 20:46:14.569 7f094dcd4700  2 osd.17 6328 ms_handle_reset con 0x55b72c797000 session 0x55b7401f1800
111
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9987> 2019-09-30 20:46:14.569 7f096da92700 10 monclient: handle_auth_request added challenge on 0x55b733bf0400
112
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9986> 2019-09-30 20:46:14.917 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _tune_cache_size target: 8485076992 heap: 398680064 unmapped: 9035776 mapped: 389644288 old cache_size: 5064831794 new cache siz
113
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9985> 2019-09-30 20:46:14.917 7f0950744700  5 bluestore.MempoolThread(0x55b72c210a88) _trim_shards cache_size: 5064831794 kv_alloc: 1979711488 kv_used: 120635251 meta_alloc: 1979711488 meta_used: 117487036 data_all
114
Sep 30 22:52:58 storage2n2-la ceph-osd-17[10770]:  -9984> 2019-09-30 20:46:15.381 7f0937f13700  5 osd.17 6328 heartbeat osd_stat(store_statfs(0x1a485594000/0x40000000/0x1bf00000000, data 0x19cfcadc2a/0x1a3aa68000, compress 0x0/0x0/0x0, omap 0xcdefc78, meta 0x3321038