Project

General

Profile

Actions
Copy link

Bug #23653

closed

tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_db

Added by Yuri Weinstein about 6 years ago. Updated almost 6 years ago.

Status:
Resolved
Priority:
Immediate
Assignee:
Kefu Chai
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
smoke
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is on rhel7.5 qst run

Run: http://pulpito.ceph.com/teuthology-2018-04-10_20:02:32-smoke-master-testing-basic-smithi/
Jobs: '2381353', '2381344', '2381333', '2381351', '2381334', '2381336', '2381331', '2381352', '2381355', '2381340', '2381335', '2381339', '2381337', '2381354', '2381350', '2381345', '2381348', '2381332', '2381347', '2381341', '2381338', '2381329', '2381342', '2381349', '2381346'

Logs:

2018-04-10T20:58:59.441 INFO:teuthology.orchestra.run.smithi011.stderr:src/tcmalloc.cc:284] Attempt to free invalid pointer 0x55de11f2a540
2018-04-10T20:58:59.442 INFO:teuthology.orchestra.run.smithi011.stderr:*** Caught signal (Aborted) **
2018-04-10T20:58:59.442 INFO:teuthology.orchestra.run.smithi011.stderr: in thread 7f2c8e12f0c0 thread_name:ceph-osd
2018-04-10T20:58:59.444 INFO:teuthology.orchestra.run.smithi011.stderr: ceph version 13.0.2-918-gbd0c68e (bd0c68e085a84d0c972925d2992ef4fb5a2d6e5f) mimic (dev)
2018-04-10T20:58:59.444 INFO:teuthology.orchestra.run.smithi011.stderr: 1: (()+0x8e84d0) [0x55de0f7324d0]
2018-04-10T20:58:59.444 INFO:teuthology.orchestra.run.smithi011.stderr: 2: (()+0xf680) [0x7f2c829c2680]
2018-04-10T20:58:59.444 INFO:teuthology.orchestra.run.smithi011.stderr: 3: (gsignal()+0x37) [0x7f2c819e3207]
2018-04-10T20:58:59.444 INFO:teuthology.orchestra.run.smithi011.stderr: 4: (abort()+0x148) [0x7f2c819e48f8]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 5: (tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem)+0x1e6) [0x7f2c840288d6]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 6: (()+0x174b4) [0x7f2c8401d4b4]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 7: (rocksdb::LRUCache::~LRUCache()+0x65) [0x55de0f75c4a5]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 8: (std::_Sp_counted_ptr<rocksdb::BlockBasedTableFactory*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x25a) [0x55de0f83f69a]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 9: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x55de0f241356]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 10: (rocksdb::ColumnFamilyOptions::~ColumnFamilyOptions()+0x1e) [0x55de0f68fd6e]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 11: (RocksDBStore::init(std::string)+0x75) [0x55de0f686865]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 12: (BlueStore::_open_db(bool, bool)+0xe48) [0x55de0f61c2e8]
2018-04-10T20:58:59.445 INFO:teuthology.orchestra.run.smithi011.stderr: 13: (BlueStore::mkfs()+0x699) [0x55de0f649fa9]
2018-04-10T20:58:59.446 INFO:teuthology.orchestra.run.smithi011.stderr: 14: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x55de0f219447]
2018-04-10T20:58:59.446 INFO:teuthology.orchestra.run.smithi011.stderr: 15: (main()+0x2adc) [0x55de0f0ed40c]
2018-04-10T20:58:59.446 INFO:teuthology.orchestra.run.smithi011.stderr: 16: (__libc_start_main()+0xf5) [0x7f2c819cf3d5]
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr: 17: (()+0x3830d0) [0x55de0f1cd0d0]
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr:2018-04-10 20:58:59.442 7f2c8e12f0c0 -1 *** Caught signal (Aborted) **
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr: in thread 7f2c8e12f0c0 thread_name:ceph-osd
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr:
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr: ceph version 13.0.2-918-gbd0c68e (bd0c68e085a84d0c972925d2992ef4fb5a2d6e5f) mimic (dev)
2018-04-10T20:58:59.454 INFO:teuthology.orchestra.run.smithi011.stderr: 1: (()+0x8e84d0) [0x55de0f7324d0]
2018-04-10T20:58:59.455 INFO:teuthology.orchestra.run.smithi011.stderr: 2: (()+0xf680) [0x7f2c829c2680]
2018-04-10T20:58:59.455 INFO:teuthology.orchestra.run.smithi011.stderr: 3: (gsignal()+0x37) [0x7f2c819e3207]
2018-04-10T20:58:59.455 INFO:teuthology.orchestra.run.smithi011.stderr: 4: (abort()+0x148) [0x7f2c819e48f8]
2018-04-10T20:58:59.455 INFO:teuthology.orchestra.run.smithi011.stderr: 5: (tcmalloc::Log(tcmalloc::LogMode, char const*, int, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem)+0x1e6) [0x7f2c840288d6]
2018-04-10T20:58:59.455 INFO:teuthology.orchestra.run.smithi011.stderr: 6: (()+0x174b4) [0x7f2c8401d4b4]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 7: (rocksdb::LRUCache::~LRUCache()+0x65) [0x55de0f75c4a5]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 8: (std::_Sp_counted_ptr<rocksdb::BlockBasedTableFactory*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x25a) [0x55de0f83f69a]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 9: (std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46) [0x55de0f241356]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 10: (rocksdb::ColumnFamilyOptions::~ColumnFamilyOptions()+0x1e) [0x55de0f68fd6e]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 11: (RocksDBStore::init(std::string)+0x75) [0x55de0f686865]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 12: (BlueStore::_open_db(bool, bool)+0xe48) [0x55de0f61c2e8]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 13: (BlueStore::mkfs()+0x699) [0x55de0f649fa9]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 14: (OSD::mkfs(CephContext*, ObjectStore*, std::string const&, uuid_d, int)+0x177) [0x55de0f219447]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 15: (main()+0x2adc) [0x55de0f0ed40c]
2018-04-10T20:58:59.456 INFO:teuthology.orchestra.run.smithi011.stderr: 16: (__libc_start_main()+0xf5) [0x7f2c819cf3d5]
2018-04-10T20:58:59.457 INFO:teuthology.orchestra.run.smithi011.stderr: 17: (()+0x3830d0) [0x55de0f1cd0d0]
2018-04-10T20:58:59.457 INFO:teuthology.orchestra.run.smithi011.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Related issues 3 (0 open — 3 closed)

Related to Ceph - Bug #21422: crash in rocksdb LRUCache destructor with tcmalloc v4.2.6 / gperf-tools v2.5.93ResolvedKefu Chai09/18/2017

Actions
Related to RADOS - Bug #35969: "symbol lookup error: ceph-osd: undefined symbol: _ZdaPvm" on centos 7.4ResolvedKefu Chai09/13/2018

Actions
Copied to bluestore - Backport #24154: mimic: tcmalloc Attempt to free invalid pointer 0x55de11f2a540 in rocksdb::LRUCache::~LRUCache during mkfs->_open_dbResolvedKefu ChaiActions
Actions
Copy link

Also available in: Atom PDF