Bug #16177
closedleveldb horrendously slow
0%
Description
Recently ran into an issue using cephfs where loading pg data on osd start (with some lightning fast ssds) took long enough to cause the osd to suicide timeout. This caused cascading suicide timeouts through more than half of my ssds (used exclusively for metadata within cephfs).
The mailing list suggested using ceph-objectstore-tool to export the pgs, delete them off the osd and re-import them. Preferably not in that order, but I digress.
Below is a complete archive of an osd (only 2GB of data compressed, and 8GB uncompressed) and a pg that took just over 3 days to extract (200MB of data).
http://people.cis.ksu.edu/~mozes/ceph-osd.16.tar.gz
http://people.cis.ksu.edu/~mozes/osd.16.32.10c.out
During the extraction, ceph-objectstore-tool pegged a single cpu for the duration. Something has to be wrong here, I cannot imagine an algorithm performing this poorly on a data structure that should be able to fit entirely in main memory. We're now at almost day 5 of extracting slow (damaged?) pgs off of 8 ssds.
My understanding is that leveldb can start to misbehave on objects with 300+million keys, (based on mailing list reports), but I don't believe I've got any objects that have more than 1 million keys. Whats more is that 300+million keys still only took 8 hours to extract based on other reports.
If I can provide any other information, I'd be glad to. More detail is available on the mailing list archives ( http://thread.gmane.org/gmane.comp.file-systems.ceph.user/30016 )
Updated by Brad Hubbard almost 8 years ago
Hey Adam,
I don't know a lot about ceph_objectstore_tool but to get the ball rolling my naive approach would be to do some profiling in order to establish where the code is spending most of its time. To do this I would load debuginfo for Ceph and then use an approach like http://poormansprofiler.org/. Multiple runs would probably give the best insight as the laws of statistics should be on our side. You could post the results here if they seem meaningful/relevant.
Updated by Adam Tygart almost 8 years ago
How about a
perf trace? http://people.cis.ksu.edu/~mozes/perftrace.objectstore-tool.out
Anyway, here's 5 minutes of "poormansprofiler" run every 10 seconds.
90 pthread_cond_wait@@GLIBC_2.3.2,Finisher::finisher_thread_entry(),start_thread,clone 60 pthread_cond_timedwait@@GLIBC_2.3.2,ThreadPool::worker(ThreadPool::WorkThread*),ThreadPool::WorkThread::entry(),start_thread,clone 30 pthread_cond_wait@@GLIBC_2.3.2,WBThrottle::get_next_should_flush(boost::tuples::tuple<ghobject_t,,WBThrottle::entry(),start_thread,clone 30 pthread_cond_wait@@GLIBC_2.3.2,SafeTimer::timer_thread(),SafeTimerThread::entry(),start_thread,clone 30 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_thread_entry(),FileJournal::Writer::entry(),start_thread,clone 30 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_finish_thread_entry(),FileJournal::WriteFinisher::entry(),start_thread,clone 30 pthread_cond_wait@@GLIBC_2.3.2,ceph::log::Log::entry(),start_thread,clone 30 pthread_cond_timedwait@@GLIBC_2.3.2,Cond::WaitInterval(CephContext*,,FileStore::sync_entry(),FileStore::SyncThread::entry(),start_thread,clone 30 pthread_cond_timedwait@@GLIBC_2.3.2,CephContextServiceThread::entry(),start_thread,clone 4 snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 std::string::_M_mutate(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 __memcmp_sse4_1,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_M_mutate(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_M_mutate(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 pthread_mutex_unlock,leveldb::port::Mutex::Unlock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1
Updated by Adam Tygart almost 8 years ago
And, to add to that, 5 and a half hours of profiling using both methods:
http://people.cis.ksu.edu/~mozes/perftrace.objectstore-tool.out.gz
6000 pthread_cond_wait@@GLIBC_2.3.2,Finisher::finisher_thread_entry(),start_thread,clone 4000 pthread_cond_timedwait@@GLIBC_2.3.2,ThreadPool::worker(ThreadPool::WorkThread*),ThreadPool::WorkThread::entry(),start_thread,clone 2000 pthread_cond_wait@@GLIBC_2.3.2,WBThrottle::get_next_should_flush(boost::tuples::tuple<ghobject_t,,WBThrottle::entry(),start_thread,clone 2000 pthread_cond_wait@@GLIBC_2.3.2,SafeTimer::timer_thread(),SafeTimerThread::entry(),start_thread,clone 2000 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_thread_entry(),FileJournal::Writer::entry(),start_thread,clone 2000 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_finish_thread_entry(),FileJournal::WriteFinisher::entry(),start_thread,clone 2000 pthread_cond_wait@@GLIBC_2.3.2,ceph::log::Log::entry(),start_thread,clone 2000 pthread_cond_timedwait@@GLIBC_2.3.2,Cond::WaitInterval(CephContext*,,FileStore::sync_entry(),FileStore::SyncThread::entry(),start_thread,clone 2000 pthread_cond_timedwait@@GLIBC_2.3.2,CephContextServiceThread::entry(),start_thread,clone 522 snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveld b::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 252 leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_regio n(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 132 std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIterat orImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 128 std::string::_M_mutate(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMa pIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 114 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMa p::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 97 __memcmp_sse4_1,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap:: DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 82 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImp l::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 57 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 50 leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 49 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DB ObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 38 leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapItera torImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_ex port(ObjectStore*,,main 37 leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete _region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 33 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 30 leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectM ap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 26 std::string::resize(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIt eratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 25 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObj ectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 23 std::string::_M_mutate(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 20 leveldb::GetVarint32PtrFallback(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 19 snappy::GetUncompressedLength(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 18 leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 16 std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 16 __memcmp_sse4_1,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::a djust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 15 tc_deletearray,leveldb::DeleteCachedBlock(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,Le velDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreT ool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 14 pthread_mutex_unlock,leveldb::port::Mutex::Unlock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeS paceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectSt ore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 14 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBOb jectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore *,,ObjectStoreTool::do_export(ObjectStore*,,main 13 std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMap IteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool:: do_export(ObjectStore*,,main 11 memcpy@plt,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObje ctMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreT ool::do_export(ObjectStore*,,main 10 std::string::_M_mutate(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBOb jectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStor eTool::do_export(ObjectStore*,,main 9 std::string::assign(char,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get _omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 9 leveldb::Block::~Block(),leveldb::DeleteCachedBlock(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(an onymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,Ob jectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 7 pthread_mutex_lock,leveldb::port::Mutex::Lock(),leveldb::(anonymous,leveldb::Iterator::~Iterator(),leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImp l>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 7 leveldb::Block::Iter::key(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region (std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 6 tc_deletearray,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,Obje ctStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 6 pthread_mutex_lock,leveldb::port::Mutex::Lock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore* ,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 6 memcmp@plt,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStore Tool::do_export(ObjectStore*,,main 5 tc_new,std::string::_Rep::_S_create(unsigned,std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::Gen ericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 5 leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapItera torImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 5 leveldb::GetVarint32PtrFallback(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,Objec tStoreTool::do_export(ObjectStore*,,main 4 std::string::resize,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImp l::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(O bjectStore*,,main 4 leveldb::Block::Iter::value(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_regi on(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 4 leveldb::Block::Iter::Valid(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 tc_newarray,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectS toreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 std::string::resize(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTo ol::do_export(ObjectStore*,,main 3 std::string::append,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(O bjectStore*,,main 3 __memcpy_ssse3_back,std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anony mous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 __memcpy_ssse3_back,snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 leveldb::Hash(char,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 3 leveldb::GetVarint32PtrFallback,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 tc_new,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool ::do_export(ObjectStore*,,main 2 std::string::_M_mutate,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::D BObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 std::string::_M_mutate,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIt eratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do _export(ObjectStore*,,main 2 std::string::assign(char,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::i n_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(Objec tStore*,,main 2 snappy::SnappyDecompressor::RefillTag(),snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveld b::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::sh ared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObje ctMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*, ,ObjectStoreTool::do_export(ObjectStore*,,main 2 __memcmp_sse4_1,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std:: string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files( ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to _last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::ex port_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek _to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool: :export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 2 leveldb::(anonymous,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceI teratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 tc_delete,leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBOb jectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore *,,ObjectStoreTool::do_export(ObjectStore*,,main 1 tc_delete,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool: :export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(O bjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 ??,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(Obje ctStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_f ile(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::s tring,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::_M_mutate,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_expor t(ObjectStore*,,main 1 std::string::_M_mutate,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_o map_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 std::string::assign,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap _batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 snappy::ByteArraySource::Skip(unsigned,snappy::SnappyDecompressor::RefillTag(),snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb ::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 snappy::ByteArraySource::~ByteArraySource,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorI mpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 operator,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTo ol::do_export(ObjectStore*,,main 1 __memcpy_ssse3_back,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_fi les(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 memcpy@plt,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap:: DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::Iterator::~Iterator(),leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::GetVarint64Ptr(char,leveldb::GetVarint64(leveldb::Slice*,,leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::GetVarint64(leveldb::Slice*,,leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::Block::NewIterator(leveldb::Comparator,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::e xport_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::D BObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectS toreTool::do_export(ObjectStore*,,main 1 leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(s td::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_fi les(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main 1 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_com plete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStor e*,,main
Updated by Kefu Chai almost 8 years ago
Adam. when did you collected the trace using "poormansprofiler". i think it was collected when you were importing the pglog right?
Updated by Kefu Chai almost 8 years ago
i will try to reproduce this with ceph-osd.16.tar.gz, following the steps at http://thread.gmane.org/gmane.comp.file-systems.ceph.user/30016.
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-16 --journal-path /var/lib/ceph/osd/ceph-16/journal --pgid 32.10c --op export --file /root/32.10c.b.export ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-100 --journal-path /var/lib/ceph/osd/ceph-100/journal --pgid 32.10c --op export --file /root/32.10c.b.export
Adam, you might want to enable "--xattrs" when archiving osd.16, we store the user xattr and system xattr using it, also you failed to include the journal (it's a symbolic link to "/dev/disk/by-partuuid/22e5b992-d21b-417a-aba9-b9d3fad59747").
Updated by Kefu Chai almost 8 years ago
- Status changed from New to Need More Info
Updated by Adam Tygart almost 8 years ago
This is what I used when compressing the osd:
tar -c -z --xattrs --selinux --acls --preserve -f /tmp/ceph-osd.16.tar.gz ./
I flushed the journal before this (ceph-osd -i 16 --flush-journal), so including the journal shouldn't be important.
And both the perf trace and poormansprofiler were collected during export of pgs.
Updated by Brad Hubbard almost 8 years ago
Hi Adam,
We're looking into why it appears to be spending so much time doing writes and why the write back throttle is being activated while you are doing an export. At first glance it seems quite odd which prompted the question about when it was collected. Not what we expected to see.
Updated by Adam Tygart almost 8 years ago
Since this export/import process is being so slow (now running for a week), and the inodes I'm working on extracting are the "stray" inodes, is there a chance that stopping the export, turning the suicide timeout options way up (which ones? does it need to be on the osd or monitors or both?), and restarting the osds will let the mds server start clearing out these leveldb entries?
How likely?
Is there any way to get an estimate on the number of keys on an inode? I'd like to be able to get some estimate of how far along in the export process we actually are. A week of extracting a single pg off each of these 8 ssds is hard to swallow. And the thought of losing all this progress now scary as heck.
Updated by Kefu Chai almost 8 years ago
Adam,
seems we are throttling when exporting the pglog to the OSD (yeah, it's a bug, imo!). you can disable it by passing "--filestore_wbthrottle_enable=false" to "ceph-objectstore-tool" along with other options. but yes, you need to terminate the ceph-objectstore-tool, and start it over.
This is what I used when compressing the osd:
tar -c -z --xattrs --selinux --acls --preserve -f /tmp/ceph-osd.16.tar.gz ./
oh, so it's me who didn't pass '--xattrs' to tar, will try again in my test bed. sorry for the noise.
Updated by Kefu Chai almost 8 years ago
- Status changed from Need More Info to New
Updated by Greg Farnum almost 7 years ago
- Project changed from Ceph to RADOS
- Category changed from leveldb to Performance/Resource Usage
- Status changed from New to Closed
- Component(RADOS) OSD added
Adam's cluster got cleaned up; the MDS doesn't allow you to generate directory omaps that large anymore; RGW is doing likewise with their sharding; and we have http://tracker.ceph.com/issues/16172 to restrict their size once that becomes realistic.