Project

General

Profile

Actions
Copy link

Bug #16177

closed

leveldb horrendously slow

Added by Adam Tygart almost 8 years ago. Updated almost 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
Performance/Resource Usage
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v10.2.1, Ceph - v9.2.1
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Recently ran into an issue using cephfs where loading pg data on osd start (with some lightning fast ssds) took long enough to cause the osd to suicide timeout. This caused cascading suicide timeouts through more than half of my ssds (used exclusively for metadata within cephfs).

The mailing list suggested using ceph-objectstore-tool to export the pgs, delete them off the osd and re-import them. Preferably not in that order, but I digress.

Below is a complete archive of an osd (only 2GB of data compressed, and 8GB uncompressed) and a pg that took just over 3 days to extract (200MB of data).
http://people.cis.ksu.edu/~mozes/ceph-osd.16.tar.gz
http://people.cis.ksu.edu/~mozes/osd.16.32.10c.out

During the extraction, ceph-objectstore-tool pegged a single cpu for the duration. Something has to be wrong here, I cannot imagine an algorithm performing this poorly on a data structure that should be able to fit entirely in main memory. We're now at almost day 5 of extracting slow (damaged?) pgs off of 8 ssds.

My understanding is that leveldb can start to misbehave on objects with 300+million keys, (based on mailing list reports), but I don't believe I've got any objects that have more than 1 million keys. Whats more is that 300+million keys still only took 8 hours to extract based on other reports.

If I can provide any other information, I'd be glad to. More detail is available on the mailing list archives ( http://thread.gmane.org/gmane.comp.file-systems.ceph.user/30016 )

Actions
Copy link
 #1

Updated by Brad Hubbard almost 8 years ago

Hey Adam,

I don't know a lot about ceph_objectstore_tool but to get the ball rolling my naive approach would be to do some profiling in order to establish where the code is spending most of its time. To do this I would load debuginfo for Ceph and then use an approach like http://poormansprofiler.org/. Multiple runs would probably give the best insight as the laws of statistics should be on our side. You could post the results here if they seem meaningful/relevant.

Actions
Copy link
 #2

Updated by Adam Tygart almost 8 years ago

How about a 

perf trace
? http://people.cis.ksu.edu/~mozes/perftrace.objectstore-tool.out

Anyway, here's 5 minutes of "poormansprofiler" run every 10 seconds.

     90 pthread_cond_wait@@GLIBC_2.3.2,Finisher::finisher_thread_entry(),start_thread,clone
     60 pthread_cond_timedwait@@GLIBC_2.3.2,ThreadPool::worker(ThreadPool::WorkThread*),ThreadPool::WorkThread::entry(),start_thread,clone
     30 pthread_cond_wait@@GLIBC_2.3.2,WBThrottle::get_next_should_flush(boost::tuples::tuple<ghobject_t,,WBThrottle::entry(),start_thread,clone
     30 pthread_cond_wait@@GLIBC_2.3.2,SafeTimer::timer_thread(),SafeTimerThread::entry(),start_thread,clone
     30 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_thread_entry(),FileJournal::Writer::entry(),start_thread,clone
     30 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_finish_thread_entry(),FileJournal::WriteFinisher::entry(),start_thread,clone
     30 pthread_cond_wait@@GLIBC_2.3.2,ceph::log::Log::entry(),start_thread,clone
     30 pthread_cond_timedwait@@GLIBC_2.3.2,Cond::WaitInterval(CephContext*,,FileStore::sync_entry(),FileStore::SyncThread::entry(),start_thread,clone
     30 pthread_cond_timedwait@@GLIBC_2.3.2,CephContextServiceThread::entry(),start_thread,clone
      4 snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 std::string::_M_mutate(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 __memcmp_sse4_1,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_M_mutate(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_M_mutate(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 pthread_mutex_unlock,leveldb::port::Mutex::Unlock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1
Actions
Copy link
 #3

Updated by Adam Tygart almost 8 years ago

And, to add to that, 5 and a half hours of profiling using both methods:

http://people.cis.ksu.edu/~mozes/perftrace.objectstore-tool.out.gz

   6000 pthread_cond_wait@@GLIBC_2.3.2,Finisher::finisher_thread_entry(),start_thread,clone
   4000 pthread_cond_timedwait@@GLIBC_2.3.2,ThreadPool::worker(ThreadPool::WorkThread*),ThreadPool::WorkThread::entry(),start_thread,clone
   2000 pthread_cond_wait@@GLIBC_2.3.2,WBThrottle::get_next_should_flush(boost::tuples::tuple<ghobject_t,,WBThrottle::entry(),start_thread,clone
   2000 pthread_cond_wait@@GLIBC_2.3.2,SafeTimer::timer_thread(),SafeTimerThread::entry(),start_thread,clone
   2000 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_thread_entry(),FileJournal::Writer::entry(),start_thread,clone
   2000 pthread_cond_wait@@GLIBC_2.3.2,Cond::Wait(Mutex&),FileJournal::write_finish_thread_entry(),FileJournal::WriteFinisher::entry(),start_thread,clone
   2000 pthread_cond_wait@@GLIBC_2.3.2,ceph::log::Log::entry(),start_thread,clone
   2000 pthread_cond_timedwait@@GLIBC_2.3.2,Cond::WaitInterval(CephContext*,,FileStore::sync_entry(),FileStore::SyncThread::entry(),start_thread,clone
   2000 pthread_cond_timedwait@@GLIBC_2.3.2,CephContextServiceThread::entry(),start_thread,clone
    522 snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveld
b::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
    252 leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_regio
n(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
    132 std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIterat
orImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
    128 std::string::_M_mutate(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMa
pIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
    114 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMa
p::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     97 __memcmp_sse4_1,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::
DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     82 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImp
l::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     57 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     50 leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     49 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DB
ObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     38 leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapItera
torImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_ex
port(ObjectStore*,,main
     37 leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete
_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     33 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     30 leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectM
ap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     26 std::string::resize(unsigned,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIt
eratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     25 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObj
ectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     23 std::string::_M_mutate(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     20 leveldb::GetVarint32PtrFallback(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     19 snappy::GetUncompressedLength(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     18 leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     16 std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     16 __memcmp_sse4_1,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::a
djust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     15 tc_deletearray,leveldb::DeleteCachedBlock(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,Le
velDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreT
ool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     14 pthread_mutex_unlock,leveldb::port::Mutex::Unlock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeS
paceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectSt
ore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
     14 __memcpy_ssse3_back,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBOb
jectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore
*,,ObjectStoreTool::do_export(ObjectStore*,,main
     13 std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMap
IteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::
do_export(ObjectStore*,,main
     11 memcpy@plt,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObje
ctMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreT
ool::do_export(ObjectStore*,,main
     10 std::string::_M_mutate(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBOb
jectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStor
eTool::do_export(ObjectStore*,,main
      9 std::string::assign(char,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get
_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      9 leveldb::Block::~Block(),leveldb::DeleteCachedBlock(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(an
onymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,Ob
jectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      7 pthread_mutex_lock,leveldb::port::Mutex::Lock(),leveldb::(anonymous,leveldb::Iterator::~Iterator(),leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImp
l>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      7 leveldb::Block::Iter::key(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region
(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      6 tc_deletearray,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,Obje
ctStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      6 pthread_mutex_lock,leveldb::port::Mutex::Lock(),leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*
,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      6 memcmp@plt,leveldb::(anonymous,leveldb::InternalKeyComparator::Compare(leveldb::Slice,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStore
Tool::do_export(ObjectStore*,,main
      5 tc_new,std::string::_Rep::_S_create(unsigned,std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::Gen
ericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      5 leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapItera
torImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      5 leveldb::GetVarint32PtrFallback(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,Objec
tStoreTool::do_export(ObjectStore*,,main
      4 std::string::resize,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImp
l::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(O
bjectStore*,,main
      4 leveldb::Block::Iter::value(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_regi
on(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      4 leveldb::Block::Iter::Valid(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 tc_newarray,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectS
toreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 std::string::resize(unsigned,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTo
ol::do_export(ObjectStore*,,main
      3 std::string::append,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(O
bjectStore*,,main
      3 __memcpy_ssse3_back,std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anony
mous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 __memcpy_ssse3_back,snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 leveldb::Hash(char,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      3 leveldb::GetVarint32PtrFallback,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 tc_new,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool
::do_export(ObjectStore*,,main
      2 std::string::_M_mutate,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::D
BObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 std::string::_M_mutate,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIt
eratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do
_export(ObjectStore*,,main
      2 std::string::assign(char,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::i
n_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(Objec
tStore*,,main
      2 snappy::SnappyDecompressor::RefillTag(),snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveld
b::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::sh
ared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 __memcpy_ssse3_back,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObje
ctMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,
,ObjectStoreTool::do_export(ObjectStore*,,main
      2 __memcmp_sse4_1,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::
string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(
ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to
_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::ex
port_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek
_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool:
:export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      2 leveldb::(anonymous,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceI
teratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 tc_delete,leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBOb
jectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore
*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 tc_delete,leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool:
:export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(O
bjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 ??,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(Obje
ctStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_Rep::_M_clone(std::allocator<char>,std::string::reserve(unsigned,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_f
ile(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_M_mutate(unsigned,std::string::_M_replace_safe(unsigned,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::s
tring,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::_M_mutate,leveldb::Block::Iter::Prev(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_expor
t(ObjectStore*,,main
      1 std::string::_M_mutate,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_o
map_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 std::string::assign,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap
_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 snappy::ByteArraySource::Skip(unsigned,snappy::SnappyDecompressor::RefillTag(),snappy::RawUncompress(snappy::Source*,,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb
::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 snappy::ByteArraySource::~ByteArraySource,snappy::RawUncompress(char,leveldb::ReadBlock(leveldb::RandomAccessFile*,,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorI
mpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 operator,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTo
ol::do_export(ObjectStore*,,main
      1 __memcpy_ssse3_back,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_fi
les(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 memcpy@plt,std::string::append(char,leveldb::Block::Iter::SeekToLast(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::
DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::Iterator::~Iterator(),leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::GetVarint64Ptr(char,leveldb::GetVarint64(leveldb::Slice*,,leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::GetVarint64(leveldb::Slice*,,leveldb::BlockHandle::DecodeFrom(leveldb::Slice*),leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::Block::NewIterator(leveldb::Comparator,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::e
xport_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::Block::Iter::~Iter(),leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::D
BObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectS
toreTool::do_export(ObjectStore*,,main
      1 leveldb::(anonymous,leveldb::(anonymous,leveldb::Table::BlockReader(void*,,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(s
td::string,DBObjectMap::DBObjectMapIteratorImpl::in_complete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_fi
les(ObjectStore*,,ObjectStoreTool::do_export(ObjectStore*,,main
      1 leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,leveldb::(anonymous,LevelDBStore::LevelDBWholeSpaceIteratorImpl::seek_to_last(std::string,DBObjectMap::DBObjectMapIteratorImpl::in_com
plete_region(std::string,DBObjectMap::DBObjectMapIteratorImpl::adjust(),get_omap_batch(std::shared_ptr<KeyValueDB::GenericIteratorImpl>&,,ObjectStoreTool::export_file(ObjectStore*,,ObjectStoreTool::export_files(ObjectStore*,,ObjectStoreTool::do_export(ObjectStor
e*,,main
Actions
Copy link
 #4

Updated by Kefu Chai almost 8 years ago

Adam. when did you collected the trace using "poormansprofiler". i think it was collected when you were importing the pglog right?

Actions
Copy link
 #5

Updated by Kefu Chai almost 8 years ago

i will try to reproduce this with ceph-osd.16.tar.gz, following the steps at http://thread.gmane.org/gmane.comp.file-systems.ceph.user/30016.

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-16 --journal-path /var/lib/ceph/osd/ceph-16/journal --pgid 32.10c --op export --file /root/32.10c.b.export
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-100 --journal-path /var/lib/ceph/osd/ceph-100/journal --pgid 32.10c --op export --file /root/32.10c.b.export

Adam, you might want to enable "--xattrs" when archiving osd.16, we store the user xattr and system xattr using it, also you failed to include the journal (it's a symbolic link to "/dev/disk/by-partuuid/22e5b992-d21b-417a-aba9-b9d3fad59747").

Actions
Copy link
 #6

Updated by Kefu Chai almost 8 years ago

  • Status changed from New to Need More Info
Actions
Copy link
 #7

Updated by Adam Tygart almost 8 years ago

This is what I used when compressing the osd:
tar -c -z --xattrs --selinux --acls --preserve -f /tmp/ceph-osd.16.tar.gz ./

I flushed the journal before this (ceph-osd -i 16 --flush-journal), so including the journal shouldn't be important.

And both the perf trace and poormansprofiler were collected during export of pgs.

Actions
Copy link
 #8

Updated by Brad Hubbard almost 8 years ago

Hi Adam,

We're looking into why it appears to be spending so much time doing writes and why the write back throttle is being activated while you are doing an export. At first glance it seems quite odd which prompted the question about when it was collected. Not what we expected to see.

Actions
Copy link
 #9

Updated by Adam Tygart almost 8 years ago

Since this export/import process is being so slow (now running for a week), and the inodes I'm working on extracting are the "stray" inodes, is there a chance that stopping the export, turning the suicide timeout options way up (which ones? does it need to be on the osd or monitors or both?), and restarting the osds will let the mds server start clearing out these leveldb entries?

How likely?

Is there any way to get an estimate on the number of keys on an inode? I'd like to be able to get some estimate of how far along in the export process we actually are. A week of extracting a single pg off each of these 8 ssds is hard to swallow. And the thought of losing all this progress now scary as heck.

Actions
Copy link
 #10

Updated by Kefu Chai almost 8 years ago

Adam,

seems we are throttling when exporting the pglog to the OSD (yeah, it's a bug, imo!). you can disable it by passing "--filestore_wbthrottle_enable=false" to "ceph-objectstore-tool" along with other options. but yes, you need to terminate the ceph-objectstore-tool, and start it over.

This is what I used when compressing the osd:
tar -c -z --xattrs --selinux --acls --preserve -f /tmp/ceph-osd.16.tar.gz ./

oh, so it's me who didn't pass '--xattrs' to tar, will try again in my test bed. sorry for the noise.

Actions
Copy link
 #11

Updated by Kefu Chai almost 8 years ago

  • Status changed from Need More Info to New
Actions
Copy link
 #12

Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to RADOS
  • Category changed from leveldb to Performance/Resource Usage
  • Status changed from New to Closed
  • Component(RADOS) OSD added

Adam's cluster got cleaned up; the MDS doesn't allow you to generate directory omaps that large anymore; RGW is doing likewise with their sharding; and we have http://tracker.ceph.com/issues/16172 to restrict their size once that becomes realistic.

Actions
Copy link

Also available in: Atom PDF