Infrastructure

The common error message between failed tests indicates that a string is getting set to `null` somewhere that it shouldn't. I was not able to reproduce this on my local environment, even when resetting to the latest version of master and updating the submodules.

terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid

Based on this:

*** Caught signal (Aborted) **
 in thread 7f7c95758480 thread_name:ceph-mon
 ceph version Development (no_version) quincy (dev)
 1: ceph-mon() [0x1b7d7e4]
 2: /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f7c9650d3c0]
 3: gsignal()
 4: abort()
 5: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xa1a31) [0x7f7c95bf7a31]
 6: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xad5dc) [0x7f7c95c035dc]
 7: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xad647) [0x7f7c95c03647]
 8: __cxa_rethrow()
 9: (void std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign<std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)::{lambda(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*)#1}>(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&, std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)::{lambda(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*)#1} const&)+0x1f9) [0x1ea7479]
 10: (std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Hashtable(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)+0xaa) [0x1ea71ea]
 11: (std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::unordered_map(std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&)+0x1d) [0x1ea6d0d]
 12: (rocksdb::ConfigurableHelper::ConfigureOptions(rocksdb::ConfigOptions const&, rocksdb::Configurable&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*)+0x40) [0x1ea37d0]
 13: (rocksdb::DBOptionsConfigurable::ConfigureOptions(rocksdb::ConfigOptions const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*)+0x64) [0x1ead8f4]
 14: (rocksdb::Configurable::ConfigureFromMap(rocksdb::ConfigOptions const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*)+0x48) [0x1ea33c8]
 15: (rocksdb::GetOptionsFromString(rocksdb::ConfigOptions const&, rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options*)+0x137) [0x1ec09a7]
 16: (rocksdb::GetOptionsFromString(rocksdb::Options const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options*)+0x5d) [0x1ec082d]
 17: (RocksDBStore::ParseOptionsFromStringStatic(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options&, std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options&)>)+0x756) [0x1b9b526]
 18: (RocksDBStore::ParseOptionsFromString(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options&)+0x63) [0x1b9ad93]
 19: (RocksDBStore::init(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)+0x81) [0x1b9bc81]
 20: (MonitorDBStore::_open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x862) [0x15b2a72]
 21: (MonitorDBStore::create_and_open(std::ostream&)+0x1e0) [0x159cb70]
 22: main()
 23: __libc_start_main()
 24: _start()

I'm suspecting a problem with this function in src/kv/RocksDBStore.cc. The core dump indicates a problem with "std::unordered_map<std::string, std::string> str_map". Based on the error message in note#1, I'm guessing that str_map is not getting parsed correctly.

int RocksDBStore::ParseOptionsFromStringStatic(
  CephContext *cct,
  const string& opt_str,
  rocksdb::Options& opt,
  function<int(const string&, const string&, rocksdb::Options&)> interp)
{
  // keep aligned with func tryInterpret
  const set<string> need_interp_keys = {"compaction_threads", "flusher_threads", "compact_on_mount", "disableWAL"};
  rocksdb::Status status;
  std::unordered_map<std::string, std::string> str_map;
  status = StringToMap(opt_str, &str_map);
  if (!status.ok()) {
     dout(5) << __func__ << " error '" << status.getState() <<
      "' while parsing options '" << opt_str << "'" << dendl;
    return -EINVAL;
  }

  for (auto it = str_map.begin(); it != str_map.end(); ++it) {
    string this_opt = it->first + "=" + it->second;
    rocksdb::Status status =
      rocksdb::GetOptionsFromString(opt, this_opt, &opt);
    int r = 0;
    if (!status.ok()) {
      if (interp != nullptr) {
        r = interp(it->first, it->second, opt);
      } else if (!need_interp_keys.count(it->first)) {
        r = -1;
      }
      if (r < 0) {
        derr << status.ToString() << dendl;
        return -EINVAL;
      }
    }
    lgeneric_dout(cct, 1) << " set rocksdb option " << it->first
      << " = " << it->second << dendl;
  }
  return 0;
}

After putting a try-catch block around the `GetOptionsFromString` line to get some more debug info, I can see that the loop does not complete any iterations before hitting an exception. The first instance of `this_opt` shows that it was correctly assigned.

int i = 0;
for (auto it = str_map.begin(); it != str_map.end(); ++it) {
  string this_opt = it->first + "=" + it->second;
   try {
    rocksdb::Status status =
      rocksdb::GetOptionsFromString(opt, this_opt, &opt);
    i++;
  }
  catch (...) {
    // Exception should reveal how many times the loop iterates, and
    // which part of str_map it was handling at the time.
    throw std::invalid_argument( std::to_string(i) + " " + this_opt );
  }

Result of the debug exception (0 iterations; `this_opt` was assigned correctly to a string.):

terminate called after throwing an instance of 'std::invalid_argument'
  what():  0 level_compaction_dynamic_level_bytes=true

At this point, it may be a good idea to check the logs on one of the nodes Jenkins is building these tests on (i.e. braggi15 https://jenkins.ceph.com/computer/172.21.2.15+braggi15/). Something may be wrong with the monitor configuration on these nodes.

Laura, thanks for looking into this issue. it's really annoying. my gut feeling is that it is a bug in clang or libstdc++.

#0  0x0000000000ce3925 in std::__detail::_Hash_node_value_base<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >::_M_v (this=0x17f3620) at /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/hashtable_policy.h:249
#1  0x0000000000fd4377 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Hashtable(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)::{lambda(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*)#1}::operator()[abi:cxx11](std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*) const (this=0x7fffffffc120, __n=0x17f3620)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/hashtable.h:1243
#2  0x0000000000fd4167 in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign<std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)::{lambda(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*)#1}>(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&, std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_M_assign(std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> > const&)::{lambda(std::__detail::_Hash_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, true> const*)#1} const&) (this=0x7fffffffc270, __ht=..., __node_gen=...)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/hashtable.h:1127
#3  0x0000000000fd404a in std::_Hashtable<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::_Hashtable (this=0x7fffffffc270, __ht=...) at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/hashtable.h:1241
#4  0x0000000000fd3b6d in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::unordered_map (this=0x7fffffffc270)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unordered_map.h:181
#4  0x0000000000fd3b6d in std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::unordered_map (this=0x7fffffffc270)
    at /usr/bin/../lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unordered_map.h:181
--Type <RET> for more, q to quit, c to continue without paging--
#5  0x0000000000fd0630 in rocksdb::ConfigurableHelper::ConfigureOptions (config_options=..., configurable=..., opts_map=std::unordered_map with 1 element = {...}, unused=0x7fffffffd190)
    at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/configurable.cc:257
#6  0x0000000000fda784 in rocksdb::DBOptionsConfigurable::ConfigureOptions (this=0x17f3680, config_options=..., opts_map=std::unordered_map with 1 element = {...}, unused=0x7fffffffd190)
    at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/db_options.cc:468
#7  0x0000000000fd0228 in rocksdb::Configurable::ConfigureFromMap (this=0x17f3680, config_options=..., opts_map=std::unordered_map with 1 element = {...}, unused=0x7fffffffd190)
    at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/configurable.cc:153
#8  0x0000000000fed977 in rocksdb::GetOptionsFromString (config_options=..., base_options=..., opts_str="compaction_options_universal={min_merge_width=4;size_ratio=2;max_size_amplification_percent=500}",
    new_options=0x7fffffffde40) at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/options_helper.cc:860
#9  0x0000000000fed7fd in rocksdb::GetOptionsFromString (base_options=..., opts_str="compaction_options_universal={min_merge_width=4;size_ratio=2;max_size_amplification_percent=500}", new_options=0x7fffffffde40)
    at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/options_helper.cc:843
#10 0x0000000000cafa56 in RocksDBStore::ParseOptionsFromStringStatic(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options&, std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::Options&)>) (
    cct=0x1755bf0,
    opt_str="write_buffer_size=536870912;create_if_missing=true;max_write_buffer_number=4;max_background_compactions=4;stats_dump_period_sec = 5;min_write_buffer_number_to_merge = 2;level0_file_num_compaction_trig"..., opt=..., interp=...) at ../src/kv/RocksDBStore.cc:378
#11 0x0000000000caf2c3 in RocksDBStore::ParseOptionsFromString (this=0x17f1010,
    opt_str="write_buffer_size=536870912;create_if_missing=true;max_write_buffer_number=4;max_background_compactions=4;stats_dump_period_sec = 5;min_write_buffer_number_to_merge = 2;level0_file_num_compaction_trig"..., opt=...) at ../src/kv/RocksDBStore.cc:351
#12 0x0000000000b65446 in RocksDBOption_simple_Test::TestBody (this=0x17ee720) at ../src/test/objectstore/TestRocksdbOptionParse.cc:31

(gdb) p _M_valptr()
$11 = (const std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > *) 0x17f3620
(gdb) p *_M_valptr()
$12 = {first = <error reading variable: Cannot create a lazy string with address 0x0, and a non-zero length.>,
  second = <error reading variable: Cannot create a lazy string with address 0x0, and a non-zero length.>}

and in hashtable.h

│   1125                    // _M_before_begin.                                                                                                                                                                      │
│   1126                    __node_type* __ht_n = __ht._M_begin();                                                                                                                                                   │
│B+>1127                    __node_type* __this_n = __node_gen(__ht_n);                                                                                                                                              │
│   1128                    this->_M_copy_code(__this_n, __ht_n);                                                                                                                                                    │
│   1129                    _M_before_begin._M_nxt = __this_n;                                                                                                                                                       │
│   1130                    _M_buckets[_M_bucket_index(__this_n)] = &_M_before_begin;

in its caller

│   1230            _Hashtable(const _Hashtable& __ht)                                                                                                                                                               │
│   1231            : __hashtable_base(__ht),                                                                                                                                                                        │
│   1232              __map_base(__ht),                                                                                                                                                                              │
│   1233              __rehash_base(__ht),                                                                                                                                                                           │
│   1234              __hashtable_alloc(                                                                                                                                                                             │
│   1235                __node_alloc_traits::_S_select_on_copy(__ht._M_node_allocator())),                                                                                                                           │
│   1236              _M_buckets(nullptr),                                                                                                                                                                           │
│   1237              _M_bucket_count(__ht._M_bucket_count),                                                                                                                                                         │
│   1238              _M_element_count(__ht._M_element_count),                                                                                                                                                       │
│   1239              _M_rehash_policy(__ht._M_rehash_policy)                                                                                                                                                        │
│   1240            {                                                                                                                                                                                                │
│  >1241              _M_assign(__ht,                                                                                                                                                                                │
│   1242                        [this](const __node_type* __n)                                                                                                                                                       │
│   1243                        { return this->_M_allocate_node(__n->_M_v()); });                                                                                                                                    │
│   1244            }

caller's caller

│   180               /// Copy constructor.                                                                                                                                                                          │
│  >181               unordered_map(const unordered_map&) = default;                                                                                                                                                 │

caller**3, i.e. in rocksdb/options/configurable.cc

│   253         Status ConfigurableHelper::ConfigureOptions(                                                                                                                                                         │
│   254             const ConfigOptions& config_options, Configurable& configurable,                                                                                                                                 │
│   255             const std::unordered_map<std::string, std::string>& opts_map,                                                                                                                                    │
│   256             std::unordered_map<std::string, std::string>* unused) {                                                                                                                                          │
│  >257           std::unordered_map<std::string, std::string> remaining = opts_map;                                                                                                                                 │
│   258           Status s = Status::OK();                                                                                                                                                                           │

(gdb) f 5
#5  0x0000000000fd0630 in rocksdb::ConfigurableHelper::ConfigureOptions (config_options=..., configurable=..., opts_map=std::unordered_map with 1 element = {...}, unused=0x7fffffffd190)
    at /home/jenkins-build/build/workspace/ceph-pull-requests/src/rocksdb/options/configurable.cc:257
(gdb) p opts_map
$13 = std::unordered_map with 1 element = {["compaction_options_universal"] = "min_merge_width=4;size_ratio=2;max_size_amplification_percent=500"}

an unordered_map<> is backed by a _Hashtable in libstdc++, when we try to copy-construct an unordered_map, we're actually copying the nodes in the _Hashtable. but it turns out the first node has a pair<string, string>. where the first and second are backed by nullptr!

while the output of "p opts_map" tells another story.

i searched around, but failed to find a bugzilla ticket filed against this issue. what i found though was that unordered_map has being updated multiple times in the past couple years. see https://github.com/gcc-mirror/gcc/blob/master/libstdc%2B%2B-v3/include/bits/hashtable.h

what makes the problem more complicated is that we are using the combination of clang-10 + gcc-11. before Casey's change of https://github.com/ceph/ceph/pull/45133, we've been using clang-10 + libstdc++-9

$ clang -v
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

probably we could try to remove libstdc++-11-dev and try again?

after removing libstdc++-11-dev:

$ clang -v
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64

i cannot reproduce the test failure using

./bin/unittest_rocksdb_option

anymore. while i could reproduce it consistently on irvingi03 before removing libstdc++-11.

Thank you very much for looking further into this, Kefu. If it helps any, I’m noticing that failed builds (as of ~3 days ago when the problem first arose) are getting built on braggi15, irvingi03, irvingi04, irvingi05, and irvingi07.

There are a couple of rare exceptions where make check is successful, and these builds have run on adami05, adami08, and braggi14. PRs for reference:
https://github.com/ceph/ceph/pull/45246
https://github.com/ceph/ceph/pull/45224
https://github.com/ceph/ceph/pull/45200
https://github.com/ceph/ceph/pull/45241
https://github.com/ceph/ceph/pull/45255

Perhaps the configurations are slightly different between these nodes.

the time line matches with the introduction of gcc-11.

as a workaround, i am implementing following change in the impacted test nodes:

sudo apt-get remove libstdc++-11-dev libgcc-11-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 40 --slave /usr/bin/g++ g++ /usr/bin/g++-9

seems the combination of clang-13 + libstdc++-11 survived the test:

$ clang-13 -v
Ubuntu clang version 13.0.1-++20220120110924+75e33f71c2da-1~exp1~20220120231001.58
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/11
Candidate multilib: .;@m64
Selected multilib: .;@m64

$ bin/unittest_rocksdb_option
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from RocksDBOption
[ RUN      ] RocksDBOption.simple
[       OK ] RocksDBOption.simple (2 ms)
[ RUN      ] RocksDBOption.interpret
[       OK ] RocksDBOption.interpret (101 ms)
[----------] 2 tests from RocksDBOption (103 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (103 ms total)
[  PASSED  ] 2 tests.

EDIT, clang-12 + libstd++-11 works, while clang-13 + libstd++-11 does not.

Josh Durgin wrote:

One more occurrence today, on irvingi05 - are there potentially other nodes we need to update?

https://jenkins.ceph.com/job/ceph-pull-requests/92005/ triggered by https://github.com/ceph/ceph/pull/45510

per my test last night, clang-13 + libstdc++-11 fails the test. i am removing all clang-13 from the test nodes.

Sighting on braggi17: https://jenkins.ceph.com/job/ceph-pull-requests/97301/

The following tests FAILED:
     36 - smoke.sh (Timeout)
    178 - mgr-dashboard-smoke.sh (Failed)
    182 - unittest_rocksdb_option (Failed)
    193 - safe-to-destroy.sh (Timeout)
    227 - unittest_rgw_lua (Failed)

• confusa13 - https://jenkins.ceph.com/job/ceph-pull-requests-arm64/28191/ --> strangely these two got marked as a successful builds (this is a separate issue tracked in #55344)
• confusa10 - https://jenkins.ceph.com/job/ceph-pull-requests-arm64/28192/ --^

https://jenkins.ceph.com/job/ceph-pull-requests/106184/consoleFull#-208872746540526d21-3511-427d-909c-dd086c0d1034