Ceph » RADOS

very good reason to drop one distro in teuthology and replace it with fedora 28, I think Brad brought this up long time back too in #sepia.

Vasu Kulkarni wrote:

very good reason to drop one distro in teuthology and replace it with fedora 28, I think Brad brought this up long time back too in #sepia.

Many times, long ago, yes.

I'm looking into this crash.

Turning up verbosity gives clues to what might be the problem.

<mock-chroot> sh-4.4# ./crushtool -v -c crushmap.txt 2>&1|head -25
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable chooseleaf_vary_r 1
tunable chooseleaf_stable 0
tunable straw_calc_version 1
tunable allowed_bucket_algs 22
type 0 'osd'
type 1 'host'
type 2 'chassis'
type 3 'rack'
type 4 'row'
type 5 'pdu'
type 6 'pod'
type 7 'room'
type 8 'datacenter'
type 9 'region'
type 10 'root'
bucket default id -1
bucket default (-1) 0 items and weight 0
/usr/include/c++/8/bits/stl_vector.h:932: std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::reference = int&; std::vector<_Tp, _Alloc>::size_type = long unsigned int]: Assertion '__builtin_expect(__n < this->size(), true)' failed.
*** Caught signal (Aborted) **
 in thread 7f64629f6540 thread_name:crushtool
 ceph version 12.2.11 (26dc3775efc7bb286a1d6d66faee0ba30ea23eee) luminous (stable)

The problem here is we have the following code in src/crush/CrushCompiler.cc

 561 int CrushCompiler::parse_bucket(iter_t const& i)                                                                                                                                                                                        
 562 {
...
 651   vector<int> items(size);                                                                                                                                                                                                              
 652   vector<int> weights(size);
...
 746   int r = crush.add_bucket(id, alg, hash, type, size,                                                                                                                                                                                   
 747                            &items[0], &weights[0], &idout);

Looking at a coredump.

(gdb) f
#4  0x00007fffefb86e85 in CrushCompiler::parse_bucket (this=0x7fffffffcfe0, i=...) at /builddir/build/BUILD/ceph-12.2.11/src/crush/CrushCompiler.cc:746
746       int r = crush.add_bucket(id, alg, hash, type, size,
(gdb) l
741       item_id[name] = id;
742       item_weight[id] = bucketweight;
743       
744       assert(id != 0);
745       int idout;
746       int r = crush.add_bucket(id, alg, hash, type, size,
747                                &items[0], &weights[0], &idout);
748       if (r < 0) {
749         if (r == -EEXIST)
750           err << "Duplicate bucket id " << id << std::endl;
(gdb) p items
$1 = std::vector of length 0, capacity 0
(gdb) p weights
$2 = std::vector of length 0, capacity 0
(gdb) down
#3  0x00007fffef936783 in std::vector<int, std::allocator<int> >::operator[] (this=this@entry=0x7fffffffc5a0, __n=__n@entry=0) at /usr/include/c++/8/bits/stl_vector.h:805
805           size() const _GLIBCXX_NOEXCEPT
(gdb) 
#2  0x00007fffef716168 in std::__replacement_assert (__file=__file@entry=0x7fffefc134c0 "/usr/include/c++/8/bits/stl_vector.h", __line=__line@entry=932, 
    __function=__function@entry=0x7fffefc47ca0 <_ZZNSt6vectorIiSaIiEEixEmE19__PRETTY_FUNCTION__> "std::vector<_Tp, _Alloc>::reference std::vector<_Tp, _Alloc>::operator[](std::vector<_Tp, _Alloc>::size_type) [with _Tp = int; _Alloc = std::allocator<int>; std::vector<_Tp, _Alloc>::reference = int&;"..., __condition=__condition@entry=0x7fffefc13490 "__builtin_expect(__n < this->size(), true)") at /usr/include/c++/8/x86_64-redhat-linux/bits/c++config.h:2391
2391        __builtin_abort();
(gdb) l
2386      __replacement_assert(const char* __file, int __line,
2387                           const char* __function, const char* __condition)
2388      {
2389        __builtin_printf("%s:%d: %s: Assertion '%s' failed.\n", __file, __line,
2390                         __function, __condition);
2391        __builtin_abort();
2392      }
2393    }
2394    #define __glibcxx_assert_impl(_Condition)                                \
2395      do
(gdb) printf "Assertion '%s' failed.\n", __condition
Assertion '__builtin_expect(__n < this->size(), true)' failed.
(gdb) up
#3  0x00007fffef936783 in std::vector<int, std::allocator<int> >::operator[] (this=this@entry=0x7fffffffc5a0, __n=__n@entry=0) at /usr/include/c++/8/bits/stl_vector.h:805
805           size() const _GLIBCXX_NOEXCEPT
(gdb) p __n
$3 = 0
(gdb) p  this->size()
$4 = 0

Well fair enough. So why here? Why now?

Due to the inclusion of _GLIBCXX_ASSERTIONS in the CXXFLAGS. The use of the address of element 0 of an empty vector is considered unsafe although it will historically do what you want. However, here we are being pulled up on it. I suspect we need to pass the data() [1] member function of vector here but I'll need to do some testing.

[1] http://www.open-std.org/jtc1/sc22/wg21/docs/lwg-defects.html#464 first line under "Rationale:"

https://bugzilla.redhat.com/show_bug.cgi?id=1515858