Project

General

Profile

Bug #17526

Updated by Loïc Dachary over 6 years ago

In the same way obj_bencher.cc verifies the metadata object exists before trying to read from the pool when required to do sequential read bench, it must verify that the hostname used to build object names is the same as the one used when writing the objects. Otherwise it will fail later with a core dump because there are no safeguard against that kind of error.

Steps to reproduce:

* rados bench -p rbd 2 write --no-cleanup
* sudo hostname somethingelse
* rados bench -p rbd 20 seq

On hammer it fails with:

<pre>
(gdb) bt
bt
#0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:69
#1 0x0000000000872829 in ObjBencher::seq_read_bench (this=0x7fffffffe730,
seconds_to_run=20, num_objects=36, concurrentios=16, pid=1587,
no_verify=false) at common/obj_bencher.cc:592
#2 0x00000000008703d5 in ObjBencher::aio_bench (this=0x7fffffffe730,
operation=2, secondsToRun=20, maxObjectsToCreate=0, concurrentios=16,
op_size=4194304, cleanup=true, run_name=0x0, no_verify=false)
at common/obj_bencher.cc:208
Python Exception <class 'IndexError'> list index out of range:
#3 0x000000000084b340 in rados_tool_common (opts=std::map with 1 elements,
nargs=std::vector of length 3, capacity 8 = {...})
at tools/rados/rados.cc:2271
#4 0x0000000000850e61 in main (argc=6, argv=0x7fffffffecd8)
at tools/rados/rados.cc:2732
(gdb) frame 1
frame 1
#1 0x0000000000872829 in ObjBencher::seq_read_bench (this=0x7fffffffe730,
seconds_to_run=20, num_objects=36, concurrentios=16, pid=1587,
no_verify=false) at common/obj_bencher.cc:592
592 if (memcmp(data.object_contents, cur_contents->c_str(), data.object_size) != 0) {
(gdb) list
list
587 // invalidate internal crc cache
588 cur_contents->invalidate_crc();
589
590 if (!no_verify) {
591 snprintf(data.object_contents, data.object_size, "I'm the %16dth object!", current_index);
592 if (memcmp(data.object_contents, cur_contents->c_str(), data.object_size) != 0) {
593 cerr << name[slot] << " is not correct!" << std::endl;
594 ++errors;
595 }
596 }
(gdb) print cur_contents->c_str()
print cur_contents->c_str()
$3 = 0x0
</pre>

on master ( 003dbc83e5cdb2c1984578f19d55383924db9ce1 ) it fails with

<pre>
$ bin/rados bench -p rbd 2 seq
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
benchmark_data_abc_707_object0 is not correct!
read got -2
error during benchmark: -2
error 2: (2) No such file or directory

</pre>

Back