Bug #17526
rados bench seq must verify the hostname
% Done:
0%
Source:
other
Tags:
Backport:
jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
In the same way obj_bencher.cc verifies the metadata object exists before trying to read from the pool when required to do sequential read bench, it must verify that the hostname used to build object names is the same as the one used when writing the objects. Otherwise it will fail later with a core dump because there are no safeguard against that kind of error.
Steps to reproduce:
- rados bench -p rbd 2 write --no-cleanup
- sudo hostname somethingelse
- rados bench -p rbd 20 seq
On hammer it fails with:
(gdb) bt bt #0 __memcmp_sse4_1 () at ../sysdeps/x86_64/multiarch/memcmp-sse4.S:69 #1 0x0000000000872829 in ObjBencher::seq_read_bench (this=0x7fffffffe730, seconds_to_run=20, num_objects=36, concurrentios=16, pid=1587, no_verify=false) at common/obj_bencher.cc:592 #2 0x00000000008703d5 in ObjBencher::aio_bench (this=0x7fffffffe730, operation=2, secondsToRun=20, maxObjectsToCreate=0, concurrentios=16, op_size=4194304, cleanup=true, run_name=0x0, no_verify=false) at common/obj_bencher.cc:208 Python Exception <class 'IndexError'> list index out of range: #3 0x000000000084b340 in rados_tool_common (opts=std::map with 1 elements, nargs=std::vector of length 3, capacity 8 = {...}) at tools/rados/rados.cc:2271 #4 0x0000000000850e61 in main (argc=6, argv=0x7fffffffecd8) at tools/rados/rados.cc:2732 (gdb) frame 1 frame 1 #1 0x0000000000872829 in ObjBencher::seq_read_bench (this=0x7fffffffe730, seconds_to_run=20, num_objects=36, concurrentios=16, pid=1587, no_verify=false) at common/obj_bencher.cc:592 592 if (memcmp(data.object_contents, cur_contents->c_str(), data.object_size) != 0) { (gdb) list list 587 // invalidate internal crc cache 588 cur_contents->invalidate_crc(); 589 590 if (!no_verify) { 591 snprintf(data.object_contents, data.object_size, "I'm the %16dth object!", current_index); 592 if (memcmp(data.object_contents, cur_contents->c_str(), data.object_size) != 0) { 593 cerr << name[slot] << " is not correct!" << std::endl; 594 ++errors; 595 } 596 } (gdb) print cur_contents->c_str() print cur_contents->c_str() $3 = 0x0
on master ( 003dbc83e5cdb2c1984578f19d55383924db9ce1 ) it fails with
$ bin/rados bench -p rbd 2 seq sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 benchmark_data_abc_707_object0 is not correct! read got -2 error during benchmark: -2 error 2: (2) No such file or directory
Related issues
History
#1 Updated by Loïc Dachary over 7 years ago
- Status changed from New to 12
#2 Updated by Loïc Dachary over 7 years ago
- Backport set to hammer,jewel
#3 Updated by Loïc Dachary over 7 years ago
- Description updated (diff)
#4 Updated by Loïc Dachary over 7 years ago
- Status changed from 12 to Fix Under Review
- Assignee set to Loïc Dachary
#5 Updated by Loïc Dachary over 7 years ago
- Status changed from Fix Under Review to Resolved
- Backport changed from hammer,jewel to jewel
#6 Updated by Loïc Dachary about 7 years ago
- Status changed from Resolved to Pending Backport
#7 Updated by Nathan Cutler about 7 years ago
- Copied to Backport #18433: jewel: rados bench seq must verify the hostname added
#8 Updated by Nathan Cutler about 7 years ago
- Status changed from Pending Backport to Resolved