Bug #10955: rest-bench - segmentation fault with concurrent writes > 64 - rgw - Ceph

Actions

Copy link

Bug #10955

closed

rest-bench - segmentation fault with concurrent writes > 64

Added by Italo Santos about 9 years ago. Updated almost 9 years ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

v0.92

% Done:

Source:

Community (user)

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

When I try to use rest-bench with more then 64 concurrent writes I face with a unexpected error, aborting the execution generating too many Errors and a segmentation fault (see attachment).

Looking to the radosgw logs (debug=20) and apache logs I can't see any error, which make me believe this could be an rest-bench issue. See the errors below:

root@host:~# rest-bench -t 128 --api-host=hostname --access-key=access_key --secret=secret_key --bucket=bench --uri_style=vhost --no-cleanup write
host=hostname
Maintaining 128 concurrent writes of 4194304 bytes for up to 10 seconds or 0 objects
Object prefix: benchmark_data_cpro24374_14928
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
0 1 1 0 0 0 - 0
1 128 128 0 0 0 - 0
2 128 129 1 1.97244 2 1.23336 1.23336
3 128 129 1 1.19953 0 - 1.23336
4 128 138 10 9.227 18 4.27861 3.86935
5 127 182 55 41.2308 180 5.25191 4.6642
ERROR: benchmark_data_cpro24374_14928_object124: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object110: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object57: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object113: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object26: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object117: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object56: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object109: ConnectionFailed
6 128 216 88 55.5478 132 1.85818 4.23068
ERROR: benchmark_data_cpro24374_14928_object119: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object71: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object10: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object111: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object17: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object80: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object104: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object122: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object127: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object16: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object65: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object47: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object90: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object121: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object108: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object115: ConnectionFailedERROR: benchmark_data_cpro24374_14928_object81: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object82: ConnectionFailed

ERROR: benchmark_data_cpro24374_14928_object114: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object21: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object79: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object49: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object116: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object73: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object123: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object94: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object52: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object24: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object103: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object118: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object125: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object40: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object38: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object35: ConnectionFailed
ERROR: ERROR: benchmark_data_cpro24374_14928_object105: ConnectionFailed
benchmark_data_cpro24374_14928_object84: ConnectionFailed
ERROR: benchmark_data_cpro24374_14928_object20: ConnectionFailed
error during benchmark: -5
common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f7b45f2d7c0 time 2015-02-25 18:46:30.393222
common/WorkQueue.cc: 68: FAILED assert(_threads.empty())
ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578)