root@host:~# rest-bench -t 128 --api-host=hostname --access-key=access_key --secret=secret_key --bucket=bench --uri_style=vhost --no-cleanup write host=hostname Maintaining 128 concurrent writes of 4194304 bytes for up to 10 seconds or 0 objects Object prefix: benchmark_data_cpro24374_14928 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 1 1 0 0 0 - 0 1 128 128 0 0 0 - 0 2 128 129 1 1.97244 2 1.23336 1.23336 3 128 129 1 1.19953 0 - 1.23336 4 128 138 10 9.227 18 4.27861 3.86935 5 127 182 55 41.2308 180 5.25191 4.6642 ERROR: benchmark_data_cpro24374_14928_object124: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object110: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object57: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object113: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object26: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object117: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object56: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object109: ConnectionFailed 6 128 216 88 55.5478 132 1.85818 4.23068 ERROR: benchmark_data_cpro24374_14928_object119: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object71: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object10: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object111: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object17: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object80: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object104: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object122: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object127: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object16: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object65: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object47: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object90: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object121: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object108: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object115: ConnectionFailedERROR: benchmark_data_cpro24374_14928_object81: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object82: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object114: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object21: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object79: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object49: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object116: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object73: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object123: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object94: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object52: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object24: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object103: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object118: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object125: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object40: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object38: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object35: ConnectionFailed ERROR: ERROR: benchmark_data_cpro24374_14928_object105: ConnectionFailed benchmark_data_cpro24374_14928_object84: ConnectionFailed ERROR: benchmark_data_cpro24374_14928_object20: ConnectionFailed error during benchmark: -5 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f7b45f2d7c0 time 2015-02-25 18:46:30.393222 common/WorkQueue.cc: 68: FAILED assert(_threads.empty()) ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x4cd9fb] 2: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 3: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 4: (main()+0x42e) [0x4b071e] 5: (__libc_start_main()+0xf5) [0x7f7b43debec5] 6: rest-bench() [0x4b1a47] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2015-02-25 18:46:30.394252 7f7b45f2d7c0 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f7b45f2d7c0 time 2015-02-25 18:46:30.393222 common/WorkQueue.cc: 68: FAILED assert(_threads.empty()) ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x4cd9fb] 2: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 3: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 4: (main()+0x42e) [0x4b071e] 5: (__libc_start_main()+0xf5) [0x7f7b43debec5] 6: rest-bench() [0x4b1a47] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- -13> 2015-02-25 18:46:21.767160 7f7b45f2d7c0 5 asok(0x4319850) register_command perfcounters_dump hook 0x4319710 -12> 2015-02-25 18:46:21.767197 7f7b45f2d7c0 5 asok(0x4319850) register_command 1 hook 0x4319710 -11> 2015-02-25 18:46:21.767289 7f7b45f2d7c0 5 asok(0x4319850) register_command perf dump hook 0x4319710 -10> 2015-02-25 18:46:21.767320 7f7b45f2d7c0 5 asok(0x4319850) register_command perfcounters_schema hook 0x4319710 -9> 2015-02-25 18:46:21.767343 7f7b45f2d7c0 5 asok(0x4319850) register_command 2 hook 0x4319710 -8> 2015-02-25 18:46:21.767351 7f7b45f2d7c0 5 asok(0x4319850) register_command perf schema hook 0x4319710 -7> 2015-02-25 18:46:21.767359 7f7b45f2d7c0 5 asok(0x4319850) register_command config show hook 0x4319710 -6> 2015-02-25 18:46:21.767382 7f7b45f2d7c0 5 asok(0x4319850) register_command config set hook 0x4319710 -5> 2015-02-25 18:46:21.767391 7f7b45f2d7c0 5 asok(0x4319850) register_command config get hook 0x4319710 -4> 2015-02-25 18:46:21.767399 7f7b45f2d7c0 5 asok(0x4319850) register_command config diff hook 0x4319710 -3> 2015-02-25 18:46:21.767434 7f7b45f2d7c0 5 asok(0x4319850) register_command log flush hook 0x4319710 -2> 2015-02-25 18:46:21.767458 7f7b45f2d7c0 5 asok(0x4319850) register_command log dump hook 0x4319710 -1> 2015-02-25 18:46:21.767467 7f7b45f2d7c0 5 asok(0x4319850) register_command log reopen hook 0x4319710 0> 2015-02-25 18:46:30.394252 7f7b45f2d7c0 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f7b45f2d7c0 time 2015-02-25 18:46:30.393222 common/WorkQueue.cc: 68: FAILED assert(_threads.empty()) ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x4cd9fb] 2: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 3: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 4: (main()+0x42e) [0x4b071e] 5: (__libc_start_main()+0xf5) [0x7f7b43debec5] 6: rest-bench() [0x4b1a47] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events --- terminate called after throwing an instance of 'ceph::FailedAssertion' *** Caught signal (Aborted) ** in thread 7f7b45f2d7c0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: rest-bench() [0x4c240a] 2: (()+0x10340) [0x7f7b45907340] 3: (gsignal()+0x39) [0x7f7b43e00bb9] 4: (abort()+0x148) [0x7f7b43e03fc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7b4470c6b5] 6: (()+0x5e836) [0x7f7b4470a836] 7: (()+0x5e863) [0x7f7b4470a863] 8: (()+0x5eaa2) [0x7f7b4470aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x4cdbe8] 10: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 11: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 12: (main()+0x42e) [0x4b071e] 13: (__libc_start_main()+0xf5) [0x7f7b43debec5] 14: rest-bench() [0x4b1a47] 2015-02-25 18:46:30.398877 7f7b45f2d7c0 -1 *** Caught signal (Aborted) ** in thread 7f7b45f2d7c0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: rest-bench() [0x4c240a] 2: (()+0x10340) [0x7f7b45907340] 3: (gsignal()+0x39) [0x7f7b43e00bb9] 4: (abort()+0x148) [0x7f7b43e03fc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7b4470c6b5] 6: (()+0x5e836) [0x7f7b4470a836] 7: (()+0x5e863) [0x7f7b4470a863] 8: (()+0x5eaa2) [0x7f7b4470aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x4cdbe8] 10: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 11: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 12: (main()+0x42e) [0x4b071e] 13: (__libc_start_main()+0xf5) [0x7f7b43debec5] 14: rest-bench() [0x4b1a47] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- begin dump of recent events --- 0> 2015-02-25 18:46:30.398877 7f7b45f2d7c0 -1 *** Caught signal (Aborted) ** in thread 7f7b45f2d7c0 ceph version 0.87 (c51c8f9d80fa4e0168aa52685b8de40e42758578) 1: rest-bench() [0x4c240a] 2: (()+0x10340) [0x7f7b45907340] 3: (gsignal()+0x39) [0x7f7b43e00bb9] 4: (abort()+0x148) [0x7f7b43e03fc8] 5: (__gnu_cxx::__verbose_terminate_handler()+0x155) [0x7f7b4470c6b5] 6: (()+0x5e836) [0x7f7b4470a836] 7: (()+0x5e863) [0x7f7b4470a863] 8: (()+0x5eaa2) [0x7f7b4470aaa2] 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x278) [0x4cdbe8] 10: (ThreadPool::~ThreadPool()+0x126) [0x4c93a6] 11: (RESTDispatcher::~RESTDispatcher()+0xe1) [0x4b6191] 12: (main()+0x42e) [0x4b071e] 13: (__libc_start_main()+0xf5) [0x7f7b43debec5] 14: rest-bench() [0x4b1a47] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 0/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 keyvaluestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs -2/-2 (syslog threshold) 99/99 (stderr threshold) max_recent 500 max_new 1000 log_file --- end dump of recent events --- Aborted