Project

General

Profile

Bug #3896

rest-bench common/WorkQueue.cc: 54: FAILED assert(_threads.empty())

Added by Bill Reid almost 6 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Low
Assignee:
-
Target version:
-
Start date:
01/22/2013
Due date:
% Done:

0%

Source:
Support
Tags:
Backport:
hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

It seems rest-bench doesn't like to exit cleanly while cleaning up after itself.... I did test at low concurrency but didn't test with the --no-cleanup option
rest-bench --seconds=600 -t 40 -b 10000000 --access-key=XXX --secret=XXX --api-host=localhost write

2013-01-22 23:59:38.967072min lat: 1.43787 max lat: 20.4884 avg lat: 3.98123
sec Cur ops started finished avg MB/s cur MB/s last lat avg lat
600 40 6046 6006 95.449 95.3674 2.79372 3.98123
601 35 6047 6012 95.3854 57.2205 5.00556 3.98158
602 33 6047 6014 95.2586 19.0735 5.79511 3.98154
Total time run: 602.862396
Total writes made: 6047
Write size: 10000000
Bandwidth (MB/sec): 95.658
Stddev Bandwidth:       23.7106
Max bandwidth (MB/sec): 181.198
Min bandwidth (MB/sec): 0
Average Latency: 3.98538
Stddev Latency: 1.30604
Max latency: 20.4884
Min latency: 1.43787
common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
common/WorkQueue.cc: 54: FAILED assert(threads.empty())
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
3: (main()+0x743) [0x41af03]
4: (
_libc_start_main()+0xfd) [0x7f5b6e586ead]
5: rest-bench() [0x41bd39]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-01-23 00:07:53.570052 7f5b70400780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
common/WorkQueue.cc: 54: FAILED assert(_threads.empty())
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
3: (main()+0x743) [0x41af03]
4: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
5: rest-bench() [0x41bd39]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
-11> 2013-01-22 23:49:38.436864 7f5b70400780 5 asok(0x1762270) register_command perfcounters_dump hook 0x1762190
-10> 2013-01-22 23:49:38.436962 7f5b70400780 5 asok(0x1762270) register_command 1 hook 0x1762190
-9> 2013-01-22 23:49:38.436969 7f5b70400780 5 asok(0x1762270) register_command perf dump hook 0x1762190
-8> 2013-01-22 23:49:38.436979 7f5b70400780 5 asok(0x1762270) register_command perfcounters_schema hook 0x1762190
-7> 2013-01-22 23:49:38.436990 7f5b70400780 5 asok(0x1762270) register_command 2 hook 0x1762190
-6> 2013-01-22 23:49:38.436994 7f5b70400780 5 asok(0x1762270) register_command perf schema hook 0x1762190
-5> 2013-01-22 23:49:38.437004 7f5b70400780 5 asok(0x1762270) register_command config show hook 0x1762190
-4> 2013-01-22 23:49:38.437013 7f5b70400780 5 asok(0x1762270) register_command config set hook 0x1762190
-3> 2013-01-22 23:49:38.437018 7f5b70400780 5 asok(0x1762270) register_command log flush hook 0x1762190
-2> 2013-01-22 23:49:38.437028 7f5b70400780 5 asok(0x1762270) register_command log dump hook 0x1762190
-1> 2013-01-22 23:49:38.437033 7f5b70400780 5 asok(0x1762270) register_command log reopen hook 0x1762190
0> 2013-01-23 00:07:53.570052 7f5b70400780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
common/WorkQueue.cc: 54: FAILED assert(_threads.empty())
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
3: (main()+0x743) [0x41af03]
4: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
5: rest-bench() [0x41bd39]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 100000
max_new 1000
log_file
--
end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
  • Caught signal (Aborted)
    in thread 7f5b70400780
    ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
    1: rest-bench() [0x42c532]
    2: (()+0xf030) [0x7f5b6ffec030]
    3: (gsignal()+0x35) [0x7f5b6e59a475]
    4: (abort()+0x180) [0x7f5b6e59d6f0]
    5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
    6: (()+0x63996) [0x7f5b6eded996]
    7: (()+0x639c3) [0x7f5b6eded9c3]
    8: (()+0x63bee) [0x7f5b6ededbee]
    9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
    10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
    11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
    12: (main()+0x743) [0x41af03]
    13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
    14: rest-bench() [0x41bd39]
    2013-01-23 00:07:53.571327 7f5b70400780 -1
    Caught signal (Aborted) *
    in thread 7f5b70400780
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: rest-bench() [0x42c532]
2: (()+0xf030) [0x7f5b6ffec030]
3: (gsignal()+0x35) [0x7f5b6e59a475]
4: (abort()+0x180) [0x7f5b6e59d6f0]
5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
6: (()+0x63996) [0x7f5b6eded996]
7: (()+0x639c3) [0x7f5b6eded9c3]
8: (()+0x63bee) [0x7f5b6ededbee]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
12: (main()+0x743) [0x41af03]
13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
14: rest-bench() [0x41bd39]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
0> 2013-01-23 00:07:53.571327 7f5b70400780 -1 ** Caught signal (Aborted) *
in thread 7f5b70400780
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
1: rest-bench() [0x42c532]
2: (()+0xf030) [0x7f5b6ffec030]
3: (gsignal()+0x35) [0x7f5b6e59a475]
4: (abort()+0x180) [0x7f5b6e59d6f0]
5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
6: (()+0x63996) [0x7f5b6eded996]
7: (()+0x639c3) [0x7f5b6eded9c3]
8: (()+0x63bee) [0x7f5b6ededbee]
9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
12: (main()+0x743) [0x41af03]
13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
14: rest-bench() [0x41bd39]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 100000
max_new 1000
log_file
--
end dump of recent events ---
Aborted

radosgw-admin bucket stats --bucket=rest-bench-bucket

{ "bucket": "rest-bench-bucket",
"pool": ".rgw.buckets2",
"id": "7701.1",
"marker": "7701.1",
"owner": "wer",
"usage": { "rgw.main": { "size_kb": 454102,
"size_kb_actual": 455700,
"num_objects": 465}}}

rest-bench.txt View (7.25 KB) Alexandre Marangone, 09/17/2013 03:18 PM


Related issues

Duplicated by rgw - Bug #10955: rest-bench - segmentation fault with concurrent writes > 64 Duplicate 02/25/2015
Copied to rgw - Backport #12504: rest-bench common/WorkQueue.cc: 54: FAILED assert(_threads.empty()) Resolved 01/22/2013

Associated revisions

Revision f3d34d8f (diff)
Added by huang jun over 3 years ago

rest_bench: drain the work queue to fix a crash
Fixes: #3896
Signed-off-by: huangjun <>

Revision 6e7358b3 (diff)
Added by huang jun over 3 years ago

rest_bench: drain the work queue to fix a crash
Fixes: #3896
Signed-off-by: huangjun <>

(cherry picked from commit f3d34d8ff921dbd2ff21f6b72af7c73bb9c6940e)

History

#1 Updated by Sage Weil about 5 years ago

  • Status changed from New to Can't reproduce

#2 Updated by Alexandre Marangone about 5 years ago

I'm seeing this for every rest-bench I run (seq, write and cleanup) on both Cuttlefish and Dumpling.

cephdrop@ceph.com:~/rest-bench/core was generated from running the rest-bench below (0.67.3):
$ rest-bench --api-host=burnupi02 --bucket=tmp2 --access-key=XXX --secret=YYY --seconds 60 write

Full output attached

#3 Updated by Alexandre Marangone about 5 years ago

  • Status changed from Can't reproduce to New
  • Target version deleted (v0.56)
  • Source changed from Community (user) to Support

#4 Updated by Loic Dachary about 4 years ago

  • Project changed from Ceph to rgw
  • Category deleted (common)

#5 Updated by Sage Weil about 4 years ago

  • Status changed from New to Resolved

#6 Updated by Kefu Chai over 3 years ago

  • Status changed from Resolved to Need Review
  • Regression set to No

it is spotted again. and the commits do not look right, and never got merged.

https://github.com/ceph/ceph/pull/4917

#7 Updated by Kefu Chai over 3 years ago

  • Status changed from Need Review to Pending Backport
  • Backport set to hammer

#8 Updated by Loic Dachary about 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF