Project

General

Profile

Actions
Copy link

Bug #3896

closed

rest-bench common/WorkQueue.cc: 54: FAILED assert(_threads.empty())

Added by Bill Reid over 11 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Low
Assignee:
-
Target version:
-
% Done:

0%

Source:
Support
Tags:
Backport:
hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

It seems rest-bench doesn't like to exit cleanly while cleaning up after itself.... I did test at low concurrency but didn't test with the --no-cleanup option
rest-bench --seconds=600 -t 40 -b 10000000 --access-key=XXX --secret=XXX --api-host=localhost write

2013-01-22 23:59:38.967072min lat: 1.43787 max lat: 20.4884 avg lat: 3.98123
       sec Cur ops   started  finished  avg MB/s  cur MB/s  last lat   avg lat
       600      40      6046      6006    95.449   95.3674   2.79372   3.98123
       601      35      6047      6012   95.3854   57.2205   5.00556   3.98158
       602      33      6047      6014   95.2586   19.0735   5.79511   3.98154
     Total time run:         602.862396
    Total writes made:      6047
    Write size:             10000000
    Bandwidth (MB/sec):     95.658
Stddev Bandwidth:       23.7106
    Max bandwidth (MB/sec): 181.198
    Min bandwidth (MB/sec): 0
    Average Latency:        3.98538
    Stddev Latency:         1.30604
    Max latency:            20.4884
    Min latency:            1.43787
    common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
    common/WorkQueue.cc: 54: FAILED assert(threads.empty())
     ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
     1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
     2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
     3: (main()+0x743) [0x41af03]
     4: (_libc_start_main()+0xfd) [0x7f5b6e586ead]
     5: rest-bench() [0x41bd39]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
    2013-01-23 00:07:53.570052 7f5b70400780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
    common/WorkQueue.cc: 54: FAILED assert(_threads.empty())
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
     1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
     2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
     3: (main()+0x743) [0x41af03]
     4: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
     5: rest-bench() [0x41bd39]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
       -11> 2013-01-22 23:49:38.436864 7f5b70400780  5 asok(0x1762270) register_command perfcounters_dump hook 0x1762190
       -10> 2013-01-22 23:49:38.436962 7f5b70400780  5 asok(0x1762270) register_command 1 hook 0x1762190
        -9> 2013-01-22 23:49:38.436969 7f5b70400780  5 asok(0x1762270) register_command perf dump hook 0x1762190
        -8> 2013-01-22 23:49:38.436979 7f5b70400780  5 asok(0x1762270) register_command perfcounters_schema hook 0x1762190
        -7> 2013-01-22 23:49:38.436990 7f5b70400780  5 asok(0x1762270) register_command 2 hook 0x1762190
        -6> 2013-01-22 23:49:38.436994 7f5b70400780  5 asok(0x1762270) register_command perf schema hook 0x1762190
        -5> 2013-01-22 23:49:38.437004 7f5b70400780  5 asok(0x1762270) register_command config show hook 0x1762190
        -4> 2013-01-22 23:49:38.437013 7f5b70400780  5 asok(0x1762270) register_command config set hook 0x1762190
        -3> 2013-01-22 23:49:38.437018 7f5b70400780  5 asok(0x1762270) register_command log flush hook 0x1762190
        -2> 2013-01-22 23:49:38.437028 7f5b70400780  5 asok(0x1762270) register_command log dump hook 0x1762190
        -1> 2013-01-22 23:49:38.437033 7f5b70400780  5 asok(0x1762270) register_command log reopen hook 0x1762190
         0> 2013-01-23 00:07:53.570052 7f5b70400780 -1 common/WorkQueue.cc: In function 'virtual ThreadPool::~ThreadPool()' thread 7f5b70400780 time 2013-01-23 00:07:53.569556
    common/WorkQueue.cc: 54: FAILED assert(_threads.empty())
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
     1: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
     2: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
     3: (main()+0x743) [0x41af03]
     4: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
     5: rest-bench() [0x41bd39]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
0/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
0/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 hadoop
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
2/-2 (syslog threshold)
99/99 (stderr threshold)
max_recent 100000
max_new 1000
log_file
-- end dump of recent events ---
terminate called after throwing an instance of 'ceph::FailedAssertion'
  • Caught signal (Aborted)
    in thread 7f5b70400780
    ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
    1: rest-bench() [0x42c532]
    2: (()+0xf030) [0x7f5b6ffec030]
    3: (gsignal()+0x35) [0x7f5b6e59a475]
    4: (abort()+0x180) [0x7f5b6e59d6f0]
    5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
    6: (()+0x63996) [0x7f5b6eded996]
    7: (()+0x639c3) [0x7f5b6eded9c3]
    8: (()+0x63bee) [0x7f5b6ededbee]
    9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
    10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
    11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
    12: (main()+0x743) [0x41af03]
    13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
    14: rest-bench() [0x41bd39]
    2013-01-23 00:07:53.571327 7f5b70400780 -1     Caught signal (Aborted) *
    in thread 7f5b70400780
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
     1: rest-bench() [0x42c532]
     2: (()+0xf030) [0x7f5b6ffec030]
     3: (gsignal()+0x35) [0x7f5b6e59a475]
     4: (abort()+0x180) [0x7f5b6e59d6f0]
     5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
     6: (()+0x63996) [0x7f5b6eded996]
     7: (()+0x639c3) [0x7f5b6eded9c3]
     8: (()+0x63bee) [0x7f5b6ededbee]
     9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
     10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
     11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
     12: (main()+0x743) [0x41af03]
     13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
     14: rest-bench() [0x41bd39]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
         0> 2013-01-23 00:07:53.571327 7f5b70400780 -1 ** Caught signal (Aborted) *
     in thread 7f5b70400780
ceph version 0.56.1 (e4a541624df62ef353e754391cbbb707f54b16f7)
     1: rest-bench() [0x42c532]
     2: (()+0xf030) [0x7f5b6ffec030]
     3: (gsignal()+0x35) [0x7f5b6e59a475]
     4: (abort()+0x180) [0x7f5b6e59d6f0]
     5: (_gnu_cxx::_verbose_terminate_handler()+0x11d) [0x7f5b6edef89d]
     6: (()+0x63996) [0x7f5b6eded996]
     7: (()+0x639c3) [0x7f5b6eded9c3]
     8: (()+0x63bee) [0x7f5b6ededbee]
     9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x40a) [0x43444a]
     10: (ThreadPool::~ThreadPool()+0x10c) [0x43177c]
     11: (RESTDispatcher::~RESTDispatcher()+0xf1) [0x420091]
     12: (main()+0x743) [0x41af03]
     13: (__libc_start_main()+0xfd) [0x7f5b6e586ead]
     14: rest-bench() [0x41bd39]
     NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
       0/ 5 none
       0/ 1 lockdep
       0/ 1 context
       1/ 1 crush
       1/ 5 mds
       1/ 5 mds_balancer
       1/ 5 mds_locker
       1/ 5 mds_log
       1/ 5 mds_log_expire
       1/ 5 mds_migrator
       0/ 1 buffer
       0/ 1 timer
       0/ 1 filer
       0/ 1 striper
       0/ 1 objecter
       0/ 5 rados
       0/ 5 rbd
       0/ 5 journaler
       0/ 5 objectcacher
       0/ 5 client
       0/ 5 osd
       0/ 5 optracker
       0/ 5 objclass
       1/ 3 filestore
       1/ 3 journal
       0/ 5 ms
       1/ 5 mon
       0/10 monc
       0/ 5 paxos
       0/ 5 tp
       1/ 5 auth
       1/ 5 crypto
       1/ 1 finisher
       1/ 5 heartbeatmap
       1/ 5 perfcounter
       1/ 5 rgw
       1/ 5 hadoop
       1/ 5 javaclient
       1/ 5 asok
       1/ 1 throttle
      2/-2 (syslog threshold)
      99/99 (stderr threshold)
      max_recent    100000
      max_new         1000
      log_file
    -- end dump of recent events ---
    Aborted

radosgw-admin bucket stats --bucket=rest-bench-bucket

{ "bucket": "rest-bench-bucket",
"pool": ".rgw.buckets2",
"id": "7701.1",
"marker": "7701.1",
"owner": "wer",
"usage": { "rgw.main": { "size_kb": 454102,
"size_kb_actual": 455700,
"num_objects": 465}}}

Files

rest-bench.txt (7.25 KB) rest-bench.txt Alexandre Marangone, 09/17/2013 03:18 PM

Related issues 2 (0 open — 2 closed)

Has duplicate rgw - Bug #10955: rest-bench - segmentation fault with concurrent writes > 64Duplicate02/25/2015

Actions
Copied to rgw - Backport #12504: rest-bench common/WorkQueue.cc: 54: FAILED assert(_threads.empty())ResolvedLoïc Dachary01/22/2013Actions
Actions
Copy link
 #1

Updated by Sage Weil over 10 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link
 #2

Updated by Alexandre Marangone over 10 years ago

I'm seeing this for every rest-bench I run (seq, write and cleanup) on both Cuttlefish and Dumpling.

cephdrop@ceph.com:~/rest-bench/core was generated from running the rest-bench below (0.67.3):
$ rest-bench --api-host=burnupi02 --bucket=tmp2 --access-key=XXX --secret=YYY --seconds 60 write

Full output attached

Actions
Copy link
 #3

Updated by Alexandre Marangone over 10 years ago

  • Status changed from Can't reproduce to New
  • Target version deleted (v0.56)
  • Source changed from Community (user) to Support
Actions
Copy link
 #4

Updated by Loïc Dachary over 9 years ago

  • Project changed from Ceph to rgw
  • Category deleted (common)
Actions
Copy link
 #5

Updated by Sage Weil over 9 years ago

  • Status changed from New to Resolved
Actions
Copy link
 #6

Updated by Kefu Chai almost 9 years ago

  • Status changed from Resolved to Fix Under Review
  • Regression set to No

it is spotted again. and the commits do not look right, and never got merged.

https://github.com/ceph/ceph/pull/4917

Actions
Copy link
 #7

Updated by Kefu Chai almost 9 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to hammer
Actions
Copy link
 #8

Updated by Loïc Dachary over 8 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF