Project

General

Profile

Actions
Copy link

Bug #5798

closed

librados: mutex assertion in SimpleMessenger::submit_message

Added by Sage Weil over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Samuel Just
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-07-29T10:59:51.538 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]: common/Mutex.cc: In function 'void Mutex::Lock(bool)' thread 7fbfc8cc0700 time 2013-07-29 11:00:31.870905
2013-07-29T10:59:51.538 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]: common/Mutex.cc: 93: FAILED assert(r == 0)
2013-07-29T10:59:51.538 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  ceph version 0.67-rc2-106-g12c1f11 (12c1f1157c7b9513a3d9f716a8ec62fce00d28f5)
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  1: (Mutex::Lock(bool)+0x1c3) [0x7fbfce2b3213]
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  2: (SimpleMessenger::submit_message(Message*, Connection*, entity_addr_t const&, int, bool)+0x58) [0x7fbfce3d5298]
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  3: (SimpleMessenger::_send_message(Message*, Connection*, bool)+0x288) [0x7fbfce3d6068]
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  4: (Objecter::tick()+0x903) [0x7fbfce25af33]
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  5: (Context::complete(int)+0xa) [0x7fbfce23164a]
2013-07-29T10:59:51.539 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  6: (SafeTimer::timer_thread()+0x425) [0x7fbfce2d3665]
2013-07-29T10:59:51.540 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  7: (SafeTimerThread::entry()+0xd) [0x7fbfce2d429d]
2013-07-29T10:59:51.540 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  8: (()+0x7e9a) [0x7fbfcdeb2e9a]
2013-07-29T10:59:51.540 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  9: (clone()+0x6d) [0x7fbfcd6c9ccd]
2013-07-29T10:59:51.540 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]:  NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2013-07-29T10:59:51.540 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]: terminate called after throwing an instance of 'ceph::FailedAssertion'
2013-07-29T10:59:51.548 INFO:teuthology.task.workunit.client.0.err:[10.214.133.35]: Aborted (core dumped)

job was
ubuntu@teuthology:/a/teuthology-2013-07-29_09:44:34-rados-next-testing-basic-plana/88497$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 88b7f22bc0e44db48a24af23e4de3653bc44b2d2
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 12c1f1157c7b9513a3d9f716a8ec62fce00d28f5
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 12c1f1157c7b9513a3d9f716a8ec62fce00d28f5
  s3tests:
    branch: next
  workunit:
    sha1: 12c1f1157c7b9513a3d9f716a8ec62fce00d28f5
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- workunit:
    clients:
      client.0:
      - rados/test.sh
teuthology_branch: next
Actions
Copy link
 #1

Updated by Samuel Just over 10 years ago

  • Status changed from 12 to In Progress
  • Assignee set to Samuel Just
Actions
Copy link
 #2

Updated by Samuel Just over 10 years ago

  • Status changed from In Progress to 12
Actions
Copy link
 #3

Updated by Samuel Just over 10 years ago

  • Assignee deleted (Samuel Just)
Actions
Copy link
 #4

Updated by Samuel Just over 10 years ago

There is a chance that this may be related to #5784.

Actions
Copy link
 #5

Updated by Samuel Just over 10 years ago

binary was ceph_test_rados_api_cmd

Actions
Copy link
 #6

Updated by Samuel Just over 10 years ago

2013-07-29T10:59:31.528 INFO:teuthology.task.workunit.client.0.out:[10.214.133.35]: [ RUN ] LibRadosCmd.PGCmd

Actions
Copy link
 #7

Updated by Ian Colle over 10 years ago

  • Assignee set to Samuel Just
Actions
Copy link
 #8

Updated by Samuel Just over 10 years ago

(gdb) print *this
$5 = {<RefCountedObject> = {_vptr.RefCountedObject = 0xd66500, nref = {val = 140461764831096}}, lock = {name = 0x7fbfce4de80a "Connection::lock", id = -1, recursive = false, lockdep = true, backtrace = false, m = {_data = {__lock = 0, _count = 0, __owner = 0, __nusers = 0, __kind = -1, __spins = 0, __list = {
__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 16 times>"\377, \377\377\377", '\000' <repeats 19 times>, __align = 0}, nlock = 0, locked_by = 0, cct = 0x0, logger = 0x0}, msgr = 0xd631e0, priv = 0x0, peer_type = 4, peer_addr = {type = 0, nonce = 29586, {addr = {ss_family = 2, __ss_align = 0,
__ss_padding = '\000' <repeats 111 times>}, addr4 = {sin_family = 2, sin_port = 38170, sin_addr = {s_addr = 595973642}, sin_zero = "\000\000\000\000\000\000\000"}, addr6 = {sin6_family = 2, sin6_port = 38170, sin6_flowinfo = 595973642, sin6_addr = {_in6_u = {__u6_addr8 = '\000' <repeats 15 times>,
_u6_addr16 = {0, 0, 0, 0, 0, 0, 0, 0}, __u6_addr32 = {0, 0, 0, 0}}}, sin6_scope_id = 0}}}, features = 0, pipe = 0x0, failed = true, rx_buffers_version = 0, rx_buffers = {_M_t = {
_M_impl = {<std::allocator<std::_Rb_tree_node<std::pair<unsigned long const, std::pair<ceph::buffer::list, int> > > >> = {<_gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<unsigned long const, std::pair<ceph::buffer::list, int> > > >> = {<No data fields>}, <No data fields>},
_M_key_compare = {<std::binary_function<unsigned long, unsigned long, bool>> = {<No data fields>}, <No data fields>}, _M_header = {_M_color = std::_S_red, _M_parent = 0x0, _M_left = 0xd66398, _M_right = 0xd66398}, _M_node_count = 0}}}}
(gdb)

Actions
Copy link
 #9

Updated by Samuel Just over 10 years ago

  • Status changed from 12 to Resolved
Actions
Copy link

Also available in: Atom PDF