Project

General

Profile

Actions

Bug #3061

closed

osd crash during shutdown

Added by Tamilarasi muthamizhan over 11 years ago. Updated over 11 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs: ubuntu@teuthology:/a/teuthology-2012-08-28_19:00:05-regression-master-testing-gcov/10931

ceph version 0.51-305-g82c62bd (commit:82c62bd977c74c22385b18791943cb2054920f47)
 1: /tmp/cephtest/binary/usr/local/bin/ceph-osd() [0x81ffba]
 2: (()+0xfcb0) [0x7f2f1f2eacb0]
 3: (tcmalloc::CentralFreeList::FetchFromSpans()+0x27) [0x7f2f1e390df7]
 4: (tcmalloc::CentralFreeList::RemoveRange(void**, void**, int)+0x107) [0x7f2f1e391167]
 5: (tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long)+0x5d) [0x7f2f1e393cad]
 6: (operator new[](unsigned long)+0x486) [0x7f2f1e3a3b26]
 7: (ceph::buffer::create(unsigned int)+0x87) [0x902cd7]
 8: (PG::write_log(ObjectStore::Transaction&)+0x1e0) [0x6c26f0]
 9: (PG::write_if_dirty(ObjectStore::Transaction&)+0x5b) [0x6c314b]
 10: (OSD::process_peering_events(std::list<PG*, std::allocator<PG*> > const&)+0x3ad) [0x656fed]
 11: (OSD::PeeringWQ::_process(std::list<PG*, std::allocator<PG*> > const&)+0x18) [0x69fb58]
 12: (ThreadPool::BatchWorkQueue<PG>::_void_process(void*)+0x12) [0x66fd42]
 13: (ThreadPool::worker()+0x4db) [0x8f5f4b]
 14: (ThreadPool::WorkThread::entry()+0x15) [0x6710c5]
 15: (Thread::_entry_func(void*)+0x12) [0x8e89f2]
 16: (()+0x7e9a) [0x7f2f1f2e2e9a]
 17: (clone()+0x6d) [0x7f2f1d6864bd]

ubuntu@teuthology:/a/teuthology-2012-08-28_19:00:05-regression-master-testing-gcov/10931/remote/ubuntu@plana38.front.sepia.ceph.com/coredump$ ls
1346215171.661.core

ubuntu@teuthology:/a/teuthology-2012-08-28_19:00:05-regression-master-testing-gcov/10931$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 995fc068ddf675260098c60591989bf2ee184338
nuke-on-error: true
overrides:
  ceph:
    conf:
      client:
        rbd cache: false
      global:
        ms inject socket failures: 5000
    coverage: true
    fs: xfs
    log-whitelist:
    - slow request
    sha1: 82c62bd977c74c22385b18791943cb2054920f47
  workunit:
    sha1: 82c62bd977c74c22385b18791943cb2054920f47
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
- - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana30.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYDpptyOTrVH3RqiwH5A//Q2CkkVz5dPTpd/s8qG/Q4EHVA4WDMu80pcDvdSewOfFJl83MEtDKKjuJOuEzI4OGn0DPptDN5wHC1OWrXqFMcIaWVe/KBYOdWEZbA7FECeXgEZR1Sid2bH7XDUE9AYalpS2/SmuuHEU1ObL6zSpAqoY6AIPCR6LgFrtxAqrYmIdpb8YfSuI5uPBv6qikl0yvam06WNerUNQ9lnZXFmFm1wBeicRvWH3jZ6w/xlQBIp/zG6k9IJa0vaLm+FqztLkDWri8Qz1dbdsz0bNjyzD6iRuDOpgmz0Kf8m2IjaJRgRgz2ARcOOdBJKmwnnW/knk5
  ubuntu@plana31.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5J4n7rTsH+IMjGAu+EfhukuK5+zScoSaPIfXDOUU8LfvuI/3x8Luiyv9eRVwZgwuLBWZ/zorBbGZ+G2Iaxy3632AG/XE7cRZA9AxzZT+Qvm9D+BW+Uletgf92cttKMk7qwK3DetQwRKKl6AMv0SDpUff+nzqnJH6LMS8zoBPVXDHFM3Lup8h9H6DYEs1F/Zn8LVSw8hNiD279rg1n1hqWdItmnKBPKyC/qkRoPa6h7gDU6FPaBiNhuhBd0016XGrVwL7Y8gqoDBiArP+NDt1lcnbeiK43bFhqW+pYovOdIA2MJC6z+bkZDlOJdxoz9mDP0cJZBdB43v3UdbS1R+WT
  ubuntu@plana38.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDARPmUWw72IMuZaJozKLrN06DeIgQacSM4fOhaa9jLxqnt8VZRSynN2sMzbKfA+JjLgz69zBawXb0TiVu0cbdqzPV94FylkRduEcZYM9zeD3B74BKZTltZgmugaEPv20olaEfseYMV52VTDNMKSdKPbYmYOVCCpzDJAJIuWJ51UdngsBMDPwTqys49Dcj9Gul71L0FDUSa+pavNG5Ricao6tJyv6rgVrUIz8UfutVB/5xYjwYnR8yFDQuKhmteY+kk3ve8nqgNR9VjuRjfP5mg5jT5e5CtDi1OSCWUi6lJKepv+IVKCSs1vc/1WpHiNLYCLNh9PImzi5GlljuS7o7x
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph:
    log-whitelist:
    - wrongly marked me down
    - objects unfound and apparently lost
- thrashosds:
    timeout: 1200
- rbd_fsx:
    clients:
    - client.0
    ops: 20000
ubuntu@teuthology:/a/teuthology-2012-08-28_19:00:05-regression-master-testing-gcov/10931$ cat summary.yaml 
ceph-sha1: 82c62bd977c74c22385b18791943cb2054920f47
client.0-kernel-sha1: 995fc068ddf675260098c60591989bf2ee184338
description: collection:rbd-thrash clusters:6-osd-3-machine.yaml fs:xfs.yaml msgr-failures:few.yaml
  thrashers:default.yaml workloads:rbd_fsx_nocache.yaml
duration: 4557.7895209789276
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
  /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper term /tmp/cephtest/binary/usr/local/bin/ceph-osd
  -f -i 4 -c /tmp/cephtest/ceph.conf'''
flavor: gcov
mds.a-kernel-sha1: 995fc068ddf675260098c60591989bf2ee184338
mon.a-kernel-sha1: 995fc068ddf675260098c60591989bf2ee184338
owner: scheduled_teuthology@teuthology
success: false

Actions #1

Updated by Tamilarasi muthamizhan over 11 years ago

Recent logs: ubuntu@teuthology:/a/teuthology-2012-09-30_19:00:06-regression-master-testing-gcov/33616

Actions #2

Updated by Sage Weil over 11 years ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF