Project

General

Profile

Actions

Bug #4308

closed

ceph-fuse crashed during blogbench test (argonaut)

Added by Tamilarasi muthamizhan about 11 years ago. Updated about 11 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Log: ubuntu@teuthology:/a/teuthology-2013-02-28_20:00:04-regression-argonaut-master-basic/14324

2013-02-28T20:10:14.346 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err:*** Caught signal (Segmentation fault) **
2013-02-28T20:10:14.346 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: in thread 7f484094e700
2013-02-28T20:10:14.354 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: ceph version 0.48.3argonaut-19-g50b4f72 (commit:50b4f72eb74c4ae8238731ac3ceb40b5a21b79b6)
2013-02-28T20:10:14.355 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 1: (ceph::BackTrace::BackTrace(int)+0x2d) [0x86caf7]
2013-02-28T20:10:14.355 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 2: /tmp/cephtest/binary/usr/local/bin/ceph-fuse() [0x94ab02]
2013-02-28T20:10:14.355 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 3: (()+0xfcb0) [0x7f48456b3cb0]
2013-02-28T20:10:14.355 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 4: (tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)+0xeb) [0x7f4844960e3b]
2013-02-28T20:10:14.356 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 5: (tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long)+0x23) [0x7f4844960ec3]
2013-02-28T20:10:14.356 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 6: (operator delete(void*)+0x28b) [0x7f48449703ab]
2013-02-28T20:10:14.357 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 7: (std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()+0x23) [0x7f48446ddc13]
2013-02-28T20:10:14.357 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 8: (object_t::~object_t()+0x18) [0x72bf50]
2013-02-28T20:10:14.357 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 9: (MOSDOpReply::~MOSDOpReply()+0x43) [0x79a641]
2013-02-28T20:10:14.360 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 10: (MOSDOpReply::~MOSDOpReply()+0x18) [0x79a6b8]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 11: (RefCountedObject::put()+0x44) [0x72e5da]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 12: (Objecter::handle_osd_op_reply(MOSDOpReply*)+0x19a9) [0x8f8889]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 13: (Client::ms_dispatch(Message*)+0x23a) [0x6f5e88]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 14: (Messenger::ms_deliver_dispatch(Message*)+0x9b) [0x85ce6f]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 15: (SimpleMessenger::DispatchQueue::entry()+0xa39) [0x8432e5]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 16: (SimpleMessenger::dispatch_entry()+0x1e) [0x84358a]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 17: (SimpleMessenger::DispatchThread::entry()+0x1c) [0x7fb96e]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 18: (Thread::_entry_func(void*)+0x23) [0x86690d]
2013-02-28T20:10:14.361 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 19: (()+0x7e9a) [0x7f48456abe9a]
2013-02-28T20:10:14.362 INFO:teuthology.task.ceph-fuse.ceph-fuse.0.err: 20: (clone()+0x6d) [0x7f4843e644bd]
ubuntu@teuthology:/a/teuthology-2013-02-28_20:00:04-regression-argonaut-master-basic/14324$ cat summary.yaml 
ceph-sha1: 50b4f72eb74c4ae8238731ac3ceb40b5a21b79b6
client.0-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
description: collection:basic clusters:fixed-3.yaml fs:btrfs.yaml tasks:cfuse_workunit_suites_blogbench.yaml
duration: 478.7611839771271
failure_reason: 'Command failed with status 1: ''mkdir -- /tmp/cephtest/mnt.0/client.0/tmp
  && cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=50b4f72eb74c4ae8238731ac3ceb40b5a21b79b6
  PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib" 
  CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret" 
  CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages" 
  /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
  /tmp/cephtest/workunit.client.0/suites/blogbench.sh && rm -rf -- /tmp/cephtest/mnt.0/client.0/tmp'''
flavor: basic
mon.a-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
mon.b-kernel-sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
owner: scheduled_teuthology@teuthology
success: false
ubuntu@teuthology:/a/teuthology-2013-02-28_20:00:04-regression-argonaut-master-basic/14324$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 83ca14fdd35821554058e5fd4fa7b118ee504a33
nuke-on-error: true
overrides:
  ceph:
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 50b4f72eb74c4ae8238731ac3ceb40b5a21b79b6
  s3tests:
    branch: argonaut
  workunit:
    sha1: 50b4f72eb74c4ae8238731ac3ceb40b5a21b79b6
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana19.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0IExVUuQEnwBzUbN6+jgEb/qLLsSDFLAe0OPv+R3Q3uUZn+4QjB+FQ3sMMEwaGpEhUfbpWn3xZbCmi4NngTsgPDjImYJPMeMecaxvVXqAJt4IM6gdBN0415lrKXtbXaGBJCmeFFDB+xN+JN6Qhk8DWN62DnJB8MS5vKn9u0S3HvtGzY/QrnutT3AX9I4097isbTepLFRC4n2CoAC9srXaxAprFgLgOIYHm386B3W7yK3yfIImWofvZxYpPCsr/7ws8DSgdRX9eDucGQfY2WcYKCEIgZclGPBhExm7q/ahHUqYKrzWY3RD93AuXJVgOJk5Yp8C1Ryx8vyJkAxBW4nb
  ubuntu@plana20.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDsgMh/6mIDSo4biGWf/J3RkwiTI024iLAWAP0eFej28LkKX9IX6zlToAQeo7dibZ1LP4O7B3D43Fx+XUrz6x2y1ue7WmR+YEeza2wa8KQqJ0jpBNnJucPvMGtEiuEaAOc+WIWEyGGdXg1tUR2j9VcU+B26JS1QVov3ttW+/0GaSLqp080EtjN2bV6L/imEBcyEiYy1nFczCGdYNWQGwn9Fk5H79PuThymM1Fyp1ALwWJIMkI7Hd+9wTqhjBacIc7YS/c3dJyrTtFKshcQYRC0ECskeXU/p5pwfb+wVpkkYeX1bar6Ma/IThnyoDtDrnMs+QYt+wuz9dmnj3ZQ/dIJp
  ubuntu@plana25.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuXajaQgHe9XnbLOzI8WWFYVz6+TnOiTzbkIJPGOZpzQEjnUtJraQIEt5ABSeovMjiEj+V4XvunfyuSmEd0H9giRSyjmCHTPGlpndfTeCdVtCBpNqf5GkUqHaEY1Hp57XPbya2rGlwtFm0NeIDYx6pfkejKnsTOUqwhgUb6950TRhjHQhMjFgyALSyfAm/4y6vGZfjm57+yyih6XgDkqWiiQ6Y/aJVR2n+iCzvqEzV7JSCU+Brn+k8IQLHho1fadYqc5PjYct5BaVlHcP6c+T8nJE/DvqGwZ4gQaVJcuWJiDfLOPPYo1g/0AFicxauLwVNJ6HFR9FjLLGtGU+2DcVN
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
    clients:
      all:
      - suites/blogbench.sh

Actions #1

Updated by Sam Lang about 11 years ago

  • Project changed from Ceph to CephFS
Actions #2

Updated by Sage Weil about 11 years ago

  • Project changed from CephFS to Ceph
  • Subject changed from ceph fuse crashed during blogbench test to ceph-osd crashed during blogbench test (argonaut)
  • Assignee deleted (Sam Lang)
Actions #3

Updated by Sage Weil about 11 years ago

  • Project changed from Ceph to CephFS
  • Subject changed from ceph-osd crashed during blogbench test (argonaut) to ceph-fuse crashed during blogbench test (argonaut)
  • Status changed from New to Won't Fix
  • Priority changed from High to Normal

this is most likely memory corruption in argonaut's ceph-fuse.

Actions

Also available in: Atom PDF