Project

General

Profile

Actions
Copy link

Bug #9771

closed

Segmentation fault after upgrade v0.80.5 -> v0.80.6

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
Sage Weil
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is new test upgrades from v0.80.4 > v0.80.5 -> v0.80.4>firefly and runs different workloads after each step.

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-10-14_08:06:20-upgrade:firefly:singleton-firefly-distro-basic-vps/548125/

2014-10-14T08:41:01.196 INFO:tasks.workunit.client.0.vpm053.stderr:+ rbd import --new-format --order 20 /tmp/sparse1
2014-10-14T08:41:01.247 INFO:tasks.workunit.client.0.vpm053.stderr:*** Caught signal (Segmentation fault) **
2014-10-14T08:41:01.248 INFO:tasks.workunit.client.0.vpm053.stderr: in thread 7f630c1df700
2014-10-14T08:41:01.392 INFO:tasks.workunit.client.0.vpm053.stderr:Segmentation fault (core dumped)
2014-10-14T08:41:01.456 INFO:tasks.workunit:Stopping rbd/import_export.sh on client.0...

archive_path: /var/lib/teuthworker/archive/teuthology-2014-10-14_08:06:20-upgrade:firefly:singleton-firefly-distro-basic-vps/548125
branch: firefly
description: upgrade:firefly:singleton/{all/versions-steps.yaml distros/ubuntu_12.04.yaml}
email: yuri.weinstein@inktank.com
job_id: '548125'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: vps
name: teuthology-2014-10-14_08:06:20-upgrade:firefly:singleton-firefly-distro-basic-vps
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    fs: xfs
    log-whitelist:
    - slow request
    - scrub
    sha1: a6544a3137aeb5066d57a6fd54208438a8cd66b8
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: a6544a3137aeb5066d57a6fd54208438a8cd66b8
  s3tests:
    branch: firefly
  workunit:
    sha1: a6544a3137aeb5066d57a6fd54208438a8cd66b8
owner: yuriw
priority: 100
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mon.c
  - osd.3
  - osd.4
  - osd.5
  - client.0
  - client.1
suite: upgrade:firefly:singleton
suite_branch: wip_9758
suite_path: /var/lib/teuthworker/src/ceph-qa-suite_wip_9758
targets:
  ubuntu@vpm053.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHHLy4mq1Jy8mfNxFtH2Tx0wqxNPAbDPqpxUcI3DKXerfGx58pXfIpxYyvtQRL49odDjaj7rXBSCM7yDaBO7hhyK6FLDKRcewxnv/4x2TH/XqwU1qq07V1qs4O6eX/ZLK40ScBVX/91A6BY75O5kHPLrU54GDnPfjDeR/Vn4jntiWQ4USRbR/zaGBTkxc5kwCjRkDIhFjRBzQb6RCsjh7zDK8cxSuTMMTapqlN3+GpCHkVJ9ayCDhgKI7NLWgTMfYwrOembKUIIdnShVa8PHACMydE4/M3rURis02Vbe8zMNYIjwW8midW1lHt3lRH12YkQpifw++usvRgftQ4lp0P
  ubuntu@vpm139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDb60ZXRkWl6fQ9Kg0sez6ScoVWCTl6DbL+9WD0tbR4d6oGGgwLxhRTdO7ZlGJPtypgBIEIsS3kvyRAOh0Ki8AnGvZzh66kFgYszlUzo/4UuLZZN7rJOVIkAR89GADq8XKaVaF7H9rzFIF9ixoRpo4ylLiCj6d6sEzCFDb94CXCfMwfDt8+f+GcRn4zvFJpIRB9r5mTvb1SIcYMjyB2cLApWVgTFRaPN/k8FlmhpGKcvJgjffULFwbSRYxje0UqWbOU2Vu3TeaaKI5yV++m8sLwDIOuk0VwX7GCx2x6m+VaQiRwIVuyquwg7+s3olbAuNGqZLfyi1mCkV+qweqcikWN
tasks:
- internal.lock_machines:
  - 2
  - vps
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.push_inventory: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    tag: v0.80.4
- print: '**** done v0.80.4 install'
- ceph:
    fs: xfs
- print: '**** done ceph xfs'
- parallel:
  - workload
- print: '**** done workload'
- parallel:
  - workload1
  - upgrade-sequence1
- print: '**** done parallel 1'
- parallel:
  - workload2
  - upgrade-sequence2
- print: '**** done parallel 2'
- parallel:
  - workload_firefly
  - upgrade-sequence_firefly
- print: '**** done parallel firefly branch'
teuthology_branch: master
tube: vps
upgrade-sequence1:
  sequential:
  - install.upgrade:
      mon.a:
        tag: v0.80.5
      mon.b:
        tag: v0.80.5
  - print: '**** done v0.80.5 install.upgrade'
  - ceph.restart:
    - mon.a
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.b
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.4
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.5
  - sleep:
      duration: 30
  - print: '**** done ceph.restart all 1'
upgrade-sequence2:
  sequential:
  - install.upgrade:
      mon.a:
        tag: v0.80.6
      mon.b:
        tag: v0.80.6
  - print: '**** done v0.80.6 install.upgrade'
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.4
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.5
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.a
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.b
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - print: '**** done ceph.restart all 2'
upgrade-sequence_firefly:
  sequential:
  - install.upgrade:
      mon.a:
        branch: firefly
      mon.b:
        branch: firefly
  - print: '**** done branch: firefly install.upgrade'
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.4
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.5
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.a
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.b
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - print: '**** done ceph.restart all firefly current branch'
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.vps.3000
workload:
  workunit:
    clients:
      client.0:
      - suites/blogbench.sh
workload1:
  sequential:
  - workunit:
      clients:
        client.0:
        - rbd/import_export.sh
      env:
        RBD_CREATE_ARGS: --new-format
  - workunit:
      clients:
        client.0:
        - cls/test_cls_rbd.sh
workload2:
  sequential:
  - workunit:
      clients:
        client.0:
        - rbd/import_export.sh
      env:
        RBD_CREATE_ARGS: --new-format
  - workunit:
      clients:
        client.0:
        - cls/test_cls_rbd.sh
workload_firefly:
  sequential:
  - rgw:
    - client.0
  - s3tests:
      client.0:
        force-branch: firefly-original
        rgw_server: client.0

description: upgrade:firefly:singleton/{all/versions-steps.yaml distros/ubuntu_12.04.yaml}
duration: 1542.9695949554443
failure_reason: 'Command failed on vpm053 with status 139: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp
  && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=a6544a3137aeb5066d57a6fd54208438a8cd66b8 TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'''
flavor: basic
owner: yuriw
success: false
Actions
Copy link
 #1

Updated by Yuri Weinstein over 9 years ago

  • Assignee set to Sage Weil
Actions
Copy link
 #2

Updated by Samuel Just over 9 years ago

  • Project changed from Ceph to rbd
Actions
Copy link
 #3

Updated by Jason Dillaman over 9 years ago

  • Subject changed from Segmentation fault after upgrade v0.80.4 -> v0.80.5 to Segmentation fault after upgrade v0.80.5 -> v0.80.6
Actions
Copy link
 #4

Updated by Jason Dillaman over 9 years ago

Commit b75f85a2 added new elements to the Thread class, breaking ABI. In this (and several other upgrade tests from v0.80.5), if librados has been updated while librbd is still in the process of being upgraded, code within librbd that uses the Finisher class will incorrectly expect the old Thread class ABI where priority isn't present but execute new Thread class ABI where priority is added. As a result, the Finisher's this pointer is now located at the same memory address as the priority values, resulting in a crash.

Actions
Copy link
 #5

Updated by Jason Dillaman over 9 years ago

  • Status changed from New to 12
Actions
Copy link
 #6

Updated by Sage Weil over 9 years ago

  • Status changed from 12 to Won't Fix
Actions
Copy link

Also available in: Atom PDF