Project

General

Profile

Actions
Copy link

Bug #9417

closed

"Segmentation fault" in upgrade:dumpling-giant-x-master-distro-basic-vps run

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
Josh Durgin
Category:
librbd
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-09-09_17:00:02-upgrade:dumpling-giant-x-master-distro-basic-vps/475490/

2014-09-09T22:51:00.367 INFO:tasks.workunit.client.1.vpm072.stdout:[ RUN      ] LibRadosIo.SimpleWrite
2014-09-09T22:51:01.680 INFO:tasks.workunit.client.1.vpm072.stdout:[       OK ] LibRadosIo.SimpleWrite (1109 ms)
2014-09-09T22:51:01.681 INFO:tasks.workunit.client.1.vpm072.stdout:[ RUN      ] LibRadosIo.ReadTimeout
2014-09-09T22:51:01.681 INFO:tasks.workunit.client.1.vpm072.stderr:Segmentation fault (core dumped)
2014-09-09T22:51:01.681 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.1...
2014-09-09T22:51:01.682 INFO:teuthology.orchestra.run.vpm072:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-09-09T22:51:01.694 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 117, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 357, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 104, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm072 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3c6e8884dfa9a7457b9f14d200d51ede44a97815 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'

archive_path: /var/lib/teuthworker/archive/teuthology-2014-09-09_17:00:02-upgrade:dumpling-giant-x-master-distro-basic-vps/475490
branch: master
description: upgrade:dumpling-giant-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml}
  3-giant-upgrade/giant.yaml 4-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml
  test_rbd_python.yaml} 5-upgrade-sequence/upgrade-by-daemon.yaml 6-final-workload/{ec-rados-default.yaml
  ec-rados-plugin=jerasure-k=3-m=1.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml
  rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_s3tests.yaml rgw_swift.yaml}
  distros/debian_7.0.yaml}
email: ceph-qa@ceph.com
job_id: '475490'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: vps
name: teuthology-2014-09-09_17:00:02-upgrade:dumpling-giant-x-master-distro-basic-vps
nuke-on-error: true
os_type: debian
os_version: '7.0'
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      global:
        osd heartbeat grace: 100
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    sha1: 3c6e8884dfa9a7457b9f14d200d51ede44a97815
  ceph-deploy:
    branch:
      dev: master
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 3c6e8884dfa9a7457b9f14d200d51ede44a97815
  rgw:
    default_idle_timeout: 1200
  s3tests:
    branch: master
    idle_timeout: 1200
  workunit:
    sha1: 3c6e8884dfa9a7457b9f14d200d51ede44a97815
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
suite: upgrade:dumpling-giant-x
suite_branch: master
suite_path: /var/lib/teuthworker/src/ceph-qa-suite_master
targets:
  ubuntu@vpm057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtyrKT5Qqp1gpWRuEG4a5FSIv7M2f7hK0pv0fMTpthjgEPuCs7EP/FphULqlCfOvdCyrq5865QNVKU6KXU2S44W7dBVaoFbE9IcqVVNv4leWI3a6vs/DVxZe0LJDtv+HroO7ZJR3HqAiYVtW85sLm5J50CrSGPRf4riLhVCzC3mpp0CCcam/Gj0mN6knV3p1gOJsVYzuZ3d8bWGSYRvJJ+1oyE5ucTMs+WDDPLFWKFXj6Ip+bAI6qpERkkw8A6Uwtk3bfFz3YZeuItt/L1a7y5/GCyjAmgR6bInpmPjV8Q6GdgX4BOEUIBuTnYx1ITib95yd0VXC36UEjeQUCXZ+Vz
  ubuntu@vpm072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCs9XFAmFJla3FlwyUafV6xtVukpQlbZ5H1oQLPB5j/F9yhzt4WwCuQtEJoXc501y2SSV/2emYkhtkf48mXolKS+e2E/G3X2slIm4g4d4UP5rPAfPb7Sx8y/ajHGtKg3+lXrwDeKqjRExA7d5knMFDhaO+tpCyN852cd3W3GFiknal1aC84tkjYRohDJHreC02uTXOK3pBFMDlHLlccAfG7k5qjfXcgQ73VHjKrC6KvRtJ6UchB+hsXm2Hu0qwwr+ISVJUsNoz7oG2ohp5qty7xZTUANm//EL5X7Yb4S2loKruk7ARbK1eLiVct0/Kipgrby5ZnNI9+WC39GKt5jy+R
  ubuntu@vpm178.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBZ1XoCWVnzrp+6PVZ01UodgwxKn4VCk9TgFZ7VgNfHbJ+VbUzkaQqb1mw6a+DA7ctkyesj4wHADNv6R9RvMXiVCxlMFUNLZZVmu9QwZ9+/n2o3GKKhC4fAXGds8/bmdA/HTQ0ScjiR2yJQ+gP1wmMDoBxAs+fgq9fLrxuBkcwlcwTc+DMqfFUOYgQyEFmetlC5qlgOfUy7m785JGetTRjkntgn0IWkQnFwpN5X0quN2dhtu/HCGbTySA9xRPhJWaFQIXf+/mqC+VcztlqEFD1Yz0InVd9MVKmbBS3xIg6HWKUyoJqQzzzEhZVGACtKnUvj/NJ4mqIZ69efnDgprkv
tasks:
- internal.lock_machines:
  - 3
  - vps
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: dumpling
- print: '**** done dumpling install'
- ceph:
    fs: xfs
- parallel:
  - workload
- print: '**** done parallel'
- install.upgrade:
    client.0:
      branch: giant
    mon.a:
      branch: giant
    mon.b:
      branch: giant
- print: '**** done install.upgrade'
- ceph.restart: null
- print: '**** done restart'
- parallel:
  - workload2
  - upgrade-sequence
- print: '**** done parallel 2'
- install.upgrade:
    client.0: null
- print: '**** done install.upgrade client.0 to the version from teuthology-suite
    arg'
- rados:
    clients:
    - client.0
    ec_pool: true
    objects: 50
    op_weights:
      append: 100
      copy_from: 50
      delete: 50
      read: 100
      rmattr: 25
      rollback: 50
      setattr: 25
      snap_create: 50
      snap_remove: 50
      write: 0
    ops: 4000
- rados:
    clients:
    - client.0
    ec_pool: true
    erasure_code_profile:
      k: 3
      m: 1
      name: jerasure31profile
      plugin: jerasure
      ruleset-failure-domain: osd
      technique: reed_sol_van
    objects: 50
    op_weights:
      append: 100
      copy_from: 50
      delete: 50
      read: 100
      rmattr: 25
      rollback: 50
      setattr: 25
      snap_create: 50
      snap_remove: 50
      write: 0
    ops: 4000
- rados:
    clients:
    - client.1
    objects: 50
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
- workunit:
    clients:
      client.1:
      - rados/load-gen-mix.sh
- sequential:
  - mon_thrash:
      revive_delay: 20
      thrash_delay: 1
  - workunit:
      clients:
        client.1:
        - rados/test.sh
  - print: '**** done rados/test.sh - 6-final-workload'
- workunit:
    clients:
      client.1:
      - cls/test_cls_rbd.sh
- workunit:
    clients:
      client.1:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format
- rgw:
  - client.1
- s3tests:
    client.1:
      rgw_server: client.1
- swift:
    client.1:
      rgw_server: client.1
teuthology_branch: master
tube: vps
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
  - print: '**** done install.upgrade mon.a to the version from teuthology-suite arg'
  - install.upgrade:
      mon.b: null
  - print: '**** done install.upgrade mon.b to the version from teuthology-suite arg'
  - ceph.restart:
      daemons:
      - mon.a
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - mon.b
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - exec:
      mon.a:
      - ceph osd crush tunables firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.vps.15623
workload:
  sequential:
  - workunit:
      branch: dumpling
      clients:
        client.0:
        - rados/test.sh
        - cls
  - print: '**** done rados/test.sh &  cls'
  - workunit:
      branch: dumpling
      clients:
        client.0:
        - rados/load-gen-big.sh
  - print: '**** done rados/load-gen-big.sh'
  - workunit:
      branch: dumpling
      clients:
        client.0:
        - rbd/test_librbd.sh
  - print: '**** done rbd/test_librbd.sh'
  - workunit:
      branch: dumpling
      clients:
        client.0:
        - rbd/test_librbd_python.sh
  - print: '**** done rbd/test_librbd_python.sh'
workload2:
  sequential:
  - workunit:
      branch: giant
      clients:
        client.0:
        - rados/test.sh
        - cls
  - print: '**** done #rados/test.sh and cls 2'
  - workunit:
      branch: giant
      clients:
        client.0:
        - rados/load-gen-big.sh
  - print: '**** done rados/load-gen-big.sh 2'
  - workunit:
      branch: giant
      clients:
        client.0:
        - rbd/test_librbd.sh
  - print: '**** done rbd/test_librbd.sh 2'
  - workunit:
      branch: giant
      clients:
        client.0:
        - rbd/test_librbd_python.sh
  - print: '**** done rbd/test_librbd_python.sh 2'

description: upgrade:dumpling-giant-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml}
  3-giant-upgrade/giant.yaml 4-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml
  test_rbd_python.yaml} 5-upgrade-sequence/upgrade-by-daemon.yaml 6-final-workload/{ec-rados-default.yaml
  ec-rados-plugin=jerasure-k=3-m=1.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml
  rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_s3tests.yaml rgw_swift.yaml}
  distros/debian_7.0.yaml}
duration: 21899.848738908768
failure_reason: 'Command failed on vpm072 with status 139: ''mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp
  && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=3c6e8884dfa9a7457b9f14d200d51ede44a97815 TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
success: false

Related issues 1 (0 open — 1 closed)

Is duplicate of Ceph - Bug #9582: librados: segmentation fault on timeoutResolvedSage Weil09/24/2014

Actions
Actions
Copy link
 #1

Updated by Ian Colle over 9 years ago

  • Project changed from devops to Ceph
Actions
Copy link
 #3

Updated by Tamilarasi muthamizhan over 9 years ago

  • Category set to librbd
  • Assignee set to Josh Durgin
  • Priority changed from Normal to Urgent
Actions
Copy link
 #4

Updated by Samuel Just over 9 years ago

  • Status changed from New to Duplicate
Actions
Copy link

Also available in: Atom PDF