Project

General

Profile

Actions
Copy link

Bug #7583

closed

"librbd::ImageCtx: error finding header" in upgrade:dumpling-x-firefly---basic-plana

Added by Yuri Weinstein about 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in qa-proxy.ceph.com/teuthology/teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana/113365/

2014-03-01T22:20:25.412 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: 2014-03-01 22:20:25.411932 7f1a3c2c4780 -1 librbd::ImageCtx: error finding header: (2) No such file or directory
2014-03-01T22:20:25.455 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: 
Removing image: 0% complete...failed.
2014-03-01T22:20:25.455 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: rbd: delete error: (2) No such file or directory
2014-03-01T22:20:25.455 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: 2014-03-01 22:20:25.454666 7f1a3c2c4780 -1 librbd: error removing img from new-style directory: (2) No such file or directory
2014-03-01T22:20:25.456 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: + true
2014-03-01T22:20:25.456 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: + rbd import --new-format /tmp/img testimg
2014-03-01T22:20:27.977 INFO:teuthology.task.workunit.client.1.err:[10.214.133.25]: 
..................
2014-03-01T22:20:54.614 DEBUG:teuthology.orchestra.run:Running [10.214.133.25]: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-03-01T22:20:54.687 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 345, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.25 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.1/rbd/import_export.sh'
2014-03-01T22:20:54.688 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 31, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 19, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 96, in task
    all_spec = True
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 345, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.25 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.1/rbd/import_export.sh'
2014-03-01T22:20:54.721 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/inktank/teuthology/search?q=3bd4807fbc614579bd383843e65b3693
CommandFailedError: Command failed on 10.214.133.25 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.1/rbd/import_export.sh'
2014-03-01T22:20:54.721 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2014-03-01T22:20:54.721 DEBUG:teuthology.orchestra.run:Running [10.214.133.25]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json'

archive_path: /var/lib/teuthworker/archive/teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana/113365
description: upgrade/dumpling-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-mds-mon-osd.yaml 4-final-workload/rbd_import_export.yaml
  distros/ubuntu_12.04.yaml}
email: null
job_id: '113365'
last_in_suite: false
machine_type: plana
name: teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
  s3tests:
    branch: master
  workunit:
    sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
targets:
  ubuntu@plana86.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUMsCERn/r+eXKzlu9W/n6BFak26ReRaNUaccpQTrqOqhm3M7LiIGAeBo6JsygU9Kdtsm3115P4odiDbQLuNm0gKPw5DY7zDVqV+YRqe3kOrgIL/rxs6l6Y7htSRvzGhz7RsaG1fQH/BuaD18+s2WguKWwgRuuU1bvRYEHu0Y7qYYqUy+xJd4CGX+LMuAJzDYj5R7xIEYlJBln/c47He8q53cUUw5w48Hm47O8xo9ov7CqHCpOTZixeseY+zEm9skoLsUpBDIOmT+xUh7sOyKFhj+CnETpv3DocxGtoUgkx42GtkDhFK/dVV95Q6EOBseyYIUvFoHBKB2WV32xyxB5
  ubuntu@plana87.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOIL0Fp0SbVsgtW9QPJ5/xsNbSgWhZFnfJaHHxopCl1pQSidALezU+amcNffEgTG5B4SKuPKMkIxLXSCM+dRwB6tp0sM7xtAv+SGOpG5emTp0lRHvo9WToef7n59irekX60A8a7HPeCnS3Yjrrk4GWKmd5Y+BAypekOxTbTKJfA7HJmVOW1Dl36KGoEvN0+z5tHCMmxP2eJgBT2GffDvhp7ThKAKw6qgMmEIBAw7M1keipdAtcX9SdzgF9Pr6yFb/aB9HddNwJ3W9gG3egKQ6nZJw1wvAUvTHbqNnDUCOHwqs0wCP0hCzWufUBeZVDtZxBQwyH9cO93v4oep3NPXrJ
  ubuntu@plana91.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC35R8gF2rOu5meqvMr/vp9m+RozQEpRG3+0KNjFYW+lm1f6vckHVP1+1xFx3fx4iGASg2aoKtnq26zR8k7ErKepFrAzT27FvEjdfEa6Vn5fJhFOtJOTaaxxQw0LNURPl21rrQaMRCVewrHVVEJiZ/Nl4fldZPWTJXBMZvFHRBae/aax+Fcqe4jf7uauPg6x8FUEErBbvIcn5Qm46qZS/WTLBAaah5o4JW1zQNaXxFvnsv3cgAu0d10GFVY1BYi23gYpgHfLLo20H4GLmb7J4KOxGuRaNxg4kR2tQv37mOJYkmZSImd+JaUFSDA9HSnQ9BA7rskAPX5jXIXRYYnI8+X
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: dumpling
- ceph:
    fs: xfs
- parallel:
  - workload
  - upgrade-sequence
- workunit:
    clients:
      client.1:
      - rbd/import_export.sh
    env:
      RBD_CREATE_ARGS: --new-format
teuthology_branch: firefly
upgrade-sequence:
  sequential:
  - install.upgrade:
      all: null
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - mon.a
      wait-for-healthy: false
      wait-for-osds-up: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - mon.b
      wait-for-healthy: false
      wait-for-osds-up: true
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.3
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11178
workload:
  sequential:
  - branch: dumpling
    clients:
      client.0:
      - rbd/test_librbd.sh
    workunit: null

description: upgrade/dumpling-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-workload/test_rbd_api.yaml 3-upgrade-sequence/upgrade-mds-mon-osd.yaml 4-final-workload/rbd_import_export.yaml
  distros/ubuntu_12.04.yaml}
duration: 262.61854004859924
failure_reason: 'Command failed on 10.214.133.25 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp
  && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="1" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  /home/ubuntu/cephtest/workunit.client.1/rbd/import_export.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=3bd4807fbc614579bd383843e65b3693
success: false
Actions
Copy link
 #1

Updated by Yuri Weinstein about 10 years ago

  • Project changed from teuthology to rbd
Actions
Copy link
 #2

Updated by Sage Weil about 10 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF