Project

General

Profile

Actions
Copy link

Bug #7582

closed

"FAIL: test_rbd.TestImage.test_remove_with_watcher" in upgrade:dumpling-x-firefly---basic-plana

Added by Yuri Weinstein about 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in qa-proxy.ceph.com/teuthology/teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana/113455/

014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: ======================================================================
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: FAIL: test_rbd.TestImage.test_remove_with_watcher
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: ----------------------------------------------------------------------
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: Traceback (most recent call last):
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:   File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:     self.test(*self.arg)
2014-03-02T00:04:41.785 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:   File "/home/ubuntu/cephtest/mnt.0/client.0/tmp/test_rbd.py", line 370, in test_remove_with_watcher
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:     eq(read, data)
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:   File "/usr/lib/python2.7/dist-packages/nose/tools.py", line 31, in eq_
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]:     assert a == b, msg or "%r != %r" % (a, b)
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: AssertionError: '\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' != '\x9c\xe9\xae\x8c\x822&2G\x8c\xfee`\x04y\xc4s\xa5\xc74\x1ea=,E\xf9\x9cY\xb4\x1c.]\xa4\xe1\xbe5\x15\xb1%\x98\x03M\xdek)FQ\x03\xde\xc0\xbc\x0e\x0b\xa9\xb5\x11\x8a_\xf9\xb9Wq\x1fZ\x17\xd4\x8b\xb9T}\xe08\x93\xe3\xd7\r\xf7\xc6R\xad\x8d{\x97\xce>\xcbI\x8e\xc1S7\x9f\xcf\x12=)\xfb\xfb\x13/\x15]\xe8K\x93\x9d]o\x0c\x1bv\xd0\x86F\xe5"\xdbD\xeb\xccN\xc0\x08!\x94~\xe7\x04\xb3\xc3\x02d\x14|U\xa3\x84g~\x16G\xc5\xb9\x86\x92p\x14\xf64\xdf\x8f\x82\xbduo\xb1\x82\xda\xc6\x8c\x02Z\xbd\xb3n\x95^!\xc1\x87\xe1\xf2\x0c\x9b\x03CA\xa6\x17\x92\xff\xd2\xc4\xe7\x10\xf9Kd\xf5l\x0c\xda\xdf\xc4\x07\xe7\x93t\xf6\x03\x96\x9bhk\xef\xbb\xe2\'1\x18\x18\xfc\xa8\xd8\x14\xa3}\x859#Ut\xa6\x1a\x93\xe3\xe8s\xad\x19\xfd\xfa\xe5C\x82\xa0&pP\xe9\x81-\xde\x94\t\xc9\xd4\x0f2\x15\x9e\xe1\r\x1e\x17\x90'
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: 
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: ----------------------------------------------------------------------
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: Ran 52 tests in 559.939s
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: 
2014-03-02T00:04:41.786 INFO:teuthology.task.workunit.client.0.err:[10.214.133.24]: FAILED (SKIP=14, failures=1)
2014-03-02T00:04:41.806 INFO:teuthology.task.workunit:Stopping rbd/test_librbd_python.sh on client.0...
2014-03-02T00:04:41.806 DEBUG:teuthology.orchestra.run:Running [10.214.133.24]: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2014-03-02T00:04:41.853 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 345, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.24 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_python.sh'
2014-03-02T00:04:41.854 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 31, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 19, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 96, in task
    all_spec = True
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 345, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.133.24 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_python.sh'
2014-03-02T00:04:41.912 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/inktank/teuthology/search?q=42d620e6ba1c46dcb36599915aabb62b
CommandFailedError: Command failed on 10.214.133.24 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_python.sh'

archive_path: /var/lib/teuthworker/archive/teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana/113455
description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/rbd-python.yaml
  distros/ubuntu_12.04.yaml}
email: null
job_id: '113455'
last_in_suite: false
machine_type: plana
name: teuthology-2014-03-01_22:00:03-upgrade:dumpling-x-firefly---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - wrongly marked me down
    - objects unfound and apparently lost
    - log bound mismatch
    sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
  s3tests:
    branch: master
  workunit:
    sha1: 32a4e903497bcf40a50e5e683c7bda17eccd11b3
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.b
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - osd.3
  - osd.4
  - osd.5
  - mon.c
- - client.0
targets:
  ubuntu@plana92.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCuveGNP9yQDx0hMt9UdUaiejKFKzqTlHrUZeaQ21bX7Aofr1BjMgWjSvZ1q5CCY0xC266hra1ga7kVe2V68IP3g8l8IVYdNa+BWIglXAcbXKYLAIPeiOPz4WHWVBV5YBMKgV9hAUsRQ3/iKQ+yJ9RAJ18h0VjMEDJKZYWXdwJVs4wzyoRe8C3Pll81QzHzgwTY/9sOGXFLG2GedJgq4mTO4yW3nLj2WUgcTcCUvGKkMNA/9hdmOirbCZFxLhY8diOYfHUZHUx+g+MEx1WJfUyxNLMhQKiblSeP+i7S3baOnjRfCn/MWnH2a+Qnz/oNh6u+8+NxFMiYtPbqf0Ia3Uqh
  ubuntu@plana93.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPzHSYgRXH5aC2Wn7m418ld5rErB4Q1Ki5Ev5up1+orbbKEzA+de4ZP3yaLfwmnpJlLxik9h6ellNs5gkJaLRCv4mQkOH9FPpUwDHuEoFnElVC10RO4uS5COmMcY4GR8tXbitQsx3mjw/++F90mENxWpgU6GDPn/KYQJfe22mpbMtiygJhHdCyOwXSpQQVNG0Xg/8QybFC9GmmiXHekigXwusIMXoSA5qqZ/fbi+j+4Y0j1bdqxHC3mpoAEXJrmNqypgHAlwB8zrlO7TypcXeIdEFVFA/GE9nygFQjw0pl1Gc6nqQr33ZzY5Iekch2Xjof3y/K8elsZ+6+yBU5u4/T
  ubuntu@plana94.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtBFZH1vJ4VyJt3ReE2X11UmD6sLcvvSk5jWtbbmTKwHPECESeIH7se+2Hm2jNoqzgvGCI6IHTgqJIRXLNJL/Z2QAC6w5cHJhVgbqI4ksZ8Z3r0QEfuJzmO4eMQccfhjmuNRSm+UwNtUg2RgH0qXvHchxuvBSgyBpQ3MxsdBCwpRvUbg17YzGGe41H/S0ObmSQecWoPoYa5VDoFNR8aURu0fV9KBYJ3b57SL3fieF/IEYbcIT1lmml7bdfEUTWrh3HmnXq6/qMaaINPcuMYERNTbRmJg/by7c0DC3qViDXFXaQCif7mXvk0+S7I3pV3GwcqxPrmmuJJGFJ08MDxur
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: dumpling
- ceph:
    fs: xfs
- install.upgrade:
    osd.0: null
- ceph.restart:
    daemons:
    - osd.0
    - osd.1
    - osd.2
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000
- ceph.restart:
    daemons:
    - mon.b
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: dumpling
    clients:
      client.0:
      - rbd/test_librbd.sh
- install.upgrade:
    mon.c: null
- ceph.restart:
    daemons:
    - mon.c
    wait-for-healthy: false
    wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
  - a
  - b
  - c
- workunit:
    clients:
      client.0:
      - rbd/test_librbd_python.sh
teuthology_branch: firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11166

description: upgrade/dumpling-x/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/readwrite.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/rbd-python.yaml
  distros/ubuntu_12.04.yaml}
duration: 1280.4398169517517
failure_reason: 'Command failed on 10.214.133.24 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp
  && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=32a4e903497bcf40a50e5e683c7bda17eccd11b3 TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  /home/ubuntu/cephtest/workunit.client.0/rbd/test_librbd_python.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=42d620e6ba1c46dcb36599915aabb62b
success: false
Actions
Copy link
 #1

Updated by Yuri Weinstein about 10 years ago

  • Project changed from teuthology to rbd
Actions
Copy link
 #2

Updated by Sage Weil about 10 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF