Project

General

Profile

Actions
Copy link

Bug #8988

closed

AssertionError(s) in upgrade:firefly-x:stress-split-next---basic-plana

Added by Yuri Weinstein almost 10 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Loïc Dachary
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Related issue

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755/
and
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387763/

2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount)
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:    cls.env.setUp()
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:    raise ResponseError(cls.conn.response)
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:ResponseError: 500: Internal Server Error
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testContainerListing (test.functional.tests.TestAccount)
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 262, in testContainerListing
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:    self.assertEquals(a, b)
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: Lists differ: ['167e6c2f7790448cb665118c0ca1... != ['167e6c2f7790448cb665118c0ca1...
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:First differing element 4:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:72c48b7e4f774b6ebae34c9e9008b690
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:525bc170f49f4e408b74dd2ca8f10b28
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:Second list contains 1 additional elements.
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:First extra element 10:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:e53eab2df63e444a8c2248f08eefbac3
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:  ['167e6c2f7790448cb665118c0ca14064',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '473c9ce271f24bb69255ea3a60f41e26',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '4ad0858c06204b71968a45d33782e90d',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '4de3250a2b3847bfb65743044ece7718',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:+  '525bc170f49f4e408b74dd2ca8f10b28',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   '72c48b7e4f774b6ebae34c9e9008b690',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   '8f49eb55dbae4f90a360e66260a29e05',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'b63f263822cb4b4d900af8a122573259',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'd1e9f200a3b8493f8f5c292dcbfb241e',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'dcf96a03c43049ffa799ea9fceffc27d',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'e53eab2df63e444a8c2248f08eefbac3']
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testLastContainerMarker (test.functional.tests.TestAccount)
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 272, in testLastContainerMarker
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:    self.assertEquals(len(containers), len(self.env.containers))
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: 11 != 10
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:Ran 137 tests in 874.291s
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:FAILED (errors=1, failures=2)
2014-07-30T23:59:17.951 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:17.952 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data'
2014-07-30T23:59:20.927 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data'
2014-07-30T23:59:21.377 INFO:teuthology.task.swift:Removing swift...
2014-07-30T23:59:21.377 INFO:teuthology.orchestra.run.plana47:Running: 'rm -rf /home/ubuntu/cephtest/swift'
2014-07-30T23:59:21.456 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:21.457 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2014-07-30T23:59:21.457 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 29, in nested
    yield vars
  File "/home/teuthworker/src/teuthology_next/teuthology/task/rgw.py", line 808, in task
    yield
  File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:21.458 INFO:teuthology.task.rgw:Stopping apache...

archive_path: /var/lib/teuthworker/archive/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755
branch: next
description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
  rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
email: ceph-qa@ceph.com
job_id: '387755'
last_in_suite: false
machine_type: plana
name: teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - wrongly marked me down
    - objects unfound and apparently lost
    - log bound mismatch
    sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
  s3tests:
    branch: next
  workunit:
    sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
  - mon.b
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - osd.3
  - osd.4
  - osd.5
  - mon.c
- - client.0
suite: upgrade:firefly-x:stress-split
suite_branch: master
targets:
  ubuntu@plana47.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIDs+Sl9mfdFX0VZy41zfHkMJlD1OtErcQsSPLq3wrQ5HPPflg8IRtS45LHsYBAyBcAl4fmeG1q45jOrHj6yibJhT/HzxO6TOikYKUgr3uWy804NsTAP/A2SJ4wA8Y+1WyiAjBs0EAnISCnIl5Arj17NYPBroDeclKqZEJpL5UbPCmMibqEmjIPNVNulukox/Dh9GJZWlxqD0aKfTRnIOM+6RWHv2h9mCYVtU2ZbMx+QR0sxji5IJzg4ITlGaR9g+U3DnOU95pkxb3x6mE+H8udB58FONty/DHaAC/rXGfjkg056R3iq8FJUpF3HKKjILLRHjLb3ueK3tN1zrEueCt
  ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFHCeGWMPGOLyScKFkduv7aJL9bpMUPZQATO9lxpWu1NtzYndPJtWcyUxgWlItu75SJwpXx/l2GhPYcDKrR1Nl37+dbgs5TeDTbr9YdQBuLPbkbIZMQqO4GqUjurEwLU3vFUZ0X7PTlUqn6qwpT+I2YJua19eF2cRQFIGYVZMzaezm47uh67cdKFh0RTA1pSJ2qM/WMn91boRWcsRQrmn4BeOzfpGfSPDRjrHXHiPx3Br4zcOi/3lOxNFcEeoBrA47PMxvxVIlbmxKDfNjHpQQT18VFWb+qcTAzf+zdBy3iDRFFS45fPrqlWjGn9sK74EbRQanDrZlrFkg2a/HIe5T
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Dgv5XfnXOHEuwwOoK+xQtHEZusRHTm9I4oBJ41RYkPq2y5GA5OrXiWVlOrwBoXXdtCeW4ynErDhqiFuL3tYmwNEYzRWqnyqZte4qfsTz93Lhv7UEkageJ2iHNaUNt+H071A8JULR2CRtIxXu6zSSKC8vwmEirxqYj3pPRVm9TCa1iPaj8R3wPmeBjwVD9IU+zAuvIi6oWcqKrxZEdEOciMa72nGO58V7Wo0yICMST6day1jxIBnNaOqGnKafMQSiLAIUSChY+Q544o0LRZO3HW6k9eZlO5yqRUJN1p2H+QxOSG/PicKR2Trode3A/tZmtYqelF2FgOfLjEgBkkKF
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: firefly
- ceph:
    fs: xfs
- install.upgrade:
    osd.0: null
- ceph.restart:
    daemons:
    - osd.0
    - osd.1
    - osd.2
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: firefly
    clients:
      client.0:
      - cls/test_cls_rbd.sh
- ceph.restart:
    daemons:
    - mon.b
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: firefly
    clients:
      client.0:
      - rbd/test_librbd.sh
- install.upgrade:
    mon.c: null
- ceph.restart:
    daemons:
    - mon.c
    wait-for-healthy: false
    wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
  - a
  - b
  - c
- workunit:
    branch: firefly
    clients:
      client.0:
      - rbd/test_librbd_python.sh
- rgw:
    client.0:
      idle_timeout: 300
- swift:
    client.0:
      rgw_server: client.0
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
teuthology_branch: next
tube: plana
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11466

description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
  rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
duration: 3735.700578927994
failure_reason: 'Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf
  /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional
  -v -a ''!fails_on_rgw''"'
flavor: basic
owner: scheduled_teuthology@teuthology
success: false

Files

error-log.txt (1.15 MB) error-log.txt error logs in one file Loïc Dachary, 08/10/2014 01:31 AM

Related issues 5 (0 open — 5 closed)

Related to rgw - Bug #7799: Errors in upgrade:dumpling-x:stress-split-firefly---basic-plana suiteCan't reproduceIan Colle03/20/2014

Actions
Related to rgw - Bug #8016: "testPrefixAndLimit (test.functional.tests.TestContainerUTF8) ... ERROR" in upgrade:dumpling-x:stress-split-firefly-distro-basic-vps suiteResolvedYuri Weinstein04/07/2014

Actions
Related to rgw - Bug #7815: Test failed in upgrade:dumpling-x:parallel-firefly-testing-basic-plana suiteCan't reproduce03/21/2014

Actions
Related to rgw - Bug #7808: "ERROR: testContainerSerializedInfo" in upgrade:dumpling-x:stress-split-firefly---basic-planaDuplicate03/20/2014

Actions
Related to rgw - Bug #7702: osd thrashing + rgw = timeoutsResolved03/12/2014

Actions
Actions
Copy link

Also available in: Atom PDF