Project

General

Profile

Actions
Copy link

Bug #8988

closed

AssertionError(s) in upgrade:firefly-x:stress-split-next---basic-plana

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Loïc Dachary
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Related issue

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755/
and
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387763/

2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount)
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:    cls.env.setUp()
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:    raise ResponseError(cls.conn.response)
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:ResponseError: 500: Internal Server Error
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testContainerListing (test.functional.tests.TestAccount)
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 262, in testContainerListing
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:    self.assertEquals(a, b)
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: Lists differ: ['167e6c2f7790448cb665118c0ca1... != ['167e6c2f7790448cb665118c0ca1...
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:First differing element 4:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:72c48b7e4f774b6ebae34c9e9008b690
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:525bc170f49f4e408b74dd2ca8f10b28
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:Second list contains 1 additional elements.
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:First extra element 10:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:e53eab2df63e444a8c2248f08eefbac3
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:  ['167e6c2f7790448cb665118c0ca14064',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '473c9ce271f24bb69255ea3a60f41e26',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '4ad0858c06204b71968a45d33782e90d',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:   '4de3250a2b3847bfb65743044ece7718',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:+  '525bc170f49f4e408b74dd2ca8f10b28',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   '72c48b7e4f774b6ebae34c9e9008b690',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   '8f49eb55dbae4f90a360e66260a29e05',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'b63f263822cb4b4d900af8a122573259',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'd1e9f200a3b8493f8f5c292dcbfb241e',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'dcf96a03c43049ffa799ea9fceffc27d',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:   'e53eab2df63e444a8c2248f08eefbac3']
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testLastContainerMarker (test.functional.tests.TestAccount)
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 272, in testLastContainerMarker
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:    self.assertEquals(len(containers), len(self.env.containers))
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: 11 != 10
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:Ran 137 tests in 874.291s
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:FAILED (errors=1, failures=2)
2014-07-30T23:59:17.951 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:17.952 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data'
2014-07-30T23:59:20.927 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data'
2014-07-30T23:59:21.377 INFO:teuthology.task.swift:Removing swift...
2014-07-30T23:59:21.377 INFO:teuthology.orchestra.run.plana47:Running: 'rm -rf /home/ubuntu/cephtest/swift'
2014-07-30T23:59:21.456 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:21.457 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2014-07-30T23:59:21.457 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 29, in nested
    yield vars
  File "/home/teuthworker/src/teuthology_next/teuthology/task/rgw.py", line 808, in task
    yield
  File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
2014-07-30T23:59:21.458 INFO:teuthology.task.rgw:Stopping apache...

archive_path: /var/lib/teuthworker/archive/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755
branch: next
description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
  rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
email: ceph-qa@ceph.com
job_id: '387755'
last_in_suite: false
machine_type: plana
name: teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - wrongly marked me down
    - objects unfound and apparently lost
    - log bound mismatch
    sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
  s3tests:
    branch: next
  workunit:
    sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
  - mon.b
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - osd.3
  - osd.4
  - osd.5
  - mon.c
- - client.0
suite: upgrade:firefly-x:stress-split
suite_branch: master
targets:
  ubuntu@plana47.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIDs+Sl9mfdFX0VZy41zfHkMJlD1OtErcQsSPLq3wrQ5HPPflg8IRtS45LHsYBAyBcAl4fmeG1q45jOrHj6yibJhT/HzxO6TOikYKUgr3uWy804NsTAP/A2SJ4wA8Y+1WyiAjBs0EAnISCnIl5Arj17NYPBroDeclKqZEJpL5UbPCmMibqEmjIPNVNulukox/Dh9GJZWlxqD0aKfTRnIOM+6RWHv2h9mCYVtU2ZbMx+QR0sxji5IJzg4ITlGaR9g+U3DnOU95pkxb3x6mE+H8udB58FONty/DHaAC/rXGfjkg056R3iq8FJUpF3HKKjILLRHjLb3ueK3tN1zrEueCt
  ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFHCeGWMPGOLyScKFkduv7aJL9bpMUPZQATO9lxpWu1NtzYndPJtWcyUxgWlItu75SJwpXx/l2GhPYcDKrR1Nl37+dbgs5TeDTbr9YdQBuLPbkbIZMQqO4GqUjurEwLU3vFUZ0X7PTlUqn6qwpT+I2YJua19eF2cRQFIGYVZMzaezm47uh67cdKFh0RTA1pSJ2qM/WMn91boRWcsRQrmn4BeOzfpGfSPDRjrHXHiPx3Br4zcOi/3lOxNFcEeoBrA47PMxvxVIlbmxKDfNjHpQQT18VFWb+qcTAzf+zdBy3iDRFFS45fPrqlWjGn9sK74EbRQanDrZlrFkg2a/HIe5T
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Dgv5XfnXOHEuwwOoK+xQtHEZusRHTm9I4oBJ41RYkPq2y5GA5OrXiWVlOrwBoXXdtCeW4ynErDhqiFuL3tYmwNEYzRWqnyqZte4qfsTz93Lhv7UEkageJ2iHNaUNt+H071A8JULR2CRtIxXu6zSSKC8vwmEirxqYj3pPRVm9TCa1iPaj8R3wPmeBjwVD9IU+zAuvIi6oWcqKrxZEdEOciMa72nGO58V7Wo0yICMST6day1jxIBnNaOqGnKafMQSiLAIUSChY+Q544o0LRZO3HW6k9eZlO5yqRUJN1p2H+QxOSG/PicKR2Trode3A/tZmtYqelF2FgOfLjEgBkkKF
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: firefly
- ceph:
    fs: xfs
- install.upgrade:
    osd.0: null
- ceph.restart:
    daemons:
    - osd.0
    - osd.1
    - osd.2
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: firefly
    clients:
      client.0:
      - cls/test_cls_rbd.sh
- ceph.restart:
    daemons:
    - mon.b
    wait-for-healthy: false
    wait-for-osds-up: true
- workunit:
    branch: firefly
    clients:
      client.0:
      - rbd/test_librbd.sh
- install.upgrade:
    mon.c: null
- ceph.restart:
    daemons:
    - mon.c
    wait-for-healthy: false
    wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
  - a
  - b
  - c
- workunit:
    branch: firefly
    clients:
      client.0:
      - rbd/test_librbd_python.sh
- rgw:
    client.0:
      idle_timeout: 300
- swift:
    client.0:
      rgw_server: client.0
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 50
      read: 100
      rollback: 50
      snap_create: 50
      snap_remove: 50
      write: 100
    ops: 4000
teuthology_branch: next
tube: plana
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11466

description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
  6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
  rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
duration: 3735.700578927994
failure_reason: 'Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf
  /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional
  -v -a ''!fails_on_rgw''"'
flavor: basic
owner: scheduled_teuthology@teuthology
success: false

Files

error-log.txt (1.15 MB) error-log.txt error logs in one file Loïc Dachary, 08/10/2014 01:31 AM

Related issues 5 (0 open — 5 closed)

Related to rgw - Bug #7799: Errors in upgrade:dumpling-x:stress-split-firefly---basic-plana suiteCan't reproduceIan Colle03/20/2014

Actions
Related to rgw - Bug #8016: "testPrefixAndLimit (test.functional.tests.TestContainerUTF8) ... ERROR" in upgrade:dumpling-x:stress-split-firefly-distro-basic-vps suiteResolvedYuri Weinstein04/07/2014

Actions
Related to rgw - Bug #7815: Test failed in upgrade:dumpling-x:parallel-firefly-testing-basic-plana suiteCan't reproduce03/21/2014

Actions
Related to rgw - Bug #7808: "ERROR: testContainerSerializedInfo" in upgrade:dumpling-x:stress-split-firefly---basic-planaDuplicate03/20/2014

Actions
Related to rgw - Bug #7702: osd thrashing + rgw = timeoutsResolved03/12/2014

Actions
Actions
Copy link
 #1

Updated by Yuri Weinstein over 9 years ago

  • Project changed from teuthology to Ceph
Actions
Copy link
 #3

Updated by Loïc Dachary over 9 years ago

The errors for each failure are different and suggests the tests are failing for an independent reason such as the cluster going down before the test complete or something similar.

2014-08-08T11:57:30.497 INFO:teuthology.orchestra.run.vpm146.stderr:======================================================================
2014-08-08T11:57:30.497 INFO:teuthology.orchestra.run.vpm146.stderr:FAIL: testCreate (test.functional.tests.TestContainerUTF8)
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:----------------------------------------------------------------------
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:Traceback (most recent call last):
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 447, in testCreate
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:    self.assert_(cont.create())
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:AssertionError: False is not true
2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:
2014-08-08T11:57:30.499 INFO:teuthology.orchestra.run.vpm146.stderr:----------------------------------------------------------------------
2014-08-08T11:57:30.499 INFO:teuthology.orchestra.run.vpm146.stderr:Ran 137 tests in 2077.215s

2014-08-08T11:38:10.693 INFO:teuthology.orchestra.run.vpm137.stderr:======================================================================
2014-08-08T11:38:10.693 INFO:teuthology.orchestra.run.vpm137.stderr:ERROR: testStructure (test.functional.tests.TestContainerPaths)
2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:----------------------------------------------------------------------
2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:Traceback (most recent call last):
2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:    cls.env.setUp()
2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 687, in setUp
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:    'application/directory'})
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 739, in write_random
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:    if not self.write(data, hdrs=hdrs, parms=parms, cfg=cfg):
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 731, in write
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:    raise ResponseError(self.conn.response)
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:ResponseError: 500: Internal Server Error
2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:
2014-08-08T11:38:10.989 INFO:teuthology.orchestra.run.vpm137.stderr:----------------------------------------------------------------------
2014-08-08T11:38:10.990 INFO:teuthology.orchestra.run.vpm137.stderr:Ran 137 tests in 3036.831s

2014-08-08T12:42:51.996 INFO:teuthology.orchestra.run.vpm134.stderr:======================================================================
2014-08-08T12:42:51.996 INFO:teuthology.orchestra.run.vpm134.stderr:ERROR: testBadHeaders (test.functional.tests.TestFileUTF8)
2014-08-08T12:42:51.997 INFO:teuthology.orchestra.run.vpm134.stderr:----------------------------------------------------------------------
2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:in_osds:  [4, 0, 3, 1, 2]  out_osds:  [5] dead_osds:  [] live_osds:  [0, 3, 2, 1, 4, 5]
2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [4, 0, 3, 1, 2]
2014-08-08T12:42:52.843 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2'
2014-08-08T12:42:53.228 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.2.
2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr:Traceback (most recent call last):
2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 122, in setUp
2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr:    super(Base2, self).setUp()
2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:    cls.env.setUp()
2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 796, in setUp
2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:    raise ResponseError(cls.conn.response)
2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:ResponseError: 500: Internal Server Error
2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:
2014-08-08T12:42:53.624 INFO:teuthology.orchestra.run.vpm134.stderr:----------------------------------------------------------------------
2014-08-08T12:42:54.731 INFO:teuthology.orchestra.run.vpm134.stderr:Ran 137 tests in 4019.450s
2014-08-08T12:42:54.731 INFO:teuthology.orchestra.run.vpm134.stderr:
2014-08-08T12:42:55.106 INFO:teuthology.orchestra.run.vpm134.stderr:FAILED (errors=1)

2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:in_osds:  [4, 2, 1, 0, 3, 5]  out_osds:  [] dead_osds:  [1] live_osds:  [0, 4, 2, 5, 3]
2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [4, 2, 1, 0, 3, 5]
2014-08-08T11:01:54.970 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0'
2014-08-08T11:01:55.002 INFO:teuthology.orchestra.run.vpm167.stderr:testZeroByteFile (test.functional.tests.TestFileUTF8) ... ok
2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr:
2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr:======================================================================
2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr:ERROR: testContainerListing (test.functional.tests.TestContainerPaths)
2014-08-08T11:01:55.005 INFO:teuthology.orchestra.run.vpm167.stderr:----------------------------------------------------------------------
2014-08-08T11:01:55.005 INFO:teuthology.orchestra.run.vpm167.stderr:Traceback (most recent call last):
2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:    cls.env.setUp()
2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 637, in setUp
2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:    raise ResponseError(cls.conn.response)
2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:ResponseError: 500: Internal Server Error
2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:
2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:----------------------------------------------------------------------
2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:Ran 137 tests in 1508.034s
2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:
2014-08-08T11:01:55.008 INFO:teuthology.orchestra.run.vpm167.stderr:FAILED (errors=1)
Actions
Copy link
 #4

Updated by Loïc Dachary over 9 years ago

In the attached file, each part separated with ----------------------------- is the output between the last successfull rgw test and the test that fail. Maybe there is a common pattern that would explain why it fails at this point ? error-log.txt

Actions
Copy link
 #5

Updated by Loïc Dachary over 9 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #6

Updated by Loïc Dachary over 9 years ago

In a few tickets it is suggested that this may be an idle timeout problem. I rescheduled a suite after changing all timeouts from 300 to 1200 to check if that shows improvement.

diff --git a/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml b/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
index e1c223c..b94dc3e 100644
--- a/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
+++ b/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
@@ -36,14 +36,14 @@ tasks:
         master zone: r1z1
         zones: [r1z1]
     client.0:
-      idle_timeout: 300
+      idle_timeout: 1200
       valgrind: [--tool=memcheck]
       system user:
         name: client0-system-user
         access key: 1te6NH5mcdcq0Tc5i8i2
         secret key: 1y4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
     client.1:
-      idle_timeout: 300
+      idle_timeout: 1200
       valgrind: [--tool=memcheck]
       system user:
         name: client1-system-user
diff --git a/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml b/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
index c324c03..c2a5eda 100644
--- a/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
+++ b/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
@@ -1,7 +1,7 @@
 tasks:
 - rgw:
     client.0:
-      idle_timeout: 300
+      idle_timeout: 1200
 - swift:
     client.0:
       rgw_server: client.0
diff --git a/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml b/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
index c324c03..c2a5eda 100644
--- a/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
+++ b/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
@@ -1,7 +1,7 @@
 tasks:
 - rgw:
     client.0:
-      idle_timeout: 300
+      idle_timeout: 1200
 - swift:
     client.0:
       rgw_server: client.0
diff --git a/tasks/rgw.py b/tasks/rgw.py
index 5cf72dc..d0ab425 100644
--- a/tasks/rgw.py
+++ b/tasks/rgw.py
@@ -86,7 +86,7 @@ def ship_apache_configs(ctx, config, role_endpoints):
         system_type = teuthology.get_system_type(remote)
         if not conf:
             conf = {}
-        idle_timeout = conf.get('idle_timeout', 30)
+        idle_timeout = conf.get('idle_timeout', 1200)
         if system_type == 'deb':
             mod_path = '/usr/lib/apache2/modules'
             print_continue = 'on'

Actions
Copy link
 #7

Updated by Loïc Dachary over 9 years ago

  • Status changed from New to Fix Under Review

Two consecutive run with the increased timeout do not show the bug (one and two)

proposed patch

Actions
Copy link
 #8

Updated by Sage Weil over 9 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #9

Updated by Sage Weil over 9 years ago

  • Status changed from Fix Under Review to In Progress
Actions
Copy link
 #10

Updated by Loïc Dachary over 9 years ago

To confirm there is a large delay requiring a large idle_timeout:

ubuntu@teuthology:/a/teuthology-2014-08-13_09:42:01-upgrade:firefly-x:stress-split-next---basic-vps/422009$ zgrep 'slow req' remote/*/log/ceph-mon* | sed -ne  's/.* > //p' | sort -n | tail -1
904.913166 secs

Actions
Copy link
 #13

Updated by Loïc Dachary over 9 years ago

Waiting for Shipping apache config with

ubuntu@teuthology:/a/loic-2014-08-14_01:02:11-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps$ grep 'Shipping apache config' */teuthology.log

before inspecting the actual configuration on a running machine.

Actions
Copy link
 #14

Updated by Loïc Dachary over 9 years ago

  • Description updated (diff)
Actions
Copy link
 #15

Updated by Yuri Weinstein over 9 years ago

Loic, we had disk failed and possibly due to that suite failed (guessing), I re-started it http://pulpito.front.sepia.ceph.com/teuthology-2014-08-13_18:26:31-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping---basic-vps/

Actions
Copy link
 #17

Updated by Loïc Dachary over 9 years ago

The reason why the suite fails with

Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/rgw.py", line 508, in create_nonregion_pools
    (remote,) = ctx.cluster.only(client).remotes.iterkeys()

is because vps.yaml creates entries in the rgw config for clients that are not defined in the role section of the file. The method loops over them and fails when it tries to acquire them

Actions
Copy link
 #18

Updated by Loïc Dachary over 9 years ago

  • Status changed from In Progress to Fix Under Review
  • Assignee set to Loïc Dachary
Actions
Copy link
 #19

Updated by Loïc Dachary over 9 years ago

the suite runs ok except for one error related to a MDSMap encoding bug that has been fixed in master today.

Actions
Copy link
 #20

Updated by Sage Weil over 9 years ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link

Also available in: Atom PDF