Project

General

Profile

Actions
Copy link

Bug #8058

closed

"LibRadosTierECPP.FlushWriteRaces" failed in upgrade:dumpling-x:parallel-firefly---basic-plana

Added by Yuri Weinstein about 10 years ago. Updated almost 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-04-09_19:33:29-upgrade:dumpling-x:parallel-firefly---basic-plana/182415/

2014-04-09T21:44:05.592 INFO:teuthology.task.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors
2014-04-09T21:44:08.767 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [       OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (21682 ms)
2014-04-09T21:44:08.767 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [ RUN      ] LibRadosTwoPoolsECPP.TryFlushReadRace
2014-04-09T21:44:14.663 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [       OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (5896 ms)
2014-04-09T21:44:17.958 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [----------] 12 tests from LibRadosTwoPoolsECPP (147497 ms total)
2014-04-09T21:44:17.958 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: 
2014-04-09T21:44:17.958 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [----------] Global test environment tear-down
2014-04-09T21:44:17.959 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [==========] 37 tests from 4 test cases ran. (679838 ms total)
2014-04-09T21:44:17.959 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [  PASSED  ] 36 tests.
2014-04-09T21:44:17.959 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [  FAILED  ] 1 test, listed below:
2014-04-09T21:44:17.959 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: [  FAILED  ] LibRadosTierECPP.FlushWriteRaces
2014-04-09T21:44:17.960 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]: 
2014-04-09T21:44:17.960 INFO:teuthology.task.workunit.client.1.out:[10.214.131.31]:  1 FAILED TEST
2014-04-09T21:44:17.961 INFO:teuthology.task.workunit:Stopping rados/test.sh on client.1...
2014-04-09T21:44:17.961 DEBUG:teuthology.orchestra.run:Running [10.214.131.31]: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-04-09T21:44:17.972 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.31 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=010dff12c38882238591bb042f8e497a1f7ba020 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'
2014-04-09T21:44:18.041 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 39, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 27, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 102, in task
    all_spec = True
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/teuthology-firefly/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/task/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.31 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=010dff12c38882238591bb042f8e497a1f7ba020 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'
2014-04-09T21:44:18.087 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/inktank/teuthology/search?q=6bf5a383c0e247689ccaa10b3f3619e1
CommandFailedError: Command failed on 10.214.131.31 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=010dff12c38882238591bb042f8e497a1f7ba020 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'
2014-04-09T21:44:18.087 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash

archive_path: /var/lib/teuthworker/archive/teuthology-2014-04-09_19:33:29-upgrade:dumpling-x:parallel-firefly---basic-plana/182415
description: upgrade/dumpling-x/parallel/{0-cluster/start.yaml 1-dumpling-install/cuttlefish-dumpling.yaml
  2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final-upgrade/client.yaml
  5-final-workload/rados_mon_thrash.yaml distros/ubuntu_14.04.yaml}
email: null
job_id: '182415'
last_in_suite: false
machine_type: plana
name: teuthology-2014-04-09_19:33:29-upgrade:dumpling-x:parallel-firefly---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '14.04'
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    sha1: 010dff12c38882238591bb042f8e497a1f7ba020
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 010dff12c38882238591bb042f8e497a1f7ba020
  s3tests:
    branch: master
  workunit:
    sha1: 010dff12c38882238591bb042f8e497a1f7ba020
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
targets:
  ubuntu@plana09.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDG4mCN35oBlJwwjbyP25p4JELF88rwP0K5+fkh0l3RhKc16Vtzm4tz+4n3cT545jZBu1/5cyH+AIfQwCjBjmo3e6+Wmu0bNqcvHUbT6huPElGAb8dQs8Zcu7G5U2V+LSEh0FK2cfmkGp76OoYlmNVdJsEEhvoh0ZmcGLC6zX1rajAZLQ0G0mBHXvS9Bv6ybyjMy7OIUvh4jz354a+5OYKSRc1ao6UdqgjKLQ6RYRMS/MZaJsvKDwlJEJgNMg9zmnR+ZLI1zA4Ilnia0LNyPsbeFtn49ZoC0hWar97GHsS2Za9UPiIVS+4UBn1qfnRTXMJXne6V6/MOW8XNW7l8zVj9
  ubuntu@plana41.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO1APNwMPhJnDeyE8W4NBl4Fdsx8dBV8IzROahOlY56SZsVVthCKUZm1cHPE6nN9L4iEZw7ibM9JpqAI/cfFoSQh4HVAwe3lfIQTO3dh7EF7vPjMNowiiPEmQcby0RNi85x33Q6m+44E+5A72ZmVdmmuOLsi7ERd+m3eAnzI4GdTLL4bJuxLMfpj8X4aGdMnopICuCOmzJGCw8+ye5pC/NdX9PBmMsg2G/Yjb54SQXELTMTgSCPOt+LoemSD1CxsPgaXVM3KMSPGQ3cRaOr3n+UpnC8bpzfvqGEIYtU40zTqLeElgkrn7lTZl35+yG3y20mcvCDiL0VITU/zToTLaD
  ubuntu@plana44.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC3D6STCdQoHZTNAdmf7wM8bnmb25JPvoZ/IPPOj3oNapI1umDOVaG4c1sdEHIipU5rSzdDSKb4nRQPhBc9oPiSMm4RkMA3PNqopWmw4A/Uazp9Oz0h1l05wfkvaLWQ/tnITGOQtO+qKsuTyjRGBwfDlGXVmsU8oLrvfE5Lhpsd/Z6Zq0LRFTMOLxZpp9i0QE5OFeVYBZEW7JfVSyO8ocPWoEK1XuKxITd9iz4bS1sUIzCYZVPDw47d1uNRBPQtVnAPlpEcO+A+UISms7L4j/AvzZTyx8nwhKukchGsP0ucPuiPSXwqSUHXG1LvPNmRhSKSJNlcbOZtSFL9kCNF05AB
tasks:
- internal.lock_machines:
  - 3
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: cuttlefish
- print: '**** done cuttlefish install'
- ceph:
    fs: xfs
- print: '**** done ceph'
- install.upgrade:
    all:
      branch: dumpling
- ceph.restart: null
- parallel:
  - workload
  - upgrade-sequence
- print: '**** done parallel'
- install.upgrade:
    client.0: null
- print: '**** done install.upgrade'
- mon_thrash:
    revive_delay: 20
    thrash_delay: 1
- workunit:
    clients:
      client.1:
      - rados/test.sh
teuthology_branch: firefly
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
      mon.b: null
  - ceph.restart:
      daemons:
      - mon.a
      wait-for-healthy: false
      wait-for-osds-up: true
  - sleep:
      duration: 60
  - ceph.restart:
      daemons:
      - mon.b
      wait-for-healthy: false
      wait-for-osds-up: true
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.16923
workload:
  sequential:
  - workunit:
      branch: dumpling
      clients:
        client.0:
        - rados/test-upgrade-firefly.sh
        - cls

description: upgrade/dumpling-x/parallel/{0-cluster/start.yaml 1-dumpling-install/cuttlefish-dumpling.yaml
  2-workload/rados_api.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final-upgrade/client.yaml
  5-final-workload/rados_mon_thrash.yaml distros/ubuntu_14.04.yaml}
duration: 2527.0889229774475
failure_reason: 'Command failed on 10.214.131.31 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp
  && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=010dff12c38882238591bb042f8e497a1f7ba020 TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="1" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  timeout 3h /home/ubuntu/cephtest/workunit.client.1/rados/test.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=6bf5a383c0e247689ccaa10b3f3619e1
success: false
Actions
Copy link
 #1

Updated by Samuel Just almost 10 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF