Project

General

Profile

Bug #10002

Errors during import_export test in upgrade:firefly-x-next-distro-basic-vps run

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Two jobs failed ['584634', '584648']

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-03_17:18:01-upgrade:firefly-x-next-distro-basic-vps/584648/

2014-11-03T17:47:14.470 INFO:tasks.workunit.client.0.vpm072.stderr:+ '[' 2048 = 3968 ']'
2014-11-03T17:47:14.485 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0...
2014-11-03T17:47:14.486 INFO:teuthology.orchestra.run.vpm072:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2014-11-03T17:47:15.002 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 364, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 105, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=firefly TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'
2014-11-03T17:47:15.003 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/workunit.py", line 105, in task
    config.get('env'), timeout=timeout)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 364, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 105, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=firefly TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'

Associated revisions

Revision e94d3c11 (diff)
Added by Josh Durgin over 9 years ago

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <>

Revision 0cb32c15 (diff)
Added by Josh Durgin over 9 years ago

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <>
(cherry picked from commit e94d3c11edb9c9cbcf108463fdff8404df79be33)

Revision 0d8ad6ad (diff)
Added by Josh Durgin over 9 years ago

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <>
(cherry picked from commit e94d3c11edb9c9cbcf108463fdff8404df79be33)

Revision e17d579f (diff)
Added by Josh Durgin over 9 years ago

qa: allow small allocation diffs for exported rbds

The local filesytem may behave slightly differently. This isn't
foolproof, but seems to be reliable enough on rhel7 rootfs, where
exact comparison was failing.

Fixes: #10002
Signed-off-by: Josh Durgin <>
(cherry picked from commit e94d3c11edb9c9cbcf108463fdff8404df79be33)

History

#2 Updated by Yuri Weinstein over 9 years ago

  • Priority changed from Normal to High

suite:upgrade:firefly:singleton
Run http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-04_18:41:21-upgrade:firefly:singleton-firefly-distro-basic-vps

Job http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-04_18:41:21-upgrade:firefly:singleton-firefly-distro-basic-vps/586511/teuthology.log

Looks like the same problem with Segmentation fault:

2014-11-04T20:04:09.594 INFO:tasks.workunit.client.0.vpm200.stderr:Segmentation fault (core dumped)
2014-11-04T20:04:09.595 INFO:tasks.workunit:Stopping rbd/import_export.sh on client.0...
2014-11-04T20:04:09.596 INFO:teuthology.orchestra.run.vpm200:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2014-11-04T20:04:09.983 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_firefly/tasks/workunit.py", line 359, in _run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 128, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 364, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 105, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm200 with status 139: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1a9d000bb679a7392b9dd115373c3827c9626694 TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'

#3 Updated by Yuri Weinstein over 9 years ago

  • Source changed from other to Q/A

suite:upgrade:dumpling-firefly-x
Run http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-04_17:25:01-upgrade:dumpling-firefly-x:stress-split-next-distro-basic-vps/

Same issue in job http://qa-proxy.ceph.com/teuthology/teuthology-2014-11-04_17:25:01-upgrade:dumpling-firefly-x:stress-split-next-distro-basic-vps/586497/teuthology.log

2014-11-04T20:15:11.149 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0...
2014-11-04T20:15:11.149 INFO:teuthology.orchestra.run.vpm081:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'

#6 Updated by Sage Weil over 9 years ago

  • Project changed from Ceph to rbd
  • Priority changed from High to Urgent

#8 Updated by Josh Durgin over 9 years ago

  • Status changed from New to Fix Under Review

#9 Updated by Josh Durgin over 9 years ago

  • Status changed from Fix Under Review to Resolved

commit:e94d3c11edb9c9cbcf108463fdff8404df79be33

Also available in: Atom PDF