Actions
Subtask #9609
closedBug #9511: Stress-split giant upgrade workunits.sh
"Exception in parallel execution" in upgrade:dumpling-giant-x:stress-split-giant-distro-basic-vps run
Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Reviewed:
Affected Versions:
Description
This is in stress-split tests.
014-09-26T17:55:21.286 INFO:tasks.workunit:Stopping ['rbd/import_export.sh'] on client.0...
2014-09-26T17:55:21.286 INFO:teuthology.orchestra.run.vpm072:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.0'
2014-09-26T17:55:21.448 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
for result in self:
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
resurrect_traceback(result)
File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
return func(*args, **kwargs)
File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/workunit.py", line 359, in _run_tests
args=args,
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 127, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 361, in run
r.wait()
File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 105, in wait
exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on vpm072 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=giant TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'
archive_path: /var/lib/teuthworker/archive/teuthology-2014-09-26_17:29:38-upgrade:dumpling-giant-x:stress-split-giant-distro-basic-vps/513909
branch: giant
description: upgrade:dumpling-giant-x:stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/{rados_api_tests.yaml
rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 6-next-mon/monb.yaml
7-workload/{rados_api_tests.yaml radosbench.yaml rbd_api.yaml} 8-next-mon/monc.yaml
9-workload/{rados_api_tests.yaml rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml}
distros/ubuntu_12.04.yaml}
email: ceph-qa@ceph.com
job_id: '513909'
kernel: &id001
kdb: true
sha1: distro
last_in_suite: false
machine_type: vps
name: teuthology-2014-09-26_17:29:38-upgrade:dumpling-giant-x:stress-split-giant-distro-basic-vps
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
admin_socket:
branch: giant
ceph:
conf:
mon:
debug mon: 20
debug ms: 1
debug paxos: 20
mon warn on legacy crush tunables: false
osd:
debug filestore: 20
debug journal: 20
debug ms: 1
debug osd: 20
log-whitelist:
- slow request
- wrongly marked me down
- objects unfound and apparently lost
- log bound mismatch
sha1: 5c2984e6e155026699c2ed1df8b08b7a33f56e9b
ceph-deploy:
branch:
dev: giant
conf:
client:
log file: /var/log/ceph/ceph-$name.$pid.log
mon:
debug mon: 1
debug ms: 20
debug paxos: 20
osd default pool size: 2
install:
ceph:
sha1: 5c2984e6e155026699c2ed1df8b08b7a33f56e9b
s3tests:
branch: giant
workunit:
sha1: 5c2984e6e155026699c2ed1df8b08b7a33f56e9b
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
- mon.b
- mds.a
- osd.0
- osd.1
- osd.2
- mon.c
- - osd.3
- osd.4
- osd.5
- - client.0
suite: upgrade:dumpling-giant-x:stress-split
suite_branch: master
suite_path: /var/lib/teuthworker/src/ceph-qa-suite_master
targets:
ubuntu@vpm072.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNnkpiHXT/tqHDGdeZNSRrIyyFf+ndYI67QQCTKahp2IwQYAW+DKaCKNFVIRBhB5jS0+chOFCtfDwTRKEUJ+x273a6JBvI+i5qf7ASmmZYGgNoSm5eyIdi6MOGokoBdSli24Ck782XSF8YS2zqv90I1Cp2ZfMTmtuzL2EzusDCL3c4G05sDO1cEda/LlfaZ9bzPL4U86rV0YsT8d595+SfVVtuRCu8mgSoWd0HQ9dD7QAKOJgnd+zIby/27/4ptrldywtZwmI4YDMjS7NjylHWUh854b6koGG0ucIgdIt1/gMLovc7ktQmrnS7a/0WmH6k0LNzMT0NK158AeOkAuWX
ubuntu@vpm139.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkNYE5XBtEjEiIv0B4uASvHhz1GrCFgUFdoi9GGc40Kwii9ugEcfLucfjsIaknemvY5CUnos13ioUbFcmxdw9uYDrLHJm5XwF0JmaZC4iY0zQe6KjJ0Mrpi30+69QtJwXE2JP/EOc+BbeHlqk1OxxHmaVlkvIceiuEG80QVyHacvKxST40RkojCPwTt07rfHYCEsffLcpCdkx3oeQUGcPLljJHwOYE5aoHzsLmpwzLGDJgrJLK9SJnPBp9/1c1WA9KYnpkJeP7iAryQa+oUXd8woBwzWVQCHasWVqiiPUX655SwZSHDskQxd3nZXyyfOxNm+Inua4ZbbdkxR+9gvDp
ubuntu@vpm195.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEXY8GcK5oABAr3A8HJ6bRwum9VxBNR17EqfSVZiY2nyX5t+OYPqS+iz1d6hdJOq2Y6QDzNmjLE1c+BUmyLcswuq689gsMPeam6i4RcpV80YtNBAA5VCgl2khYCvuUbmlUHIkt/MsSqI9QiLgj+32QCyMMsVF/m/9/ZS3ZDxUseMojr15tPm2St3WWw/+wbEaLmWXtDLgYA4m/2M71b5+rgZLSrMiUrldzjkjANZpyLkWWfI/qoslhcWt7IQhI5LzB0fsTPZBxIMyFkRZTcl5LzyCWrgj2Lcm3C2sSZsXiue1XhgHu7dnIlph4P/hoHK1uveRobeXfQqqeYy5uH7WT
tasks:
- internal.lock_machines:
- 3
- vps
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.push_inventory: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
branch: dumpling
- ceph:
fs: xfs
- install.upgrade:
osd.0: null
- ceph.restart:
daemons:
- osd.0
- osd.1
- osd.2
- thrashosds:
chance_pgnum_grow: 1
chance_pgpnum_fix: 1
thrash_primary_affinity: false
timeout: 1200
- ceph.restart:
daemons:
- mon.a
wait-for-healthy: false
wait-for-osds-up: true
- workunit:
branch: dumpling
clients:
client.0:
- rados/test-upgrade-firefly.sh
- workunit:
branch: giant
clients:
client.0:
- cls/test_cls_rbd.sh
- workunit:
branch: giant
clients:
client.0:
- rbd/import_export.sh
env:
RBD_CREATE_ARGS: --new-format
- rados:
clients:
- client.0
objects: 500
op_weights:
delete: 10
read: 45
write: 45
ops: 4000
- rados:
clients:
- client.0
objects: 50
op_weights:
delete: 50
read: 100
rollback: 50
snap_create: 50
snap_remove: 50
write: 100
ops: 4000
- ceph.restart:
daemons:
- mon.b
wait-for-healthy: false
wait-for-osds-up: true
- workunit:
branch: dumpling
clients:
client.0:
- rados/test-upgrade-firefly.sh
- radosbench:
clients:
- client.0
time: 1800
- workunit:
branch: giant
clients:
client.0:
- rbd/test_librbd.sh
- install.upgrade:
mon.c: null
- ceph.restart:
daemons:
- mon.c
wait-for-healthy: false
wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
- a
- b
- c
- workunit:
branch: dumpling
clients:
client.0:
- rados/test-upgrade-firefly.sh
- workunit:
branch: giant
clients:
client.0:
- rbd/test_librbd_python.sh
- rgw:
client.0: null
default_idle_timeout: 300
- swift:
client.0:
rgw_server: client.0
- rados:
clients:
- client.0
objects: 500
op_weights:
delete: 50
read: 100
rollback: 50
snap_create: 50
snap_remove: 50
write: 100
ops: 4000
teuthology_branch: master
tube: vps
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.vps.3045
description: upgrade:dumpling-giant-x:stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/{rados_api_tests.yaml
rbd-cls.yaml rbd-import-export.yaml readwrite.yaml snaps-few-objects.yaml} 6-next-mon/monb.yaml
7-workload/{rados_api_tests.yaml radosbench.yaml rbd_api.yaml} 8-next-mon/monc.yaml
9-workload/{rados_api_tests.yaml rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml}
distros/ubuntu_12.04.yaml}
duration: 1011.2013618946075
failure_reason: 'Command failed on vpm072 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp
&& cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
CEPH_REF=giant TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format
adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
success: false
Updated by Tamilarasi muthamizhan over 9 years ago
- Assignee set to Josh Durgin
- Priority changed from Normal to Urgent
hey Josh, can you please take a look at this one?
Updated by Anonymous over 9 years ago
- Assignee changed from Josh Durgin to Anonymous
I think that the workunits versions are messed up.
I'll check this
Updated by Anonymous over 9 years ago
- Status changed from New to In Progress
I have a made a change for this -- we'll see what happens overnight.
Updated by Anonymous over 9 years ago
mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c2984e6e155026699c2ed1df8b08b7a33f56e9b TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh
Commands like the above were failing because of failed to run command ‘/home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh’: No such file or directory
I believe the ceph-deploy giant: clause is incorrect. I am currently testing this, but I probably will not check it in until tomorrow (I believe the overnight suites have started already).
The test is running right now. All of the above statements assume that I am correct.
Updated by Anonymous over 9 years ago
mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c2984e6e155026699c2ed1df8b08b7a33f56e9b TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh
Commands like the above were failing because of:
failed to run command ‘/home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh’: No such file or directory
I believe the ceph-deploy giant: clause is incorrect. I am currently testing this, but I probably will not check it in until tomorrow (I believe the overnight suites have started already).
The test is running right now. All of the above statements assume that I am correct.
Updated by Anonymous over 9 years ago
mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=5c2984e6e155026699c2ed1df8b08b7a33f56e9b TESTDIR="/home/ubuntu/cephtest" CEPH_ID="0" RBD_CREATE_ARGS=--new-format adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh
Commands like the above were failing because of:
failed to run command ‘/home/ubuntu/cephtest/workunit.client.0/rbd/import_export.sh’: No such file or directory
I believe the ceph-deploy giant: clause is incorrect. I am currently testing this, but I probably will not check it in until tomorrow (I believe the overnight suites have started already).
The test is running right now. All of the above statements assume that I am correct.
Updated by Anonymous over 9 years ago
- Status changed from In Progress to Rejected
- Priority changed from Urgent to Normal
- Translation missing: en.field_remaining_hours set to 0.0
Upgrade tests from dumpling-giant-x are not going to be used. dumpling-firefly and firefly-giant-x will be.
Actions