Project

General

Profile

Actions
Copy link

Bug #12336

closed

adjust-ulimits ceph-coverage command fails in two different ways

Added by Nathan Cutler almost 9 years ago. Updated almost 9 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  • http://pulpito.ceph.com/loic-2015-07-09_16:57:58-rbd-firefly-backports---basic-multi
    2015-07-14T02:59:32.940 INFO:teuthology.orchestra.run.mira077.stdout: * Starting NFS kernel daemon
2015-07-14T02:59:32.992 INFO:teuthology.orchestra.run.mira077.stdout:   ...done.
2015-07-14T02:59:32.995 INFO:tasks.qemu:starting qemu...
2015-07-14T02:59:32.995 INFO:teuthology.orchestra.run.mira077:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term qemu-system-x86_64 -enable-kvm -nographic -m 4096 -drive file=/home/ubuntu/cephtest/qemu/base.client.0.qcow2,format=qcow2,if=virtio -cdrom /home/ubuntu/cephtest/qemu/client.0.iso -drive file=rbd:rbd/client.0.0:id=0,format=raw,if=virtio,cache=none'
2015-07-14T02:59:32.997 DEBUG:teuthology.run_tasks:Unwinding manager qemu
2015-07-14T02:59:32.998 INFO:tasks.qemu:waiting for qemu tests to finish...
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:Traceback (most recent call last):
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:  File "/usr/bin/daemon-helper", line 66, in <module>
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:    preexec_fn=os.setsid,
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:    errread, errwrite)
2015-07-14T02:59:33.096 INFO:tasks.qemu.client.0.mira077.stderr:  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
2015-07-14T02:59:33.097 INFO:tasks.qemu.client.0.mira077.stderr:    raise child_exception
2015-07-14T02:59:33.097 INFO:tasks.qemu.client.0.mira077.stderr:OSError: [Errno 2] No such file or directory
  • http://pulpito.ceph.com/loic-2015-07-09_21:20:15-upgrade:firefly-x-firefly-backports-distro-basic-multi/967470/
    2015-07-14T12:55:00.560 INFO:teuthology.orchestra.run.burnupi19:Running: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool ecbase'
2015-07-14T12:55:00.584 INFO:tasks.rados.rados.0.burnupi19.stdout:adding op weight read -> 100
2015-07-14T12:55:00.584 INFO:tasks.rados.rados.0.burnupi19.stdout:adding op weight write -> 50
2015-07-14T12:55:00.584 INFO:tasks.rados.rados.0.burnupi19.stdout:adding op weight copy_from -> 50
2015-07-14T12:55:00.584 INFO:tasks.rados.rados.0.burnupi19.stderr:unknown op write_excl
2015-07-14T12:55:00.586 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 41, in run_one_task
    return fn(**kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/sequential.py", line 55, in task
    mgr.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/rados.py", line 246, in task
    running.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 331, in get
    raise self._exception
CommandFailedError: Command failed on burnupi19 with status 1: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool ecbase'
2015-07-14T12:55:00.673 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/sepia/teuthology/search?q=40e29050ba53440c8893d9976ee7783e
CommandFailedError: Command failed on burnupi19 with status 1: 'CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 500 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op copy_from 50 --op write_excl 50 --op delete 50 --pool ecbase'

Files

teuthology.log (785 KB) teuthology.log Nathan Cutler, 07/15/2015 01:54 PM
Actions
Copy link
 #2

Updated by Nathan Cutler almost 9 years ago

This happened while running teuthology suites on the 0.80.11 firefly integration branch. For more details see the attached teuthology log file and the 0.80.11 stable releases tracker issue: http://tracker.ceph.com/issues/11644

Actions
Copy link
 #3

Updated by Nathan Cutler almost 9 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Nathan Cutler almost 9 years ago

  • Subject changed from os.setsid fails when teuthology test rbd/librbd attempts to execute "adjust-ulimits ceph-coverage ..." command to adjust-ulimits ceph-coverage command fails in two different ways
Actions
Copy link
 #5

Updated by Loïc Dachary almost 9 years ago

Could you re-run just these two to confirm it's not a temporary failure ?

Actions
Copy link
 #6

Updated by Loïc Dachary almost 9 years ago

It maybe be the case that -k testing needs to be included in the teuthology-suite command because the test relies on features that are specific to this kernel and not available in the distro kernel. This is just a hunch based on a vague recollection though.

Actions
Copy link
 #7

Updated by Nathan Cutler almost 9 years ago

re-running all the failed tests: http://tracker.ceph.com/issues/11644

Actions
Copy link
 #8

Updated by Nathan Cutler almost 9 years ago

  • Status changed from New to Rejected
This was caused by running an inappropriate teuthology command:
  • (10:46:41) loicd: smithfarm: we want something like ./virtualenv/bin/teuthology-suite -v -c hammer -k distro -m plana,burnupi,mira -s upgrade/firefly-backport-x --filter ubuntu_14.04 --email
  • (10:46:52) loicd: except firefly-backport-x does not exist
Actions
Copy link

Also available in: Atom PDF