Actions
Bug #9805
closedError in "objectstore_tool.\\$pid.log --op list-pgs" in upgrade:firefly-x-giant-distro-basic-multi run
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):
Description
Jobs failed: ['552357', '552359', '552360', '552361', '552362', '552363', '552364', '552365', '552366', '552367', '552368', '552369', '552370', '552371', '552372']
For example one: logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-10-16_19:20:01-upgrade:firefly-x-giant-distro-basic-multi/552357/
2014-10-17T01:07:15.967 INFO:teuthology.orchestra.run.plana50.stdout:successfully deleted pool unique_pool_0 2014-10-17T01:07:15.969 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart 2014-10-17T01:07:15.969 DEBUG:teuthology.run_tasks:Unwinding manager ceph.restart 2014-10-17T01:07:15.969 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2014-10-17T01:07:15.970 INFO:tasks.thrashosds:joining thrashosds 2014-10-17T01:07:15.970 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 117, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__ self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/thrashosds.py", line 172, in task thrash_proc.do_join() File "/var/lib/teuthworker/src/ceph-qa-suite_giant/tasks/ceph_manager.py", line 275, in do_join self.thread.get() File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get raise self._exception CommandFailedError: Command failed on plana84 with status 1: 'sudo ceph_objectstore_tool --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
archive_path: /var/lib/teuthworker/archive/teuthology-2014-10-16_19:20:01-upgrade:firefly-x-giant-distro-basic-multi/552357 branch: giant description: upgrade:firefly-x/stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml 6-next-mon/monb.yaml 7-workload/radosbench.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} email: ceph-qa@ceph.com job_id: '552357' kernel: &id001 kdb: true sha1: distro last_in_suite: false machine_type: plana,burnupi,mira name: teuthology-2014-10-16_19:20:01-upgrade:firefly-x-giant-distro-basic-multi nuke-on-error: true os_type: ubuntu os_version: '12.04' overrides: admin_socket: branch: giant ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on legacy crush tunables: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 log-whitelist: - slow request - wrongly marked me down - objects unfound and apparently lost - log bound mismatch sha1: 5558afa03dbd1b20766b76e9410ef5bc3e73784f ceph-deploy: branch: dev: giant conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: 5558afa03dbd1b20766b76e9410ef5bc3e73784f s3tests: branch: giant workunit: sha1: 5558afa03dbd1b20766b76e9410ef5bc3e73784f owner: scheduled_teuthology@teuthology priority: 1000 roles: - - mon.a - mon.b - mon.c - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - - osd.7 - osd.8 - osd.9 - osd.10 - osd.11 - osd.12 - osd.13 - - client.0 suite: upgrade:firefly-x suite_branch: giant suite_path: /var/lib/teuthworker/src/ceph-qa-suite_giant targets: ubuntu@plana36.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnlCi7I5djMdXIWh2QWEUWqsIiYIodhFxMZ938OxrukJrayf6FNgyIxafKyf0WDnacoivI0sWpUwjjdzUKh8BauyY2P1dUZpxWuaXwFyMCQiYrGYK/IXsNIg6SwXc2T0WYojYTAj2M5w/e+3Xu0jV3zJ9HSFGmK8P1u/u89xIGdB7YvG/ypT2Fc7B22M1+DMDbhx2vTT3ZLtXR0zh/rUZ1lCb3mzNijmbb1Dlm1Ox0wigtTS9D0glnBNZaw3vhXiotmULCruLAW2WsmISA2UBRnUCqcaMWal1/dgIeqWBv84j1iyE2aXnh5rQHhZks8pZ36qrTqu5usrgi4zXFLMNv ubuntu@plana50.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVWbmfptOM9BWnec9CUwypAVMmQj9hiPspeKJDAm5jY8ZImNEAS5KFNz+MPJraFvVryfLjZ9dHWDUTtBZHGWiirLCH8KeIZMwZSOUFJODzGbjN6VXUNAAWkRrEdSyksxHixsN6iyJa00D0K6rN5/7OgkJN36U+wsnavWlL+LJZlSd2KOQT7I3po2jSqqwZYqJ5WWMzwDTt+4x6N9mZ2dDvRiUEX7RPAfkG8+RDXoQiekG37EqAKNtI2oEkD7q/EpEOf9eATDFhrsTfyxHkvEIXz5mcpQqfus/JvDi+lUCjL0MQLYr9cCEVDTUJznXcq1EmvbmLpVNiBqqdDYsgOWlx ubuntu@plana84.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDexpeRrYmIKkV6aq0rRfg6+NRZpsnOZhoxACSD+5KajxFukcJEY65z4rOobKcpVxacwm3Q27OndHGWbRmda0FpE5Ggo4+18lcXmYeUGL/s+g+Z970tY0OOQGhZdIafkDdd/JuoZCl+G9HyouVS4OQtbW6z7pgdoc9MjCuupp5ht9bY9msBMasDYXCUHTab5BrhdhodGGfn2zfPX2OUTsPFKnacRn9cX3f3X7wJ7eVbnmZ0IEVvegNgFUSz+uj1ZXRUpHO36ZjMBYc5kRtJPIg/V92QHsdGXE5/BkMlJH1toftXov0gU94xZMD0e+AnDlzGVYqHX/FvICna0QoMV2lB tasks: - internal.lock_machines: - 3 - plana,burnupi,mira - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.push_inventory: null - internal.serialize_remote_roles: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: branch: firefly - ceph: fs: xfs - install.upgrade: osd.0: null - ceph.restart: daemons: - osd.0 - osd.1 - osd.2 - osd.3 - osd.4 - osd.5 - osd.6 - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - ceph.restart: daemons: - mon.a wait-for-healthy: false wait-for-osds-up: true - workunit: branch: firefly clients: client.0: - cls/test_cls_rbd.sh - ceph.restart: daemons: - mon.b wait-for-healthy: false wait-for-osds-up: true - radosbench: clients: - client.0 time: 1800 - install.upgrade: mon.c: null - ceph.restart: daemons: - mon.c wait-for-healthy: false wait-for-osds-up: true - ceph.wait_for_mon_quorum: - a - b - c - workunit: branch: firefly clients: client.0: - rbd/test_librbd_python.sh - rgw: client.0: null default_idle_timeout: 300 - swift: client.0: rgw_server: client.0 - rados: clients: - client.0 objects: 500 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master tube: multi verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.multi.3168
description: upgrade:firefly-x/stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml 6-next-mon/monb.yaml 7-workload/radosbench.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml rgw-swift.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} duration: 4893.873789072037 failure_reason: 'Command failed on plana84 with status 1: ''sudo ceph_objectstore_tool --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs''' flavor: basic owner: scheduled_teuthology@teuthology success: false
Updated by Sage Weil over 9 years ago
ceph_objectstore_tool: false in the thrasher section of the yaml
Updated by Sage Weil over 9 years ago
- Assignee set to Yuri Weinstein
- Priority changed from Normal to Urgent
Updated by Yuri Weinstein over 9 years ago
Adding ceph_objectstore_tool: false
to thrashosds
.
- thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 ceph_objectstore_tool: false
to stress-split/3-thrash
and in stress-split-erasure-code/3-thrash
Updated by Yuri Weinstein over 9 years ago
- Status changed from New to In Progress
Updated by David Zafman over 9 years ago
- Status changed from In Progress to Resolved
a295c18a808033dfbdacf8801c26fed137ccab35
Ignore the ceph_objectstore_test if the binary is present.
Actions