Bug #9549
closedError running "ceph osd erasure-code-profile" command on upgrade:dumpling-firefly-x-giant-distro-basic-multi run
100%
Description
All tests failed in http://pulpito.front.sepia.ceph.com/teuthology-2014-09-19_19:20:01-upgrade:dumpling-firefly-x-giant-distro-basic-multi/
One log for example in http://qa-proxy.ceph.com/teuthology/teuthology-2014-09-19_19:20:01-upgrade:dumpling-firefly-x-giant-distro-basic-multi/499091/teuthology.log
2014-09-20T01:15:09.373 INFO:teuthology.orchestra.run.burnupi41:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd erasure-code-profile set teuthologyprofile k=2 m=1 ruleset-failure-domain=osd' 2014-09-20T01:15:09.529 INFO:teuthology.orchestra.run.burnupi41.stderr:no valid command found; 10 closest matches: 2014-09-20T01:15:09.529 INFO:teuthology.orchestra.run.burnupi41.stderr:osd dump {<int[0-]>} 2014-09-20T01:15:09.529 INFO:teuthology.orchestra.run.burnupi41.stderr:osd thrash <int[0-]> 2014-09-20T01:15:09.529 INFO:teuthology.orchestra.run.burnupi41.stderr:osd pool set <poolname> size|min_size|crash_replay_interval|pg_num|pgp_num|crush_ruleset|hashpspool <int> 2014-09-20T01:15:09.530 INFO:teuthology.orchestra.run.burnupi41.stderr:osd pool get <poolname> size|min_size|crash_replay_interval|pg_num|pgp_num|crush_ruleset 2014-09-20T01:15:09.530 INFO:teuthology.orchestra.run.burnupi41.stderr:osd reweight-by-utilization {<int[100-]>} 2014-09-20T01:15:09.530 INFO:teuthology.orchestra.run.burnupi41.stderr:osd pool set-quota <poolname> max_objects|max_bytes <val> 2014-09-20T01:15:09.530 INFO:teuthology.orchestra.run.burnupi41.stderr:osd pool delete <poolname> <poolname> --yes-i-really-really-mean-it 2014-09-20T01:15:09.531 INFO:teuthology.orchestra.run.burnupi41.stderr:osd pool rename <poolname> <poolname> 2014-09-20T01:15:09.531 INFO:teuthology.orchestra.run.burnupi41.stderr:osd rm <ids> [<ids>...] 2014-09-20T01:15:09.531 INFO:teuthology.orchestra.run.burnupi41.stderr:osd reweight <int[0-]> <float[0.0-1.0]> 2014-09-20T01:15:09.532 INFO:teuthology.orchestra.run.burnupi41.stderr:Error EINVAL: invalid command
archive_path: /var/lib/teuthworker/archive/teuthology-2014-09-19_19:20:01-upgrade:dumpling-firefly-x-giant-distro-basic-multi/499091 branch: giant description: upgrade:dumpling-firefly-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml 2-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-firefly-upgrade/firefly.yaml 4-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml} 5-upgrade-sequence/upgrade-by-daemon.yaml 6-final-workload/{ec-rados-default.yaml ec-rados-plugin=jerasure-k=3-m=1.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_s3tests.yaml rgw_swift.yaml} distros/ubuntu_12.04.yaml} email: ceph-qa@ceph.com job_id: '499091' kernel: &id001 kdb: true sha1: distro last_in_suite: false machine_type: plana,burnupi,mira name: teuthology-2014-09-19_19:20:01-upgrade:dumpling-firefly-x-giant-distro-basic-multi nuke-on-error: true os_type: ubuntu os_version: '12.04' overrides: admin_socket: branch: giant ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on legacy crush tunables: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 log-whitelist: - slow request - scrub mismatch - ScrubResult sha1: 2a2711daf86534ece11cad4527d69d43ec91d661 ceph-deploy: branch: dev: giant conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: 2a2711daf86534ece11cad4527d69d43ec91d661 s3tests: branch: giant workunit: sha1: 2a2711daf86534ece11cad4527d69d43ec91d661 owner: scheduled_teuthology@teuthology priority: 1000 roles: - - mon.a - mds.a - osd.0 - osd.1 - - mon.b - mon.c - osd.2 - osd.3 - - client.0 - client.1 suite: upgrade:dumpling-firefly-x suite_branch: master suite_path: /var/lib/teuthworker/src/ceph-qa-suite_master targets: ubuntu@burnupi16.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDY0z5p4byNFm/XI01BmGE1VJ5WIUyFhyukFvp4jIvSP7juOexMTWFSc3OhCNN7tTGMovcm3mNOFSZibaj3wBSCSjlmTAca9Jz2jS1x/5eeOVNroor9NrVf5ho+54kQHbtsAy3Q9kGRz9zpqky5etpK84I3oU74pV7FMWcwKQ0ym2j3UcaYmr7mSPUSvwd2LL5HZWZXoS+oxS8Zp8GjSdglFjRlfPa6UZzs8QrRpiElsD6fIf4yg/MMtlrQRrUMct/gY5i1Lgk0/slD3hkvtH9xCCLy27lI1waDt5+uKV9ANR3GnyIpZThSckLIheYW30qwMEp3FwkKZepNLSKISdQx ubuntu@burnupi18.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGVGWmHvMbNjuD58UIAxEszzZrUwr5xCPX9FPE8r30VYn3mmwn6P6FYl4JaEO7KnoHiaSRqUZVgutib/c/F2fYzVqE5OXKUEgOSFg98uiJQ4w7cOThiTkJ1MtXZW3+2vO8pG4YnsURlEqynlaTn88lyoRqUi9HWrAS614jqfh4RYBuwjWnn4YKdax6lsoiHj+LdT9fcuUCwLZoAf/0Y3W3Aa30RyQLaJwx/sHGHJm8m/jxLEksPoiJB7OCkY5U42QDweFxMH40+PszWgKvfuc7YaSEcwYaztx8K77P5bqLRV3Uhr5VGWCsUVcisRlsiHG4APKI0QorL7N9RaXFJZHF ubuntu@burnupi41.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHl9qOFK7gaAkB5pv19Ia1NQhJAy0sfo7XN2kHYu4ifgqTLHWkiYjQiXq9nP/Y/wOH+QQvTanhBYOsSRckfG9aTkc3ojM9qicFsyI/nLjRGcKMcMQYuayDSx3CNwxEj5Y3/wvaJvZtewZ3VNU8ddXkx+XCvAa922sgN5AUYreh7NaDoSJc0XRcQnDCNfTGnXVz9kx21S0Ybg1bt8k8eBuG5qvP8stamvo1ollFWo9khbaSPWu44XT1/ejem0tgqlkEUhHRUcdOCyCrW8Gvg08ajYRhafuxObFCrxfzdQJWIsCeCv3xfYcxb7iaCOTe3xdCj2sQZxCdbbPgyKgf3JAf tasks: - internal.lock_machines: - 3 - plana,burnupi,mira - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.push_inventory: null - internal.serialize_remote_roles: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: branch: dumpling - print: '**** done dumpling install' - ceph: fs: xfs - parallel: - workload - print: '**** done parallel' - install.upgrade: client.0: branch: firefly mon.a: branch: firefly mon.b: branch: firefly - print: '**** done install.upgrade' - ceph.restart: null - print: '**** done restart' - parallel: - workload2 - upgrade-sequence - print: '**** done parallel' - install.upgrade: client.0: null - print: '**** done install.upgrade client.0 to the version from teuthology-suite arg' - rados: clients: - client.0 ec_pool: true erasure_code_profile: k: 3 m: 1 name: jerasure31profile plugin: jerasure ruleset-failure-domain: osd technique: reed_sol_van objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 - rados: clients: - client.1 objects: 50 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 - workunit: clients: client.1: - rados/load-gen-mix.sh - sequential: - mon_thrash: revive_delay: 20 thrash_delay: 1 - workunit: clients: client.1: - rados/test.sh - print: '**** done rados/test.sh - 6-final-workload' - workunit: clients: client.1: - cls/test_cls_rbd.sh - workunit: clients: client.1: - rbd/import_export.sh env: RBD_CREATE_ARGS: --new-format - rgw: - client.1 - s3tests: client.1: rgw_server: client.1 - swift: client.1: rgw_server: client.1 teuthology_branch: master tube: multi upgrade-sequence: sequential: - install.upgrade: mon.a: null - print: '**** done install.upgrade mon.a to the version from teuthology-suite arg' - install.upgrade: mon.b: null - print: '**** done install.upgrade mon.b to the version from teuthology-suite arg' - ceph.restart: daemons: - mon.a - sleep: duration: 60 - ceph.restart: daemons: - mon.b - sleep: duration: 60 - ceph.restart: - mon.c - sleep: duration: 60 - ceph.restart: - osd.0 - sleep: duration: 60 - ceph.restart: - osd.1 - sleep: duration: 60 - ceph.restart: - osd.2 - sleep: duration: 60 - ceph.restart: - osd.3 - sleep: duration: 60 - ceph.restart: - mds.a - exec: mon.a: - ceph osd crush tunables firefly verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.multi.3179 workload: sequential: - workunit: branch: dumpling clients: client.0: - rados/test.sh - cls - print: '**** done rados/test.sh & cls' - workunit: branch: dumpling clients: client.0: - rados/load-gen-big.sh - print: '**** done rados/load-gen-big.sh' - workunit: branch: dumpling clients: client.0: - rbd/test_librbd.sh - print: '**** done rbd/test_librbd.sh' - workunit: branch: dumpling clients: client.0: - rbd/test_librbd_python.sh - print: '**** done rbd/test_librbd_python.sh' - rados: clients: - client.0 ec_pool: true objects: 50 op_weights: append: 100 copy_from: 50 delete: 50 read: 100 rmattr: 25 rollback: 50 setattr: 25 snap_create: 50 snap_remove: 50 write: 0 ops: 4000 workload2: sequential: - workunit: branch: firefly clients: client.0: - rados/test.sh - cls - print: '**** done #rados/test.sh and cls 2' - workunit: branch: firefly clients: client.0: - rados/load-gen-big.sh - print: '**** done rados/load-gen-big.sh 2' - workunit: branch: firefly clients: client.0: - rbd/test_librbd.sh - print: '**** done rbd/test_librbd.sh 2' - workunit: branch: firefly clients: client.0: - rbd/test_librbd_python.sh - print: '**** done rbd/test_librbd_python.sh 2'
description: upgrade:dumpling-firefly-x/parallel/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml 2-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml} 3-firefly-upgrade/firefly.yaml 4-workload/{rados_api.yaml rados_loadgenbig.yaml test_rbd_api.yaml test_rbd_python.yaml} 5-upgrade-sequence/upgrade-by-daemon.yaml 6-final-workload/{ec-rados-default.yaml ec-rados-plugin=jerasure-k=3-m=1.yaml rados-snaps-few-objects.yaml rados_loadgenmix.yaml rados_mon_thrash.yaml rbd_cls.yaml rbd_import_export.yaml rgw_s3tests.yaml rgw_swift.yaml} distros/ubuntu_12.04.yaml} duration: 2468.2781608104706 failure_reason: 'Command failed on burnupi41 with status 22: ''adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd erasure-code-profile set teuthologyprofile k=2 m=1 ruleset-failure-domain=osd''' flavor: basic owner: scheduled_teuthology@teuthology success: false
Updated by Loïc Dachary over 9 years ago
I'm not sure what the consequence of this indentation mistake was https://github.com/ceph/ceph-qa-suite/commit/cdcb315c5dc3a3e8719661197be036f15c52e2f6 . Maybe it means that this never actually ran before ?
https://github.com/ceph/ceph-qa-suite/tree/master/suites/upgrade/dumpling-firefly-x/parallel contains https://github.com/ceph/ceph-qa-suite/blob/master/suites/upgrade/dumpling-firefly-x/parallel/6-final-workload/ec-rados-default.yaml which is supposed to run only after the cluster is fully upgraded. The only way for the above error to happen is if a dumpling client/daemon is still around. I must be missing something ...
Updated by Loïc Dachary over 9 years ago
- Project changed from Ceph to teuthology
- Status changed from New to Fix Under Review
- Assignee set to Loïc Dachary
- Priority changed from Normal to Urgent
https://github.com/ceph/ceph-qa-suite/blob/master/suites/upgrade/dumpling-firefly-x/parallel/6-final-workload/ec-rados-default.yaml although added in the 6-final-workload directory defines a workload that is used during the upgrade from dumpling, hence the error
https://github.com/ceph/ceph-qa-suite/pull/145 should fix it
Updated by Yuri Weinstein over 9 years ago
Test run http://pulpito.front.sepia.ceph.com/teuthology-2014-09-21_10:14:47-upgrade:dumpling-firefly-x-giant-distro-basic-multi/ off dachary:wip-erasure-code-workloads
branch
Updated by Yuri Weinstein over 9 years ago
Updated by Loïc Dachary over 9 years ago
- Status changed from Fix Under Review to Resolved
- % Done changed from 0 to 100