Bug #38384
Updated by David Zafman about 5 years ago
http://qa-proxy.ceph.com/teuthology/dzafman-2019-02-19_08:38:36-rados-wip-zafman-testing-38344-distro-basic-smithi/3612917/teuthology.log I also ran ceph --help mon manually after a vstart and it produced the same error. This is blocking other tests in test.sh. This may have been caused by: <pre> commit 19fbdb3ffde5bcb73792b1d822fe3f8d0407d490 Author: Patrick Donnelly <pdonnell@redhat.com> Date: Mon Jan 21 14:45:07 2019 -0800 ceph: omit hidden commands from suggestions Fixes: http://tracker.ceph.com/issues/37955 Signed-off-by: Patrick Donnelly <pdonnell@redhat.com> </pre> <pre> 2019-02-19T17:46:45.314 INFO:tasks.workunit.client.0.smithi053.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1156: test_mon_mon: ceph --help mon 2019-02-19T17:46:45.456 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.456 INFO:tasks.workunit.client.0.smithi053.stdout: General usage: 2019-02-19T17:46:45.457 INFO:tasks.workunit.client.0.smithi053.stdout: ============== 2019-02-19T17:46:45.457 INFO:tasks.workunit.client.0.smithi053.stdout:usage: ceph [-h] [-c CEPHCONF] [-i INPUT_FILE] [-o OUTPUT_FILE] 2019-02-19T17:46:45.457 INFO:tasks.workunit.client.0.smithi053.stdout: [--id CLIENT_ID] [--name CLIENT_NAME] [--cluster CLUSTER] 2019-02-19T17:46:45.457 INFO:tasks.workunit.client.0.smithi053.stdout: [--admin-daemon ADMIN_SOCKET] [-s] [-w] [--watch-debug] 2019-02-19T17:46:45.458 INFO:tasks.workunit.client.0.smithi053.stdout: [--watch-info] [--watch-sec] [--watch-warn] [--watch-error] 2019-02-19T17:46:45.458 INFO:tasks.workunit.client.0.smithi053.stdout: [--watch-channel {cluster,audit,*}] [--version] [--verbose] 2019-02-19T17:46:45.458 INFO:tasks.workunit.client.0.smithi053.stdout: [--concise] [-f {json,json-pretty,xml,xml-pretty,plain}] 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout: [--connect-timeout CLUSTER_TIMEOUT] [--block] [--period PERIOD] 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout:Ceph administration tool 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout:optional arguments: 2019-02-19T17:46:45.459 INFO:tasks.workunit.client.0.smithi053.stdout: -h, --help request mon help 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: -c CEPHCONF, --conf CEPHCONF 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: ceph configuration file 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: -i INPUT_FILE, --in-file INPUT_FILE 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: input file, or "-" for stdin 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: -o OUTPUT_FILE, --out-file OUTPUT_FILE 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: output file, or "-" for stdout 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: --id CLIENT_ID, --user CLIENT_ID 2019-02-19T17:46:45.460 INFO:tasks.workunit.client.0.smithi053.stdout: client id for authentication 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: --name CLIENT_NAME, -n CLIENT_NAME 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: client name for authentication 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: --cluster CLUSTER cluster name 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: --admin-daemon ADMIN_SOCKET 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: submit admin-socket commands ("help" for help 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: -s, --status show cluster status 2019-02-19T17:46:45.461 INFO:tasks.workunit.client.0.smithi053.stdout: -w, --watch watch live cluster changes 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-debug watch debug events 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-info watch info events 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-sec watch security events 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-warn watch warn events 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-error watch error events 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: --watch-channel {cluster,audit,*} 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: which log channel to follow when using -w/--watch. One 2019-02-19T17:46:45.462 INFO:tasks.workunit.client.0.smithi053.stdout: of ['cluster', 'audit', '*'] 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --version, -v display version 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --verbose make verbose 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --concise make less verbose 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: -f {json,json-pretty,xml,xml-pretty,plain}, --format {json,json-pretty,xml,xml-pretty,plain} 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --connect-timeout CLUSTER_TIMEOUT 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: set a timeout for connecting to the cluster 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --block block until completion (scrub and deep-scrub only) 2019-02-19T17:46:45.463 INFO:tasks.workunit.client.0.smithi053.stdout: --period PERIOD, -p PERIOD 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: polling period, default 1.0 second (for polling 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: commands only) 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: Local commands: 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: =============== 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.464 INFO:tasks.workunit.client.0.smithi053.stdout:ping <mon.id> Send simple presence/life test to a mon 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout: <mon.id> may be 'mon.*' for all mons 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout:daemon {type.id|path} <cmd> 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout: Same as --admin-daemon, but auto-find admin socket 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout:daemonperf {type.id | path} [stat-pats] [priority] [<interval>] [<count>] 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout:daemonperf {type.id | path} list|ls [stat-pats] [priority] 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout: Get selected perf stats from daemon/admin socket 2019-02-19T17:46:45.465 INFO:tasks.workunit.client.0.smithi053.stdout: Optional shell-glob comma-delim match string stat-pats 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: Optional selection priority (can abbreviate name): 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: critical, interesting, useful, noninteresting, debug 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: List shows a table of all available stats 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: Run <count> times (default forever), 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: once per <interval> seconds (default 1) 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: 2019-02-19T17:46:45.466 INFO:tasks.workunit.client.0.smithi053.stdout: Monitor commands: 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stdout: ================= 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr:Traceback (most recent call last): 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 1242, in <module> 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr: retval = main() 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 1039, in main 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr: return do_extended_help(parser, childargs, ('mon', ''), ' '.join(childargs)) 2019-02-19T17:46:45.467 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 407, in do_extended_help 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: return help_for_target(target, partial) 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 404, in help_for_target 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: return help_for_sigs(outbuf.decode('utf-8'), partial) 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 383, in help_for_sigs 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: partial=partial)) 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: File "/usr/bin/ceph", line 474, in format_help 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr: if flags & (argparse.FLAG_OBSOLETE | argparse.FLAG_DEPRECATED | argparse.FLAG_HIDDEN): 2019-02-19T17:46:45.468 INFO:tasks.workunit.client.0.smithi053.stderr:AttributeError: 'module' object has no attribute 'FLAG_OBSOLETE' 2019-02-19T17:46:45.469 INFO:tasks.workunit.client.0.smithi053.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_mon_mon: rm -fr /tmp/cephtool.Lv4 2019-02-19T17:46:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-02-19T17:46:45.470 INFO:tasks.workunit:Stopping ['cephtool'] on client.0... 2019-02-19T17:46:45.470 INFO:teuthology.orchestra.run.smithi053:Running: 2019-02-19T17:46:45.470 INFO:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2019-02-19T17:46:45.593 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-testing-38344/qa/tasks/workunit.py", line 136, in task cleanup=cleanup) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-testing-38344/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-testing-38344/qa/tasks/workunit.py", line 409, in _run_tests label="workunit test {workunit}".format(workunit=workunit) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 193, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 435, in run r.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status node=self.hostname, label=self.label CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi053 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=de422f77a7271f8af9861c940b1c7879d2e2ee10 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh' </pre>