Ceph » RADOS

Run: http://pulpito.ceph.com/yuriw-2019-08-13_15:08:38-rados-wip-yuri2-testing-2019-08-12-1528-nautilus-distro-basic-smithi/
Job: 4213035
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2019-08-13_15:08:38-rados-wip-yuri2-testing-2019-08-12-1528-nautilus-distro-basic-smithi/4213035/teuthology.log

2019-08-13T17:15:38.648 INFO:tasks.workunit.client.0.smithi009.stderr:1
2019-08-13T17:15:38.648 INFO:tasks.workunit.client.0.smithi009.stderr:2'
2019-08-13T17:15:38.649 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1961: flush_pg_stats:  seqs=
2019-08-13T17:15:38.649 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1962: flush_pg_stats:  for osd in '$ids'
2019-08-13T17:15:38.649 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  ceph tell osd.0 flush_pg_stats
2019-08-13T17:15:38.808 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  seq=932007903471
2019-08-13T17:15:38.808 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1964: flush_pg_stats:  seqs=' 0-932007903471'
2019-08-13T17:15:38.809 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1962: flush_pg_stats:  for osd in '$ids'
2019-08-13T17:15:38.809 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  ceph tell osd.1 flush_pg_stats
2019-08-13T17:15:38.972 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  seq=34359738607
2019-08-13T17:15:38.972 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1964: flush_pg_stats:  seqs=' 0-932007903471 1-34359738607'
2019-08-13T17:15:38.973 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1962: flush_pg_stats:  for osd in '$ids'
2019-08-13T17:15:38.973 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  ceph tell osd.2 flush_pg_stats
2019-08-13T17:15:39.115 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1963: flush_pg_stats:  seq=34359738607
2019-08-13T17:15:39.116 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1964: flush_pg_stats:  seqs=' 0-932007903471 1-34359738607 2-34359738607'
2019-08-13T17:15:39.116 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1967: flush_pg_stats:  for s in '$seqs'
2019-08-13T17:15:39.116 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  echo 0-932007903471
2019-08-13T17:15:39.116 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  cut -d - -f 1
2019-08-13T17:15:39.117 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  osd=0
2019-08-13T17:15:39.118 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  echo 0-932007903471
2019-08-13T17:15:39.118 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  cut -d - -f 2
2019-08-13T17:15:39.118 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  seq=932007903471
2019-08-13T17:15:39.118 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1970: flush_pg_stats:  echo 'waiting osd.0 seq 932007903471'
2019-08-13T17:15:39.118 INFO:tasks.workunit.client.0.smithi009.stdout:waiting osd.0 seq 932007903471
2019-08-13T17:15:39.119 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  ceph osd last-stat-seq 0
2019-08-13T17:15:39.518 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  test 932007903469 -lt 932007903471
2019-08-13T17:15:39.519 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1972: flush_pg_stats:  sleep 1
2019-08-13T17:15:40.519 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1973: flush_pg_stats:  '[' 300 -eq 0 ']'
2019-08-13T17:15:40.520 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  ceph osd last-stat-seq 0
2019-08-13T17:15:41.035 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  test 932007903471 -lt 932007903471
2019-08-13T17:15:41.036 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1967: flush_pg_stats:  for s in '$seqs'
2019-08-13T17:15:41.036 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  echo 1-34359738607
2019-08-13T17:15:41.037 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  cut -d - -f 1
2019-08-13T17:15:41.039 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  osd=1
2019-08-13T17:15:41.040 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  echo 1-34359738607
2019-08-13T17:15:41.041 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  cut -d - -f 2
2019-08-13T17:15:41.042 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  seq=34359738607
2019-08-13T17:15:41.042 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1970: flush_pg_stats:  echo 'waiting osd.1 seq 34359738607'
2019-08-13T17:15:41.043 INFO:tasks.workunit.client.0.smithi009.stdout:waiting osd.1 seq 34359738607
2019-08-13T17:15:41.043 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  ceph osd last-stat-seq 1
2019-08-13T17:15:41.586 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  test 34359738607 -lt 34359738607
2019-08-13T17:15:41.586 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1967: flush_pg_stats:  for s in '$seqs'
2019-08-13T17:15:41.586 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  echo 2-34359738607
2019-08-13T17:15:41.586 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  cut -d - -f 1
2019-08-13T17:15:41.587 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1968: flush_pg_stats:  osd=2
2019-08-13T17:15:41.588 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  echo 2-34359738607
2019-08-13T17:15:41.589 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  cut -d - -f 2
2019-08-13T17:15:41.590 INFO:tasks.workunit.client.0.smithi009.stdout:waiting osd.2 seq 34359738607
2019-08-13T17:15:41.590 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1969: flush_pg_stats:  seq=34359738607
2019-08-13T17:15:41.591 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1970: flush_pg_stats:  echo 'waiting osd.2 seq 34359738607'
2019-08-13T17:15:41.592 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  ceph osd last-stat-seq 2
2019-08-13T17:15:42.071 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/../../standalone/ceph-helpers.sh:1971: flush_pg_stats:  test 34359738607 -lt 34359738607
2019-08-13T17:15:42.071 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2596: test_mon_cephdf_commands:  local 'jq_filter=.pools | .[] | select(.name == "cephdf_for_test") | .stats'
2019-08-13T17:15:42.072 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2597: test_mon_cephdf_commands:  ceph df detail --format=json
2019-08-13T17:15:42.072 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2597: test_mon_cephdf_commands:  jq '.pools | .[] | select(.name == "cephdf_for_test") | .stats.stored * 2'
2019-08-13T17:15:42.497 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2597: test_mon_cephdf_commands:  stored=8192
2019-08-13T17:15:42.498 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2598: test_mon_cephdf_commands:  ceph df detail --format=json
2019-08-13T17:15:42.499 INFO:tasks.workunit.client.0.smithi009.stderr://home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2598: test_mon_cephdf_commands:  jq '.pools | .[] | select(.name == "cephdf_for_test") | .stats.stored_raw'
2019-08-13T17:15:42.917 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2598: test_mon_cephdf_commands:  stored_raw=8192
2019-08-13T17:15:42.917 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2600: test_mon_cephdf_commands:  ceph osd pool delete cephdf_for_test cephdf_for_test --yes-i-really-really-mean-it
2019-08-13T17:15:44.396 INFO:tasks.workunit.client.0.smithi009.stderr:pool 'cephdf_for_test' does not exist
2019-08-13T17:15:44.408 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2601: test_mon_cephdf_commands:  rm ./cephdf_for_test
2019-08-13T17:15:44.410 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2603: test_mon_cephdf_commands:  expect_false test 8192 '!=' 8192
2019-08-13T17:15:44.410 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:48: expect_false:  set -x
2019-08-13T17:15:44.410 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:49: expect_false:  test 8192 '!=' 8192
2019-08-13T17:15:44.410 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:49: expect_false:  return 0
2019-08-13T17:15:44.411 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2851: main:  set +x
2019-08-13T17:15:44.884 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2850: main:  test_mon_tell_help_command
2019-08-13T17:15:44.884 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:2649: test_mon_tell_help_command:  ceph tell mon.a help
2019-08-13T17:15:55.019 INFO:tasks.workunit.client.0.smithi009.stderr:Traceback (most recent call last):
2019-08-13T17:15:55.019 INFO:tasks.workunit.client.0.smithi009.stderr:  File "/usr/bin/ceph", line 1263, in <module>
2019-08-13T17:15:55.020 INFO:tasks.workunit.client.0.smithi009.stderr:    retval = main()
2019-08-13T17:15:55.020 INFO:tasks.workunit.client.0.smithi009.stderr:  File "/usr/bin/ceph", line 1050, in main
2019-08-13T17:15:55.020 INFO:tasks.workunit.client.0.smithi009.stderr:    return do_extended_help(parser, childargs, target, None)
2019-08-13T17:15:55.020 INFO:tasks.workunit.client.0.smithi009.stderr:  File "/usr/bin/ceph", line 412, in do_extended_help
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:    return help_for_target(target, partial)
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:  File "/usr/bin/ceph", line 396, in help_for_target
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:    timeout=10)
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:  File "/usr/lib/python2.7/dist-packages/ceph_argparse.py", line 1463, in json_command
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:    raise RuntimeError('"{0}": exception {1}'.format(argdict, e))
2019-08-13T17:15:55.021 INFO:tasks.workunit.client.0.smithi009.stderr:RuntimeError: "None": exception "['{"prefix": "get_command_descriptions"}']": exception timed out
2019-08-13T17:15:55.031 INFO:tasks.workunit.client.0.smithi009.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/cephtool/test.sh:1: test_mon_tell_help_command:  rm -fr /tmp/cephtool.UPb
2019-08-13T17:15:55.036 DEBUG:teuthology.orchestra.run:got remote process result: 1
2019-08-13T17:15:55.036 INFO:tasks.workunit:Stopping ['cephtool'] on client.0...
2019-08-13T17:15:55.036 INFO:teuthology.orchestra.run.smithi009:Running:
2019-08-13T17:15:55.037 INFO:teuthology.orchestra.run.smithi009:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0
2019-08-13T17:15:55.244 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 86, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 65, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri2-testing-2019-08-12-1528-nautilus/qa/tasks/workunit.py", line 136, in task
    cleanup=cleanup)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri2-testing-2019-08-12-1528-nautilus/qa/tasks/workunit.py", line 286, in _spawn_on_all_clients
    timeout=timeout)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 85, in __exit__
    for result in self:
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 99, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/parallel.py", line 22, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-yuri2-testing-2019-08-12-1528-nautilus/qa/tasks/workunit.py", line 409, in _run_tests
    label="workunit test {workunit}".format(workunit=workunit)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 205, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 437, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 162, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 184, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed (workunit test cephtool/test.sh) on smithi009 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=faf806db5dace30d4806cddced17688be30cf432 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h