Project

General

Profile

Bug #20771

ceph-objectstore-tool logging to stderr (didn't before)

Added by Sage Weil 3 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
Start date:
07/25/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

2017-07-25T23:22:12.341 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'

then ~20 minutes of
2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.377845 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.379327 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380695 7fb1944fd280 10 bluestore(/var/lib/ceph/osd/ceph-3) set_cache_shards 1
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380718 7fb1944fd280  1 bluestore(/var/lib/ceph/osd/ceph-3) _mount path /var/lib/ceph/osd/ceph-3
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380735 7fb1944fd280  1 bluestore(/var/lib/ceph/osd/ceph-3) fsck (deep) start
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380765 7fb1944fd280  1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380774 7fb1944fd280  1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380894 7fb1944fd280 20 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open devname nvme0n1
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380921 7fb1944fd280 10 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) _aio_start
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380970 7fb1944fd280  1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) non-rotational
...

then
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783088 7fb1944fd280  1 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) close
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783101 7fb1944fd280 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_stop
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.984820 7fb1852da700 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_thread end
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.8
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.69
...
 2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:2.3d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.12
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.2d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.0
2017-07-25T23:42:37.360 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 1.5f --file /home/ubuntu/cephtest/ceph.data/exp.1.5f.3'
2017-07-25T23:42:37.361 INFO:teuthology.orchestra.run.smithi040:Running: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 'osd.*' injectargs --osd_enable_op_tracker=false" 
2017-07-25T23:42:37.493 INFO:teuthology.orchestra.run.smithi040:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub'
2017-07-25T23:42:37.494 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/rados.py", line 257, in thread
    manager.wait_snap_trimming_complete(pool);
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/ceph_manager.py", line 1925, in wait_snap_trimming_complete
    'failed to complete snap trimming before timeout'
AssertionError: failed to complete snap trimming before timeout

/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442898/

and several other failures in that same batch:

/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442967
and i think more

History

#1 Updated by Sage Weil 3 months ago

  • Description updated (diff)

#2 Updated by Sage Weil 3 months ago

  • Status changed from Verified to Need Review

#3 Updated by Sage Weil 3 months ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF