Bug #20771: ceph-objectstore-tool logging to stderr (didn't before) - Ceph - Ceph

Actions

Copy link

Bug #20771

closed

ceph-objectstore-tool logging to stderr (didn't before)

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Immediate

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

2017-07-25T23:22:12.341 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'

then ~20 minutes of

2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.377845 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.379327 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380695 7fb1944fd280 10 bluestore(/var/lib/ceph/osd/ceph-3) set_cache_shards 1
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380718 7fb1944fd280  1 bluestore(/var/lib/ceph/osd/ceph-3) _mount path /var/lib/ceph/osd/ceph-3
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380735 7fb1944fd280  1 bluestore(/var/lib/ceph/osd/ceph-3) fsck (deep) start
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380765 7fb1944fd280  1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380774 7fb1944fd280  1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380894 7fb1944fd280 20 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open devname nvme0n1
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380921 7fb1944fd280 10 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) _aio_start
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380970 7fb1944fd280  1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) non-rotational
...

then

2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783088 7fb1944fd280  1 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) close
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783101 7fb1944fd280 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_stop
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.984820 7fb1852da700 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_thread end
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.8
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.69
...
 2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:2.3d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.12
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.2d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.0
2017-07-25T23:42:37.360 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 1.5f --file /home/ubuntu/cephtest/ceph.data/exp.1.5f.3'
2017-07-25T23:42:37.361 INFO:teuthology.orchestra.run.smithi040:Running: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 'osd.*' injectargs --osd_enable_op_tracker=false" 
2017-07-25T23:42:37.493 INFO:teuthology.orchestra.run.smithi040:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub'
2017-07-25T23:42:37.494 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/rados.py", line 257, in thread
    manager.wait_snap_trimming_complete(pool);
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/ceph_manager.py", line 1925, in wait_snap_trimming_complete
    'failed to complete snap trimming before timeout'
AssertionError: failed to complete snap trimming before timeout

/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442898/

and several other failures in that same batch:

/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442967
and i think more