Bug #20771
Updated by Sage Weil over 6 years ago
<pre>
2017-07-25T23:22:12.341 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
</pre>
then ~20 minutes of
<pre>
2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.377845 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.187 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.379327 7fb1944fd280 -1 WARNING: all dangerous and experimental features are enabled.
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380695 7fb1944fd280 10 bluestore(/var/lib/ceph/osd/ceph-3) set_cache_shards 1
2017-07-25T23:22:26.188 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380718 7fb1944fd280 1 bluestore(/var/lib/ceph/osd/ceph-3) _mount path /var/lib/ceph/osd/ceph-3
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380735 7fb1944fd280 1 bluestore(/var/lib/ceph/osd/ceph-3) fsck (deep) start
2017-07-25T23:22:26.288 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380765 7fb1944fd280 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380774 7fb1944fd280 1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380894 7fb1944fd280 20 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open devname nvme0n1
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380921 7fb1944fd280 10 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) _aio_start
2017-07-25T23:22:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:12.380970 7fb1944fd280 1 bdev(0x7fb1957c4a00 /var/lib/ceph/osd/ceph-3/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) non-rotational
...
</pre>
then
<pre>
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783088 7fb1944fd280 1 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) close
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.783101 7fb1944fd280 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_stop
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stderr:2017-07-25 23:22:25.984820 7fb1852da700 10 bdev(0x7fb195c45400 /var/lib/ceph/osd/ceph-3/block) _aio_thread end
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.8
2017-07-25T23:42:37.350 INFO:teuthology.orchestra.run.smithi076.stdout:2.69
...
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:2.3d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.12
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.2d
2017-07-25T23:42:37.359 INFO:teuthology.orchestra.run.smithi076.stdout:1.0
2017-07-25T23:42:37.360 INFO:teuthology.orchestra.run.smithi076:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 1.5f --file /home/ubuntu/cephtest/ceph.data/exp.1.5f.3'
2017-07-25T23:42:37.361 INFO:teuthology.orchestra.run.smithi040:Running: "sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell 'osd.*' injectargs --osd_enable_op_tracker=false"
2017-07-25T23:42:37.493 INFO:teuthology.orchestra.run.smithi040:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub'
2017-07-25T23:42:37.494 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/rados.py", line 257, in thread
manager.wait_snap_trimming_complete(pool);
File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-sage-testing/qa/tasks/ceph_manager.py", line 1925, in wait_snap_trimming_complete
'failed to complete snap trimming before timeout'
AssertionError: failed to complete snap trimming before timeout
</pre>
/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442898/
and several other failures in that same batch:
/a/sage-2017-07-25_20:13:42-rados-wip-sage-testing-distro-basic-smithi/1442967
and i think more