Ceph

If you spin up a Kraken cluster on teuthology VPS, stop an osd daemon and flush its journal you see the following output.

# ceph-osd -i 1 --flush-journal
2017-10-04 04:30:01.006318 7f002103e940 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2017-10-04 04:30:01.011028 7f002103e940 -1 flushed journal /var/lib/ceph/osd/ceph-1/journal for object store /var/lib/ceph/osd/ceph-1

Try the same on a post-Luminous cluster and the output is truncated.

# ceph-osd -i 1 --flush-journal
2017-10-04 04:43:51.599960 7f1bc7721d00 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway

The simple workaround is to pass '--log_flush_on_exit'

# ceph-osd -i 1 --flush-journal --log_flush_on_exit
2017-10-04 04:44:49.030263 7f80c1643d00 -1 journal FileJournal::_open: disabling aio for non-block journal.  Use journal_force_aio to force use of aio anyway
2017-10-04 04:44:49.032395 7f80c1643d00 -1 flushed journal /var/lib/ceph/osd/ceph-1/journal for object store /var/lib/ceph/osd/ceph-1

Note that this only happens on VPS (VMs) and not bare metal and that running the binary under strace clearly shows the write going to the stderr file descriptor.

Given the simplicity of the workaround and the potential difficulty in tracking this down and rectifying it (since potentially many utility commands across multiple daemons may exhibit this behaviour and the fix may require a non-trivial amount of effort) it's not clear this is worth fixing but I am documenting it and the workaround here nonetheless.