Ceph » CephFS

Venky Shankar wrote:

Dhairya,

Please take a look at this. I think there is some sort of race that is causing this crash while iterating the directory list in cephfs-mirror daemon.

Have we faced this issue anytime recently?

Dhairya Parmar wrote:

Venky Shankar wrote:

Dhairya,

Please take a look at this. I think there is some sort of race that is causing this crash while iterating the directory list in cephfs-mirror daemon.

Have we faced this issue anytime recently?

Not in our teuthology tests, but nothing major has changes in cephfs-mirror, so the race most likely still exists.

Issue seems to be at:

std::rotate(m_directories.begin(), m_directories.begin() + 1, m_directories.end());

if m_directories is empty, rotate tries to access the index that doesn't exist

From https://en.cppreference.com/w/cpp/algorithm/rotate:

Return value
An iterator that is equal to:
last, if first middle is true,
first, if middle last is true,
first + (last - middle)[1] otherwise, i.e. the new location of the element pointed by first.

It performs swaps to make element at index middle (here m_directories.begin() + 1) the first element of vector with range [first, last), where first = m_directories.begin() and last = m_directories.end(), if there is nothing in the vector m_directories, it basically tries to access value at an index that does not exist/doesn't belong to it and thus the segmentation fault. This is easy to fix, we only rotate when the vector size is more than 1, else don't touch it.

Dhairya Parmar wrote:

Issue seems to be at:
[...]
@ https://github.com/ceph/ceph/blob/main/src/tools/cephfs_mirror/PeerReplayer.cc#L315

if m_directories is empty, rotate tries to access the index that doesn't exist

From https://en.cppreference.com/w/cpp/algorithm/rotate:

Return value
An iterator that is equal to:
last, if first middle is true,
first, if middle last is true,
first + (last - middle)[1] otherwise, i.e. the new location of the element pointed by first.

It performs swaps to make element at index middle (here m_directories.begin() + 1) the first element of vector with range [first, last), where first = m_directories.begin() and last = m_directories.end(), if there is nothing in the vector m_directories, it basically tries to access value at an index that does not exist/doesn't belong to it and thus the segmentation fault. This is easy to fix, we only rotate when the vector size is more than 1, else don't touch it.

This doesn't make sense as m_directories is only iterated if its size() is non-zero.