Bug #50670
closed[pwl ssd] head / tail pointer corruption
0%
Description
After filling the ssd cache to capacity and maintaining the load (so that the cache is constantly retiring old and generating new log entries), I have observed that the head (m_first_valid_entry) and tail (m_first_free_entry) pointers can get corrupted. Since the cache is at capacity, they are expected to cycle through the log space very close to each other, with the invariant that head > tail (head - tail = 8-12K, the rest is allocated). However, the tail pointer sometimes "jumps" the head pointer, violating the invariant and making the full cache appear almost empty (only 8-12K used).
Updated by Ilya Dryomov almost 3 years ago
- Assignee set to Ilya Dryomov
I suspect this is related to AbstractWriteLog::check_allocation() logic and may be addressed by https://github.com/ceph/ceph/pull/41101, at least partially. Yet to test.
Updated by Ilya Dryomov almost 3 years ago
- Related to Bug #50675: [pwl ssd] cache larger than 4G will corrupt itself added
Updated by CONGMIN YIN almost 3 years ago
In check_allocation(), "if (m_bytes_allocated + bytes_allocated >= m_bytes_allocated_cap)" control ssd allocation in ssd mode. But after this, lock is released. before "m_bytes_allocated += bytes_allocated;", m_bytes_allocated might plus again and exceed m_bytes_allocated_cap. so tail entry might exceed tail. see https://github.com/ceph/ceph/pull/41968/commits/91b8862cd2b09b4555236dc06088f68339ff4f1c
Updated by Ilya Dryomov almost 3 years ago
- Status changed from New to Fix Under Review
- Assignee changed from Ilya Dryomov to CONGMIN YIN
- Pull request ID set to 41968
Updated by Ilya Dryomov over 2 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot over 2 years ago
- Copied to Backport #52310: pacific: [pwl ssd] head / tail pointer corruption added
Updated by Ilya Dryomov about 2 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".