PR #16172 causing performance regression
This is most obvious during 4K random writes to NVMe backed bluestore RBD volumes, but it may be present in other tests as well. A git bisection very clearly points to PR #16172 (introduced with v12.1.2) as the culprit.
Performance decreases in the above described test from 30K IOPS to 15K IOPS for a single OSD. A wallclock profile shows extra time spent in pg_log_dup_t get_key_name (~0.7%) and encode (~1.7%) per tp_osd_tp thread. Greg hypothesized that we might be doing unnecessary string manipulation in get_key_name and indeed it looks like there may be extra string manipulation and memory copying going on. Given that we are spending about 1.7% of the time in each tp_osd_tp thread doing pg_log_dup_t encode however, I suspect the bigger issue is that are writing a lot more pglog data to the KV store now and this is less about CPU overhead than simply using a greater percentage of our available KV store throughput for pglog. The column family PR might give us a better hint if this is correct.
#2 Updated by Mark Nelson about 2 years ago
After discussion with Eric, we can avoid the new code path entirely by increasing the osd_min_pg_log_entries to the same value as osd_pg_log_dups_tracks (ie currently 1500 -> 3000), however this doesn't fix the problem, only avoids it by not invoking the new code. This has been verified to increase performance to near levels prior to PR #16172.