Ceph » RADOS

Looks like a RADOS bug. Reassigning.

I was looking for num_rd_kb in the OSD code. It looks to me it almost never goes down but there is the logic in split:

  void split(std::vector<object_stat_sum_t> &out) const {
#define SPLIT(PARAM)                            \
    for (unsigned i = 0; i < out.size(); ++i) { \
      out[i].PARAM = PARAM / out.size();        \
      if (i < (PARAM % out.size())) {           \
        out[i].PARAM++;                         \
      }                                         \
    }
#define SPLIT_PRESERVE_NONZERO(PARAM)           \
    for (unsigned i = 0; i < out.size(); ++i) { \
      if (PARAM)                                \
        out[i].PARAM = 1 + PARAM / out.size();  \
      else                                      \
        out[i].PARAM = 0;                       \
    }

Do you have the autoscaler turned on? Do the freshly added OSDs map into the affected pool? Is the issue reproducible?

We've turned off the autoscaler in all clusters. Yes, freshly added OSDs map into the affected pool. Maybe it is related that we've added OSDs and then increased pg_num on pool?

The issue seems reproducible when increasing pg_num. For now, one cluster backfilling without increased pg_num. I will come back with information after  backfills.

I've added screenshots, backfills done and metrics then wrong, but after some days metrics back to normal(decreasing value little by little) 

Yes, I can confirm that metrics went wrong only in this scenario:

1. Added new osds nodes to pool
2. Increased pg_num
3, Done backfills
4. Wrong metrics ~2 days (attached screenshot)

It is not a critical, but why two days metrics abnormal?