Ceph » RADOS

Thanks for the report! The log comes from there:

int CrushWrapper::verify_upmap(CephContext *cct,
                               int rule_id,
                               int pool_size,
                               const vector<int>& up)
{
    // ...
      {
        int numrep = curstep->arg1;
        int type = curstep->arg2;
        if (numrep <= 0)
          numrep += pool_size;
        type_stack.emplace(type, numrep);
        if (type == 0) // osd
          break;
        map<int, set<int>> osds_by_parent; // parent_of_desired_type -> osds
        for (auto osd : up) {
          auto parent = get_parent_of_type(osd, type, rule_id);
          if (parent < 0) {
            osds_by_parent[parent].insert(osd);
          } else {
            ldout(cct, 1) << __func__ << " unable to get parent of osd." << osd
                          << ", skipping for now" 
                          << dendl;
          }
        }

It looks the verify_upmap was looking for parents for those OSDs (which should always be CRUSH buckets) but got something with non-negative ID (which is weird).

Could you please provide dump the CRUSH map as well as the output ceph osd tree?

Hints:
*https://docs.ceph.com/en/pacific/man/8/crushtool/ * ceph osd getcrushmap

Radoslaw,

Yes, I saw that piece of code too. But i think I figured it out just a short time ago. I had the crush hierarchy backwards. My crush rule has: pick racks(4)->pods(2)->host(1)(leaf). (It is a 6+2) EC Pool. So I get 8 chunks. But the hierarchy is that pods are HIGHER than racks. So I extracted the osdmap, and I ran: osdmaptool osdmap --upmap-cleanup. Doing so gives me the exact same errors as in the ceph-mon log for verify_upmap.

If I extract the crushmap from the osdmap, modify it to pick racks(4)->host(2)(leaf), put the crushmap back into the osdmap and run osdmaptool osdmap --upmap-cleanup, the verify_upmap messages do not occur, (but i get other upmap add/rm

My question is, if I actually deploy the crushmap without the pod choice (I can live without it), will I be ok or will it cause more problems given the current state. I am surprised that crush let me choose such a rule to begin with. The PGs look fine as to their OSDs and such.

Thanks

See the attached message I sent to ceph-users, that has what you asked.