Ceph » CephFS

Hmm, so this is happening because volume client creates a pool, then tries to use it as a layout at a time before its libcephfs Client instance's Objecter has seen the osdmap epoch that contains the pool. The Server::check_layout_vxattr logic waits for the client's osdmap epoch before concluding ENOENT -- if it waited for the global latest osdmap epoch then it would work.

I'm not sure the Server behaviour makes sense: setting up pools is almost always an out-of-band thing with respect to cephfs clients, so why would we assume that a client setting a layout xattr has the right osdmap epoch for the newly set up pool?

That's odd; I thought clients validated pools before passing them to the mds. Maybe that's wrong or undesirable for other reasons, though.

Oh yeah, the client does have code that's meant to be doing that, and on the client side it's a wait_for_latest. So don't know why this is happening.

The thing that's confusing me is that Client::ll_setxattr has this block:

    if (r == -ENOENT) {
      C_SaferCond ctx;
      objecter->wait_for_latest_osdmap(&ctx);
      ctx.wait();
    }

Which should be handling this case, but isn't for some reason.

I think cephfs python bind calls ceph_setxattr instead of ceph_ll_setxattr. There is no such code in Client::setxattr

Of course, you're right.