Ceph » CephFS

Part of the bug appears to be in ceph, where the following returns an error, causing an infinite loop in get_key():

sudo ceph --cluster=ceph --name=mon. --keyring=/var/lib/ceph/mon/ceph-a/keyring auth get-or-create client.admin mon a

llow * osd allow * mds allow
access denied

More details? I'm not sure how the title relates to the bug description or MDS log. The log is crashing on the SessionMap decode, and my money is on all the encoding changes.
I'm hoping that this system just followed an unusual upgrade path that customers won't actually see and got busted up on the weird encoding versions we did. Can you provide more precise details on the upgrades and exact versions this system ran as?

It's quite possible the upgrade was corrupted somewhere along the line. Prior to the issues the system was on 0.56.3, 0.57.667, and now 58.500

I moved from bobtail release to the master branch and ran into the issue in 4424. Gary fixed that in wip-4424 and now I am seeing this issue. Sam has taken a look at it and the title reflects what he saw so I filed the bug to track it.

I am not seeing this issue on the system tracking along the NEXT branch or on a fresh install of Master. It may very well be something no one will ever create in the real world. my concern is that it started from a "released" distribution and might happen between bobtail and cuttlefish.

Just to make sure I'm tracking these upgrades correctly:
It was created on v0.56.3? (Not a branch.) Then it moved to master branch when it was following .57 (I can't translate .57.667 into a commit or a release)? And then you saw the issue when you upgraded to a fresher master following 0.58?

This path would indicate that you ran into the encoding stuff I'm hoping for. If you can get me the precise commit post-.57 commit then I can make sure.

This is what was captured at the time the test was run successfully: Ceph Version: 0.57-667-g6a9cda7
The next instance installed from master was: Ceph Version: 0.58-500-gaf3b163
The current installation is sitting at: ceph version 0.58-501-g66be33a (wip-4424)

Is this what you are looking for or is there something else we should be capturing in the test log?

Yep! This says that you ran a branch which included an unreleased set of encoding rules on the MDS which would have collided horribly with what ended up in 0.58. (af3b163 includes the merge of "wip-mds-encode-rebased" but not any of the follow-on conflict cleanup which came as a result of our needing to backport some stuff to bobtail.)

Hurray, not a problem for users, and not an unexpected collision of doom. :)