Ceph

2016-10-03 21:30:50.027588 7f31d11eb700  1 -- 172.21.15.68:6790/0 <== client.? 172.21.15.68:0/1828017859 3 ==== auth(proto 2 165 bytes epoch 0) v1 ==== 195+0+0 (688865921 0 0) 0x7f31e49e5600 con 0x7f31e4be0800
2016-10-03 21:30:50.027608 7f31d11eb700 20 mon.b@1(peon) e1 _ms_dispatch existing session 0x7f31e48e9de0 for client.? 172.21.15.68:0/1828017859
2016-10-03 21:30:50.027613 7f31d11eb700 20 mon.b@1(peon) e1  caps allow *
2016-10-03 21:30:50.027624 7f31d11eb700 10 mon.b@1(peon).paxosservice(auth 1..7) dispatch 0x7f31e49e5600 auth(proto 2 165 bytes epoch 0) v1 from client.? 172.21.15.68:0/1828017859 con 0x7f31e4be0800
2016-10-03 21:30:50.027634 7f31d11eb700  5 mon.b@1(peon).paxos(paxos active c 1..224) is_readable = 1 - now=2016-10-03 21:30:50.027635 lease_expire=2016-10-03 21:30:55.025940 has v0 lc 224
2016-10-03 21:30:50.027640 7f31d11eb700 10 mon.b@1(peon).auth v7 preprocess_query auth(proto 2 165 bytes epoch 0) v1 from client.? 172.21.15.68:0/1828017859
2016-10-03 21:30:50.027644 7f31d11eb700 10 mon.b@1(peon).auth v7 prep_auth() blob_size=165
2016-10-03 21:30:50.027836 7f31d11eb700  2 mon.b@1(peon) e1 send_reply 0x7f31e4ec5ce0 0x7f31e49e5840 auth_reply(proto 2 -1 (1) Operation not permitted) v1
2016-10-03 21:30:50.027848 7f31d11eb700  1 -- 172.21.15.68:6790/0 --> 172.21.15.68:0/1828017859 -- auth_reply(proto 2 -1 (1) Operation not permitted) v1 -- 0x7f31e49e5840 con 0

/a/sage-2016-10-03_21:06:11-rados-master---basic-smithi/450863

I think this is a race with the new election and AuthMonitor / CephxKeyServer setup and rotating keys?? Maybe. Turning on debug auth = 20 for mons in teuthology.