Project

General

Profile

Bug #5427

mon: could not get service secret for auth subsystem

Added by Sage Weil almost 11 years ago. Updated almost 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
Monitor
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
cuttlefish
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-06-22 11:11:09.779485 7f084cad3700  1 -- 10.214.131.22:6791/0 <== client.? 10.214.132.16:0/1025494 2 ==== auth(proto 2 32 bytes epoch 0) v1 ==== 62+0+0 (232334284 0 0) 0x3083d80 con 0x3067b00
2013-06-22 11:11:09.779506 7f084cad3700 20 mon.f@4(peon) e1 have connection
2013-06-22 11:11:09.779506 7f084acc3700 10 -- 10.214.131.22:6791/0 >> 10.214.132.16:0/1025494 pipe(0x30c7780 sd=28 :6791 s=2 pgs=1 cs=1 l=1).write_ack 2
2013-06-22 11:11:09.779512 7f084cad3700 20 mon.f@4(peon) e1 ms_dispatch existing session MonSession: client.? 10.214.132.16:0/1025494 is open for client.? 10.214.132.16:0/1025494
2013-06-22 11:11:09.779526 7f084cad3700 20 mon.f@4(peon) e1  caps 
2013-06-22 11:11:09.779529 7f084acc3700 10 -- 10.214.131.22:6791/0 >> 10.214.132.16:0/1025494 pipe(0x30c7780 sd=28 :6791 s=2 pgs=1 cs=1 l=1).writer: state = open policy.server=1
2013-06-22 11:11:09.779532 7f084cad3700 10 mon.f@4(peon).paxosservice(auth) dispatch auth(proto 2 32 bytes epoch 0) v1 from client.? 10.214.132.16:0/1025494
2013-06-22 11:11:09.779541 7f084acc3700 20 -- 10.214.131.22:6791/0 >> 10.214.132.16:0/1025494 pipe(0x30c7780 sd=28 :6791 s=2 pgs=1 cs=1 l=1).writer sleeping
2013-06-22 11:11:09.779580 7f084cad3700 10 mon.f@4(peon).auth v2 update_from_paxos
2013-06-22 11:11:09.779599 7f084cad3700 10 mon.f@4(peon).auth v2 preprocess_query auth(proto 2 32 bytes epoch 0) v1 from client.? 10.214.132.16:0/1025494
2013-06-22 11:11:09.779614 7f084cad3700 10 mon.f@4(peon).auth v2 prep_auth() blob_size=32
2013-06-22 11:11:09.779674 7f084cad3700  0 cephx server client.admin:  could not get service secret for auth subsystem
...

job was
kernel:
  kdb: true
  sha1: 2dd322b42d608a37f3e5beed57a8fbc673da6e32
machine_type: plana
nuke-on-error: true
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 20
        debug paxos: 20
      mon.b:
        clock offset: 10
    log-whitelist:
    - slow request
    sha1: 94eada40460cc6010be23110ef8ce0e3d92691af
  install:
    ceph:
      sha1: 94eada40460cc6010be23110ef8ce0e3d92691af
  s3tests:
    branch: next
  workunit:
    sha1: 94eada40460cc6010be23110ef8ce0e3d92691af
roles:
- - mon.a
  - mon.c
  - mon.e
  - osd.0
- - mon.b
  - mon.d
  - mon.f
  - osd.1
  - mds.a
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    log-whitelist:
    - slow request
    - .*clock.*skew.*
    - clocks not synchronized
    wait-for-healthy: false
- mon_clock_skew_check:
    expect-skew: true

Associated revisions

Revision cd98eb0c (diff)
Added by Sage Weil almost 11 years ago

mon/AuthMonitor: make initial auth include rotating keys

This closes a very narrow race during mon creation where there are no
service keys.

Fixes: #5427
Signed-off-by: Sage Weil <>

Revision 4474a0cc (diff)
Added by Sage Weil almost 11 years ago

mon/AuthMonitor: make initial auth include rotating keys

This closes a very narrow race during mon creation where there are no
service keys.

Fixes: #5427
Signed-off-by: Sage Weil <>
(cherry picked from commit cd98eb0c651d9ee62e19c2cc92eadae9bed678cd)

History

#1 Updated by Sage Weil almost 11 years ago

the leader never ticked while paxos was healthy, i think because of the clock skew.

see wip-5427

#2 Updated by Sage Weil almost 11 years ago

  • Status changed from 12 to Fix Under Review

#3 Updated by Ian Colle almost 11 years ago

  • Assignee set to Joao Eduardo Luis

Joao - please review.

#4 Updated by Joao Eduardo Luis almost 11 years ago

  • Status changed from Fix Under Review to 4
  • Assignee changed from Joao Eduardo Luis to Sage Weil

Looked at it yesterday, pull request 374. I believe it to be correct.

#5 Updated by Sage Weil almost 11 years ago

  • Status changed from 4 to Pending Backport
  • Backport set to cuttlefish

#6 Updated by Sage Weil almost 11 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF