Project

General

Profile

Bug #3563

osd crashed with error "auth: could not find secret_id=2"

Added by Tamilarasi muthamizhan over 11 years ago. Updated about 11 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Test : collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:many.yaml tasks:rbd_cli_copy_old_format.yaml

Log: ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540

an excerpt from the osd log:
2012-11-30 09:38:12.112440 7fc4d678a700  0 monclient: hunting for new mon
2012-11-30 09:38:13.055487 7fc4d3481700  0 auth: could not find secret_id=2
2012-11-30 09:38:13.055499 7fc4d3481700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:13.055504 7fc4d3481700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:13.055507 7fc4d3481700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x23fe6c0 sd=29 :6804 pgs=0 cs=0 l=0).accept bad authorizer
2012-11-30 09:38:13.056119 7fc4d3380700  0 auth: could not find secret_id=2
2012-11-30 09:38:13.056137 7fc4d3380700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:13.056143 7fc4d3380700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:13.056146 7fc4d3380700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x23feb40 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer
2012-11-30 09:38:13.056926 7fc4d3380700  0 auth: could not find secret_id=2
2012-11-30 09:38:13.056939 7fc4d3380700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
.....
....
2012-11-30 09:38:16.404413 7fc4d2b78700  0 auth: could not find secret_id=2
   1/ 5 rgw2012-11-30 09:38:16.404427 7fc4d2b78700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2

2012-11-30 09:38:16.404432 7fc4d2b78700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
   1/ 5 hadoop
   1/ 5 asok
   1/ 5 throttle
  -2/-2 (syslog threshold)2012-11-30 09:38:16.404540 7fc4d2976700  0 auth: could not find secret_id=2

2012-11-30 09:38:16.404435 7fc4d2b78700  0   -1/-1 (stderr threshold)-- 10.214.131.10:6804/29058 >> 10.214.131.7:6807/1896 pipe(0x280d000 sd=32 :6804 pgs=0 cs=0 l=0).accept bad authorizer

2012-11-30 09:38:16.404550 7fc4d2976700  0   max_recent     10000cephx: verify_authorizer could not get service secret for service osd secret_id=2

2012-11-30 09:38:16.404555 7fc4d2976700  1   max_new      1000000CephxAuthorizeHandler::verify_authorizer isvalid=0

2012-11-30 09:38:16.404557 7fc4d2976700  0   log_file /tmp/cephtest/archive/log/osd.5.log-- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x280dd80 sd=33 :6804 pgs=0 cs=0 l=0).accept bad authorizer

--- end dump of recent events ---
2012-11-30 09:38:16.404763 7fc4d2a77700  0 auth: could not find secret_id=2
2012-11-30 09:38:16.404771 7fc4d2a77700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:16.404774 7fc4d2a77700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:16.404776 7fc4d2a77700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6804/1859 pipe(0x280d900 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer
2012-11-30 09:38:16.405087 7fc4d2b78700  0 auth: could not find secret_id=2
2012-11-30 09:38:16.405094 7fc4d2b78700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:16.405094 7fc4d2976700  0 auth: could not find secret_id=2
2012-11-30 09:38:16.405099 7fc4d2b78700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:16.405102 7fc4d2976700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:16.405106 7fc4d2976700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:16.405109 7fc4d2976700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x2758900 sd=32 :6804 pgs=0 cs=0 l=0).accept bad authorizer
2012-11-30 09:38:16.405104 7fc4d2b78700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6807/1896 pipe(0x2758d80 sd=31 :6804 pgs=0 cs=0 l=0).accept bad authorizer
2012-11-30 09:38:16.405183 7fc4d2a77700  0 auth: could not find secret_id=2
2012-11-30 09:38:16.405189 7fc4d2a77700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=2
2012-11-30 09:38:16.405192 7fc4d2a77700  1 CephxAuthorizeHandler::verify_authorizer isvalid=0
2012-11-30 09:38:16.405195 7fc4d2a77700  0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6804/1859 pipe(0x2758b40 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer
(END)

ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66
nuke-on-error: true
overrides:
  ceph:
    conf:
      global:
        ms inject socket failures: 500
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150
  s3tests:
    branch: testing
  workunit:
    sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana28.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnMB6Wmqywp9d2Ns8PYeySKtDaQCkkfldLWV57qoMTDC9+ZoIJcSUE9OshK8pwO7C14cU+IiFIjY4a/fctylCuopSnneI7tsTq2er/Te2XOA+4Q1L4K2twa0pTwTAPyqqsZODqM4O/QG/voJjKMvrazp85VsmQ7mM1bOdR4wCNbPZoydg1/iO7cxSx8iSb2LdWTx+mRFSrWzHWhqvxwfHDZXtim3U/dej+7dFDZrM8Cq1PsvOXkE+zEkmSAMU64JUPZBTSdYl8Dh1aMMz+BWfIn5ZUFzcKt/7LLims85X4rBGyp1CAmhyTtTDBmNrvemZI6BEl+xA1kamhG+Cp/Z+n
  ubuntu@plana30.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYDpptyOTrVH3RqiwH5A//Q2CkkVz5dPTpd/s8qG/Q4EHVA4WDMu80pcDvdSewOfFJl83MEtDKKjuJOuEzI4OGn0DPptDN5wHC1OWrXqFMcIaWVe/KBYOdWEZbA7FECeXgEZR1Sid2bH7XDUE9AYalpS2/SmuuHEU1ObL6zSpAqoY6AIPCR6LgFrtxAqrYmIdpb8YfSuI5uPBv6qikl0yvam06WNerUNQ9lnZXFmFm1wBeicRvWH3jZ6w/xlQBIp/zG6k9IJa0vaLm+FqztLkDWri8Qz1dbdsz0bNjyzD6iRuDOpgmz0Kf8m2IjaJRgRgz2ARcOOdBJKmwnnW/knk5
  ubuntu@plana33.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZv0weNolvyE+m05bRiGKWyjAbZs/nYL1/5DAdr8QYbmjTSrEjycJ2iZCHVPXHHPTkFN4vWtWQzfEE2Ra8T/Ti0w65C+H6HNwckWvLk0RRYWSFNLjfvTR+0OeCNBbpTcCBaeGpIdJhMcM9k6eek5GGm1djc7ZgG11jepzVe6HiKKbh2roc/EZGuADs8sY/bBf0cRbhsPc/1EJ2sxd8CUmnWXrGfCj5izW/1bAdyBQcAZPpMp5OJkuAT2OznMVYWxkg54JM8TlKKj1T8nccEC5+c01Dbe0vAxuIPCeU2obkxr+VvQN3oJbhUXFDYv9PCNaS0LReuVBKVN9bRYn97xXB
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
    clients:
      client.0:
      - rbd/copy.sh
ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540$ cat summary.yaml 
ceph-sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150
client.0-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66
description: collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:many.yaml
  tasks:rbd_cli_copy_old_format.yaml
duration: 794.4466111660004
failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage
  /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd
  -f -i 5 -c /tmp/cephtest/ceph.conf'''
flavor: basic
mon.a-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66
mon.b-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66
owner: scheduled_teuthology@teuthology
success: false


Related issues

Related to Ceph - Bug #3591: auth: could not find secret_id=0 Closed 12/07/2012

History

#1 Updated by Sage Weil over 11 years ago

  • Priority changed from Normal to High

#2 Updated by Dan Mick over 11 years ago

  • Project changed from rbd to Ceph

#3 Updated by Ian Colle about 11 years ago

  • Status changed from New to Resolved
  • Assignee set to Sage Weil

Resolved by fix to 3591

#4 Updated by Ian Colle about 11 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF