Actions
Bug #3563
closedosd crashed with error "auth: could not find secret_id=2"
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Test : collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:many.yaml tasks:rbd_cli_copy_old_format.yaml
Log: ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540
an excerpt from the osd log: 2012-11-30 09:38:12.112440 7fc4d678a700 0 monclient: hunting for new mon 2012-11-30 09:38:13.055487 7fc4d3481700 0 auth: could not find secret_id=2 2012-11-30 09:38:13.055499 7fc4d3481700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:13.055504 7fc4d3481700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:13.055507 7fc4d3481700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x23fe6c0 sd=29 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:13.056119 7fc4d3380700 0 auth: could not find secret_id=2 2012-11-30 09:38:13.056137 7fc4d3380700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:13.056143 7fc4d3380700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:13.056146 7fc4d3380700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x23feb40 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:13.056926 7fc4d3380700 0 auth: could not find secret_id=2 2012-11-30 09:38:13.056939 7fc4d3380700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 ..... .... 2012-11-30 09:38:16.404413 7fc4d2b78700 0 auth: could not find secret_id=2 1/ 5 rgw2012-11-30 09:38:16.404427 7fc4d2b78700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.404432 7fc4d2b78700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 1/ 5 hadoop 1/ 5 asok 1/ 5 throttle -2/-2 (syslog threshold)2012-11-30 09:38:16.404540 7fc4d2976700 0 auth: could not find secret_id=2 2012-11-30 09:38:16.404435 7fc4d2b78700 0 -1/-1 (stderr threshold)-- 10.214.131.10:6804/29058 >> 10.214.131.7:6807/1896 pipe(0x280d000 sd=32 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:16.404550 7fc4d2976700 0 max_recent 10000cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.404555 7fc4d2976700 1 max_new 1000000CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:16.404557 7fc4d2976700 0 log_file /tmp/cephtest/archive/log/osd.5.log-- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x280dd80 sd=33 :6804 pgs=0 cs=0 l=0).accept bad authorizer --- end dump of recent events --- 2012-11-30 09:38:16.404763 7fc4d2a77700 0 auth: could not find secret_id=2 2012-11-30 09:38:16.404771 7fc4d2a77700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.404774 7fc4d2a77700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:16.404776 7fc4d2a77700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6804/1859 pipe(0x280d900 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:16.405087 7fc4d2b78700 0 auth: could not find secret_id=2 2012-11-30 09:38:16.405094 7fc4d2b78700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.405094 7fc4d2976700 0 auth: could not find secret_id=2 2012-11-30 09:38:16.405099 7fc4d2b78700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:16.405102 7fc4d2976700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.405106 7fc4d2976700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:16.405109 7fc4d2976700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.10:6801/29056 pipe(0x2758900 sd=32 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:16.405104 7fc4d2b78700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6807/1896 pipe(0x2758d80 sd=31 :6804 pgs=0 cs=0 l=0).accept bad authorizer 2012-11-30 09:38:16.405183 7fc4d2a77700 0 auth: could not find secret_id=2 2012-11-30 09:38:16.405189 7fc4d2a77700 0 cephx: verify_authorizer could not get service secret for service osd secret_id=2 2012-11-30 09:38:16.405192 7fc4d2a77700 1 CephxAuthorizeHandler::verify_authorizer isvalid=0 2012-11-30 09:38:16.405195 7fc4d2a77700 0 -- 10.214.131.10:6804/29058 >> 10.214.131.7:6804/1859 pipe(0x2758b40 sd=30 :6804 pgs=0 cs=0 l=0).accept bad authorizer (END) ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540$ cat config.yaml kernel: &id001 kdb: true sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66 nuke-on-error: true overrides: ceph: conf: global: ms inject socket failures: 500 fs: btrfs log-whitelist: - slow request sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150 s3tests: branch: testing workunit: sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150 roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - - client.0 targets: ubuntu@plana28.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnMB6Wmqywp9d2Ns8PYeySKtDaQCkkfldLWV57qoMTDC9+ZoIJcSUE9OshK8pwO7C14cU+IiFIjY4a/fctylCuopSnneI7tsTq2er/Te2XOA+4Q1L4K2twa0pTwTAPyqqsZODqM4O/QG/voJjKMvrazp85VsmQ7mM1bOdR4wCNbPZoydg1/iO7cxSx8iSb2LdWTx+mRFSrWzHWhqvxwfHDZXtim3U/dej+7dFDZrM8Cq1PsvOXkE+zEkmSAMU64JUPZBTSdYl8Dh1aMMz+BWfIn5ZUFzcKt/7LLims85X4rBGyp1CAmhyTtTDBmNrvemZI6BEl+xA1kamhG+Cp/Z+n ubuntu@plana30.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYDpptyOTrVH3RqiwH5A//Q2CkkVz5dPTpd/s8qG/Q4EHVA4WDMu80pcDvdSewOfFJl83MEtDKKjuJOuEzI4OGn0DPptDN5wHC1OWrXqFMcIaWVe/KBYOdWEZbA7FECeXgEZR1Sid2bH7XDUE9AYalpS2/SmuuHEU1ObL6zSpAqoY6AIPCR6LgFrtxAqrYmIdpb8YfSuI5uPBv6qikl0yvam06WNerUNQ9lnZXFmFm1wBeicRvWH3jZ6w/xlQBIp/zG6k9IJa0vaLm+FqztLkDWri8Qz1dbdsz0bNjyzD6iRuDOpgmz0Kf8m2IjaJRgRgz2ARcOOdBJKmwnnW/knk5 ubuntu@plana33.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZv0weNolvyE+m05bRiGKWyjAbZs/nYL1/5DAdr8QYbmjTSrEjycJ2iZCHVPXHHPTkFN4vWtWQzfEE2Ra8T/Ti0w65C+H6HNwckWvLk0RRYWSFNLjfvTR+0OeCNBbpTcCBaeGpIdJhMcM9k6eek5GGm1djc7ZgG11jepzVe6HiKKbh2roc/EZGuADs8sY/bBf0cRbhsPc/1EJ2sxd8CUmnWXrGfCj5izW/1bAdyBQcAZPpMp5OJkuAT2OznMVYWxkg54JM8TlKKj1T8nccEC5+c01Dbe0vAxuIPCeU2obkxr+VvQN3oJbhUXFDYv9PCNaS0LReuVBKVN9bRYn97xXB tasks: - internal.lock_machines: 3 - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.syslog: null - internal.timer: null - chef: null - clock: null - ceph: null - ceph-fuse: null - workunit: clients: client.0: - rbd/copy.sh ubuntu@teuthology:/a/teuthology-2012-11-30_07:00:03-regression-testing-master-basic/6540$ cat summary.yaml ceph-sha1: 60b84b095b1009a305d4d6a5b16f88571cbd3150 client.0-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66 description: collection:rbd-basic clusters:fixed-3.yaml fs:btrfs.yaml msgr-failures:many.yaml tasks:rbd_cli_copy_old_format.yaml duration: 794.4466111660004 failure_reason: 'Command failed with status 1: ''/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/daemon-helper kill /tmp/cephtest/binary/usr/local/bin/ceph-osd -f -i 5 -c /tmp/cephtest/ceph.conf''' flavor: basic mon.a-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66 mon.b-kernel-sha1: 35152979e6181b1fbb4b61c3ff641c14df53ad66 owner: scheduled_teuthology@teuthology success: false
Updated by Ian Colle over 11 years ago
- Status changed from New to Resolved
- Assignee set to Sage Weil
Resolved by fix to 3591
Actions