Project

General

Profile

Bug #12081

KeyNotFoundError: Could not find keyring file: /var/lib/ceph/bootstrap-mds/ceph.keyring on hos

Added by Vasu Kulkarni about 6 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

gatherkeys doesn't work on plana and complains about misskey keyring and works on vps system which is identical 12.04 (precise) or 14.04

More logs at: http://qa-proxy.ceph.com/teuthology/teuthology-2015-06-18_05:00:05-smoke-master-distro-basic-multi/939578/teuthology.log

    2015-06-18T05:20:19.001 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] Running command: sudo ceph-mon --cluster ceph --mkfs -i plana29 --keyring /var/lib/ceph/tmp/ceph-plana29.mon.keyring
    2015-06-18T05:20:19.121 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] ceph-mon: mon.noname-a 10.214.131.11:6789/0 is local, renaming to mon.plana29
    2015-06-18T05:20:19.121 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] ceph-mon: set fsid to 687febf7-25d0-4c05-923f-c513047fb273
    2015-06-18T05:20:19.385 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] ceph-mon: created monfs at /var/lib/ceph/mon/ceph-plana29 for mon.plana29
    2015-06-18T05:20:19.385 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-plana29.mon.keyring
    2015-06-18T05:20:19.386 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] create a done file to avoid re-doing the mon deployment
    2015-06-18T05:20:19.386 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] create the init path if it does not exist
    2015-06-18T05:20:19.387 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] locating the `service` executable...
    2015-06-18T05:20:19.389 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] Running command: sudo initctl emit ceph-mon cluster=ceph id=plana29
    2015-06-18T05:20:21.415 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana29.asok mon_status
    2015-06-18T05:20:21.534 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] ********************************************************************************
    2015-06-18T05:20:21.534 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] status for monitor: mon.plana29
    2015-06-18T05:20:21.535 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] {
    2015-06-18T05:20:21.535 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "election_epoch": 2,
    2015-06-18T05:20:21.535 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "extra_probe_peers": [],
    2015-06-18T05:20:21.535 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "monmap": {
    2015-06-18T05:20:21.535 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     "created": "0.000000",
    2015-06-18T05:20:21.536 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     "epoch": 1,
    2015-06-18T05:20:21.536 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     "fsid": "687febf7-25d0-4c05-923f-c513047fb273",
    2015-06-18T05:20:21.536 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     "modified": "0.000000",
    2015-06-18T05:20:21.536 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     "mons": [
    2015-06-18T05:20:21.536 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]       {
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]         "addr": "10.214.131.11:6789/0",
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]         "name": "plana29",
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]         "rank": 0
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]       }
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     ]
    2015-06-18T05:20:21.537 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   },
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "name": "plana29",
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "outside_quorum": [],
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "quorum": [
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]     0
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   ],
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "rank": 0,
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "state": "leader",
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ]   "sync_provider": []
    2015-06-18T05:20:21.538 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] }
    2015-06-18T05:20:21.539 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] ********************************************************************************
    2015-06-18T05:20:21.539 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] monitor: mon.plana29 is running
    2015-06-18T05:20:21.539 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana29.asok mon_status
    2015-06-18T05:20:21.652 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.mon][INFO  ] processing monitor mon.plana29
    2015-06-18T05:20:21.678 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connection detected need for sudo
    2015-06-18T05:20:21.704 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connected to host: plana29
    2015-06-18T05:20:21.711 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana29.asok mon_status
    2015-06-18T05:20:21.827 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.mon][INFO  ] mon.plana29 monitor has reached quorum!
    2015-06-18T05:20:21.828 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
    2015-06-18T05:20:21.828 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.mon][INFO  ] Running gatherkeys...
    2015-06-18T05:20:21.828 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Checking plana29 for /etc/ceph/ceph.client.admin.keyring
    2015-06-18T05:20:21.856 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connection detected need for sudo
    2015-06-18T05:20:21.881 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connected to host: plana29
    2015-06-18T05:20:21.882 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect platform information from remote host
    2015-06-18T05:20:21.910 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect machine type
    2015-06-18T05:20:21.913 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] fetch remote file
    2015-06-18T05:20:21.914 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Got ceph.client.admin.keyring key from plana29.
    2015-06-18T05:20:21.915 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
    2015-06-18T05:20:21.915 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Checking plana29 for /var/lib/ceph/bootstrap-osd/ceph.keyring
    2015-06-18T05:20:21.941 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connection detected need for sudo
    2015-06-18T05:20:21.967 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connected to host: plana29
    2015-06-18T05:20:21.967 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect platform information from remote host
    2015-06-18T05:20:21.995 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect machine type
    2015-06-18T05:20:21.999 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] fetch remote file
    2015-06-18T05:20:21.999 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Got ceph.bootstrap-osd.keyring key from plana29.
    2015-06-18T05:20:22.000 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][DEBUG ] Checking plana29 for /var/lib/ceph/bootstrap-mds/ceph.keyring
    2015-06-18T05:20:22.026 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connection detected need for sudo
    2015-06-18T05:20:22.051 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] connected to host: plana29
    2015-06-18T05:20:22.052 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect platform information from remote host
    2015-06-18T05:20:22.080 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] detect machine type
    2015-06-18T05:20:22.083 INFO:teuthology.orchestra.run.plana29.stderr:[plana29][DEBUG ] fetch remote file
    2015-06-18T05:20:22.084 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy.gatherkeys][WARNING] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on plana29
    2015-06-18T05:20:22.084 INFO:teuthology.orchestra.run.plana29.stderr:[ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file: /var/lib/ceph/bootstrap-mds/ceph.keyring on host plana29
    2015-06-18T05:20:22.085 INFO:teuthology.orchestra.run.plana29.stderr:
    2015-06-18T05:20:22.110 ERROR:teuthology.contextutil:Saw exception from nested tasks
    Traceback (most recent call last):
      File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
        vars.append(enter())
      File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
        return self.gen.next()
      File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 509, in cli_test
        execute_cdeploy(admin,create_initial,path)
      File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 465, in execute_cdeploy
        raise RuntimeError ("failed during ceph-deploy cmd: {cmd} , ec={ec}".format(cmd=cmd,ec=ec))
    RuntimeError: failed during ceph-deploy cmd: --overwrite-conf mon create-initial plana29 , ec=1
    2015-06-18T05:20:22.167 INFO:tasks.ceph_deploy:Removing ceph-deploy ...
    2015-06-18T05:20:22.167 INFO:teuthology.orchestra.run.plana29:Running: 'rm -rf /home/ubuntu/cephtest/ceph-deploy'
    2015-06-18T05:20:22.210 INFO:teuthology.task.install:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits...
    2015-06-18T05:20:22.210 INFO:teuthology.orchestra.run.plana29:Running: 'sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits'
    2015-06-18T05:20:22.223 ERROR:teuthology.run_tasks:Saw exception from tasks.
    Traceback (most recent call last):
      File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
        manager.__enter__()
      File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
        return self.gen.next()
      File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 579, in single_node_test
        lambda: cli_test(ctx=ctx,config=config),
      File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
        return self.gen.next()
      File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
        vars.append(enter())
      File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
        return self.gen.next()
      File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 509, in cli_test
        execute_cdeploy(admin,create_initial,path)
      File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 465, in execute_cdeploy
        raise RuntimeError ("failed during ceph-deploy cmd: {cmd} , ec={ec}".format(cmd=cmd,ec=ec))
    RuntimeError: failed during ceph-deploy cmd: --overwrite-conf mon create-initial plana29 , ec=1

History

#1 Updated by Vasu Kulkarni about 6 years ago

No issue on mira

http://pulpito.ceph.com/teuthology-2015-06-19_05:00:05-smoke-master-distro-basic-multi/940472/

2015-06-19T05:03:15.443 DEBUG:teuthology.task.internal:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2015-06-19 12:03:14.047072', u'locked_by': u'scheduled_teuthology@teuthology', u'up': True, u'mac_address': u'00:25:90:09:e2:e8', u'name': u'mira041.front.sepia.ceph.com', u'os_version': u'14.04', u'machine_type': u'mira', u'vm_host': None, u'os_type': u'ubuntu', u'arch': u'x86_64', u'ssh_pub_key': u'ssh-rsa 

#2 Updated by Vasu Kulkarni almost 6 years ago

This is also seen now on plana with CentOS 7.1

http://qa-proxy.ceph.com/teuthology/teuthology-2015-07-01_05:00:08-smoke-master-distro-basic-multi/957005/teuthology.log

2015-07-01T05:08:19.751 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][WARNING]    D-Bus, udev, scripted systemctl call, ...).
2015-07-01T05:08:21.753 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana63.asok mon_status
2015-07-01T05:08:21.921 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] ********************************************************************************
2015-07-01T05:08:21.921 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] status for monitor: mon.plana63
2015-07-01T05:08:21.921 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] {
2015-07-01T05:08:21.924 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "election_epoch": 2,
2015-07-01T05:08:21.924 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "extra_probe_peers": [],
2015-07-01T05:08:21.925 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "monmap": {
2015-07-01T05:08:21.925 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     "created": "0.000000",
2015-07-01T05:08:21.925 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     "epoch": 1,
2015-07-01T05:08:21.925 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     "fsid": "ccf2f911-d4ae-4317-9c47-a031d919a719",
2015-07-01T05:08:21.926 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     "modified": "0.000000",
2015-07-01T05:08:21.926 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     "mons": [
2015-07-01T05:08:21.926 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]       {
2015-07-01T05:08:21.926 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]         "addr": "10.214.132.15:6789/0",
2015-07-01T05:08:21.926 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]         "name": "plana63",
2015-07-01T05:08:21.927 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]         "rank": 0
2015-07-01T05:08:21.927 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]       }
2015-07-01T05:08:21.927 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     ]
2015-07-01T05:08:21.927 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   },
2015-07-01T05:08:21.928 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "name": "plana63",
2015-07-01T05:08:21.928 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "outside_quorum": [],
2015-07-01T05:08:21.928 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "quorum": [
2015-07-01T05:08:21.928 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]     0
2015-07-01T05:08:21.928 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   ],
2015-07-01T05:08:21.929 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "rank": 0,
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "state": "leader",
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ]   "sync_provider": []
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] }
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] ********************************************************************************
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][INFO  ] monitor: mon.plana63 is running
2015-07-01T05:08:21.930 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana63.asok mon_status
2015-07-01T05:08:22.093 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.mon][INFO  ] processing monitor mon.plana63
2015-07-01T05:08:22.128 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] connection detected need for sudo
2015-07-01T05:08:22.165 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] connected to host: plana63
2015-07-01T05:08:22.176 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][INFO  ] Running command: sudo ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.plana63.asok mon_status
2015-07-01T05:08:22.343 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.mon][INFO  ] mon.plana63 monitor has reached quorum!
2015-07-01T05:08:22.344 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
2015-07-01T05:08:22.344 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.mon][INFO  ] Running gatherkeys...
2015-07-01T05:08:22.344 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.gatherkeys][DEBUG ] Have ceph.client.admin.keyring
2015-07-01T05:08:22.344 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.gatherkeys][DEBUG ] Have ceph.mon.keyring
2015-07-01T05:08:22.345 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.gatherkeys][DEBUG ] Have ceph.bootstrap-osd.keyring
2015-07-01T05:08:22.345 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.gatherkeys][DEBUG ] Checking plana63 for /var/lib/ceph/bootstrap-mds/ceph.keyring
2015-07-01T05:08:22.379 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] connection detected need for sudo
2015-07-01T05:08:22.410 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] connected to host: plana63
2015-07-01T05:08:22.411 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] detect platform information from remote host
2015-07-01T05:08:22.450 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] detect machine type
2015-07-01T05:08:22.454 INFO:teuthology.orchestra.run.plana63.stderr:[plana63][DEBUG ] fetch remote file
2015-07-01T05:08:22.455 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy.gatherkeys][WARNING] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on plana63
2015-07-01T05:08:22.456 INFO:teuthology.orchestra.run.plana63.stderr:[ceph_deploy][ERROR ] KeyNotFoundError: Could not find keyring file: /var/lib/ceph/bootstrap-mds/ceph.keyring on host plana63
2015-07-01T05:08:22.456 INFO:teuthology.orchestra.run.plana63.stderr:
2015-07-01T05:08:22.470 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 509, in cli_test
    execute_cdeploy(admin,create_initial,path)
  File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/ceph_deploy.py", line 465, in execute_cdeploy
    raise RuntimeError ("failed during ceph-deploy cmd: {cmd} , ec={ec}".format(cmd=cmd,ec=ec))
RuntimeError: failed during ceph-deploy cmd: --overwrite-conf mon create-initial plana63 , ec=1
2015-07-01T05:08:22.491 INFO:tasks.ceph_deploy:Removing ceph-deploy ...
2015-07-01T05:08:22.491 INFO:teuthology.orchestra.run.plana63:Running: 'rm -rf /home/ubuntu/cephtest/ceph-deploy'
2015-07-01T05:08:22.625 INFO:teuthology.task.install:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits...
2015-07-01T05:08:22.625 INFO:teuthology.orchestra.run.plana63:Running: 'sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits'

#3 Updated by Travis Rhoden almost 6 years ago

This has every indication of being a race condition. It's intermittent. I just hopped onto a machine that just hit this, and found that the key was actually there!

The node I hopped on was actually running a single node test -- this is fortuitous because it means that there is only one node to deal with and the timestamps are reliable. Here is what I see:

From the Teuthology log:

2015-07-24 14:09:21,326.326 INFO:teuthology.orchestra.run.magna074.stderr:[ceph_deploy.gatherkeys][DEBUG ] Checking magna074 for /var/lib/ceph/bootstrap-mds/ceph.keyring
2015-07-24 14:09:21,345.345 INFO:teuthology.orchestra.run.magna074.stderr:[magna074][DEBUG ] connection detected need for sudo
2015-07-24 14:09:21,370.370 INFO:teuthology.orchestra.run.magna074.stderr:[magna074][DEBUG ] connected to host: magna074
2015-07-24 14:09:21,371.371 INFO:teuthology.orchestra.run.magna074.stderr:[magna074][DEBUG ] detect platform information from remote host
2015-07-24 14:09:21,390.390 INFO:teuthology.orchestra.run.magna074.stderr:[magna074][DEBUG ] detect machine type
2015-07-24 14:09:21,393.393 INFO:teuthology.orchestra.run.magna074.stderr:[magna074][DEBUG ] fetch remote file
2015-07-24 14:09:21,394.394 INFO:teuthology.orchestra.run.magna074.stderr:[ceph_deploy.gatherkeys][WARNING] Unable to find /var/lib/ceph/bootstrap-mds/ceph.keyring on magna074

But here is the timestamp on the file:

ubuntu@magna074:/var/lib/ceph/bootstrap-mds$ ls -la --time-style=full-iso *
-rw------- 1 root root 71 2015-07-24 14:09:21.419978318 -0400 ceph.keyring

Not that the time on the file is about 25 milliseconds after ceph-deploy says it couldn't find it.

I'll have to look a bit more at what creates the keyrings, and what we are keying off of to go look for them. I'd like to avoid fixing this with a sleep or anything like that.

#4 Updated by Travis Rhoden almost 6 years ago

  • Status changed from New to 12
  • Assignee set to Travis Rhoden
  • Target version set to 1.5.27
  • Source changed from other to Q/A

#5 Updated by Travis Rhoden almost 6 years ago

Seen again today, found in a completely different state. No keys were present at all, but the monitor was running.

sudo ceph-create-keys -v --id `hostname -s`
INFO:ceph-create-keys:Key exists already: /etc/ceph/ceph.client.admin.keyring
INFO:ceph-create-keys:Talking to monitor...
2015-07-28 17:27:56.961107 7f26d5cd0700  0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
INFO:ceph-create-keys:Cannot get or create bootstrap key for osd, permission denied
INFO:ceph-create-keys:Talking to monitor...
2015-07-28 17:27:57.072085 7fdc8ba43700  0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
INFO:ceph-create-keys:Cannot get or create bootstrap key for rgw, permission denied
INFO:ceph-create-keys:Talking to monitor...
2015-07-28 17:27:57.181655 7f0e87faa700  0 librados: client.admin authentication error (1) Operation not permitted
Error connecting to cluster: PermissionError
INFO:ceph-create-keys:Cannot get or create bootstrap key for mds, permission denied

This indicates that the admin key isn't good for the cluster.

removing /etc/ceph/ceph.client.admin.keyring and re-running ceph-create-keys fixed things, but it makes me wonder if there is a scenario where the admin key gathered by ceph-deploy does not match what gets generated by the monitors? Or maybe something is overwriting the key somewhere? very strange.

#6 Updated by Travis Rhoden almost 6 years ago

  • Target version changed from 1.5.27 to 1.5.28

#7 Updated by Travis Rhoden almost 6 years ago

  • Target version changed from 1.5.28 to 1.5.29

#8 Updated by Patrick Donnelly over 1 year ago

  • Status changed from 12 to New

Also available in: Atom PDF