Project

General

Profile

Fix #7380

Diamond not getting set up after bootstrap

Added by John Spray over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Backend (packaging/deployment)
Target version:
% Done:

0%

Source:
other
Tags:
Backport:
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

(wip-2.0 HEAD) This is weird, and possibly a salt bug.

We have a reactor (start.sls) for syncing modules and invoking state.highstate when a minion starts. state.highstate should be setting up diamond, and when run by hand with "salt-call state.highstate" does exactly that.

What's actually happening on minion startup is that saltutil.sync_modules is getting run, but highstate seems not to be run, although I can see the top.sls template getting rendered.

Debug log from a minion starting up is below:

[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/calamari.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/calamari.conf
[INFO    ] Using cached minion ID from /etc/salt/minion_id: mira002.front.sepia.ceph.com
[DEBUG   ] loading log_handlers in ['/var/cache/salt/minion/extmods/log_handlers', '/usr/lib/pymodules/python2.7/salt/log/handlers']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/log_handlers, it is not a directory
[DEBUG   ] Configuration file path: /etc/salt/minion
[INFO    ] Setting up the Salt Minion "mira002.front.sepia.ceph.com" 
[DEBUG   ] Created pidfile: /var/run/salt-minion.pid
[DEBUG   ] Chowned pidfile: /var/run/salt-minion.pid to user: root
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/calamari.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/calamari.conf
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/pymodules/python2.7/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] Attempting to authenticate with the Salt Master at 10.214.134.132
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Authentication with master successful!
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/calamari.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/calamari.conf
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/pymodules/python2.7/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] loading module in ['/var/cache/salt/minion/extmods/modules', '/usr/lib/pymodules/python2.7/salt/modules']
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded apt as virtual pkg
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded upstart as virtual service
[DEBUG   ] Loaded htpasswd as virtual webutil
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded dpkg as virtual lowpkg
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[DEBUG   ] loading returner in ['/var/cache/salt/minion/extmods/returners', '/usr/lib/pymodules/python2.7/salt/returners']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/returners, it is not a directory
[DEBUG   ] Loaded syslog_return as virtual syslog
[DEBUG   ] Loaded couchdb_return as virtual couchdb
[DEBUG   ] Loaded carbon_return as virtual carbon
[DEBUG   ] Loaded sqlite3_return as virtual sqlite3
[INFO    ] Minion is starting as user 'root'
[DEBUG   ] Minion "mira002.front.sepia.ceph.com" trying to tune in
[DEBUG   ] Minion PUB socket URI: ipc:///var/run/salt/minion/minion_event_b4207aaa2d31ed6468c95149ddb595c4_pub.ipc
[DEBUG   ] Minion PULL socket URI: ipc:///var/run/salt/minion/minion_event_b4207aaa2d31ed6468c95149ddb595c4_pull.ipc
[DEBUG   ] Setting zmq_reconnect_ivl to '100ms'
[DEBUG   ] Setting zmq_reconnect_ivl_max to '5100ms'
[DEBUG   ] Running scheduled job: ceph.heartbeat
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: This job was scheduled with a max number of 1
[DEBUG   ] Running scheduled job: __mine_interval
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun': 'ceph.heartbeat', 'jid': '20140210092309477868', 'pid': 9621, 'id': 'mira002.front.sepia.ceph.com'}
[DEBUG   ] schedule: This job was scheduled with a max number of 2
[DEBUG   ] schedule.handle_func: Checking job against fun mine.update: {'fun': 'ceph.heartbeat', 'jid': '20140210092309477868', 'pid': 9621, 'id': 'mira002.front.sepia.ceph.com'}
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun': 'mine.update', 'jid': '20140210092309484432', 'pid': 9624, 'id': 'mira002.front.sepia.ceph.com'}
[WARNING ] /usr/lib/pymodules/python2.7/salt/minion.py:594: DeprecationWarning: Master pub message signing is disabled but we received a signature for this message.  Most likely this means that your masters and minions are not the same version.  After Salt 0.17.6 this situation will throw an exception.
  salt.utils.warn_until((0, 17, 6), 'Master pub message signing is disabled but we '

[INFO    ] User root Executing command saltutil.sync_modules with jid 20140210092309019467
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20140210092309019467', 'tgt': 'mira002.front.sepia.ceph.com', 'ret': '', 'user': 'root', 'arg': [], 'fun': 'saltutil.sync_modules'}
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Parsed args: []
[DEBUG   ] Parsed kwargs: {}
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Including configuration from '/etc/salt/minion.d/calamari.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/calamari.conf
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/pymodules/python2.7/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
{'fun': 'ceph.heartbeat', 'return': ({'ceph-osd.4': {'cluster': 'ceph', 'type': 'osd', 'id': '4', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-mon.mira002': {'cluster': 'ceph', 'type': 'mon', 'id': 'mira002', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-osd.3': {'cluster': 'ceph', 'type': 'osd', 'id': '3', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-osd.2': {'cluster': 'ceph', 'type': 'osd', 'id': '2', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}}, {u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36': {'name': 'ceph', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36', 'versions': {'pg_brief': '937f44a37e51f6be62c4ae0e21ee1324', 'mds_map': 1, 'osd_map': 33, 'mon_status': 8, 'health': 'dd39f7e34a34092cc36c1fb763b2e63b', 'mon_map': 1, 'config': '46fcf73ec37abea8129c3d578e4b4200'}}}), 'jid': '20140210092309477868', 'pid': 9621, 'id': 'mira002.front.sepia.ceph.com'}
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] loading module in ['/var/cache/salt/minion/extmods/modules', '/usr/lib/pymodules/python2.7/salt/modules']
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded apt as virtual pkg
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded upstart as virtual service
[DEBUG   ] Loaded htpasswd as virtual webutil
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded dpkg as virtual lowpkg
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[DEBUG   ] loading states in ['/var/cache/salt/minion/extmods/states', '/usr/lib/pymodules/python2.7/salt/states']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/states, it is not a directory
[DEBUG   ] Loaded saltmod as virtual salt
[DEBUG   ] Loaded pip_state as virtual pip
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] loading render in ['/var/cache/salt/minion/extmods/renderers', '/usr/lib/pymodules/python2.7/salt/renderers']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/renderers, it is not a directory
[DEBUG   ] loading module in ['/var/cache/salt/minion/extmods/modules', '/usr/lib/pymodules/python2.7/salt/modules']
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded apt as virtual pkg
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded upstart as virtual service
[DEBUG   ] Loaded htpasswd as virtual webutil
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded dpkg as virtual lowpkg
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[INFO    ] Fetching file ** skipped **, latest already in cache 'salt://top.sls'
[DEBUG   ] Jinja search path: '['/var/cache/salt/minion/files/base']'
[DEBUG   ] Rendered data from file: /var/cache/salt/minion/files/base/top.sls:
base:
    '*':
        - diamond

[DEBUG   ] Results of YAML rendering: 
OrderedDict([('base', OrderedDict([('*', ['diamond'])]))])
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Syncing modules for environment 'base'
[INFO    ] Loading cache from salt://_modules, for base)
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[INFO    ] Caching directory '_modules/' for environment 'base'
[INFO    ] Fetching file ** skipped **, latest already in cache 'salt://_modules/ceph.py'
[INFO    ] Fetching file ** skipped **, latest already in cache 'salt://_modules/log_tail.py'
[DEBUG   ] Local cache dir: '/var/cache/salt/minion/files/base/_modules'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/ceph.py' to '/var/cache/salt/minion/extmods/modules/ceph.py'
[INFO    ] Copying '/var/cache/salt/minion/files/base/_modules/log_tail.py' to '/var/cache/salt/minion/extmods/modules/log_tail.py'
[DEBUG   ] MinionEvent PUB socket URI: ipc:///var/run/salt/minion/minion_event_b4207aaa2d31ed6468c95149ddb595c4_pub.ipc
[DEBUG   ] MinionEvent PULL socket URI: ipc:///var/run/salt/minion/minion_event_b4207aaa2d31ed6468c95149ddb595c4_pull.ipc
[INFO    ] Returning information for job: 20140210092309019467
[DEBUG   ] Reading configuration from /etc/salt/minion
[DEBUG   ] Including configuration from '/etc/salt/minion.d/calamari.conf'
[DEBUG   ] Reading configuration from /etc/salt/minion.d/calamari.conf
[DEBUG   ] loading grain in ['/var/cache/salt/minion/extmods/grains', '/usr/lib/pymodules/python2.7/salt/grains']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/grains, it is not a directory
[DEBUG   ] loading module in ['/var/cache/salt/minion/extmods/modules', '/usr/lib/pymodules/python2.7/salt/modules']
[DEBUG   ] Loaded localemod as virtual locale
[DEBUG   ] Loaded groupadd as virtual group
[DEBUG   ] Loaded linux_sysctl as virtual sysctl
[DEBUG   ] Loaded parted as virtual partition
[DEBUG   ] Loaded linux_acl as virtual acl
[DEBUG   ] Loaded sysmod as virtual sys
[DEBUG   ] Loaded apt as virtual pkg
[DEBUG   ] Loaded djangomod as virtual django
[DEBUG   ] Loaded upstart as virtual service
[DEBUG   ] Loaded htpasswd as virtual webutil
[DEBUG   ] Loaded useradd as virtual user
[DEBUG   ] Loaded dpkg as virtual lowpkg
[DEBUG   ] Loaded debconfmod as virtual debconf
[DEBUG   ] Loaded virtualenv_mod as virtual virtualenv
[DEBUG   ] Loaded cmdmod as virtual cmd
[DEBUG   ] loading returner in ['/var/cache/salt/minion/extmods/returners', '/usr/lib/pymodules/python2.7/salt/returners']
[DEBUG   ] Skipping /var/cache/salt/minion/extmods/returners, it is not a directory
[DEBUG   ] Loaded syslog_return as virtual syslog
[DEBUG   ] Loaded couchdb_return as virtual couchdb
[DEBUG   ] Loaded carbon_return as virtual carbon
[DEBUG   ] Loaded sqlite3_return as virtual sqlite3
[DEBUG   ] Running scheduled job: ceph.heartbeat
[DEBUG   ] schedule: This job was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: This job was scheduled with a max number of 1
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'fun': 'ceph.heartbeat', 'jid': '20140210092339995746', 'pid': 9797, 'id': 'mira002.front.sepia.ceph.com'}
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Decrypting the current master AES key
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
[DEBUG   ] Loaded minion key: /etc/salt/pki/minion/minion.pem
{'fun': 'ceph.heartbeat', 'return': ({'ceph-osd.4': {'cluster': 'ceph', 'type': 'osd', 'id': '4', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-mon.mira002': {'cluster': 'ceph', 'type': 'mon', 'id': 'mira002', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-osd.3': {'cluster': 'ceph', 'type': 'osd', 'id': '3', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}, 'ceph-osd.2': {'cluster': 'ceph', 'type': 'osd', 'id': '2', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36'}}, {u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36': {'name': 'ceph', 'fsid': u'f10a5c02-9407-4a2b-af4d-8d85cc1bcf36', 'versions': {'pg_brief': 'ce8bd3309e9bf1066c2a95ccba6f3d42', 'mds_map': 1, 'osd_map': 41, 'mon_status': 8, 'health': 'abb3b1e0fd451dd9452a82452615040e', 'mon_map': 1, 'config': '46fcf73ec37abea8129c3d578e4b4200'}}}), 'jid': '20140210092339995746', 'pid': 9797, 'id': 'mira002.front.sepia.ceph.com'}

Associated revisions

Revision cad70dcf (diff)
Added by John Spray over 7 years ago

salt: Fix reactor declaration

Turns out it just wasn't legal to specify two commands in one
'chunk', but two chunks is fine. The diamond install part
gets called now.

Fixes: #7380

History

#1 Updated by John Spray over 7 years ago

  • Target version changed from v1.2 Backlog to v1.2-dev4

#2 Updated by John Spray over 7 years ago

  • Status changed from New to Fix Under Review

#3 Updated by John Spray over 7 years ago

  • Assignee set to John Spray

#4 Updated by Christina Meno over 7 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF