CI

The database got upgraded and caused this:

 [2017-11-16 06:29:42,659: ERROR/ForkPoolWorker-1] Task async.check_orphaned[5a404a78-faeb-4877-99ed-07e01c0e14a4] raised unexpected: OperationalError('(psycopg2.OperationalError) terminating connection due to administrator command        \nserver closed the connection unexpectedly\n\tThis probably means the server terminated abnormally\n\tbefore or while processing the request.\n',)
4524552 Traceback (most recent call last):
4524553   File "/opt/mita/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
4524554     R = retval = fun(*args, **kwargs)
4524555   File "/opt/mita/local/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
4524556     return self.run(*args, **kwargs)
4524557   File "/opt/mita/src/mita/mita/async.py", line 254, in check_orphaned
4524558     nodes = models.Node.query.all()
4524559   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2703, in all
4524560     return list(self)
4524561   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
4524562     return self._execute_and_instances(context)
4524563   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
4524564     result = conn.execute(querycontext.statement, self._params)
4524565   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
4524566     return meth(self, multiparams, params)
4524567   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
4524568     return connection._execute_clauseelement(self, multiparams, params)
4524569   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
4524570     compiled_sql, distilled_params
4524571   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1189, in _execute_context
4524572     context)
4524573   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
4524574     exc_info
4524575   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
4524576     reraise(type(exception), exception, tb=exc_tb, cause=cause)
4524577   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1182, in _execute_context
4524578     context)
4524579   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 470, in do_execute
4524580     cursor.execute(statement, parameters)
4524581 OperationalError: (psycopg2.OperationalError) terminating connection due to administrator command
4524582 server closed the connection unexpectedly
4524583         This probably means the server terminated abnormally
4524584         before or while processing the request.
4524585  [SQL: 'SELECT nodes.id AS nodes_id, nodes.name AS nodes_name, nodes.created AS nodes_created, nodes.keyname AS nodes_keyname, nodes.image_name AS nodes_image_name, nodes.size AS nodes_size, nodes.identifier AS nodes_identifier, n        odes.idle_since AS nodes_idle_since, nodes.provider AS nodes_provider \nFROM nodes']

The side-effect meant sqlalchemy refusing to do anything:

4524736 [2017-11-16 06:33:42,868: ERROR/ForkPoolWorker-2] Task async.check_orphaned[4506d5a0-0826-4376-9a65-06743783bc65] raised unexpected: StatementError("(sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is         rolled back",)
4524737 Traceback (most recent call last):
4524738   File "/opt/mita/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
4524739     R = retval = fun(*args, **kwargs)
4524740   File "/opt/mita/local/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
4524741     return self.run(*args, **kwargs)
4524742   File "/opt/mita/src/mita/mita/async.py", line 254, in check_orphaned
4524743     nodes = models.Node.query.all()
4524744   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2703, in all
4524745     return list(self)
4524746   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2855, in __iter__
4524747     return self._execute_and_instances(context)
4524748   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in _execute_and_instances
4524749     result = conn.execute(querycontext.statement, self._params)
4524750   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 945, in execute
4524751     return meth(self, multiparams, params)
4524752   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
4524753     return connection._execute_clauseelement(self, multiparams, params)
4524754   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
4524755     compiled_sql, distilled_params
4524756   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
4524757     None, None)
4524758   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
4524759     exc_info
4524760   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
4524761     reraise(type(exception), exception, tb=exc_tb, cause=cause)
4524762   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1114, in _execute_context
4524763     conn = self._revalidate_connection()
4524764   File "/opt/mita/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 424, in _revalidate_connection
4524765     "Can't reconnect until invalid " 
4524766 StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back [SQL: u'SELECT nodes.id AS nodes_id, nodes.name AS nodes_name, nodes.created AS nodes_created, nodes.keyname AS nodes_ke        yname, nodes.image_name AS nodes_image_name, nodes.size AS nodes_size, nodes.identifier AS nodes_identifier, nodes.idle_since AS nodes_idle_since, nodes.provider AS nodes_provider \nFROM nodes'] [parameters: [{}]]

This means that a rollback() needs to be done here by mita when these pop up