Project

General

Profile

Actions
Copy link

Bug #10651

closed

stale pycs causing issues

Added by Andrew Schoen over 9 years ago. Updated almost 9 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Andrew Schoen
Category:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

We saw this failure after merging new code to master on this run and others: http://qa-proxy.ceph.com/teuthology/teuthology-2015-01-25_23:00:02-rbd-next-distro-basic-multi/722486/teuthology.log

Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 53, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 35, in run_one_task
    task = import_task(submod)
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 15, in import_task
    0)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/kernel.py", line 16, in <module>
    from ..exceptions import UnsupportedPackageTypeError, ConfigError
ImportError: cannot import name ConfigError

This is curious because the same commit that introduces ConfigError is the one that uses it, https://github.com/ceph/teuthology/commit/ec81c6bdd11caf7b8e6781e3bd547b35f75ce597

It seems as though a relative import does not create a new pyc file. A new pyc for kernel.py and a stale one for exceptions.py is the only explanation that makes sense for this bug.

We have a function already in repo_utils.py to remove pycs on a teuthology checkout, we need to reenable that.

Actions
Copy link
 #1

Updated by Andrew Schoen over 9 years ago

This issue seemed to resolve itself. As a precaution we're going to set PYTHONDONTWRITEBYTECODE=1 in the teuthology code that runs the jobs so pycs won't be created anymore. Once that is set and is proven to not be causing errors we'll delete all the existing pycs in the labs.

However, I'm starting to suspect that this issue was caused by clock skew. In my research on this I saw instances where people saw this behavior when ntp was acting up on their box and python couldn't correctly determine if it should generate a new pyc or not.

Actions
Copy link
 #2

Updated by Andrew Schoen over 9 years ago

  • Status changed from New to Need More Info

This seems to have been related to ntp problems in the lab (#10675). The clock had drifted enough that python couldn't decide correctly on if to create a new pyc or not and used an old stale pyc when it should not have.

We're gonna wait and see if this happens again before making any teuthology changes related to clearing or not creating pycs.

Actions
Copy link
 #3

Updated by Andrew Schoen about 9 years ago

  • Status changed from Need More Info to Closed

Yeah, this was related to clock skew. Closing this now as we've resolved those issues for now.

Actions
Copy link
 #4

Updated by Alfredo Deza almost 9 years ago

  • Regression set to No

A tiny bit of extra info just in case this causes trouble down the road: if PYTHONDONTWRITEBYTECODE=1 is set it can cause issues when installing Python packages as some
depend on that feature:

New python executable in test/bin/python
Installing setuptools............done.
Installing pip.............
  Complete output from command /Users/alfredo/test/bin/python /Users/alfredo/test/bin/easy_install /Library/Python/2.7/...ort/pip-1.0.2.tar.gz:
  Processing pip-1.0.2.tar.gz
Running pip-1.0.2/setup.py -q bdist_egg --dist-dir /var/folders/61/08st3n995qn_w6rsjqgv1cx00000gn/T/easy_install-Za9IA8/pip-1.0.2/egg-dist-tmp-lyVJEQ
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'
warning: build_py: byte-compiling is disabled, skipping.

warning: install_lib: byte-compiling is disabled, skipping.

error: byte-compiling is disabled.
Actions
Copy link

Also available in: Atom PDF