Bug #6469
closedteuthology locker is causing apache to use ~30% CPU
0%
Description
While testing the new teuthology-report by importing all previous test runs on the teuthology machine, I noticed that an apache process was using a large amount of cpu. It turns out it's the teuthology lock server. I peeked into /var/log/teuthology-locker/error.log, which is currently 285MB and dates back to 10/23/12.
[Thu Oct 03 08:23:33 2013] [error] Exception AttributeError: AttributeError("'_DummyThread' object has no attribute '_Thread__block'",) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
There are 408789 of these messages and counting. I found the first occurrence on 6/27/13. Conveniently, the last meaningful commit to teuthology/locker/api.py was on that date:
https://github.com/ceph/teuthology/commit/c22b941ed982c23c7a9d3b2f263302ebec945553
Then again, /locker/lock is being hit ten times per second by a python/httplib2 process on localhost. Maybe that's the cause of the high load.
Oh... access.log is 16GB and also dates back to 10/23/12.