Bug #1935
teuthology: readwrite/roundtrip jobs run manually, but not in suite
0%
Description
I'm not sure what's going on here.. roundtrip and readwrite are failing when scheduled, but when i copy/paste the same yaml and run ti manually, they work just fine. The errors look like
2012-01-12T17:53:18.292 DEBUG:teuthology.orchestra.run:Running: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /tmp/cephtest/archive/s3readwrite.client.0.config.yaml'
2012-01-12T17:53:18.361 DEBUG:teuthology.orchestra.run:Running: 'cat -- /tmp/cephtest/archive/s3readwrite.client.0.config.yaml'
2012-01-12T17:53:18.407 DEBUG:teuthology.orchestra.run:Running: '/tmp/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite'
2012-01-12T17:53:18.694 INFO:teuthology.orchestra.run.err:Traceback (most recent call last):
2012-01-12T17:53:18.695 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite", line 9, in <module>
2012-01-12T17:53:18.695 INFO:teuthology.orchestra.run.err: load_entry_point('s3tests==0.0.1', 'console_scripts', 's3tests-test-readwrite')()
2012-01-12T17:53:18.695 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/s3tests/readwrite.py", line 131, in main
2012-01-12T17:53:18.696 INFO:teuthology.orchestra.run.err: config = common.read_config(sys.stdin)
2012-01-12T17:53:18.696 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/s3tests/common.py", line 83, in read_config
2012-01-12T17:53:18.696 INFO:teuthology.orchestra.run.err: for new in g:
2012-01-12T17:53:18.696 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/__init__.py", line 83, in load_all
2012-01-12T17:53:18.696 INFO:teuthology.orchestra.run.err: yield loader.get_data()
2012-01-12T17:53:18.697 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 33, in get_data
2012-01-12T17:53:18.697 INFO:teuthology.orchestra.run.err: return self.construct_document(self.get_node())
2012-01-12T17:53:18.697 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 48, in construct_document
2012-01-12T17:53:18.697 INFO:teuthology.orchestra.run.err: for dummy in generator:
2012-01-12T17:53:18.699 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 398, in construct_yaml_map
2012-01-12T17:53:18.699 INFO:teuthology.orchestra.run.err: value = self.construct_mapping(node)
2012-01-12T17:53:18.699 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 208, in construct_mapping
2012-01-12T17:53:18.699 INFO:teuthology.orchestra.run.err: return BaseConstructor.construct_mapping(self, node, deep=deep)
2012-01-12T17:53:18.700 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 133, in construct_mapping
2012-01-12T17:53:18.700 INFO:teuthology.orchestra.run.err: value = self.construct_object(value_node, deep=deep)
2012-01-12T17:53:18.700 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 88, in construct_object
2012-01-12T17:53:18.700 INFO:teuthology.orchestra.run.err: data = constructor(self, node)
2012-01-12T17:53:18.700 INFO:teuthology.orchestra.run.err: File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/constructor.py", line 414, in construct_undefined
2012-01-12T17:53:18.701 INFO:teuthology.orchestra.run.err: node.start_mark)
2012-01-12T17:53:18.701 INFO:teuthology.orchestra.run.err:yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:python/unicode'
2012-01-12T17:53:18.701 INFO:teuthology.orchestra.run.err: in "<stdin>", line 14, column 9
That's from ubuntu@teuthology:/var/lib/teuthworker/archive/wip-backfill-2012-01-12_13:09:04/7281 (happening on master too)
History
#1 Updated by Anonymous almost 12 years ago
s3tests/functional (run via nosetests) reads a .ini -style configuration.
This was not flexible enough for all the needs the benchmarks (readwrite, roundtrip) had, so we switched those to a yaml config format. There's a sample in config.yaml.SAMPLE.
At some point, s3tests/functional should be ported to read the newer, more flexible format also. This wasn't seen as critical enough to warrant work at the time, but clearly it would help with the confusion.
#2 Updated by Anonymous almost 12 years ago
More specific plan for change:
- change s3tests/functional/__init__.py to read env var S3TEST_YAML, raise if it is not there (explicit error to prompt users to migrate)
- make it read the configuration via s3tests/common.py read_config(), etc; share more code between them
- there seems to be a lot of copy-paste code between s3tests/common.py and s3tests/functional/__init__.py, clean up:
- s3tests/common.py setup() seems only used by s3tests/fuzz (and the debug runner in s3tests/generate_objects.py), refactor for clarity so it's not confused with s3tests/functional/__init__.py setup()
- choose_bucket_prefix
- nuke_prefixed_buckets
- get_new_bucket
#3 Updated by Anonymous almost 12 years ago
I'm wrong, ignore me for a while.
#4 Updated by Sage Weil almost 12 years ago
- Assignee set to Anonymous
#5 Updated by Anonymous almost 12 years ago
- Status changed from New to Resolved
Alright, I was reading the wrong file. s3readwrite.py and s3roundtrip.py do use the yaml format. The above cleanups would still be valid.
Bug found & squished (still needs to be deployed on teuthworker I think):
commit 3bfa41cf6ad945c21cf8499f94dd71cf07b3cb04
Author: Tommi Virtanen <tommi.virtanen@dreamhost.com>
Date: 2012-01-13 11:26:36 -0800
Use yaml.safe_dump so unicode doesn't mess up the yaml files.In general, yaml.dump is comparable to pickle, and my personal
coding standard says never use it. yaml.safe_dump is much nicer.
yaml.dump should have been named yaml.unsafe_dump, yaml.safe_dump
should have been named yaml.dump :(#6 Updated by Josh Durgin almost 12 years ago
teuthworker is updated.