Bug #1935

teuthology: readwrite/roundtrip jobs run manually, but not in suite

Added by Sage Weil over 8 years ago. Updated over 8 years ago.

Target version:
% Done:


3 - minor
Affected Versions:
Pull request ID:
Crash signature:


I'm not sure what's going on here.. roundtrip and readwrite are failing when scheduled, but when i copy/paste the same yaml and run ti manually, they work just fine. The errors look like

2012-01-12T17:53:18.292 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /tmp/cephtest/archive/s3readwrite.client.0.config.yaml'
2012-01-12T17:53:18.361 'cat -- /tmp/cephtest/archive/s3readwrite.client.0.config.yaml'
2012-01-12T17:53:18.407 '/tmp/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite'
2012-01-12T17:53:18.694 (most recent call last):
2012-01-12T17:53:18.695  File "/tmp/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite", line 9, in <module>
2012-01-12T17:53:18.695    load_entry_point('s3tests==0.0.1', 'console_scripts', 's3tests-test-readwrite')()
2012-01-12T17:53:18.695  File "/tmp/cephtest/s3-tests/s3tests/", line 131, in main
2012-01-12T17:53:18.696    config = common.read_config(sys.stdin)
2012-01-12T17:53:18.696  File "/tmp/cephtest/s3-tests/s3tests/", line 83, in read_config
2012-01-12T17:53:18.696    for new in g:
2012-01-12T17:53:18.696  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 83, in load_all
2012-01-12T17:53:18.696    yield loader.get_data()
2012-01-12T17:53:18.697  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 33, in get_data
2012-01-12T17:53:18.697    return self.construct_document(self.get_node())
2012-01-12T17:53:18.697  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 48, in construct_document
2012-01-12T17:53:18.697    for dummy in generator:
2012-01-12T17:53:18.699  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 398, in construct_yaml_map
2012-01-12T17:53:18.699    value = self.construct_mapping(node)
2012-01-12T17:53:18.699  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 208, in construct_mapping
2012-01-12T17:53:18.699    return BaseConstructor.construct_mapping(self, node, deep=deep)
2012-01-12T17:53:18.700  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 133, in construct_mapping
2012-01-12T17:53:18.700    value = self.construct_object(value_node, deep=deep)
2012-01-12T17:53:18.700  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 88, in construct_object
2012-01-12T17:53:18.700    data = constructor(self, node)
2012-01-12T17:53:18.700  File "/tmp/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/yaml/", line 414, in construct_undefined
2012-01-12T17:53:18.701    node.start_mark)
2012-01-12T17:53:18.701 could not determine a constructor for the tag ',2002:python/unicode'
2012-01-12T17:53:18.701  in "<stdin>", line 14, column 9

That's from ubuntu@teuthology:/var/lib/teuthworker/archive/wip-backfill-2012-01-12_13:09:04/7281 (happening on master too)


#1 Updated by Anonymous over 8 years ago

s3tests/functional (run via nosetests) reads a .ini -style configuration.

This was not flexible enough for all the needs the benchmarks (readwrite, roundtrip) had, so we switched those to a yaml config format. There's a sample in config.yaml.SAMPLE.

At some point, s3tests/functional should be ported to read the newer, more flexible format also. This wasn't seen as critical enough to warrant work at the time, but clearly it would help with the confusion.

#2 Updated by Anonymous over 8 years ago

More specific plan for change:

- change s3tests/functional/ to read env var S3TEST_YAML, raise if it is not there (explicit error to prompt users to migrate)
- make it read the configuration via s3tests/ read_config(), etc; share more code between them

- there seems to be a lot of copy-paste code between s3tests/ and s3tests/functional/, clean up:
- s3tests/ setup() seems only used by s3tests/fuzz (and the debug runner in s3tests/, refactor for clarity so it's not confused with s3tests/functional/ setup()
- choose_bucket_prefix
- nuke_prefixed_buckets
- get_new_bucket

#3 Updated by Anonymous over 8 years ago

I'm wrong, ignore me for a while.

#4 Updated by Sage Weil over 8 years ago

  • Assignee set to Anonymous

#5 Updated by Anonymous over 8 years ago

  • Status changed from New to Resolved

Alright, I was reading the wrong file. and do use the yaml format. The above cleanups would still be valid.

Bug found & squished (still needs to be deployed on teuthworker I think):

commit 3bfa41cf6ad945c21cf8499f94dd71cf07b3cb04
Author: Tommi Virtanen <>
Date: 2012-01-13 11:26:36 -0800

Use yaml.safe_dump so unicode doesn't mess up the yaml files.
In general, yaml.dump is comparable to pickle, and my personal
coding standard says never use it. yaml.safe_dump is much nicer.
yaml.dump should have been named yaml.unsafe_dump, yaml.safe_dump
should have been named yaml.dump :(

#6 Updated by Josh Durgin over 8 years ago

teuthworker is updated.

Also available in: Atom PDF