Ceph » rgw

Running teuthology using the following yaml file failed:

roles:
- [mon.a, mon.c, osd.0, osd.1, osd.2, client.0]
- [mon.b, osd.3, osd.4, osd.5, client.1]
overrides:
  rgw:
    frontend: apache
overrides:
  ceph:
    fs: xfs
  rgw:
    ec-data-pool: false
tasks:
- chef:
- install:
- ceph:
- rgw: [client.0]
- s3readwrite:
    client.0:
      rgw_server: client.0
      readwrite:
        bucket: rwtest
        readers: 10
        writers: 3
        duration: 50
        verbose: True
        files:
          num: 10
          size: 2000
          stddev: 500 
targets:
  vpm011.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAy8s/Xwpliqi41wHPASb6UR0hX4pBPWO3wdDBb45oLToMpXrRBtLW8J92q5kOi8fhTUjhdbADjneN8I5kdY7dFaHSkqzsRffH1cS74F6YXyFuKi/+DDTl50nTvio/lkzzspdaMRyuMRD/QqkfGYpLZy8AWYV6x0woQ2UIMfsD2PbuohVwmSDlEWEwAh8Wm9+az5eHygZeYELMvFJuWAR5kgnKt0fFrHX586h4Rr8a5CP5SghaucRQ6Y9BNrpJnpc+V8elatbf3V8KlCNCpJ44S87BoJqR1VIopQkWOug850tCtgA2B61nHgbZmPVEwRjvC1BVP74Z7mqiUD4RC+usLQ==
  vpm199.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA1BclM6Q7SsAKy2mSjSzVz6N0NQTedZd/xAaKlCc1dA1qOgXKiaxWKA5GTMrntdIECX8dgdkvt+WN5OiFIc4N14OTKQu+73DkKIhZgYs3aKLCRe5tshlSqD/9meKsqblnyXRotDAYBbqoOPwGcaZv2AKEk4vOqDK9CIt40H4jC+aSny9oVfjKrVfHSqXk7J/X6iEj/ZeQBjKczUgvSeygvxxW7t9kDoSaNVL7XF+NBG6MjROjWNLKiJbU/bN7wolUN19j8YvYrI4VHQjxc3R1dL0HkfqlndAlxKWiJiSmek6NKXS2tjtS1VJOZx5EIneIB7MHKFTJf88ey5EUOt8z6Q==

So in runtests() in ceph-qa-suite/tasks/s3readwrite.py, I inserted a pdb.set_trace() call.

def run_tests(ctx, config):
    """ 
    Run the s3readwrite tests after everything is set up.

    :param ctx: Context passed to task
    :param config: specific configuration information
    """ 
    assert isinstance(config, dict)
    testdir = teuthology.get_testdir(ctx)
    for client, client_config in config.iteritems():
        (remote,) = ctx.cluster.only(client).remotes.keys()
        conf = teuthology.get_file(remote, '{tdir}/archive/s3readwrite.{client}.config.yaml'.format(tdir=testdir, client=client))
        args = [
                '{tdir}/s3-tests/virtualenv/bin/s3tests-test-readwrite'.format(tdir=testdir),
                ]
        if client_config is not None and 'extra_args' in client_config:
            args.extend(client_config['extra_args'])

        import pdb; pdb.set_trace()
        ctx.cluster.only(client).run(
            args=args,
            stdin=conf,
            )
    yield
<pre>

After I hit the breakpoint, conf was:
<pre>
>readwrite:
>   bucket: rwtest
>   duration: 50
>   files:
>     num: 10
>     size: 2000
>     stddev: 500
>   readers: 10
>   verbose: true
>   writers: 3
> s3:
>   access_key: VVGIYASDBHOEHUHNNNLR
>   display_name: Mr. foo.client.0
>   email: foo.client.0+test@test.test
>   host: vpm011.front.sepia.ceph.com
>   is_secure: false
>   port: 7280
>   secret_key: apKZhthB1gXhq0qjUEODU3ARllIIIBqcnbf/yUqGg51Q6M33RdBjUw==
>   user_id: foo.client.0

</pre>

And args was:
<pre>
[/home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite]
</pre>

So I ran echo '<above conf value>' | /home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite

and I got:

<pre>
Using random seeds: {'writer': 2240982845, 'main': 428369860, 'names': 3931836287, 'contents': 2335577820, 'reader': 522803006}
Traceback (most recent call last):
  File "./s3tests-test-readwrite", line 9, in <module>
    load_entry_point('s3tests==0.0.1', 'console_scripts', 's3tests-test-readwrite')()
  File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 166, in main
    bucket = conn.create_bucket(bucket_name)
  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/boto/s3/connection.py", line 612, in create_bucket
    data=data)
  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/boto/s3/connection.py", line 664, in make_request
    retry_handler=retry_handler
  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/boto/connection.py", line 1068, in make_request
    retry_handler=retry_handler)
  File "/home/ubuntu/cephtest/s3-tests/virtualenv/lib/python2.6/site-packages/boto/connection.py", line 1025, in _mexe
    raise BotoServerError(response.status, response.reason, body)
boto.exception.BotoServerError: BotoServerError: 500 Internal Server Error
None
Exception KeyError: KeyError(18252656,) in <module 'threading' from '/usr/lib64/python2.6/threading.pyc'> ignored
</pre>