Project

General

Profile

Actions
Copy link

Bug #6051

closed

rgw: 404 during readwrite test

Added by Sage Weil over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-08-19T03:24:43.188 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: '/home/ubuntu/cephtest/1163/s3-tests/virtualenv/bin/s3tests-test-readwrite'
2013-08-19T03:24:43.468 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Using random seeds: {'writer': 290687965, 'main': 1682398903, 'names': 393224963, 'contents': 2785472079, 'reader': 2104947738}
2013-08-19T03:24:44.191 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Created bucket: testbucket
2013-08-19T03:24:44.192 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Creating file names that are deterministic
2013-08-19T03:24:44.192 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Starting main worker loop.
2013-08-19T03:24:44.192 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Using file size: 2000 +- 500
2013-08-19T03:24:44.192 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Spawning 0 writers and 2 readers...
2013-08-19T03:25:14.192 INFO:teuthology.orchestra.run.err:[10.214.132.32]: post-join, queue size 6914
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Traceback (most recent call last):
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/bin/s3tests-test-readwrite", line 9, in <module>
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     load_entry_point('s3tests==0.0.1', 'console_scripts', 's3tests-test-readwrite')()
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/s3tests/readwrite.py", line 254, in main
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     trace=temp_dict['error']['traceback'])
2013-08-19T03:25:14.582 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Exception: exception:
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]:      S3ResponseError: 404 Not Found
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code></Error>
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]:      Traceback (most recent call last):
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/s3tests/readwrite.py", line 34, in reader
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     key.get_contents_to_file(fp)
2013-08-19T03:25:14.583 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1563, in get_contents_to_file
2013-08-19T03:25:14.584 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     response_headers=response_headers)
2013-08-19T03:25:14.584 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1403, in get_file
2013-08-19T03:25:14.584 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     query_args=None)
2013-08-19T03:25:14.584 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1435, in _get_file_internal
2013-08-19T03:25:14.584 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     override_num_retries=override_num_retries)
2013-08-19T03:25:14.585 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 322, in open
2013-08-19T03:25:14.585 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     override_num_retries=override_num_retries)
2013-08-19T03:25:14.585 INFO:teuthology.orchestra.run.err:[10.214.132.32]:   File "/home/ubuntu/cephtest/1163/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 271, in open_read
2013-08-19T03:25:14.585 INFO:teuthology.orchestra.run.err:[10.214.132.32]:     self.resp.reason, body)
2013-08-19T03:25:14.586 INFO:teuthology.orchestra.run.err:[10.214.132.32]: S3ResponseError: S3ResponseError: 404 Not Found
2013-08-19T03:25:14.586 INFO:teuthology.orchestra.run.err:[10.214.132.32]: <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code></Error>
2013-08-19T03:25:14.586 INFO:teuthology.orchestra.run.err:[10.214.132.32]: 
2013-08-19T03:25:14.586 INFO:teuthology.orchestra.run.err:[10.214.132.32]: Exception KeyError: KeyError(39458128,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored

ubuntu@teuthology:/a/teuthology-2013-08-19_01:01:14-rgw-master-testing-basic-plana/1163$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 546140dd51e9ec7e34fe0b0a5814240828f68f7d
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: master
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
    log-whitelist:
    - slow request
    sha1: 4677041da309234feb7b7df28e1d73db99d27c72
  ceph-deploy:
    branch:
      dev: master
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 4677041da309234feb7b7df28e1d73db99d27c72
  s3readwrite:
    readwrite:
      bucket: testbucket
      deterministic_file_names: true
      duration: 30
      files:
        num: 10
        size: 2000
        stddev: 500
    s3:
      access_key: 2te6NH5mcdcq0Tc5i8i4
      display_name: test user for the s3readwrite tests
      email: tester@inktank
      secret_key: Qy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXx
      user_id: s3readwrite-test-user
  s3tests:
    branch: master
  workunit:
    sha1: 4677041da309234feb7b7df28e1d73db99d27c72
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
  - client.0
- - mon.b
  - mon.c
  - osd.2
  - osd.3
  - client.1
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    conf:
      client.1:
        rgw gc pool: .rgw.gc
        rgw region: default
        rgw region root pool: .rgw
        rgw user keys pool: .users
        rgw user uid pool: .users.uid
        rgw zone: r1z1
        rgw zone root pool: .rgw
- rgw:
    client.0:
      system user:
        access key: 0te6NH5mcdcq0Tc5i8i2
        name: nr-system
        secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
- s3readwrite:
    client.0:
      extra_args:
      - --no-cleanup
      readwrite:
        readers: 0
        writers: 1
      s3:
        delete_user: false
- rgw:
    client.1:
      system user:
        access key: 1te6NH5mcdcq0Tc5i8i3
        name: r2-system
        secret key: Py4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXw
    regions:
      default:
        api name: api1
        is master: true
        master zone: r1z1
        zones:
        - r1z1
- s3readwrite:
    client.1:
      readwrite:
        readers: 2
        writers: 0
      s3:
        create_user: false
teuthology_branch: master
Actions
Copy link
 #1

Updated by Sage Weil over 10 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #2

Updated by Ian Colle over 10 years ago

  • Assignee set to Yehuda Sadeh
Actions
Copy link
 #3

Updated by Anonymous over 10 years ago

  • Assignee changed from Yehuda Sadeh to Anonymous
Actions
Copy link
 #4

Updated by Anonymous over 10 years ago

Missed adding the traceback entry to an error dict in one codepath. I've pushed a one-line fix to the s3-tests branch wip-buck-6051 and will be pushing it to master shortly.

Actions
Copy link
 #5

Updated by Anonymous over 10 years ago

  • Status changed from New to Resolved

The issue was that the tests were not specifying the 'domain root pool' and the pool name generated by the rgw.py task was not aligning to the default location of that pool, so tests were failing. The fix was to specify the 'domain root pool' to be '.rgw' so that it lines up with the default pool.

This went in as ceph-qa-suite commit #ceb1d7bb4b7c85b6c2f5a2d45d9972606a2e1a53

Actions
Copy link

Also available in: Atom PDF