Actions
Bug #15727
closedEIO during readwrite test
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2016-05-04 08:08:27.295583 7fb935ff3700 1 civetweb: 0x7fb99c0008e0: 172.21.15.67 - - [04/May/2016:08:08:27 +0000] "PUT /rwtest/sxpbzfkbuplsnbgnck HTTP/1.1" 200 0 - Boto/2.40.0 Python/2.7.6 Linux/3.13.0-85-generic 2016-05-04 08:08:27.299134 7fb931feb700 1 ====== starting new request req=0x7fb931fe57d0 ===== 2016-05-04 08:08:27.300350 7fb931feb700 0 WARNING: set_req_state_err err_no=5 resorting to 500 2016-05-04 08:08:27.300369 7fb931feb700 0 ERROR: s->cio->send_content_length() returned err=-5 2016-05-04 08:08:27.300372 7fb931feb700 0 ERROR: s->cio->print() returned err=-5 2016-05-04 08:08:27.300373 7fb931feb700 0 ERROR: STREAM_IO(s)->print() returned err=-5 2016-05-04 08:08:27.300377 7fb931feb700 0 ERROR: STREAM_IO(s)->complete_header() returned err=-5
teuthology fails like so:
2016-05-04T01:08:29.974 INFO:teuthology.orchestra.run.smithi067.stdout: worker: 1}
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout:{bucket: rwtest, duration: 31138897, key: zurtbyfljkhier, start: 1462349138.280538,
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout: type: r, worker: 7}
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout:{bucket: rwtest, duration: 16851902, key: abjsffwivpbq, start: 1462349138.30514, type: r,
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout: worker: 1}
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout:{bucket: rwtest, duration: 14437199, key: adssaryckkj, start: 1462349138.316177, type: r,
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout: worker: 7}
2016-05-04T01:08:29.975 INFO:teuthology.orchestra.run.smithi067.stdout:{bucket: rwtest, duration: 19171953, key: hjzv, start: 1462349138.333237, type: r,
2016-05-04T01:08:29.976 INFO:teuthology.orchestra.run.smithi067.stdout: worker: 7}
2016-05-04T01:08:29.976 INFO:teuthology.orchestra.run.smithi067.stdout:{bucket: rwtest, duration: 17049789, key: nfgyaikfhe, start: 1462349138.357968, type: r,
2016-05-04T01:08:29.976 INFO:teuthology.orchestra.run.smithi067.stdout: worker: 7}
2016-05-04T01:08:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,abjsffwivpbq>
2016-05-04T01:08:29.976 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,adihtqwcrtnwnxsjke>
2016-05-04T01:08:29.976 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 0, 3] out_osds: [5, 4] dead_osds: [] live_osds: [4, 0, 1, 3, 5, 2]
2016-05-04T01:08:29.977 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2016-05-04T01:08:29.977 INFO:tasks.thrashosds.thrasher:fixing pg num pool rbd
2016-05-04T01:08:29.977 INFO:teuthology.orchestra.run.smithi067:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format=json'
2016-05-04T01:08:29.978 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,adssaryckkj>
2016-05-04T01:08:29.978 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,ddsxbmwmbfiejywsq>
2016-05-04T01:08:29.978 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,hjzv>
2016-05-04T01:08:29.978 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,mluirfqd>
2016-05-04T01:08:29.992 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,nfgyaikfhe>
2016-05-04T01:08:30.008 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,odobxwoxcllsrjtoryn>
2016-05-04T01:08:30.023 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,sxpbzfkbuplsnbgnck>
2016-05-04T01:08:30.037 INFO:teuthology.orchestra.run.smithi067.stderr:Cleaning bucket <Bucket: rwtest> key <Key: rwtest,zurtbyfljkhier>
2016-05-04T01:08:30.109 INFO:teuthology.orchestra.run.smithi067.stderr:Traceback (most recent call last):
2016-05-04T01:08:30.110 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite", line 9, in <module>
2016-05-04T01:08:30.110 INFO:teuthology.orchestra.run.smithi067.stderr: load_entry_point('s3tests', 'console_scripts', 's3tests-test-readwrite')()
2016-05-04T01:08:30.110 INFO:teuthology.orchestra.run.smithi067.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 256, in main
2016-05-04T01:08:30.111 INFO:teuthology.orchestra.run.smithi067.stderr: trace=temp_dict['error']['traceback'])
2016-05-04T01:08:30.111 INFO:teuthology.orchestra.run.smithi067.stderr:Exception: exception:
2016-05-04T01:08:30.111 INFO:teuthology.orchestra.run.smithi067.stderr: timed out
...
/a/sage-2016-05-03_05:21:18-rados-wip-sage-testing2---basic-smithi/165816
I've seen several of these in the last week.
Actions