Bug #5815
rest/test.py failure: FAILURE: url http://localhost:5000/api/v0.1/mds/setmap?epoch=11
Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2013-07-31T02:01:51.721 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:33] "GET /api/v0.1/mds/compat/show HTTP/1.1" 200 - 2013-07-31T02:01:51.890 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:33] "PUT /api/v0.1/mds/deactivate?who=2 HTTP/1.1" 400 - 2013-07-31T02:01:51.895 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:33] "GET /api/v0.1/mds/dump.json HTTP/1.1" 200 - 2013-07-31T02:01:51.901 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:33] "GET /api/v0.1/mds/dump.xml HTTP/1.1" 200 - 2013-07-31T02:01:51.906 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:33] "GET /api/v0.1/mds/getmap HTTP/1.1" 200 - 2013-07-31T02:01:52.074 INFO:teuthology.task.rest-api.client.rest0.err:[10.214.131.3]: 127.0.0.1 - - [31/Jul/2013 02:02:34] "PUT /api/v0.1/mds/setmap?epoch=11 HTTP/1.1" 400 - 2013-07-31T02:01:52.075 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: FAILURE: url http://localhost:5000/api/v0.1/mds/setmap?epoch=11 2013-07-31T02:01:52.076 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: expected 200, got 400 2013-07-31T02:01:52.076 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: Response content: 2013-07-31T02:01:52.076 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: Error: next mdsmap epoch 12 != 11unrecognized command (-22) 2013-07-31T02:01:52.076 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: 2013-07-31T02:01:52.077 INFO:teuthology.task.workunit.client.0.err:[10.214.131.3]: Headers: {'date': 'Wed, 31 Jul 2013 09:02:34 GMT', 'content-length': '61', 'content-type': 'text/html; charset=utf-8', 'server': 'Werkzeug/0.8.1 Python/2.7.3'} 2013-07-31T02:01:52.077 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export: plain 200 2013-07-31T02:01:52.077 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export.json: json 200 2013-07-31T02:01:52.077 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export.xml: xml 200 2013-07-31T02:01:52.077 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export: json 200 2013-07-31T02:01:52.078 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export: xml 200 2013-07-31T02:01:52.078 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/add?entity=client.xx&caps=mon&caps=allow&caps=osd&caps=allow+*: json 200 2013-07-31T02:01:52.078 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export?entity=client.xx: plain 200 2013-07-31T02:01:52.078 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/add?entity=client.xx: plain 200 2013-07-31T02:01:52.078 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/list: plain 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/list.json: json 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/get-key?entity=client.xx: json 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/print-key?entity=client.xx: json 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/print_key?entity=client.xx: json 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/caps?entity=client.xx&caps=osd&caps=allow+rw: json 200 2013-07-31T02:01:52.079 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/list.json: json 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export: plain 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/import: plain 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/export: plain 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: auth/del?entity=client.xx: json 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: osd/dump: json 200 2013-07-31T02:01:52.080 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df: plain 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df?detail=detail: plain 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df?detail: plain 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df: json 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df: xml 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df?detail: json 200 2013-07-31T02:01:52.081 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: df?detail: xml 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: fsid: json 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: health: json 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: health?detail: json 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: health?detail: plain 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/cluster_down: 200 2013-07-31T02:01:52.082 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/cluster_down: 200 2013-07-31T02:01:52.083 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/cluster_up: 200 2013-07-31T02:01:52.083 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/cluster_up: 200 2013-07-31T02:01:52.083 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/compat/rm_incompat?feature=4: 200 2013-07-31T02:01:52.083 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/compat/rm_incompat?feature=4: 200 2013-07-31T02:01:52.083 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/compat/show: json 200 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/compat/show: xml 200 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/deactivate?who=2: 400 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/dump.json: json 200 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/dump.xml: xml 200 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/getmap: 200 2013-07-31T02:01:52.084 INFO:teuthology.task.workunit.client.0.out:[10.214.131.3]: mds/setmap?epoch=11: plain 400 2013-07-31T02:01:52.086 INFO:teuthology.task.workunit:Stopping rest/test.py on client.0... 2013-07-31T02:01:52.086 DEBUG:teuthology.orchestra.run:Running [10.214.131.3]: 'rm -rf -- /home/ubuntu/cephtest/91048/workunits.list /home/ubuntu/cephtest/91048/workunit.client.0'
Associated revisions
rest/test.py: retry mds setmap in case epoch has changed
Loop 10 times retrying mds dump/getmap/setmap.
Fixes: #5815
Signed-off-by: Dan Mick <dan.mick@inktank.com>
History
#1 Updated by Samuel Just over 10 years ago
ubuntu@teuthology:/a/teuthology-2013-07-31_01:00:23-rados-next-testing-basic-plana/91048
#2 Updated by Dan Mick over 10 years ago
OK, this is mds setmap racing against the epoch number we got from mds dump. Possible fixes: retry N times or just skip setmap.
#3 Updated by Dan Mick over 10 years ago
Also, as Sage points out, "unrecognized command" isn't really useful here.
#4 Updated by Dan Mick over 10 years ago
- Status changed from New to Fix Under Review
Fix test failure by adding 10x retry to "get epoch, set epoch+1".
wip-5815
#5 Updated by Dan Mick over 10 years ago
- Description updated (diff)
#6 Updated by Ian Colle over 10 years ago
- Status changed from Fix Under Review to Resolved
wip-5815 merged into next