Project

General

Profile

Bug #20115

rest.py: pool delete failure

Added by Sage Weil over 1 year ago. Updated over 1 year ago.

Status:
Resolved
Priority:
Immediate
Assignee:
-
Category:
-
Target version:
-
Start date:
05/30/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description


2017-05-30T19:51:22.996 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:22] "GET /api/v0.1/tell/osd.0/version HTTP/1.1" 200 -
2017-05-30T19:51:26.595 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:26] "GET /api/v0.1/tell/osd.999/version HTTP/1.1" 400 -
2017-05-30T19:51:26.601 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:26] "GET /api/v0.1/tell/osd.foo/version HTTP/1.1" 400 -
2017-05-30T22:50:06.911 INFO:tasks.workunit.client.0.smithi190.stdout:PUT osd/pool/delete?pool=data3&pool2=data3&sure=--yes-i-really-really-m

this is probably a mgr stats lag?

/a/sage-2017-05-30_17:22:40-rados-wip-sage-testing---basic-smithi/1244710

History

#1 Updated by Kefu Chai over 1 year ago

2017-05-30 19:51:14.618198 7f6d9a2bf700 10 mon.c@2(peon).osd e41 preprocess_query mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 from client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.618250 7f6d9a2bf700 10 mon.c@2(peon) e2 forward_request 282 request mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 features 1152323339925389307
2017-05-30 19:51:14.618259 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 --> mon.0 172.21.15.190:6789/0 -- forward(mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 caps allow * tid 282 con_features 1152323339925389307) v3 -- ?+0 0x5619c3922300
..
2017-05-30 19:51:14.617996 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 20 ==== mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 ==== 197+0+0 (2769755992 0 0) 0x5619c3e08400 con 0x5619c39d2d80
..
2017-05-30 19:51:14.828063 7f6d9a2bf700 10 mon.c@2(peon) e2 ms_handle_reset 0x5619c39d2d80 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828088 7f6d9a2bf700 10 mon.c@2(peon) e2 reset/close on session client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828096 7f6d9a2bf700 10 mon.c@2(peon) e2 remove_session 0x5619c3d4fd40 client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828102 7f6d9a2bf700 10 mon.c@2(peon) e2  dropping routed request 282
..
2017-05-30 19:51:15.763034 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== mon.0 172.21.15.190:6789/0 891 ==== route(mon_command_ack([{"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"}]=0 pool 'data3' removed v42) v1 tid 282) v3 ==== 356+0+0 (2620051293 0 0)0x5619c3e06900 con 0x5619c3226d00
..
2017-05-30 19:51:15.763082 7f6d9a2bf700 10 mon.c@2(peon) e2  don't have routed request tid 282
...
2017-05-30 19:51:15.820165 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 1 ==== auth(proto 0 30 bytes epoch 2) v1 ==== 60+0+0 (2980480377 0 0) 0x5619c3ec4000 con 0x5619c3d0e9a0

ceph-rest-api reset the connection before the reply was routed back to it.

          ms inject socket failures: 500
          ms type: random

i guess, the connection from ceph-rest-api was reset, and its session was removed right away, hence the reply from leader was dropped on the floor. after that, it reconnected to mon.

#2 Updated by Kefu Chai over 1 year ago

  • Status changed from New to Verified

#3 Updated by Kefu Chai over 1 year ago

  • Status changed from Verified to Need Review

#4 Updated by Sage Weil over 1 year ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF