Bug #20115: rest.py: pool delete failure - Ceph - Ceph

Actions

Copy link

Bug #20115

closed

rest.py: pool delete failure

Added by Sage Weil almost 7 years ago. Updated almost 7 years ago.

Status:

Resolved

Priority:

Immediate

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description


2017-05-30T19:51:22.996 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:22] "GET /api/v0.1/tell/osd.0/version HTTP/1.1" 200 -
2017-05-30T19:51:26.595 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:26] "GET /api/v0.1/tell/osd.999/version HTTP/1.1" 400 -
2017-05-30T19:51:26.601 INFO:tasks.rest_api.client.rest0.smithi190.stderr:127.0.0.1 - - [30/May/2017 19:51:26] "GET /api/v0.1/tell/osd.foo/version HTTP/1.1" 400 -
2017-05-30T22:50:06.911 INFO:tasks.workunit.client.0.smithi190.stdout:PUT osd/pool/delete?pool=data3&pool2=data3&sure=--yes-i-really-really-m

this is probably a mgr stats lag?

/a/sage-2017-05-30_17:22:40-rados-wip-sage-testing---basic-smithi/1244710

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

2017-05-30 19:51:14.618198 7f6d9a2bf700 10 mon.c@2(peon).osd e41 preprocess_query mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 from client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.618250 7f6d9a2bf700 10 mon.c@2(peon) e2 forward_request 282 request mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 features 1152323339925389307
2017-05-30 19:51:14.618259 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 --> mon.0 172.21.15.190:6789/0 -- forward(mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 caps allow * tid 282 con_features 1152323339925389307) v3 -- ?+0 0x5619c3922300
..
2017-05-30 19:51:14.617996 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 20 ==== mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 ==== 197+0+0 (2769755992 0 0) 0x5619c3e08400 con 0x5619c39d2d80
..
2017-05-30 19:51:14.828063 7f6d9a2bf700 10 mon.c@2(peon) e2 ms_handle_reset 0x5619c39d2d80 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828088 7f6d9a2bf700 10 mon.c@2(peon) e2 reset/close on session client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828096 7f6d9a2bf700 10 mon.c@2(peon) e2 remove_session 0x5619c3d4fd40 client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828102 7f6d9a2bf700 10 mon.c@2(peon) e2  dropping routed request 282
..
2017-05-30 19:51:15.763034 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== mon.0 172.21.15.190:6789/0 891 ==== route(mon_command_ack([{"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"}]=0 pool 'data3' removed v42) v1 tid 282) v3 ==== 356+0+0 (2620051293 0 0)0x5619c3e06900 con 0x5619c3226d00
..
2017-05-30 19:51:15.763082 7f6d9a2bf700 10 mon.c@2(peon) e2  don't have routed request tid 282
...
2017-05-30 19:51:15.820165 7f6d9a2bf700  1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 1 ==== auth(proto 0 30 bytes epoch 2) v1 ==== 60+0+0 (2980480377 0 0) 0x5619c3ec4000 con 0x5619c3d0e9a0

ceph-rest-api reset the connection before the reply was routed back to it.

          ms inject socket failures: 500
          ms type: random

i guess, the connection from ceph-rest-api was reset, and its session was removed right away, hence the reply from leader was dropped on the floor. after that, it reconnected to mon.

Actions

Copy link