2017-05-30 19:51:14.618198 7f6d9a2bf700 10 mon.c@2(peon).osd e41 preprocess_query mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 from client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.618250 7f6d9a2bf700 10 mon.c@2(peon) e2 forward_request 282 request mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 features 1152323339925389307
2017-05-30 19:51:14.618259 7f6d9a2bf700 1 -- 172.21.15.190:6791/0 --> mon.0 172.21.15.190:6789/0 -- forward(mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 caps allow * tid 282 con_features 1152323339925389307) v3 -- ?+0 0x5619c3922300
..
2017-05-30 19:51:14.617996 7f6d9a2bf700 1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 20 ==== mon_command({"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"} v 0) v1 ==== 197+0+0 (2769755992 0 0) 0x5619c3e08400 con 0x5619c39d2d80
..
2017-05-30 19:51:14.828063 7f6d9a2bf700 10 mon.c@2(peon) e2 ms_handle_reset 0x5619c39d2d80 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828088 7f6d9a2bf700 10 mon.c@2(peon) e2 reset/close on session client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828096 7f6d9a2bf700 10 mon.c@2(peon) e2 remove_session 0x5619c3d4fd40 client.4157 172.21.15.190:0/2690047051
2017-05-30 19:51:14.828102 7f6d9a2bf700 10 mon.c@2(peon) e2 dropping routed request 282
..
2017-05-30 19:51:15.763034 7f6d9a2bf700 1 -- 172.21.15.190:6791/0 <== mon.0 172.21.15.190:6789/0 891 ==== route(mon_command_ack([{"prefix": "osd pool delete", "sure": "--yes-i-really-really-mean-it", "module": "osd", "format": "plain", "pool": "data3", "pool2": "data3", "perm": "rw"}]=0 pool 'data3' removed v42) v1 tid 282) v3 ==== 356+0+0 (2620051293 0 0)0x5619c3e06900 con 0x5619c3226d00
..
2017-05-30 19:51:15.763082 7f6d9a2bf700 10 mon.c@2(peon) e2 don't have routed request tid 282
...
2017-05-30 19:51:15.820165 7f6d9a2bf700 1 -- 172.21.15.190:6791/0 <== client.4157 172.21.15.190:0/2690047051 1 ==== auth(proto 0 30 bytes epoch 2) v1 ==== 60+0+0 (2980480377 0 0) 0x5619c3ec4000 con 0x5619c3d0e9a0
ceph-rest-api reset the connection before the reply was routed back to it.
ms inject socket failures: 500
ms type: random
i guess, the connection from ceph-rest-api was reset, and its session was removed right away, hence the reply from leader was dropped on the floor. after that, it reconnected to mon.