Project

General

Profile

Actions
Copy link

Bug #36764

closed

ceph-mgr: blocking requests sent to restful api server hangs sometimes

Added by Jerry Lee over 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Ceph - v12.2.10
% Done:

0%

Source:
Tags:
Backport:
luminous,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We setup a ceph cluster (v12.2.2) with restful api plugin running, but sometimes requests got blocked forever without responding. While stucking in such condition, we checked the netstat output and it showed some packets were queued in the Recv-Q forever:

[~] netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address   State       PID/Program name
tcp      129      0 192.168.2.1:8003        0.0.0.0:*         LISTEN      1885/ceph-mgr

The issue could be easily re-produced by aggressively and concurrently issuing lots of blocking restful api requests to the rest api server.

Related issues 2 (0 open — 2 closed)

Copied to Ceph - Backport #37271: mimic: ceph-mgr: blocking requests sent to restful api server hangs sometimesResolvedNathan CutlerActions
Copied to Ceph - Backport #37272: luminous: ceph-mgr: blocking requests sent to restful api server hangs sometimesResolvedActions
Actions
Copy link
 #2

Updated by John Spray over 5 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #3

Updated by Kefu Chai over 5 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to luminous,mimic
Actions
Copy link
 #5

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #37271: mimic: ceph-mgr: blocking requests sent to restful api server hangs sometimes added
Actions
Copy link
 #6

Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #37272: luminous: ceph-mgr: blocking requests sent to restful api server hangs sometimes added
Actions
Copy link
 #7

Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #8

Updated by Greg Farnum about 5 years ago

  • Project changed from Ceph to Messengers
  • Category deleted (msgr)
Actions
Copy link
 #9

Updated by Mingqiao Wu about 5 years ago

I met this issue again in luminous 12.2.11

[root@cube-dev ceph]# curl -k https://127.0.0.1:10103/
curl: (35) TCP connection reset by peer
[root@cube-dev ceph]# ceph-mgr --version
ceph version 12.2.11-5-g4b9d97e (4b9d97eec3137bc41531c5077ba44a8de8e83b5a) luminous (stable)
[root@cube-dev ceph]# netstat -tupln | grep ceph-mgr
tcp      129      0 0.0.0.0:10103           0.0.0.0:*               LISTEN      38123/ceph-mgr    
tcp        0      0 10.20.8.130:6808        0.0.0.0:*               LISTEN      38123/ceph-mgr      
tcp6       0      0 :::9283                 :::*                    LISTEN      38123/ceph-mgr

Could someone recheck it?

Actions
Copy link
 #10

Updated by Jiang Yu over 4 years ago

ceph version 12.2.12 luminous (stable) Found the same problem, Restart mgr manually to recover
```
[root@dev ~]# curl -k https://localhost:10103/ -v
  • About to connect() to 10.20.8.130 port 10103 (#0)
  • Trying 10.20.8.130...
  • Connected to 10.20.8.130 (10.20.8.130) port 10103 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
    openssl x509 -in* NSS error -5961 (PR_CONNECT_RESET_ERROR)
  • TCP connection reset by peer
  • Closing connection 0
    curl: (35) TCP connection reset by peer

[root@dev ~]# openssl s_client -connect 10.20.8.130:10103 -msg
CONNECTED

TLS 1.2 [length 0005]

16 03 01 01 1c

TLS 1.2 Handshake [length 011c], ClientHello

01 00 01 18 03 03 55 2f b8 d0 52 5b 4e a4 0b 68
37 e9 84 f3 b9 46 63 b9 c7 83 9c 23 97 ac ee 33
9e 72 f8 5c 2f 73 00 00 ac c0 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 a5 00 a3 00 a1 00 9f 00 6b 00
6a 00 69 00 68 00 39 00 38 00 37 00 36 00 88 00
87 00 86 00 85 c0 32 c0 2e c0 2a c0 26 c0 0f c0
05 00 9d 00 3d 00 35 00 84 c0 2f c0 2b c0 27 c0
23 c0 13 c0 09 00 a4 00 a2 00 a0 00 9e 00 67 00
40 00 3f 00 3e 00 33 00 32 00 31 00 30 00 9a 00
99 00 98 00 97 00 45 00 44 00 43 00 42 c0 31 c0
2d c0 29 c0 25 c0 0e c0 04 00 9c 00 3c 00 2f 00
96 00 41 c0 12 c0 08 00 16 00 13 00 10 00 0d c0
0d c0 03 00 0a 00 07 c0 11 c0 07 c0 0c c0 02 00
05 00 04 00 ff 01 00 00 43 00 0b 00 04 03 00 01
02 00 0a 00 0a 00 08 00 17 00 19 00 18 00 16 00
23 00 00 00 0d 00 20 00 1e 06 01 06 02 06 03 05
01 05 02 05 03 04 01 04 02 04 03 03 01 03 02 03
03 02 01 02 02 02 03 00 0f 00 01 01
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 289 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1573448808
Timeout : 300 (sec)
Verify return code: 0 (ok)
---

[root@dev ceph]# openssl x509 -in ceph-mgr-restful.crt -noout -enddate
notAfter=Sep 29 05:32:30 2119 GMT

[root@dev ~]# curl -k https://10.20.8.130:10103 -v
  • About to connect() to 10.20.8.130 port 10103 (#0)
  • Trying 10.20.8.130...
  • Connected to 10.20.8.130 (10.20.8.130) port 10103 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • skipping SSL peer certificate verification
  • SSL connection using TLS_RSA_WITH_AES_256_GCM_SHA384
  • Server certificate:
  • subject: CN=ceph-restful,O=IT
  • start date: Nov 11 04:04:27 2019 GMT
  • expire date: Nov 08 04:04:27 2029 GMT
  • common name: ceph-restful
  • issuer: CN=ceph-restful,O=IT

GET / HTTP/1.1
User-Agent: curl/7.29.0
Host: 10.20.8.130:10103
Accept: */*

  • HTTP 1.0, assume close after body
    < HTTP/1.0 200 OK
    < Content-Length: 221
    < Content-Type: application/json; charset=UTF-8
    < Server: Werkzeug/0.9.1 Python/2.7.5
    < Date: Mon, 11 Nov 2019 05:41:58 GMT
    < {
    "api_version": 1,
    "auth": "Use \"ceph restful create-key <key>\" to create a key pair, pass it as HTTP Basic auth to authenticate",
    "doc": "See /doc endpoint",
    "info": "Ceph Manager RESTful API server"
  • Closing connection 0
    }

```

Actions
Copy link

Also available in: Atom PDF