Bug #21704: jewel->luminous: mon: statfs hangs on luminous mon - Ceph - Ceph

Actions

Copy link

Bug #21704

closed

jewel->luminous: mon: statfs hangs on luminous mon

Added by Sage Weil over 6 years ago. Updated over 4 years ago.

Status:

Can't reproduce

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The jewel ceph-fuse is blocked

Thread 2 (Thread 0x7f6b98d40700 (LWP 8021)):
#0  pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1  0x00007f6ba7be7cc1 in Client::statfs(char const*, statvfs*) ()
#2  0x00007f6ba7b95630 in ?? ()
#3  0x00007f6ba75aec88 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#4  0x00007f6ba75ae22b in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#5  0x00007f6ba75aae49 in ?? () from /lib/x86_64-linux-gnu/libfuse.so.2
#6  0x00007f6ba6cfe184 in start_thread (arg=0x7f6b98d40700) at pthread_create.c:312
#7  0x00007f6ba566c37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

the statfs arrived at mon.b

2017-10-06 12:01:30.743125 7f163605c700  1 -- 172.21.15.174:6789/0 <== client.14098 172.21.15.171:0/373621297 17 ==== statfs(2 pool -1 v25) v1 ==== 34+0+0 (933241214 0 0) 0x7f164c516640 con 0x7f164bba3000
2017-10-06 12:01:30.743156 7f163605c700 20 mon.b@1(peon) e2 _ms_dispatch existing session 0x7f164b664c00 for client.? 172.21.15.171:0/373621297
2017-10-06 12:01:30.743162 7f163605c700 20 mon.b@1(peon) e2  caps allow rw
2017-10-06 12:01:30.743164 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164c516640 statfs(2 pool -1 v25) v1 from client.14098 172.21.15.171:0/373621297 con 0x7f164bba3000
2017-10-06 12:01:30.743171 7f163605c700  5 mon.b@1(peon).paxos(paxos active c 11296..11810) is_readable = 1 - now=2017-10-06 12:01:30.743171 lease_expire=2017-10-06 12:01:34.714855 has v0 lc 11810
2017-10-06 12:01:30.743183 7f163605c700 20 is_capable service=pg command= read on cap allow rw
2017-10-06 12:01:30.743186 7f163605c700 20  allow so far , doing grant allow rw
2017-10-06 12:01:30.743189 7f163605c700 20  match
2017-10-06 12:01:30.743191 7f163605c700 10 mon.b@1(peon).mgrstat preprocess_statfs statfs(2 pool -1 v25) v1 from client.14098
2017-10-06 12:01:30.743196 7f163605c700  2 mon.b@1(peon) e2 send_reply 0x7f164d2bc0a0 0x7f164c7373c0 statfs_reply(2) v1
2017-10-06 12:01:30.743200 7f163605c700  1 -- 172.21.15.174:6789/0 --> 172.21.15.171:0/373621297 -- statfs_reply(2) v1 -- 0x7f164c7373c0 con 0
2017-10-06 12:01:30.745187 7f163605c700  1 -- 172.21.15.174:6789/0 <== client.14107 172.21.15.171:0/381782085 20 ==== statfs(4 pool -1 v27) v1 ==== 34+0+0 (3393795804 0 0) 0x7f164c7373c0 con 0x7f164bd8c800
2017-10-06 12:01:30.745218 7f163605c700 20 mon.b@1(peon) e2 _ms_dispatch existing session 0x7f164b667180 for client.? 172.21.15.171:0/381782085
2017-10-06 12:01:30.745224 7f163605c700 20 mon.b@1(peon) e2  caps allow rw
2017-10-06 12:01:30.745226 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164c7373c0 statfs(4 pool -1 v27) v1 from client.14107 172.21.15.171:0/381782085 con 0x7f164bd8c800
2017-10-06 12:01:30.745232 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26)  waiting for paxos -> readable (v27)

but doesn't ever get processed...

2017-10-06 12:01:33.714675 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164b6c2f40 statfs(3 pool -1 v27) v1 from client.14107 172.21.15.171:0/381782085 con 0x7f164bd8c800
2017-10-06 12:01:33.714687 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26)  waiting for paxos -> readable (v27)
2017-10-06 12:01:33.714694 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164c7373c0 statfs(4 pool -1 v27) v1 from client.14107 172.21.15.171:0/381782085 con 0x7f164bd8c800
2017-10-06 12:01:33.714704 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26)  waiting for paxos -> readable (v27)
2017-10-06 12:01:33.714709 7f163605c700 10 mon.b@1(peon).paxosservice(health 1..1) post_refresh
...
2017-10-06 12:01:43.716283 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164b6c2f40 statfs(3 pool -1 v27) v1 from client.14107 172.21.15.171:0/381782085 con 0x7f164bd8c800
2017-10-06 12:01:43.716292 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26)  waiting for paxos -> readable (v27)
2017-10-06 12:01:43.716296 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26) dispatch 0x7f164c7373c0 statfs(4 pool -1 v27) v1 from client.14107 172.21.15.171:0/381782085 con 0x7f164bd8c800
2017-10-06 12:01:43.716301 7f163605c700 10 mon.b@1(peon).paxosservice(mgrstat 26..26)  waiting for paxos -> readable (v27)
2017-10-06 12:01:43.716304 7f163605c700 10 mon.b@1(peon).paxosservice(health 1..1) post_refresh
...

/a/sage-2017-10-06_03:13:18-upgrade:jewel-x-wip-sage-testing2-2017-10-05-1735-distro-basic-smithi/1708536