Actions
Bug #41799
closedclient: FAILED assert(cap == in->auth_cap)
% Done:
0%
Source:
Community (dev)
Tags:
Backport:
nautilus,mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
crash, multimds, offline
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
below log explains the issue clearly, the auth_caps was set to NULL in previous remove_caps, and when add_update_cap get called from handle_import_cap, the seq == cap.seq holds.
2019-09-12 07:22:12.142637 7f200bee1700 1 -- 10.190.184.174:0/3460862904 <== mds.1 10.166.123.131:6800/3395465192 10 ==== client_caps(export ino 0x200000511ba 2782052306 seq 0 caps=- dirty=- wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (557563979 0 0) 0x564b9c90f600 con 0x564b9cc13000
2019-09-12 07:22:12.142721 7f200bee1700 10 client.138394001 mds.1 seq now 1
2019-09-12 07:22:12.142734 7f200bee1700 5 client.138394001 handle_cap_export ino 0x200000511ba mseq 0 EXPORT from mds.1
2019-09-12 07:22:12.142742 7f200bee1700 10 client.138394001 _open_mds_session mds.0
2019-09-12 07:22:12.143001 7f200bee1700 1 -- 10.190.184.174:0/3460862904 --> 10.212.160.177:6800/169156538 -- client_session(request_open) v2 -- 0x564b9c95aac0 con 0
2019-09-12 07:22:12.143087 7f200bee1700 10 client.138394001 add_update_cap issued - -> pAsLsXsFs from mds.0 on 0x200000511ba.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2019-09-12 07:22:04.840341 caps=pAsLsXsFs(0=pAsLsXsFs,1=pAsLsXsFs,3=pAsLsXsFs) COMPLETE quota(max_bytes = 6597069766656 max_files = 0) 0x564b9cc2a000)
2019-09-12 07:22:12.143131 7f200bee1700 10 client.138394001 remove_cap mds.1 on 0x200000511ba.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2019-09-12 07:22:04.840341 caps=pAsLsXsFs(0=pAsLsXsFs,1=pAsLsXsFs,3=pAsLsXsFs) COMPLETE quota(max_bytes = 6597069766656 max_files = 0) 0x564b9cc2a000)
2019-09-12 07:22:12.143159 7f200bee1700 1 -- 10.190.184.174:0/3460862904 <== mds.3 10.212.160.176:6800/3474105568 9 ==== client_caps(export ino 0x200000511ba 3772052911 seq 0 caps=- dirty=- wanted=- follows 0 size 0/0 mtime 0.000000) v11 ==== 252+0+0 (3141920983 0 0) 0x564b9c90f180 con 0x564b9cc10000
2019-09-12 07:22:12.143170 7f200bee1700 10 client.138394001 mds.3 seq now 3
2019-09-12 07:22:12.143172 7f200bee1700 5 client.138394001 handle_cap_export ino 0x200000511ba mseq 0 EXPORT from mds.3
2019-09-12 07:22:12.143174 7f200bee1700 10 client.138394001 remove_cap mds.3 on 0x200000511ba.head(faked_ino=0 ref=3 ll_ref=1 cap_refs={} open={} mode=40755 size=0/0 nlink=1 mtime=2019-09-12 07:22:04.840341 caps=pAsLsXsFs(0=pAsLsXsFs,3=pAsLsXsFs) COMPLETE quota(max_bytes = 6597069766656 max_files = 0) 0x564b9cc2a000)
2019-09-12 07:22:12.173783 7f200bee1700 10 client.138394001 ms_handle_connect on 10.212.160.177:6800/169156538
2019-09-12 07:22:12.181461 7f200bee1700 1 -- 10.190.184.174:0/3460862904 <== mds.0 10.212.160.177:6800/169156538 1 ==== client_session(open) v1 ==== 28+0+0 (2719320884 0 0) 0x564b9c958fc0 con 0x564b9cd92000
2019-09-12 07:22:12.181485 7f200bee1700 10 client.138394001 handle_client_session client_session(open) v1 from mds.0
2019-09-12 07:22:12.181489 7f200bee1700 10 client.138394001 renew_caps mds.0
2019-09-12 07:22:12.181491 7f200bee1700 1 -- 10.190.184.174:0/3460862904 --> 10.212.160.177:6800/169156538 -- client_session(request_renewcaps seq 1) v2 -- 0x564b9c95b840 con 0
2019-09-12 07:22:12.181544 7f200bee1700 10 client.138394001 connect_mds_targets for mds.0
2019-09-12 07:22:12.181553 7f200bee1700 1 -- 10.190.184.174:0/3460862904 <== mds.0 10.212.160.177:6800/169156538 2 ==== client_caps(import ino 0x200000511ba 5277779549 seq 1 caps=pAsLsXsFs dirty=- wanted=AsLsXsFs follows 0 size 0/0 ts 1/18446744073709551615 mtime 2019-09-12 07:22:04.840341 xattrs(v=1 l=4)) v11 ==== 321+4+0 (2210101217 0 0) 0x564b9c90ed00 con 0x564b9cd92000
2019-09-12 07:22:12.181571 7f200bee1700 10 client.138394001 mds.0 seq now 1
2019-09-12 07:22:12.181575 7f200bee1700 5 client.138394001 handle_cap_import ino 0x200000511ba mseq 0 IMPORT from mds.0
2019-09-12 07:22:12.181577 7f200bee1700 10 client.138394001 update_snap_trace len 48
2019-09-12 07:22:12.181580 7f200bee1700 20 client.138394001 get_snap_realm 0x1 0x564b9c934f70 14 -> 15
2019-09-12 07:22:12.181582 7f200bee1700 10 client.138394001 update_snap_trace snaprealm(0x1 nref=15 c=0 seq=1 parent=0x0 my_snaps=[] cached_snapc=1=[]) seq 1 <= 1 and same parent, SKIPPING
2019-09-12 07:22:12.182866 7f200bee1700 -1 /build/ceph-12.2.12/src/client/Client.cc: In function 'void Client::add_update_cap(Inode*, MetaSession*, uint64_t, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, const UserPerm&)' thread 7f200bee1700 time 2019-09-12 07:22:12.181592
/build/ceph-12.2.12/src/client/Client.cc: 3963: FAILED assert(cap == in->auth_cap)
(gdb) bt
#0 0x00007fbe58509269 in raise () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000056372c54477e in reraise_fatal (signum=6) at /build/ceph-12.2.12/src/global/signal_handler.cc:74
#2 handle_fatal_signal (signum=6) at /build/ceph-12.2.12/src/global/signal_handler.cc:138
#3 <signal handler called>
#4 0x00007fbe57295428 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fbe5729702a in abort () from /lib/x86_64-linux-gnu/libc.so.6
#6 0x000056372c15099e in ceph::__ceph_assert_fail (assertion=assertion@entry=0x56372c5d0e08 "cap == in->auth_cap", file=file@entry=0x56372c5cc6f8 "/build/ceph-12.2.12/src/client/Client.cc", line=line@entry=3963,
func=func@entry=0x56372c5d4860 <Client::add_update_cap(Inode*, MetaSession*, unsigned long, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, UserPerm const&)::__PRETTY_FUNCTION__> "void Client::add_update_cap(Inode*, MetaSession*, uint64_t, unsigned int, unsigned int, unsigned int, unsigned int, inodeno_t, int, const UserPerm&)") at /build/ceph-12.2.12/src/common/assert.cc:66
#7 0x000056372c0b570d in Client::add_update_cap (this=this@entry=0x563735d12000, in=in@entry=0x563735d54000, mds_session=mds_session@entry=0x563735b10d00, cap_id=5277779602, issued=341, wanted=340, seq=1, mseq=0, realm=..., flags=1,
cap_perms=...) at /build/ceph-12.2.12/src/client/Client.cc:3963
#8 0x000056372c0b73c6 in Client::handle_cap_import (this=this@entry=0x563735d12000, session=session@entry=0x563735b10d00, in=in@entry=0x563735d54000, m=m@entry=0x563735a40d00) at /build/ceph-12.2.12/src/client/Client.cc:4866
#9 0x000056372c0df721 in Client::handle_caps (this=this@entry=0x563735d12000, m=m@entry=0x563735a40d00) at /build/ceph-12.2.12/src/client/Client.cc:4821
#10 0x000056372c0e33cb in Client::ms_dispatch (this=0x563735d12000, m=0x563735a40d00) at /build/ceph-12.2.12/src/client/Client.cc:2571
#11 0x000056372c4d81ea in Messenger::ms_deliver_dispatch (m=0x563735a40d00, this=0x563735c6a000) at /build/ceph-12.2.12/src/msg/Messenger.h:668
#12 DispatchQueue::entry (this=0x563735c6a180) at /build/ceph-12.2.12/src/msg/DispatchQueue.cc:197
#13 0x000056372c1f37dd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /build/ceph-12.2.12/src/msg/DispatchQueue.h:101
#14 0x00007fbe584ff6ba in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#15 0x00007fbe5736741d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) f 7
#7 0x000056372c0b570d in Client::add_update_cap (this=this@entry=0x563735d12000, in=in@entry=0x563735d54000, mds_session=mds_session@entry=0x563735b10d00, cap_id=5277779602, issued=341, wanted=340, seq=1, mseq=0, realm=..., flags=1,
cap_perms=...) at /build/ceph-12.2.12/src/client/Client.cc:3963
3963 assert(cap == in->auth_cap);
(gdb) p in->auth_cap
$1 = (Cap *) 0x0
(gdb) p seq
$2 = 1
(gdb) p cap.seq
$3 = 1
Actions