Bug #15467
closedAfter "mount -l", ceph-fuse does not work
0%
Description
Since MDS stopped working because of running out ram, I did "mount -l" to unmount cephfs but did not restart the client machine.
After that I tried to mount cephfs, but it ended up with being failure.
It's probably because the session between server and client is never terminated.
[Command]
ceph-fuse -k /tmp/keyring -m 172.16.0.4:6789 /cephfs
[Client]
2016-04-12 12:23:27.012569 7fda9665b740 -1 ceph-fuse11461: starting ceph client
init, newargv = 0x7fda9a6b18b0 newargc=11
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what(): buffer::malformed_input: void pg_pool_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
- Caught signal (Aborted)
in thread 7fda8b7fe700
ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
1: (()+0x26579a) [0x7fda968e479a]
2: (()+0xf100) [0x7fda95bf6100]
3: (gsignal()+0x37) [0x7fda9470c5f7]
4: (abort()+0x148) [0x7fda9470dce8]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
6: (()+0x5e946) [0x7fda9500e946]
7: (()+0x5e973) [0x7fda9500e973]
8: (()+0x5eb93) [0x7fda9500eb93]
9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
17: (()+0x7dc5) [0x7fda95beedc5]
18: (clone()+0x6d) [0x7fda947cd28d]
2016-04-12 12:23:27.023329 7fda8b7fe700 -1 Caught signal (Aborted) *
in thread 7fda8b7fe700ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
1: (()+0x26579a) [0x7fda968e479a]
2: (()+0xf100) [0x7fda95bf6100]
3: (gsignal()+0x37) [0x7fda9470c5f7]
4: (abort()+0x148) [0x7fda9470dce8]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
6: (()+0x5e946) [0x7fda9500e946]
7: (()+0x5e973) [0x7fda9500e973]
8: (()+0x5eb93) [0x7fda9500eb93]
9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
17: (()+0x7dc5) [0x7fda95beedc5]
18: (clone()+0x6d) [0x7fda947cd28d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.-41> 2016-04-12 12:23:27.012569 7fda9665b740 -1 init, newargv = 0x7fda9a6b18b0 newargc=11
0> 2016-04-12 12:23:27.023329 7fda8b7fe700 -1 ** Caught signal (Aborted) *
in thread 7fda8b7fe700ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
1: (()+0x26579a) [0x7fda968e479a]
2: (()+0xf100) [0x7fda95bf6100]
3: (gsignal()+0x37) [0x7fda9470c5f7]
4: (abort()+0x148) [0x7fda9470dce8]
5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
6: (()+0x5e946) [0x7fda9500e946]
7: (()+0x5e973) [0x7fda9500e973]
8: (()+0x5eb93) [0x7fda9500eb93]
9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
17: (()+0x7dc5) [0x7fda95beedc5]
18: (clone()+0x6d) [0x7fda947cd28d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
ceph-fuse11459: mount failed: (33) Numerical argument out of domain
[MON@Sever]
2016-04-11 23:35:58.079300 7fa1da6a2700 10 mon.a@0(leader) e1 ms_handle_reset 0x7fa1edc2da80 172.16.0.3:0/1513410916
2016-04-11 23:35:58.079382 7fa1da6a2700 10 mon.a@0(leader) e1 reset/close on session client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.079420 7fa1da6a2700 10 mon.a@0(leader) e1 remove_session 0x7fa1ecebf9c0 client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.080060 7fa1d7f97700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x7fa1ed887400 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1edc2dd80).accept sd=25 172.16.0.3:58699/0
2016-04-11 23:35:58.080113 7fa1d7f97700 0 -- 172.16.0.4:6789/0 >> 172.16.0.3:0/1513410916 pipe(0x7fa1ed887400 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1edc2dd80).accept peer addr is really 172.16.0.3:0/1513410916 (socket is 172.16.0.3:58699/0)
2016-04-11 23:35:58.080809 7fa1d7f97700 10 mon.a@0(leader) e1 ms_verify_authorizer 172.16.0.3:0/1513410916 client protocol 0
2016-04-11 23:35:58.081319 7fa1da6a2700 1 -- 172.16.0.4:6789/0 <== client.4121 172.16.0.3:0/1513410916 1 ==== auth(proto 0 34 bytes epoch 0) ==== 60+0+0 (2519504818 0 0) 0x7fa1ed345680 con 0x7fa1edc2dd80
2016-04-11 23:35:58.081421 7fa1da6a2700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7fa1ecebf9c0 MonSession(client.4121 172.16.0.3:0/1513410916 is open)
2016-04-11 23:35:58.081463 7fa1da6a2700 10 mon.a@0(leader).auth v8 preprocess_query auth(proto 0 34 bytes epoch 0) from client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.081498 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- mon_map magic: 0 v1 -- ?+0 0x7fa1ed345f80 con 0x7fa1edc2dd80
2016-04-11 23:35:58.081626 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x7fa1ed344fc0 con 0x7fa1edc2dd80
2016-04-11 23:36:08.095451 7fa1da6a2700 10 mon.a@0(leader) e1 ms_handle_reset 0x7fa1edc2dd80 172.16.0.3:0/1513410916
2016-04-11 23:36:08.095525 7fa1da6a2700 10 mon.a@0(leader) e1 reset/close on session client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.095558 7fa1da6a2700 10 mon.a@0(leader) e1 remove_session 0x7fa1ecebf9c0 client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.096198 7fa1d7f97700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x7fa1ed1a8000 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1ed512d00).accept sd=25 172.16.0.3:58706/0
2016-04-11 23:36:08.096299 7fa1d7f97700 0 -- 172.16.0.4:6789/0 >> 172.16.0.3:0/1513410916 pipe(0x7fa1ed1a8000 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1ed512d00).accept peer addr is really 172.16.0.3:0/1513410916 (socket is 172.16.0.3:58706/0)
2016-04-11 23:36:08.097109 7fa1d7f97700 10 mon.a@0(leader) e1 ms_verify_authorizer 172.16.0.3:0/1513410916 client protocol 0
2016-04-11 23:36:08.098082 7fa1da6a2700 1 -- 172.16.0.4:6789/0 <== client.4121 172.16.0.3:0/1513410916 1 ==== auth(proto 0 34 bytes epoch 0) ==== 60+0+0 (2519504818 0 0) 0x7fa1ed345680 con 0x7fa1ed512d00
2016-04-11 23:36:08.098176 7fa1da6a2700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7fa1ecebf9c0 MonSession(client.4121 172.16.0.3:0/1513410916 is open)
2016-04-11 23:36:08.098240 7fa1da6a2700 10 mon.a@0(leader).auth v8 preprocess_query auth(proto 0 34 bytes epoch 0) from client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.098278 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- mon_map magic: 0 v1 -- ?+0 0x7fa1ed345f80 con 0x7fa1ed512d00
2016-04-11 23:36:08.098368 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x7fa1ed3458c0 con 0x7fa1ed512d00
Updated by Greg Farnum about 7 years ago
- Project changed from Ceph to CephFS
- Category deleted (
Monitor)
Updated by John Spray about 7 years ago
- Status changed from New to Won't Fix
This appears to have happened on pre-jewel code, so it's unlikely anyone is interested in investigating.