Project

General

Profile

Bug #15467

After "mount -l", ceph-fuse does not work

Added by Shinobu Kinjo almost 8 years ago. Updated almost 7 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Since MDS stopped working because of running out ram, I did "mount -l" to unmount cephfs but did not restart the client machine.

After that I tried to mount cephfs, but it ended up with being failure.
It's probably because the session between server and client is never terminated.

[Command]
ceph-fuse -k /tmp/keyring -m 172.16.0.4:6789 /cephfs

[Output]
[Client]

2016-04-12 12:23:27.012569 7fda9665b740 -1 ceph-fuse11461: starting ceph client
init, newargv = 0x7fda9a6b18b0 newargc=11
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what(): buffer::malformed_input: void pg_pool_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding
  • Caught signal (Aborted)
    in thread 7fda8b7fe700
    ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
    1: (()+0x26579a) [0x7fda968e479a]
    2: (()+0xf100) [0x7fda95bf6100]
    3: (gsignal()+0x37) [0x7fda9470c5f7]
    4: (abort()+0x148) [0x7fda9470dce8]
    5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
    6: (()+0x5e946) [0x7fda9500e946]
    7: (()+0x5e973) [0x7fda9500e973]
    8: (()+0x5eb93) [0x7fda9500eb93]
    9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
    10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
    11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
    12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
    13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
    14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
    15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
    16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
    17: (()+0x7dc5) [0x7fda95beedc5]
    18: (clone()+0x6d) [0x7fda947cd28d]
    2016-04-12 12:23:27.023329 7fda8b7fe700 -1
    Caught signal (Aborted) *
    in thread 7fda8b7fe700

    ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
    1: (()+0x26579a) [0x7fda968e479a]
    2: (()+0xf100) [0x7fda95bf6100]
    3: (gsignal()+0x37) [0x7fda9470c5f7]
    4: (abort()+0x148) [0x7fda9470dce8]
    5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
    6: (()+0x5e946) [0x7fda9500e946]
    7: (()+0x5e973) [0x7fda9500e973]
    8: (()+0x5eb93) [0x7fda9500eb93]
    9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
    10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
    11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
    12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
    13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
    14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
    15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
    16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
    17: (()+0x7dc5) [0x7fda95beedc5]
    18: (clone()+0x6d) [0x7fda947cd28d]
    NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

    -41> 2016-04-12 12:23:27.012569 7fda9665b740 -1 init, newargv = 0x7fda9a6b18b0 newargc=11
    0> 2016-04-12 12:23:27.023329 7fda8b7fe700 -1 ** Caught signal (Aborted) *
    in thread 7fda8b7fe700

    ceph version 9.0.3-1845-gf1ead76 (f1ead76f80cc3d078778080c35a6427256874062)
    1: (()+0x26579a) [0x7fda968e479a]
    2: (()+0xf100) [0x7fda95bf6100]
    3: (gsignal()+0x37) [0x7fda9470c5f7]
    4: (abort()+0x148) [0x7fda9470dce8]
    5: (_gnu_cxx::_verbose_terminate_handler()+0x165) [0x7fda950109d5]
    6: (()+0x5e946) [0x7fda9500e946]
    7: (()+0x5e973) [0x7fda9500e973]
    8: (()+0x5eb93) [0x7fda9500eb93]
    9: (pg_pool_t::decode(ceph::buffer::list::iterator&)+0x1211) [0x7fda96a81151]
    10: (void decode<long, pg_pool_t>(std::map<long, pg_pool_t, std::less<long>, std::allocator<std::pair<long const, pg_pool_t> > >&, ceph::buffer::list::iterator&)+0xf1) [0x7fda9699d461]
    11: (OSDMap::decode(ceph::buffer::list::iterator&)+0x372) [0x7fda96991952]
    12: (OSDMap::decode(ceph::buffer::list&)+0x3f) [0x7fda96993f2f]
    13: (Objecter::handle_osd_map(MOSDMap*)+0x1964) [0x7fda968c8824]
    14: (Objecter::ms_dispatch(Message*)+0x23f) [0x7fda968cd27f]
    15: (DispatchQueue::entry()+0x63a) [0x7fda96ac1cfa]
    16: (DispatchQueue::DispatchThread::entry()+0xd) [0x7fda969ae92d]
    17: (()+0x7dc5) [0x7fda95beedc5]
    18: (clone()+0x6d) [0x7fda947cd28d]
    NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

ceph-fuse11459: mount failed: (33) Numerical argument out of domain

[MON@Sever]
2016-04-11 23:35:58.079300 7fa1da6a2700 10 mon.a@0(leader) e1 ms_handle_reset 0x7fa1edc2da80 172.16.0.3:0/1513410916
2016-04-11 23:35:58.079382 7fa1da6a2700 10 mon.a@0(leader) e1 reset/close on session client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.079420 7fa1da6a2700 10 mon.a@0(leader) e1 remove_session 0x7fa1ecebf9c0 client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.080060 7fa1d7f97700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x7fa1ed887400 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1edc2dd80).accept sd=25 172.16.0.3:58699/0
2016-04-11 23:35:58.080113 7fa1d7f97700 0 -- 172.16.0.4:6789/0 >> 172.16.0.3:0/1513410916 pipe(0x7fa1ed887400 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1edc2dd80).accept peer addr is really 172.16.0.3:0/1513410916 (socket is 172.16.0.3:58699/0)
2016-04-11 23:35:58.080809 7fa1d7f97700 10 mon.a@0(leader) e1 ms_verify_authorizer 172.16.0.3:0/1513410916 client protocol 0
2016-04-11 23:35:58.081319 7fa1da6a2700 1 -- 172.16.0.4:6789/0 <== client.4121 172.16.0.3:0/1513410916 1 ==== auth(proto 0 34 bytes epoch 0) ==== 60+0+0 (2519504818 0 0) 0x7fa1ed345680 con 0x7fa1edc2dd80
2016-04-11 23:35:58.081421 7fa1da6a2700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7fa1ecebf9c0 MonSession(client.4121 172.16.0.3:0/1513410916 is open)
2016-04-11 23:35:58.081463 7fa1da6a2700 10 mon.a@0(leader).auth v8 preprocess_query auth(proto 0 34 bytes epoch 0) from client.4121 172.16.0.3:0/1513410916
2016-04-11 23:35:58.081498 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- mon_map magic: 0 v1 -- ?+0 0x7fa1ed345f80 con 0x7fa1edc2dd80
2016-04-11 23:35:58.081626 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x7fa1ed344fc0 con 0x7fa1edc2dd80
2016-04-11 23:36:08.095451 7fa1da6a2700 10 mon.a@0(leader) e1 ms_handle_reset 0x7fa1edc2dd80 172.16.0.3:0/1513410916
2016-04-11 23:36:08.095525 7fa1da6a2700 10 mon.a@0(leader) e1 reset/close on session client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.095558 7fa1da6a2700 10 mon.a@0(leader) e1 remove_session 0x7fa1ecebf9c0 client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.096198 7fa1d7f97700 1 -- 172.16.0.4:6789/0 >> :/0 pipe(0x7fa1ed1a8000 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1ed512d00).accept sd=25 172.16.0.3:58706/0
2016-04-11 23:36:08.096299 7fa1d7f97700 0 -- 172.16.0.4:6789/0 >> 172.16.0.3:0/1513410916 pipe(0x7fa1ed1a8000 sd=25 :6789 s=0 pgs=0 cs=0 l=0 c=0x7fa1ed512d00).accept peer addr is really 172.16.0.3:0/1513410916 (socket is 172.16.0.3:58706/0)
2016-04-11 23:36:08.097109 7fa1d7f97700 10 mon.a@0(leader) e1 ms_verify_authorizer 172.16.0.3:0/1513410916 client protocol 0
2016-04-11 23:36:08.098082 7fa1da6a2700 1 -- 172.16.0.4:6789/0 <== client.4121 172.16.0.3:0/1513410916 1 ==== auth(proto 0 34 bytes epoch 0) ==== 60+0+0 (2519504818 0 0) 0x7fa1ed345680 con 0x7fa1ed512d00
2016-04-11 23:36:08.098176 7fa1da6a2700 10 mon.a@0(leader) e1 _ms_dispatch new session 0x7fa1ecebf9c0 MonSession(client.4121 172.16.0.3:0/1513410916 is open)
2016-04-11 23:36:08.098240 7fa1da6a2700 10 mon.a@0(leader).auth v8 preprocess_query auth(proto 0 34 bytes epoch 0) from client.4121 172.16.0.3:0/1513410916
2016-04-11 23:36:08.098278 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- mon_map magic: 0 v1 -- ?+0 0x7fa1ed345f80 con 0x7fa1ed512d00
2016-04-11 23:36:08.098368 7fa1da6a2700 1 -- 172.16.0.4:6789/0 --> 172.16.0.3:0/1513410916 -- auth_reply(proto 2 0 (0) Success) v1 -- ?+0 0x7fa1ed3458c0 con 0x7fa1ed512d00

History

#1 Updated by Shinobu Kinjo almost 8 years ago

- mount -l
+ unmount -l

#2 Updated by Greg Farnum almost 7 years ago

  • Project changed from Ceph to CephFS
  • Category deleted (Monitor)

#3 Updated by John Spray almost 7 years ago

  • Status changed from New to Won't Fix

This appears to have happened on pre-jewel code, so it's unlikely anyone is interested in investigating.

Also available in: Atom PDF