Actions
Bug #41512
closed"*** Caught signal (Segmentation fault) **" in upgrade:luminous-x-nautilus in PyModuleRunner::log
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/luminous-x, upgrade/mimic-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2019-08-26_02:25:03-upgrade:luminous-x-nautilus-distro-basic-smithi/
Jobs: many
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2019-08-26_02:25:03-upgrade:luminous-x-nautilus-distro-basic-smithi/4252328/teuthology.log
zgrep "^ ceph version" /a/teuthology-2019-08-26_02:25:03-upgrade:luminous-x-nautilus-distro-basic-smithi/4252328/remote/smithi050/log/ceph-mgr.x.log.gz -b20 -a30
2325623-2019-08-26 03:00:34.412 7fc1f57e9700 4 mgr.server handle_open from 0x557e84b72c00 client,x 2325716-2019-08-26 03:00:34.412 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] --> 172.21.15.50:0/1183373898 -- mgrconfigure(period=5, threshold=5) v3 -- 0x557e81617a40 con 0x557e84b72c00 2325924-2019-08-26 03:00:34.412 7fc1e7f4e700 1 -- 172.21.15.50:0/1873085746 --> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] -- mgrreport(unknown.x +16-0 packed 134) v7 -- 0x557e84d3f800 con 0x557e84d26800 2326134-2019-08-26 03:00:34.412 7fc1eff9e700 1 -- 172.21.15.50:0/1183373898 <== mgr.74108 v2:172.21.15.50:6800/16906 1 ==== mgrconfigure(period=5, threshold=5) v3 ==== 12+0+0 (crc 0 0 0) 0x557e81617a40 con 0x557e84d26000 2326348-2019-08-26 03:00:34.412 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] <== client.74144 172.21.15.50:0/1873085746 2 ==== mgrreport(client.x +16-0 packed 134) v7 ==== 1272+0+0 (crc 0 0 0) 0x557e84e67800 con 0x557e84b72800 2326597-2019-08-26 03:00:34.412 7fc1f57e9700 4 mgr.server handle_report from 0x557e84b72800 client,x 2326691-2019-08-26 03:00:34.412 7fc1f57e9700 4 mgr.server handle_report rejecting report from non-daemon client x 2326798-2019-08-26 03:00:34.412 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1873085746 conn(0x557e84b72800 msgr2=0x557e84e80000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2327021-2019-08-26 03:00:34.412 7fc1f57e9700 1 --2- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1873085746 conn(0x557e84b72800 0x557e84e80000 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2327233-2019-08-26 03:00:34.412 7fc1eff9e700 1 -- 172.21.15.50:0/1183373898 --> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] -- mgrreport(unknown.x +16-0 packed 134) v7 -- 0x557e84cb5800 con 0x557e84d26000 2327443-2019-08-26 03:00:34.412 7fc2121f7700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 52 2327696-2019-08-26 03:00:34.413 7fc2121f7700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2327932-2019-08-26 03:00:34.413 7fc2121f7700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2328240-2019-08-26 03:00:34.413 7fc2121f7700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).stop 2328453-2019-08-26 03:00:34.413 7fc1e7f4e700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2328664-2019-08-26 03:00:34.413 7fc1e7f4e700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2328882-2019-08-26 03:00:34.413 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] <== client.74138 172.21.15.50:0/1183373898 2 ==== mgrreport(client.x +16-0 packed 134) v7 ==== 1272+0+0 (crc 0 0 0) 0x557e84cb5800 con 0x557e84b72c00 2329131-2019-08-26 03:00:34.413 7fc1f57e9700 4 mgr.server handle_report from 0x557e84b72c00 client,x 2329225-2019-08-26 03:00:34.413 7fc1f57e9700 4 mgr.server handle_report rejecting report from non-daemon client x 2329332-2019-08-26 03:00:34.413 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1183373898 conn(0x557e84b72c00 msgr2=0x557e84e80580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2329555-2019-08-26 03:00:34.413 7fc1f57e9700 1 --2- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1183373898 conn(0x557e84b72c00 0x557e84e80580 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2329767-2019-08-26 03:00:34.413 7fc2129f8700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 53 2330020-2019-08-26 03:00:34.413 7fc2129f8700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2330256-2019-08-26 03:00:34.413 7fc2129f8700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2330564-2019-08-26 03:00:34.413 7fc2129f8700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).stop 2330777-2019-08-26 03:00:34.413 7fc1eff9e700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2330988-2019-08-26 03:00:34.413 7fc1eff9e700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2331206-2019-08-26 03:00:34.426 7fc1f0fa0700 -1 *** Caught signal (Segmentation fault) ** 2331288- in thread 7fc1f0fa0700 thread_name:mgr-fin 2331332- 2331333: ceph version 14.2.2-471-g49a6c9a (49a6c9aaf29cfb947ac384277ebed4d7897eaf01) nautilus (stable) 2331428- 1: (()+0xf5d0) [0x7fc216d3c5d0] 2331461- 2: (Mutex::lock(bool)+0x9) [0x7fc2194ace59] 2331506- 3: (PyModuleRunner::log(int, std::string const&)+0x100) [0x557e7f31eb90] 2331580- 4: (()+0x132826) [0x557e7f285826] 2331615- 5: (PyEval_EvalFrameEx()+0x6df0) [0x7fc218efccf0] 2331666- 6: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2331717- 7: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2331768- 8: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2331819- 9: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2331870- 10: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2331920- 11: (()+0x70a6d) [0x7fc218e88a6d] 2331955- 12: (PyObject_Call()+0x43) [0x7fc218e63a63] 2332000- 13: (PyEval_EvalFrameEx()+0x17fd) [0x7fc218ef76fd] 2332052- 14: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2332102- 15: (PyEval_EvalFrameEx()+0x663c) [0x7fc218efc53c] 2332154- 16: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2332204- 17: (()+0x70a6d) [0x7fc218e88a6d] 2332239- 18: (PyObject_Call()+0x43) [0x7fc218e63a63] 2332284- 19: (PyEval_EvalFrameEx()+0x17fd) [0x7fc218ef76fd] 2332336- 20: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2332388- 21: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2332440- 22: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2332490- 23: (()+0x70978) [0x7fc218e88978] 2332525- 24: (PyObject_Call()+0x43) [0x7fc218e63a63] 2332570- 25: (()+0x5aa55) [0x7fc218e72a55] 2332605- 26: (PyObject_Call()+0x43) [0x7fc218e63a63] 2332650- 27: (PyEval_CallObjectWithKeywords()+0x47) [0x7fc218ef58f7] 2332711- 28: (()+0x115822) [0x7fc218f2d822] 2332747- 29: (()+0x7dd5) [0x7fc216d34dd5] 2332781- 30: (clone()+0x6d) [0x7fc2159e502d] -- 2932433- -27> 2019-08-26 03:00:34.412 7fc1f57e9700 4 mgr.server handle_report rejecting report from non-daemon client x 2932548- -26> 2019-08-26 03:00:34.412 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1873085746 conn(0x557e84b72800 msgr2=0x557e84e80000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2932779- -25> 2019-08-26 03:00:34.412 7fc1f57e9700 1 --2- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1873085746 conn(0x557e84b72800 0x557e84e80000 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2932999- -24> 2019-08-26 03:00:34.412 7fc1eff9e700 1 -- 172.21.15.50:0/1183373898 --> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] -- mgrreport(unknown.x +16-0 packed 134) v7 -- 0x557e84cb5800 con 0x557e84d26000 2933217- -23> 2019-08-26 03:00:34.412 7fc2121f7700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 52 2933478- -22> 2019-08-26 03:00:34.413 7fc2121f7700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2933722- -21> 2019-08-26 03:00:34.413 7fc2121f7700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2934038- -20> 2019-08-26 03:00:34.413 7fc2121f7700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 crc :-1 s=READY pgs=36 cs=0 l=1 rx=0 tx=0).stop 2934259- -19> 2019-08-26 03:00:34.413 7fc1e7f4e700 4 mgrc ms_handle_reset ms_handle_reset con 0x557e84d26800 2934363- -18> 2019-08-26 03:00:34.413 7fc1e7f4e700 4 mgrc reconnect Terminating session with v2:172.21.15.50:6800/16906 2934478- -17> 2019-08-26 03:00:34.413 7fc1e7f4e700 1 -- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 msgr2=0x557e84cfdb80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2934697- -16> 2019-08-26 03:00:34.413 7fc1e7f4e700 1 --2- 172.21.15.50:0/1873085746 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26800 0x557e84cfdb80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rx=0 tx=0).stop 2934923- -15> 2019-08-26 03:00:34.413 7fc1e7f4e700 4 mgrc reconnect waiting to retry connect until 2019-08-26 03:00:35.412172 2935044- -14> 2019-08-26 03:00:34.413 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] <== client.74138 172.21.15.50:0/1183373898 2 ==== mgrreport(client.x +16-0 packed 134) v7 ==== 1272+0+0 (crc 0 0 0) 0x557e84cb5800 con 0x557e84b72c00 2935301- -13> 2019-08-26 03:00:34.413 7fc1f57e9700 4 mgr.server handle_report from 0x557e84b72c00 client,x 2935403- -12> 2019-08-26 03:00:34.413 7fc1f57e9700 4 mgr.server handle_report rejecting report from non-daemon client x 2935518- -11> 2019-08-26 03:00:34.413 7fc1f57e9700 1 -- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1183373898 conn(0x557e84b72c00 msgr2=0x557e84e80580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2935749- -10> 2019-08-26 03:00:34.413 7fc1f57e9700 1 --2- [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] >> 172.21.15.50:0/1183373898 conn(0x557e84b72c00 0x557e84e80580 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2935969- -9> 2019-08-26 03:00:34.413 7fc2129f8700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 53 2936230- -8> 2019-08-26 03:00:34.413 7fc2129f8700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2936474- -7> 2019-08-26 03:00:34.413 7fc2129f8700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2936790- -6> 2019-08-26 03:00:34.413 7fc2129f8700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 crc :-1 s=READY pgs=37 cs=0 l=1 rx=0 tx=0).stop 2937011- -5> 2019-08-26 03:00:34.413 7fc1eff9e700 4 mgrc ms_handle_reset ms_handle_reset con 0x557e84d26000 2937115- -4> 2019-08-26 03:00:34.413 7fc1eff9e700 4 mgrc reconnect Terminating session with v2:172.21.15.50:6800/16906 2937230- -3> 2019-08-26 03:00:34.413 7fc1eff9e700 1 -- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 msgr2=0x557e84cfd600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2937449- -2> 2019-08-26 03:00:34.413 7fc1eff9e700 1 --2- 172.21.15.50:0/1183373898 >> [v2:172.21.15.50:6800/16906,v1:172.21.15.50:6818/16906] conn(0x557e84d26000 0x557e84cfd600 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2937675- -1> 2019-08-26 03:00:34.413 7fc1eff9e700 4 mgrc reconnect waiting to retry connect until 2019-08-26 03:00:35.412143 2937796- 0> 2019-08-26 03:00:34.426 7fc1f0fa0700 -1 *** Caught signal (Segmentation fault) ** 2937886- in thread 7fc1f0fa0700 thread_name:mgr-fin 2937930- 2937931: ceph version 14.2.2-471-g49a6c9a (49a6c9aaf29cfb947ac384277ebed4d7897eaf01) nautilus (stable) 2938026- 1: (()+0xf5d0) [0x7fc216d3c5d0] 2938059- 2: (Mutex::lock(bool)+0x9) [0x7fc2194ace59] 2938104- 3: (PyModuleRunner::log(int, std::string const&)+0x100) [0x557e7f31eb90] 2938178- 4: (()+0x132826) [0x557e7f285826] 2938213- 5: (PyEval_EvalFrameEx()+0x6df0) [0x7fc218efccf0] 2938264- 6: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2938315- 7: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2938366- 8: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2938417- 9: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2938468- 10: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2938518- 11: (()+0x70a6d) [0x7fc218e88a6d] 2938553- 12: (PyObject_Call()+0x43) [0x7fc218e63a63] 2938598- 13: (PyEval_EvalFrameEx()+0x17fd) [0x7fc218ef76fd] 2938650- 14: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2938700- 15: (PyEval_EvalFrameEx()+0x663c) [0x7fc218efc53c] 2938752- 16: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2938802- 17: (()+0x70a6d) [0x7fc218e88a6d] 2938837- 18: (PyObject_Call()+0x43) [0x7fc218e63a63] 2938882- 19: (PyEval_EvalFrameEx()+0x17fd) [0x7fc218ef76fd] 2938934- 20: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2938986- 21: (PyEval_EvalFrameEx()+0x67bd) [0x7fc218efc6bd] 2939038- 22: (PyEval_EvalCodeEx()+0x7ed) [0x7fc218eff03d] 2939088- 23: (()+0x70978) [0x7fc218e88978] 2939123- 24: (PyObject_Call()+0x43) [0x7fc218e63a63] 2939168- 25: (()+0x5aa55) [0x7fc218e72a55] 2939203- 26: (PyObject_Call()+0x43) [0x7fc218e63a63] 2939248- 27: (PyEval_CallObjectWithKeywords()+0x47) [0x7fc218ef58f7] 2939309- 28: (()+0x115822) [0x7fc218f2d822] 2939345- 29: (()+0x7dd5) [0x7fc216d34dd5] 2939379- 30: (clone()+0x6d) [0x7fc2159e502d]
Actions