Project

General

Profile

Actions
Copy link

Bug #19802

closed

mgr: Exception KeyError: KeyError(140194794858240,) in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored

Added by Sage Weil almost 7 years ago. Updated almost 7 years ago.

Status:
Can't reproduce
Priority:
Immediate
Assignee:
Kefu Chai
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-04-28T00:29:18.734 INFO:tasks.ceph.mgr.x.smithi149.stderr:2017-04-28 00:29:18.734622 7f81a4f26700 -1 mgr handle_mgr_map I was active but no longer am
2017-04-28T00:29:18.735 INFO:tasks.ceph.mgr.x.smithi149.stderr:Exception KeyError: KeyError(140194794858240,) in <module 'threading' from '/usr/lib64/python2.7/threading.pyc'> ignored
2017-04-28T00:29:19.094 INFO:tasks.ceph.mgr.x.smithi149.stderr:*** Caught signal (Segmentation fault) **
2017-04-28T00:29:19.095 INFO:tasks.ceph.mgr.x.smithi149.stderr: in thread 7f81a1f20700 thread_name:mgr-fin
</prE>
/a/sage-2017-04-28_00:12:30-rados-wip-sage-testing2---basic-smithi/1075127
Actions
Copy link
 #1

Updated by Sage Weil almost 7 years ago

also /a/sage-2017-04-28_00:12:30-rados-wip-sage-testing2---basic-smithi/1075374

this was the last mgr pr to merge before this started: https://github.com/ceph/ceph/pull/14802

Actions
Copy link
 #2

Updated by Sage Weil almost 7 years ago

hmm on this run /a/sage-2017-04-28_16:57:50-rados-wip-sage-testing2---basic-smithi/1077063 we got 

2017-04-28T19:08:50.983 INFO:tasks.ceph.mgr.x.smithi084.stderr:2017-04-28 19:08:50.985319 7f74c8eb2700 -1 mgr handle_mgr_map I was active but no longer am
2017-04-28T19:08:50.984 INFO:tasks.ceph.mgr.x.smithi084.stderr:Exception KeyError: KeyError(140139563788032,) in <module 'threading' from '/usr/lib/python2.7/threading.pyc'> ignored
2017-04-28T19:08:51.935 INFO:tasks.workunit.client.0.smithi084.stderr:set-quota max_objects = 1 for pool 3dc56cf5-1f43-47b8-958c-a77504d5b65e
2017-04-28T19:08:51.970 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:08:51.970086 7fbb82185980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:08:51.971 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:08:51.970258 7fbb82185980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:08:51.971 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:08:51.971940 7fbb82185980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:02.067 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:02.067201 7f345f780700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:02.083 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:02.084908 7f345f780700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:02.227 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:02.229331 7f344ffff700  0 -- 172.21.15.84:0/2802664066 >> 172.21.15.84:6789/0 conn(0x7f3438019910 :-1 s=STATE_OPEN_MESSAGE_HEADER pgs=415 cs=1 l=1).read_until injecting socket failure
2017-04-28T19:09:04.323 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:04.324924 7fc118915700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:04.340 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:04.342436 7fc118915700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:06.586 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:06.587510 7f07e335c700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:06.595 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:06.597383 7f07e335c700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:08.821 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:08.822578 7f703a496700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:08.838 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:08.839940 7f703a496700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:11.066 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:11.067639 7f3780b41700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:11.079 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:11.081094 7f3780b41700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:11.230 INFO:tasks.workunit.client.0.smithi084.stdout:pool 8 '3dc56cf5-1f43-47b8-958c-a77504d5b65e' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 1 pgp_num 1 last_change 34 flags hashpspool,full max_objects 1 stripe_width 0
2017-04-28T19:09:11.293 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:11.241750 7f185e39f980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:11.293 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:11.241910 7f185e39f980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:11.293 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:11.242887 7f185e39f980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:12.628 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:12.629705 7f8f42614980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:12.628 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:12.629902 7f8f42614980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:12.630 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:12.631693 7f8f42614980 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:13.802 INFO:tasks.workunit.client.0.smithi084.stdout:successfully deleted pool 3dc56cf5-1f43-47b8-958c-a77504d5b65e
2017-04-28T19:09:13.820 INFO:tasks.workunit.client.0.smithi084.stdout:Testing rados ls command
2017-04-28T19:09:13.892 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:13.894453 7fb75cd50700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:13.919 INFO:tasks.workunit.client.0.smithi084.stderr:2017-04-28 19:09:13.904859 7fb75cd50700 -1 WARNING: all dangerous and experimental features are enabled.
2017-04-28T19:09:15.170 INFO:tasks.workunit.client.0.smithi084.stderr:pool '7ab01f94-5d53-4e99-be43-a50f45c2046f' already exists
2017-04-28T19:09:50.984 INFO:tasks.ceph.mgr.x.smithi084.stderr:2017-04-28 19:09:50.985870 7f74c8eb2700 -1 mgr handle_mgr_map I was active but no longer am
2017-04-28T19:09:50.985 INFO:tasks.ceph.mgr.x.smithi084.stderr:*** Caught signal (Segmentation fault) **
2017-04-28T19:09:50.985 INFO:tasks.ceph.mgr.x.smithi084.stderr: in thread 7f74c8eb2700 thread_name:ms_dispatch
2017-04-28T19:09:51.003 INFO:tasks.ceph.mgr.x.smithi084.stderr: ceph version 12.0.1-1689-gcd86908 (cd86908220578fbe091355ce3d8d18166e4cb352)
2017-04-28T19:09:51.003 INFO:tasks.ceph.mgr.x.smithi084.stderr: 1: (()+0x29ef22) [0x55ff12553f22]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 2: (()+0x113e0) [0x7f74ceb6f3e0]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 3: (PyThreadState_New()+0x102) [0x7f74cfcdaba2]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 4: (PyGILState_Ensure()+0x3c) [0x7f74cfcbcc0c]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 5: (PyModules::shutdown()+0x888) [0x55ff124e6658]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 6: (Mgr::shutdown()+0x2f) [0x55ff124fd1df]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 7: (MgrStandby::handle_mgr_map(MMgrMap*)+0x3bb) [0x55ff124f62bb]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 8: (MgrStandby::ms_dispatch(Message*)+0x216) [0x55ff124f6b36]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 9: (DispatchQueue::entry()+0xf4a) [0x55ff127d0ada]
2017-04-28T19:09:51.004 INFO:tasks.ceph.mgr.x.smithi084.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x55ff12641e1d]
2017-04-28T19:09:51.005 INFO:tasks.ceph.mgr.x.smithi084.stderr: 11: (()+0x770a) [0x7f74ceb6570a]
2017-04-28T19:09:51.005 INFO:tasks.ceph.mgr.x.smithi084.stderr: 12: (clone()+0x6d) [0x7f74cdbd182d]
2017-04-28T19:09:51.005 INFO:tasks.ceph.mgr.x.smithi084.stderr:2017-04-28 19:09:51.005147 7f74c8eb2700 -1 *** Caught signal (Segmentation fault) **
2017-04-28T19:09:51.006 INFO:tasks.ceph.mgr.x.smithi084.stderr: in thread 7f74c8eb2700 thread_name:ms_dispatch
2017-04-28T19:09:51.006 INFO:tasks.ceph.mgr.x.smithi084.stderr:
2017-04-28T19:09:51.006 INFO:tasks.ceph.mgr.x.smithi084.stderr: ceph version 12.0.1-1689-gcd86908 (cd86908220578fbe091355ce3d8d18166e4cb352)
2017-04-28T19:09:51.006 INFO:tasks.ceph.mgr.x.smithi084.stderr: 1: (()+0x29ef22) [0x55ff12553f22]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 2: (()+0x113e0) [0x7f74ceb6f3e0]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 3: (PyThreadState_New()+0x102) [0x7f74cfcdaba2]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 4: (PyGILState_Ensure()+0x3c) [0x7f74cfcbcc0c]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 5: (PyModules::shutdown()+0x888) [0x55ff124e6658]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 6: (Mgr::shutdown()+0x2f) [0x55ff124fd1df]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 7: (MgrStandby::handle_mgr_map(MMgrMap*)+0x3bb) [0x55ff124f62bb]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 8: (MgrStandby::ms_dispatch(Message*)+0x216) [0x55ff124f6b36]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 9: (DispatchQueue::entry()+0xf4a) [0x55ff127d0ada]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 10: (DispatchQueue::DispatchThread::entry()+0xd) [0x55ff12641e1d]
2017-04-28T19:09:51.007 INFO:tasks.ceph.mgr.x.smithi084.stderr: 11: (()+0x770a) [0x7f74ceb6570a]
2017-04-28T19:09:51.008 INFO:tasks.ceph.mgr.x.smithi084.stderr: 12: (clone()+0x6d) [0x7f74cdbd182d]
2017-04-28T19:09:51.008 INFO:tasks.ceph.mgr.x.smithi084.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2017-04-28T19:09:51.008 INFO:tasks.ceph.mgr.x.smithi084.stderr:

Actions
Copy link
 #3

Updated by Kefu Chai almost 7 years ago

  • Assignee set to Kefu Chai
Actions
Copy link
 #5

Updated by Kefu Chai almost 7 years ago

  • Status changed from New to Can't reproduce

all 10 runs passed. i was testing using

teuthology-suite --ceph master --filter='rados/singleton/{all/radostool.yaml msgr-failures/many.yaml msgr/async.yaml objectstore/filestore-xfs.yaml rados.yaml}' --suite rados --machine-type=mira --ceph-repo https://github.com/ceph/ceph.git  --suite-repo https://github.com/ceph/ceph.git --suite-branch master -N 10

Actions
Copy link

Also available in: Atom PDF