Project

General

Profile

Bug #15564

ceph-mon partially broken by lttng due to crushtool

Added by David Zafman over 3 years ago. Updated over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
LTTNG
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Monitor
Pull request ID:
Crash signature:

Description

$ lttng create
$ lttng enable-event -u -a
$ lttng start
$ MDS=0 MON=1 OSD=3 LD_PRELOAD=/usr/lib/x86_64-linux-gnu/liblttng-ust-fork.so ./vstart.sh -d -l -n
$ rados mkpool test
error creating pool test: (22) Invalid argument

2016-04-21 16:15:00.812655 7f459dc79700  1 -- 127.0.0.1:6789/0 <== client.4109 127.0.0.1:0/1254741436 6 ==== pool_op(create pool 0 auid 0 tid 1 name test v0) v4 ==== 69+0+0 (2499892046 0 0) 0x7f4558000bd0 con 0x7f456c0091d0
2016-04-21 16:15:00.812707 7f459dc79700 20 mon.a@0(leader) e1 _ms_dispatch existing session 0x7f4578114190 for client.? 127.0.0.1:0/1254741436
2016-04-21 16:15:00.812712 7f459dc79700 20 mon.a@0(leader) e1  caps allow *
2016-04-21 16:15:00.812719 7f459dc79700 10 mon.a@0(leader).paxosservice(osdmap 1..9) dispatch 0x7f4558000bd0 pool_op(create pool 0 auid 0 tid 1 name test v0) v4 from client.4109 127.0.0.1:0/1254741436 con 0x7f456c0091d0
2016-04-21 16:15:00.812758 7f459dc79700  5 mon.a@0(leader).paxos(paxos active c 1..35) is_readable = 1 - now=2016-04-21 16:15:00.812758 lease_expire=0.000000 has v0 lc 35
2016-04-21 16:15:00.812767 7f459dc79700 10 mon.a@0(leader).osd e9 preprocess_query pool_op(create pool 0 auid 0 tid 1 name test v0) v4 from client.4109 127.0.0.1:0/1254741436
2016-04-21 16:15:00.812780 7f459dc79700 20 is_capable service=osd command= write on cap allow *
2016-04-21 16:15:00.812782 7f459dc79700 20  allow so far , doing grant allow *
2016-04-21 16:15:00.812783 7f459dc79700 20  allow all
2016-04-21 16:15:00.812788 7f459dc79700  7 mon.a@0(leader).osd e9 prepare_update pool_op(create pool 0 auid 0 tid 1 name test v0) v4 from client.4109 127.0.0.1:0/1254741436
2016-04-21 16:15:00.812794 7f459dc79700 10 mon.a@0(leader).osd e9 prepare_pool_op pool_op(create pool 0 auid 0 tid 1 name test v0) v4
2016-04-21 16:15:00.812803 7f459dc79700 10 mon.a@0(leader).osd e9 prepare_new_pool from 0x7f456c0091d0
2016-04-21 16:15:00.859267 7f459dc79700 10 mon.a@0(leader).osd e9  tester.test_with_crushtool returns -22: libust[97157/97159]: Error: recvmsg: Bad file descriptor (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:292)
libust[97157/97159]: Error: Socket shutdown error (in ustcomm_recv_unix_sock() at lttng-ust-comm.c:299)
libringbuffer[97157/97159]: Error: close: Bad file descriptor (in shmp_object_destroy() at shm.c:439)
ceph-mon: shm.c:440: shmp_object_destroy: Assertion `0' failed.
*** Caught signal (Aborted) **
 in thread 7f459cc77700 thread_name:ms_dispatch
remove lseek failed (9) Bad file descriptor
 ceph version 10.2.0-20-g99afe98 (99afe98ff822d7713a5392f7ea0aaa78d0d46f3e)
 1: (()+0x4f2b92) [0x7f45d3083b92]
 2: (()+0x10340) [0x7f45d1cdf340]
 3: (gsignal()+0x39) [0x7f45cff67cc9]
 4: (abort()+0x148) [0x7f45cff6b0d8]
 5: (()+0x2fb86) [0x7f45cff60b86]
 6: (()+0x2fc32) [0x7f45cff60c32]
 7: (()+0x3701c) [0x7f45cfd1801c]
 8: (shm_object_table_destroy()+0x3f) [0x7f45cfd1a06f]
 9: (channel_destroy()+0x2a) [0x7f45cfd14b3a]
 10: (lttng_session_destroy()+0x1f8) [0x7f45cfcfb8b8]
 11: (()+0x15e96) [0x7f45cfcf6e96]
 12: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 13: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 14: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 15: (lttng_ust_objd_table_owner_cleanup()+0x7c) [0x7f45cfcf7adc]
 16: (()+0x1233b) [0x7f45cfcf333b]
 17: (()+0x8182) [0x7f45d1cd7182]
 18: (clone()+0x6d) [0x7f45d002b47d]
problem writing to /home/dzafman/ceph/src/out/mon.a.log: (9) Bad file descriptor
2016-04-21 16:15:00.834101 7f459cc77700 -1 *** Caught signal (Aborted) **
 in thread 7f459cc77700 thread_name:ms_dispatch

 ceph version 10.2.0-20-g99afe98 (99afe98ff822d7713a5392f7ea0aaa78d0d46f3e)
 1: (()+0x4f2b92) [0x7f45d3083b92]
 2: (()+0x10340) [0x7f45d1cdf340]
 3: (gsignal()+0x39) [0x7f45cff67cc9]
 4: (abort()+0x148) [0x7f45cff6b0d8]
 5: (()+0x2fb86) [0x7f45cff60b86]
 6: (()+0x2fc32) [0x7f45cff60c32]
 7: (()+0x3701c) [0x7f45cfd1801c]
 8: (shm_object_table_destroy()+0x3f) [0x7f45cfd1a06f]
 9: (channel_destroy()+0x2a) [0x7f45cfd14b3a]
 10: (lttng_session_destroy()+0x1f8) [0x7f45cfcfb8b8]
 11: (()+0x15e96) [0x7f45cfcf6e96]
 12: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 13: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 14: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 15: (lttng_ust_objd_table_owner_cleanup()+0x7c) [0x7f45cfcf7adc]
 16: (()+0x1233b) [0x7f45cfcf333b]
 17: (()+0x8182) [0x7f45d1cd7182]
 18: (clone()+0x6d) [0x7f45d002b47d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

problem writing to /home/dzafman/ceph/src/out/mon.a.log: (9) Bad file descriptor
 -6698> 2016-04-21 16:14:47.892676 7f45d364a500 -1 WARNING: the following dangerous and experimental features are enabled: *
 -6696> 2016-04-21 16:14:47.893526 7f45d364a500 -1 WARNING: the following dangerous and experimental features are enabled: *
 -6694> 2016-04-21 16:14:47.899272 7f45d364a500 -1 WARNING: the following dangerous and experimental features are enabled: *
     0> 2016-04-21 16:15:00.834101 7f459cc77700 -1 *** Caught signal (Aborted) **
 in thread 7f459cc77700 thread_name:ms_dispatch

 ceph version 10.2.0-20-g99afe98 (99afe98ff822d7713a5392f7ea0aaa78d0d46f3e)
 1: (()+0x4f2b92) [0x7f45d3083b92]
 2: (()+0x10340) [0x7f45d1cdf340]
 3: (gsignal()+0x39) [0x7f45cff67cc9]
 4: (abort()+0x148) [0x7f45cff6b0d8]
 5: (()+0x2fb86) [0x7f45cff60b86]
 6: (()+0x2fc32) [0x7f45cff60c32]
 7: (()+0x3701c) [0x7f45cfd1801c]
 8: (shm_object_table_destroy()+0x3f) [0x7f45cfd1a06f]
 9: (channel_destroy()+0x2a) [0x7f45cfd14b3a]
 10: (lttng_session_destroy()+0x1f8) [0x7f45cfcfb8b8]
 11: (()+0x15e96) [0x7f45cfcf6e96]
 12: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 13: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 14: (lttng_ust_objd_unref()+0xdf) [0x7f45cfcf733f]
 15: (lttng_ust_objd_table_owner_cleanup()+0x7c) [0x7f45cfcf7adc]
 16: (()+0x1233b) [0x7f45cfcf333b]
 17: (()+0x8182) [0x7f45d1cd7182]
 18: (clone()+0x6d) [0x7f45d002b47d]

problem writing to /home/dzafman/ceph/src/out/mon.a.log: (9) Bad file descriptor
<REPEATS>
problem writing to /home/dzafman/ceph/src/out/mon.a.log: (9) Bad file descriptor
./crushtool: got signal: 6

History

#1 Updated by Greg Farnum over 2 years ago

  • Project changed from Ceph to RADOS
  • Category set to LTTNG
  • Component(RADOS) Monitor added

Also available in: Atom PDF