Project

General

Profile

Actions
Copy link

Bug #41818

closed

crimson run error due to ___longjmp_chk()

Added by YongSheng Zhang over 4 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Kefu Chai
Category:
-
Target version:
Ceph - v15.0.0
% Done:

0%

Source:
Community (dev)
Tags:
crimson
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
36261
Crash signature (v1):
Crash signature (v2):

Description

ceph_master branch :git commit d8251aa2139b75452371409c1bd27fec2f7f1774

run command:

MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x --without-dashboard --memstore --crimson --nodaemon --redirect-output --osd-args "--memory 1G --smp 1 --cpuset 0"

"ps -ef|grep ceph" output:

[root@centos7 usr]# ps -ef|grep ceph
root 2107 1 0 21:14 pts/0 00:00:00 /bin/sh /root/ceph/src/ceph-run /usr/bin/ceph-mon -i a -c /root/ceph/src/ceph.conf -f
root 2110 2107 1 21:14 pts/0 00:00:00 /usr/bin/ceph-mon -i a -c /root/ceph/src/ceph.conf -f
root 2243 1 0 21:14 pts/0 00:00:00 /bin/sh /root/ceph/src/ceph-run /usr/bin/ceph-mgr -i x -c /root/ceph/src/ceph.conf -f
root 2245 2243 0 21:14 pts/0 00:00:00 /usr/bin/ceph-mgr -i x -c /root/ceph/src/ceph.conf -f
root 2344 1404 0 21:15 pts/0 00:00:00 grep --color=auto ceph ==========================================
error dump:

creating /root/ceph/src/dev/mgr.x/keyring
/usr/bin/ceph -c /root/ceph/src/ceph.conf -k /root/ceph/src/keyring -i /root/ceph/src/dev/mgr.x/keyring auth add mgr.x mon 'allow profile mgr' mds 'allow *' osd 'allow *'
added key for mgr.x
/usr/bin/ceph -c /root/ceph/src/ceph.conf -k /root/ceph/src/keyring config set mgr mgr/restful/x/server_port 42656 --force
Starting mgr.x
/usr/bin/ceph -c /root/ceph/src/ceph.conf -k /root/ceph/src/keyring tell mgr restful create-self-signed-cert
no valid command found; 10 closest matches:
osd df {plain|tree} {class|name} {<filter>}
osd perf
pg force-backfill <pgid> [<pgid>...]
pg force-recovery <pgid> [<pgid>...]
pg cancel-force-backfill <pgid> [<pgid>...]
pg cancel-force-recovery <pgid> [<pgid>...]
pg scrub <pgid>
pg debug unfound_objects_exist|degraded_pgs_exist
pg repair <pgid>
pg deep-scrub <pgid>
Error EINVAL: invalid command
MGR Restful is not working, perhaps the package is not installed?
add osd0 c12bdaeb-9ca6-49ee-a9e5-01e4acbc4578
/usr/bin/ceph -c /root/ceph/src/ceph.conf -k /root/ceph/src/keyring osd new c12bdaeb-9ca6-49ee-a9e5-01e4acbc4578 -i /root/ceph/src/dev/osd0/new.json
0
  • longjmp causes uninitialized stack frame ***: /usr/bin/crimson-osd terminated ======= Backtrace: =========
    /lib64/libc.so.6(_fortify_fail+0x37)[0x7fc7b34e9b67]
    /lib64/libc.so.6(+0x117a7d)[0x7fc7b34e9a7d]
    /lib64/libc.so.6(    _longjmp_chk+0x29)[0x7fc7b34e99d9]
    /usr/bin/crimson-osd[0xe03658]
    /usr/bin/crimson-osd[0xd52e74]
    /usr/bin/crimson-osd[0xd531e1]
    /usr/bin/crimson-osd[0xdb6b0e]
    /usr/bin/crimson-osd[0xd23b2d]
    /usr/bin/crimson-osd(main+0x368)[0x70e7e8]
    /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc7b33f4495]
    /usr/bin/crimson-osd[0x71d6e5] ======= Memory map: ========
    00400000-0117e000 r-xp 00000000 fd:01 288714 /usr/bin/crimson-osd
    0137e000-013b1000 r--p 00d7e000 fd:01 288714 /usr/bin/crimson-osd
    013b1000-013ba000 rw-p 00db1000 fd:01 288714 /usr/bin/crimson-osd
    013ba000-013e5000 rw-p 00000000 00:00 0
    600000000000-600040000000 rw-p 00000000 00:00 0
    600040000000-700000000000 ---p 00000000 00:00 0
    7fc7b17b7000-7fc7b17b8000 ---p 00000000 00:00 0
    7fc7b17b8000-7fc7b1fb8000 rw-p 00000000 00:00 0
    7fc7b1fb8000-7fc7b2000000 ---p 00000000 00:00 0
    7fc7b2000000-7fc7b2200000 rw-p 00000000 00:00 0
    7fc7b2200000-7fc7b25b8000 ---p 00000000 00:00 0
    7fc7b25b8000-7fc7b25bf000 r-xp 00000000 fd:01 265919 /usr/lib64/libffi.so.6.0.1
    7fc7b25bf000-7fc7b27be000 ---p 00007000 fd:01 265919 /usr/lib64/libffi.so.6.0.1
    7fc7b27be000-7fc7b27bf000 r--p 00006000 fd:01 265919 /usr/lib64/libffi.so.6.0.1
    7fc7b27bf000-7fc7b27c0000 rw-p 00007000 fd:01 265919 /usr/lib64/libffi.so.6.0.1
    7fc7b27c0000-7fc7b2836000 r-xp 00000000 fd:01 266096 /usr/lib64/libgmp.so.10.2.0
    7fc7b2836000-7fc7b2a35000 ---p 00076000 fd:01 266096 /usr/lib64/libgmp.so.10.2.0
    7fc7b2a35000-7fc7b2a37000 r--p 00075000 fd:01 266096 /usr/lib64/libgmp.so.10.2.0
    7fc7b2a37000-7fc7b2a38000 rw-p 00077000 fd:01 266096 /usr/lib64/libgmp.so.10.2.0
    7fc7b2a38000-7fc7b2a5e000 r-xp 00000000 fd:01 287105 /usr/lib64/libhogweed.so.2.5
    7fc7b2a5e000-7fc7b2c5d000 ---p 00026000 fd:01 287105

Related issues 1 (0 open — 1 closed)

Related to crimson - Subtask #45535: crimson: crimson-osd failure in ceph-containerClosed

Actions
Actions
Copy link
 #1

Updated by Kefu Chai over 4 years ago

i cannot reproduce this issue. next time, if you run into it, could you launch

../src/seastar/scripts/seastar-addr2line

and paste the stack trace to the tool?

Actions
Copy link
 #2

Updated by Kefu Chai over 4 years ago

  • Project changed from Ceph to RADOS
Actions
Copy link
 #3

Updated by YongSheng Zhang over 4 years ago

Kefu Chai wrote:

i cannot reproduce this issue. next time, if you run into it, could you launch

[...]

and paste the stack trace to the tool?

Hi Kefu

Thanks your reply, sorry my fault

trace info:
Backtrace:
0x0000000000dd19a8
0x0000000000d551a8
0x0000000000d55253
0x0000000000d55325
/lib64/libpthread.so.0+0x000000000000f5cf
/lib64/libc.so.6+0x00000000000362c6
/lib64/libc.so.6+0x00000000000379b7
/lib64/libc.so.6+0x0000000000078e16
/lib64/libc.so.6+0x0000000000117b66
/lib64/libc.so.6+0x0000000000117a7c
/lib64/libc.so.6+0x00000000001179d8
0x0000000000e03657
0x0000000000d52e73
0x0000000000d531e0
0x0000000000db6b0d
0x0000000000d23b2c
0x000000000070e7e7
/lib64/libc.so.6+0x0000000000022494
0x000000000071d6e4

dump info by tool:

[root@centos7 src]# ../src/seastar/scripts/seastar-addr2line -e /usr/bin/crimson-osd
/usr/bin/crimson-osd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=7764410a4e887f332d79b0d8a98a1792db1fb189, not stripped

0x0000000000dd19a8
  0x0000000000d551a8
  0x0000000000d55253
  0x0000000000d55325

[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/include/seastar/util/backtrace.hh:55
seastar::backtrace_buffer::append_backtrace() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:1199
(inlined by) print_with_backtrace at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:1220
seastar::print_with_backtrace(char const*) at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:1227
sigabrt_action at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:5268
(inlined by) operator() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:5250
(inlined by) _FUN at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:5246

==============================================================

[root@centos7 src]# ../src/seastar/scripts/seastar-addr2line -e /usr/bin/crimson-osd
/usr/bin/crimson-osd: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=7764410a4e887f332d79b0d8a98a1792db1fb189, not stripped

0x0000000000e03657
  0x0000000000d52e73
  0x0000000000d531e0
  0x0000000000db6b0d
  0x0000000000d23b2c
  0x000000000070e7e7

[Backtrace #0]
seastar::jmp_buf_link::switch_in() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/thread.cc:127
seastar::reactor::run_tasks(seastar::reactor::task_queue&) at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:3825
seastar::reactor::run_some_tasks() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:4252
seastar::reactor::run_some_tasks() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:4235
(inlined by) seastar::reactor::run() at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/reactor.cc:4396
seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/seastar/src/core/app-template.cc:199
main at /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/crimson/osd/main.cc:137 ===========================================

mon.a.stdout content:

/root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/ceph-run /usr/bin/ceph-mon -i a -c /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/ceph.conf -f &
2019-09-16T10:11:45.704+0800 7f56c00c1380 -1 mon.a@0(electing) e0 failed to get devid for vda: fallback method has serial ''but no model
2019-09-16T10:11:45.724+0800 7f56a830d700 -1 mon.a@0(electing) e1 failed to get devid for vda: fallback method has serial ''but no model
2019-09-16T10:12:07.097+0800 7f56a9b10700 -1 log_channel(cluster) log [ERR] : Health check failed: 10 mgr modules have failed (MGR_MODULE_ERROR)

=======================================================
mgr.x.stdout content:

/root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/ceph-run /usr/bin/ceph-mgr -i x -c /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/ceph.conf -f &
2019-09-16T10:11:47.086+0800 7f2a1e7c2b80 -1 log_channel(cluster) log [ERR] : No ceph-mgr modules found in

============================================
run " MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x --without-dashboard --memstore --crimson --nodaemon --redirect-output --osd-args "--memory 1G --smp 1 --cpuset 0"" command can output following info, i do not know wether it is error.

......
/usr/bin/ceph -c /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/ceph.conf -k /root/rpmbuild/BUILD/ceph-15.0.0-4673-gd8251aa/src/keyring tell mgr restful create-self-signed-cert
no valid command found; 10 closest matches:
osd df {plain|tree} {class|name} {<filter>}
osd perf
pg force-backfill <pgid> [<pgid>...]
pg force-recovery <pgid> [<pgid>...]
pg cancel-force-backfill <pgid> [<pgid>...]
pg cancel-force-recovery <pgid> [<pgid>...]
pg scrub <pgid>
pg debug unfound_objects_exist|degraded_pgs_exist
pg repair <pgid>
pg deep-scrub <pgid>
Error EINVAL: invalid command
MGR Restful is not working, perhaps the package is not installed?
.....

Actions
Copy link
 #4

Updated by YongSheng Zhang over 4 years ago

Before running vstart.sh, will set some environment variables?

Actions
Copy link
 #5

Updated by Neha Ojha over 4 years ago

  • Assignee set to Kefu Chai
Actions
Copy link
 #6

Updated by Kefu Chai over 4 years ago

Before running vstart.sh, will set some environment variables?

@YongSheng sorta. but i don't think this issue is related. could you try to run seastar's unittests like

./configure.py --c++-dialect gnu++17 --mode debug
ninja -C build/debug -j1
./test.py --mode debug
Actions
Copy link
 #7

Updated by YongSheng Zhang over 4 years ago

Kefu Chai wrote:

Before running vstart.sh, will set some environment variables?

@YongSheng sorta. but i don't think this issue is related. could you try to run seastar's unittests like

[...]

Ceph version 15 require which version seastar?

Actions
Copy link
 #8

Updated by Radoslaw Zarzynski almost 4 years ago

  • Related to Subtask #45535: crimson: crimson-osd failure in ceph-container added
Actions
Copy link
 #9

Updated by Kefu Chai over 3 years ago

  • Project changed from RADOS to crimson
  • Subject changed from crimson run error to crimson run error due to ___longjmp_chk()
  • Pull request ID set to 36261

@YongSheng sorry for the latency.

currently we are using commit of 0523b0fac4123f4998f3c7af0a3d8110fc1bc675. i think you ran into an issue reported by AddressSanitizer. i believe it should have been fixed by https://github.com/ceph/ceph/pull/36261

Actions
Copy link
 #10

Updated by Matan Breizman 11 months ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF