Project

General

Profile

Bug #46596

ceph-osd --mkfs: *** longjmp causes uninitialized stack frame ***: /usr/bin/ceph-osd terminated

Added by Sebastian Wagner over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This is likely a regression that was merged yesterday into master (July 16th).

2020-07-17T08:16:37.646 INFO:teuthology.orchestra.run.smithi117.stderr:Error EINVAL: Traceback (most recent call last):
2020-07-17T08:16:37.647 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 1170, in _handle_command
2020-07-17T08:16:37.647 INFO:teuthology.orchestra.run.smithi117.stderr:    return self.handle_command(inbuf, cmd)
2020-07-17T08:16:37.647 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 116, in handle_command
2020-07-17T08:16:37.648 INFO:teuthology.orchestra.run.smithi117.stderr:    return dispatch[cmd['prefix']].call(self, cmd, inbuf)
2020-07-17T08:16:37.648 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/mgr_module.py", line 310, in call
2020-07-17T08:16:37.648 INFO:teuthology.orchestra.run.smithi117.stderr:    return self.func(mgr, **kwargs)
2020-07-17T08:16:37.649 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 78, in <lambda>
2020-07-17T08:16:37.649 INFO:teuthology.orchestra.run.smithi117.stderr:    wrapper_copy = lambda *l_args, **l_kwargs: wrapper(*l_args, **l_kwargs)
2020-07-17T08:16:37.649 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 69, in wrapper
2020-07-17T08:16:37.650 INFO:teuthology.orchestra.run.smithi117.stderr:    return func(*args, **kwargs)
2020-07-17T08:16:37.650 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/orchestrator/module.py", line 838, in _daemon_add_osd
2020-07-17T08:16:37.650 INFO:teuthology.orchestra.run.smithi117.stderr:    raise_if_exception(completion)
2020-07-17T08:16:37.651 INFO:teuthology.orchestra.run.smithi117.stderr:  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 636, in raise_if_exception
2020-07-17T08:16:37.651 INFO:teuthology.orchestra.run.smithi117.stderr:    raise e
2020-07-17T08:16:37.651 INFO:teuthology.orchestra.run.smithi117.stderr:RuntimeError: cephadm exited with an error code: 1, stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-authtool --gen-print-key
2020-07-17T08:16:37.652 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ea0b3bfb-c92d-4c0d-a6c1-dbaea1d2d9ea
2020-07-17T08:16:37.652 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-authtool --gen-print-key
2020-07-17T08:16:37.652 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
2020-07-17T08:16:37.652 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -h ceph:ceph /dev/vg_nvme/lv_4
2020-07-17T08:16:37.653 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3
2020-07-17T08:16:37.653 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ln -s /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0/block
2020-07-17T08:16:37.653 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
2020-07-17T08:16:37.654 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: got monmap epoch 3
2020-07-17T08:16:37.654 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQBkXhFfHrtPAhAAPVWYuBP8G0vUDHj9YQlpOQ==
2020-07-17T08:16:37.655 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stdout: creating /var/lib/ceph/osd/ceph-0/keyring
2020-07-17T08:16:37.655 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr added entity osd.0 auth(key=AQBkXhFfHrtPAhAAPVWYuBP8G0vUDHj9YQlpOQ==)
2020-07-17T08:16:37.656 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
2020-07-17T08:16:37.656 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
2020-07-17T08:16:37.656 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osdspec-affinity None --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid ea0b3bfb-c92d-4c0d-a6c1-dbaea1d2d9ea --setuser ceph --setgroup ceph
2020-07-17T08:16:37.656 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: WARN  2020-07-17 08:16:36,830 [shard 0] seastar - Unable to set SCHED_FIFO scheduling policy for timer thread; latency impact possible. Try adding CAP_SYS_NICE
2020-07-17T08:16:37.657 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: *** longjmp causes uninitialized stack frame ***: /usr/bin/ceph-osd terminated
2020-07-17T08:16:37.657 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: Aborting on shard 0.
2020-07-17T08:16:37.657 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: Backtrace:
2020-07-17T08:16:37.658 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e8060c
2020-07-17T08:16:37.658 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e46040
2020-07-17T08:16:37.658 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e46108
2020-07-17T08:16:37.659 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e461d5
2020-07-17T08:16:37.659 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libpthread.so.0+0x0000000000012dcf
2020-07-17T08:16:37.660 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000003770e
2020-07-17T08:16:37.660 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x0000000000021b24
2020-07-17T08:16:37.660 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000007a896
2020-07-17T08:16:37.660 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000010d904
2020-07-17T08:16:37.661 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000010d936
2020-07-17T08:16:37.661 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000010d7b0
2020-07-17T08:16:37.661 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x000000000010d70e
2020-07-17T08:16:37.661 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000eb4cdf
2020-07-17T08:16:37.662 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x00000000005eb475
2020-07-17T08:16:37.662 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e42aa0
2020-07-17T08:16:37.671 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e42df3
2020-07-17T08:16:37.672 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e6d51d
2020-07-17T08:16:37.672 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x0000000000e19237
2020-07-17T08:16:37.672 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x000000000058b61d
2020-07-17T08:16:37.672 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: /lib64/libc.so.6+0x00000000000236a2
2020-07-17T08:16:37.673 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr  stderr: 0x00000000005d133d
2020-07-17T08:16:37.673 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr --> Was unable to complete a new OSD, will rollback changes
2020-07-17T08:16:37.673 INFO:teuthology.orchestra.run.smithi117.stderr:INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it

https://pulpito.ceph.com/swagner-2020-07-17_08:02:08-rados:cephadm-wip-swagner3-testing-2020-07-16-1151-distro-basic-smithi/

https://pulpito.ceph.com/swagner-2020-07-17_08:13:18-rados:cephadm-wip-swagner2-testing-2020-07-16-1151-distro-basic-smithi/

https://pulpito.ceph.com/varsha-2020-07-17_06:11:35-rados-wip-varsha-testing-distro-basic-smithi/

History

#1 Updated by Sebastian Wagner over 3 years ago

there is a small possibility that this is related to https://github.com/ceph/ceph/pull/33770

#2 Updated by Kefu Chai over 3 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Kefu Chai

#3 Updated by Kefu Chai over 3 years ago

  • Status changed from Fix Under Review to Resolved

Also available in: Atom PDF