Bug #50997: octopus arm64 build broken since 15.2.8 - Ceph - Ceph

Actions

Copy link

Bug #50997

open

octopus arm64 build broken since 15.2.8

Added by Andreas Elvers almost 3 years ago. Updated over 2 years ago.

Status:

New

Priority:

Normal

Assignee:

Category:

build

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

v15.2.10, v15.2.11, v15.2.12, v15.2.13, v15.2.8, v15.2.9

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

I have recreated this ticket, since the old one was falsely closed as duplicated. The Packages for bionic on http://download.ceph.com/debian-octopus still contain this error.

Since ceph 15.2.8 on ARM64 the osd binary (most probably other binaries as well) fails to start with the error message:

load: jerasure load: lrc load dlopen(/usr/lib/ceph/erasure-code/libec_isa.so): /usr/lib/ceph/erasure-code/libec_isa.so: cannot restore segment prot after reloc: Operation not permitted

According to https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1917414 this is due to a problem with isa-l library that seems to be incorporated into the binaries.
Release: Ubuntu Bionic / Ceph 15.2.8 through 15.2.13
Architecture: ARM64

Expected behaviour:
OSD should startup
Actual behaviour:
OSD startup fails

OSD log:
2021-05-27T09:53:51.524+0000 7f8ff43010 0 ceph version 15.2.13 (c44bc49e7a57a87d84dfff2a077a2058aa2172e2) octopus (stable), process ceph-osd, pid 1757
2021-05-27T09:53:51.524+0000 7f8ff43010 0 pidfile_write: ignore empty --pid-file
2021-05-27T09:53:51.528+0000 7f8ff43010 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2021-05-27T09:53:51.528+0000 7f8ff43010 1 bdev(0x55bf988000 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2021-05-27T09:53:51.528+0000 7f8ff43010 0 bdev(0x55bf988000 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bdev(0x55bf988000 /var/lib/ceph/osd/ceph-3/block) open size 14000486088704 (0xcbbbe000000, 13 TiB) block_size 4096 (4 KiB) rotational discard not supported
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bluestore(/var/lib/ceph/osd/ceph-3) _set_cache_sizes cache_size 1073741824 meta 0.4 kv 0.4 data 0.2
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bdev create path /var/lib/ceph/osd/ceph-3/block type kernel
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bdev(0x55bf988380 /var/lib/ceph/osd/ceph-3/block) open path /var/lib/ceph/osd/ceph-3/block
2021-05-27T09:53:51.532+0000 7f8ff43010 0 bdev(0x55bf988380 /var/lib/ceph/osd/ceph-3/block) ioctl(F_SET_FILE_RW_HINT) on /var/lib/ceph/osd/ceph-3/block failed: (22) Invalid argument
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bdev(0x55bf988380 /var/lib/ceph/osd/ceph-3/block) open size 14000486088704 (0xcbbbe000000, 13 TiB) block_size 4096 (4 KiB) rotational discard not supported
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-3/block size 13 TiB
2021-05-27T09:53:51.532+0000 7f8ff43010 1 bdev(0x55bf988380 /var/lib/ceph/osd/ceph-3/block) close
2021-05-27T09:53:51.808+0000 7f8ff43010 1 bdev(0x55bf988000 /var/lib/ceph/osd/ceph-3/block) close
2021-05-27T09:53:52.060+0000 7f8ff43010 0 starting osd.3 osd_data /var/lib/ceph/osd/ceph-3 /var/lib/ceph/osd/ceph-3/journal
2021-05-27T09:53:52.084+0000 7f8ff43010 -1 load: jerasure load: lrc load dlopen(/usr/lib/ceph/erasure-code/libec_isa.so): /usr/lib/ceph/erasure-code/libec_isa.so: cannot restore segment prot after reloc: Operation not permitted

Actions

Copy link

Updated by Andreas Elvers almost 3 years ago

This issue supersedes ticket #50435

Actions

Copy link

Updated by Andreas Elvers over 2 years ago

release 15.2.14 for bionic still exhibits the crashing on startup.

For the record:

It is possible to make the binary executable again by tweaking the systemd unit files. This is taken from: https://bugs.launchpad.net/ubuntu/hirsute/+source/ceph/+bug/1917414

I was able to get past this issue.
In my case, the ceph-mon was invoked by systemctl service file. So, I had to change the below property under the "[Service]" node of file: /lib/systemd/system/ceph-mon@.service

- MemoryDenyWriteExecute=true
+ MemoryDenyWriteExecute=false

As per the system manpage⁰:

MemoryDenyWriteExecute=
Takes a boolean argument. If set, attempts to create memory mappings that are writable and executable at the same time, or to change existing memory mappings to become executable, or mapping shared memory segments as executable are prohibited. Specifically, a system call filter is added that rejects mmap(2) system calls with both PROT_EXEC and PROT_WRITE set, mprotect(2) or pkey_mprotect(2) system calls with PROT_EXEC set and shmat(2) system calls with SHM_EXEC set. Note that this option is incompatible with programs and libraries that generate program code dynamically at runtime, including JIT execution engines, executable stacks, and code "trampoline" feature of various C compilers.

This particular property was preventing the ceph-mon executable to run due to the above prohibitions.
After making the above change, I had to run
$ systemctl daemon-reload
And then restarting the service worked for me!