Ceph » RADOS

I forgot to add that I tried to diff code I thought was relevant between tags v15.2.9 and v16.2.1 and thought I saw some win32 related changes that looked "close" to the potentially problematic code, I don't think I saw anything that stood out as code changes that would have broken this, which makes me wonder if it was a compiler issue - which I only suggest because I did find bug reports for segfaults on push_back that seemed to be caused by some buggy compilers, but I know that may be a long shot. I don't normally suggest it's the compiler's fault.

Here's a bit more info that may be useful. Only because it's a volume already exported to the container out of the box, the crush location hook I am using lives under what the container sees as /var/log/ceph (on the host it's /var/log/ceph/$FSID). Maybe something about that location is problematic? Though as I noted earlier, trying something under /bin, which is part of the container, produced the same results.

FYI I tried with ceph/daemon-base:master-24e1f91-pacific-centos-8-x86_64 (the latest non-devel build at this time) just to see if somehow something was different there, since that build was newer (even though it should be and is still 16.2.4), and the problem persists there too.

I just wanted to note that I see the status is listed as "Need More Info", but I think I have provided everything I have been asked for, and anything I can think of additionally. This is not me being a nag, just wanted to be clear about my perspective on the status of this ticket as it pertains to my input.

I saw that 16.2.5 was released. Though I didn't expect it to address this issue, I tested with it anyway just to be sure. The issue persists with 16.2.5.

Based on the progress here it seems like I'm probably the only person to have reported this. I still can't figure out why that'd be. I wonder if you have had a chance to look at the core dump and/or reproduce this and if you have an idea of what's going on here? It may help me mitigate on my end if nothing else. Thanks.

I dug into this more today and I am wondering if it has something to do with `_conf->cluster` not being set right (to the default of `ceph`). Unfortunately editing the OSD's `unit.run` to include `--cluster ceph` in the arg list didn't change the behavior, so no additional clue provided there.

I'm also seeing this issue:

*** Caught signal (Segmentation fault) **
 in thread 7f6a7e69c700 thread_name:ceph-mon
 ceph version 16.2.6 (ee28fb57e47e9f88813e24bbf4c14496ca299d31) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12b20) [0x7f6a73562b20]
 2: /lib64/libc.so.6(+0x15d5f5) [0x7f6a722e95f5]
 3: (SubProcess::add_cmd_arg(char const*)+0x4c) [0x7f6a75a5cbfc]
 4: (SubProcess::add_cmd_args(char const*, ...)+0x75) [0x7f6a75a5cd95]
 5: (ceph::crush::CrushLocation::update_from_hook()+0x2d4) [0x7f6a75de45e4]
 6: (ceph::crush::CrushLocation::init_on_startup()+0x385) [0x7f6a75de56c5]
 7: (global_init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::alloca
tor<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const*, std::vector<char const*, std::allocator<char
const*> >&, unsigned int, code_environment_t, int, bool)+0xcd1) [0x55eadd92b941]
 8: main()
 9: __libc_start_main()
 10: _start()

And I also use a bash script as crush hook. The script works fine, and is used in Nautilus clusters with no issue:

$ /usr/local/bin/cf-crush-hook
root=default datacenter=dc1 rack=A2 host=hostxxx

Thank you for this fix. It is very much appreciated.

This present in 16.2.7. Any reason why the linked PR wasn't merged into that release?

This issue bit us in our upgrade to 16.2.7 from 15.2.15. We have a manual cluster (non-cephadm). We followed the procedure at:

https://docs.ceph.com/en/latest/releases/pacific/ under "Upgrading non-cephadm clusters"

We have several mixed servers (hdd and ssd), and used the hook functionality to segregate the ssds into their own virtual 'hosts', and then created rules that two pools were using for their crush rule. When we discovered the SIGSEGV after upgrading the mons, the data in the dump led us to this bug. Luckily, after removing the crush hook and getting the mons to start, the osds had not yet been restarted, and were still using the old crush map that had some osds remapped to their own virtual 'hosts'. We were able to change the crush rule of the pools in question before restarting the OSDs, and as such did not lose data.

https://docs.ceph.com/en/latest/releases/pacific/
https://ceph.io/en/news/blog/2021/v16-2-7-pacific-released/
https://docs.ceph.com/en/latest/cephadm/upgrade/

Nowhere that I can find in any of the above is this bug mentioned. This bug MUST be mentioned in the upgrade process, "Don't upgrade if you rely on crush hooks and cannot remap" just like the "bluestore-quick-fix-on-mount" issue is mentioned for a 16.2.6 upgrade.

Chris Durham wrote:

This issue bit us in our upgrade to 16.2.7 from 15.2.15. We have a manual cluster (non-cephadm). We followed the procedure at:

https://docs.ceph.com/en/latest/releases/pacific/ under "Upgrading non-cephadm clusters"

We have several mixed servers (hdd and ssd), and used the hook functionality to segregate the ssds into their own virtual 'hosts', and then created rules that two pools were using for their crush rule. When we discovered the SIGSEGV after upgrading the mons, the data in the dump led us to this bug. Luckily, after removing the crush hook and getting the mons to start, the osds had not yet been restarted, and were still using the old crush map that had some osds remapped to their own virtual 'hosts'. We were able to change the crush rule of the pools in question before restarting the OSDs, and as such did not lose data.

https://docs.ceph.com/en/latest/releases/pacific/
https://ceph.io/en/news/blog/2021/v16-2-7-pacific-released/
https://docs.ceph.com/en/latest/cephadm/upgrade/

Nowhere that I can find in any of the above is this bug mentioned. This bug MUST be mentioned in the upgrade process, "Don't upgrade if you rely on crush hooks and cannot remap" just like the "bluestore-quick-fix-on-mount" issue is mentioned for a 16.2.6 upgrade.

Note this was on CentOS 8.5

This seems to be a pretty high priority issue, we just hit it upgrading from nautilus to 16.2.7 on a cluster with 100+ osds in various configurations. Is it going to be fixed in the next Pacific update?

I appreciate the work to get this bug squashed but I wonder if there's a schedule published somewhere that might indicate when it could be merged into another Pacific release?

Andrew Davidoff wrote:

I appreciate the work to get this bug squashed but I wonder if there's a schedule published somewhere that might indicate when it could be merged into another Pacific release?

We are planning a 16.2.8 release in the next few weeks, which will include this fix.