Ceph » CephFS

I"ve used https://github.com/ceph/ceph/blob/main/src/script/ceph-debug-docker.sh for deploying the build on container.
This is the bt of the crash, it happen in the beginning and seems to be related to parameters to 'rbd_features_from_string(const std::string& orig_value,std::ostream *err)'
I need to figure how to get symbols of /usr/lib/ceph/libceph-common.so.2 for watching rbd_features_from_string parameters

(gdb) bt full
#0  0x00007fc83f6053ee in std::locale::operator==(std::locale const&) const () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#1  0x00007fc83f4c6798 in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned long, char>::convert() () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#2  0x00007fc83f4c59e4 in librbd::rbd_features_from_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::ostream*) ()
   from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#3  0x00007fc83f150e0f in ?? () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#4  0x00007fc83f0acfb6 in Option::pre_validate(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#5  0x00007fc83f0af5cb in Option::parse_value(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#6  0x00007fc83f07efa2 in md_config_t::_set_val(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Option const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#7  0x00007fc83f07f417 in md_config_t::set_val_default(ConfigValues&, ConfigTracker const&, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#8  0x00007fc83f08ef0a in md_config_t::md_config_t(ConfigValues&, ConfigTracker const&, bool) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#9  0x00007fc83f035bdc in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#10 0x00007fc83f036c61 in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#11 0x00007fc83f075675 in common_preinit(CephInitParameters const&, code_environment_t, int) () from /usr/lib/ceph/libceph-common.so.2
No symbol table info available.
#12 0x00007fc83fe4a7f9 in ceph_create () from /lib/libcephfs.so.2
No symbol table info available.
#13 0x0000555a90b7882a in update_root_mode () at ./src/test/libcephfs/reclaim.cc:149
        admin = 0xe90000007f
        r = <optimized out>
        admin = <optimized out>
        r = <optimized out>
#14 main (argc=<optimized out>, argv=0x7ffc0ba75948) at ./src/test/libcephfs/reclaim.cc:149
        r = <optimized out>

The same crash reported for rgw - https://tracker.ceph.com/issues/57050
I"ll go over it to get the details.

Tamar,

Were you able to go through the changes for the rgw fix here: https://github.com/ceph/ceph/pull/47504 to see if we'd need to do something similar?

Cheers,
Venky

This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>

Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.

Milind Changire wrote:

This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>

Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.

Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?

Venky Shankar wrote:

Milind Changire wrote:

This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>

Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.

Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?

The rgw solution is to stop building radosgw as a shared library and instead build an executable binary.
There were also some references on the internet which discussed about avoiding linking against static libstdc++ when building a shared library.
Nothing conclusive so far. If the problem was indeed about linking a shared library against static libstdc++, then the problem should've been 100% reproducible in a vstart cluster as well.

Milind Changire wrote:

Venky Shankar wrote:

Milind Changire wrote:

This doesn't crash on my local ubuntu focal vstart cluster.
The stack trace points to a boost::lexical_cast<>

Hypothesis:
I'm not sure if boost::lexical_cast<> has any runtime requirements that may not match with those on the build environment.
Hence the crash.

Its not reproducible always. Did you check what changes were done in https://github.com/ceph/ceph/pull/47504 to mitigate this crash in rgw?

The rgw solution is to stop building radosgw as a shared library and instead build an executable binary.
There were also some references on the internet which discussed about avoiding linking against static libstdc++ when building a shared library.
Nothing conclusive so far. If the problem was indeed about linking a shared library against static libstdc++, then the problem should've been 100% reproducible in a vstart cluster as well.

Probably depends on the version of libstdc++ I guess. Do you see the version in the logs or maybe another linked (copy) version?

Seeing this in me recent run - https://pulpito.ceph.com/vshankar-2023-01-25_07:57:32-fs-wip-vshankar-testing-20230125.055346-testing-default-smithi/7138090/

file ./remote/smithi148/coredump/1675013313.79003.core
./remote/smithi148/coredump/1675013313.79003.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_libcephfs_reclaim', real uid: 1000, effective uid: 1000, real gid: 1267, effective gid: 1267, execfn: '/usr/bin/ceph_test_libcephfs_reclaim', platform: 'x86_64'

Didn't see this segfault in my last run, but, I've been doing fewer runs due to teuthology infra issues.

Milind, Did we RCA this?

http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7194030

Rishabh Dave wrote:

http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7194030

This shows up once in a while. We didn't RCA this, did we?

reef - https://pulpito.ceph.com/yuriw-2023-05-09_19:37:41-fs-wip-yuri10-testing-2023-05-08-0849-reef-distro-default-smithi/7268815

Venky Shankar wrote:

Rishabh Dave wrote:

http://pulpito.front.sepia.ceph.com/rishabh-2023-03-03_21:39:49-fs-wip-rishabh-2023Mar03-2316-testing-default-smithi/7194030

This shows up once in a while. We didn't RCA this, did we?

No RCA yet.

reef:
http://pulpito.front.sepia.ceph.com/yuriw-2023-06-23_16:20:29-fs-wip-yuri11-testing-2023-06-19-1232-reef-distro-default-smithi/7313672

https://pulpito.ceph.com/yuriw-2023-07-28_14:23:59-fs-wip-yuri-testing-2023-07-25-0833-reef-distro-default-smithi/7355227/

This pops in once in a while.

Found an instance of this in the smoke suite and got some more information from the coredump here: https://tracker.ceph.com/issues/62228#note-2

(gdb) bt
#0  0x00007fd656f8243e in std::locale::operator==(std::locale const&) const () from /usr/lib/ceph/libceph-common.so.2
#1  0x00007fd656e45358 in boost::detail::lcast_ret_unsigned<std::char_traits<char>, unsigned long, char>::convert (this=this@entry=0x7ffd1855c530)
    at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/lcast_unsigned_converters.hpp:184
#2  0x00007fd656e44594 in boost::detail::lexical_ostream_limited_src<char, std::char_traits<char> >::shr_unsigned<unsigned long> (output=@0x7ffd1855c510: 1, this=<synthetic pointer>)
    at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/lcast_unsigned_converters.hpp:167
#3  boost::detail::lexical_ostream_limited_src<char, std::char_traits<char> >::operator>> (output=@0x7ffd1855c510: 1, this=<synthetic pointer>)
    at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/converter_lexical_streams.hpp:641
#4  boost::detail::lexical_converter_impl<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::try_convert (arg="61", arg="61", result=@0x7ffd1855c510: 1)
    at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/detail/converter_lexical.hpp:485
#5  boost::conversion::detail::try_lexical_convert<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (result=@0x7ffd1855c510: 1, arg="61")
    at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast/try_lexical_convert.hpp:201
#6  boost::lexical_cast<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > (arg="61") at ./obj-x86_64-linux-gnu/boost/include/boost/lexical_cast.hpp:41
#7  librbd::rbd_features_from_string (orig_value="61", err=err@entry=0x7ffd1855c8d0) at ./src/librbd/Features.cc:67
#8  0x00007fd656abd49f in operator() (value=0x7ffd1855cb80, error_message=<optimized out>, __closure=<optimized out>) at ./obj-x86_64-linux-gnu/src/common/options/rbd_options.cc:44
#9  0x00007fd656a14c26 in std::function<int (std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)>::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const (__args#1=0x7ffd1855cd60, __args#0=0x7ffd1855cb80, this=<optimized out>)
    at /usr/include/c++/11/bits/std_function.h:586
#10 Option::pre_validate (this=<optimized out>, new_value=<optimized out>, err=<optimized out>) at ./src/common/options.cc:93
#11 0x00007fd656a1721d in Option::parse_value (this=0x7fd654ea8420, raw_val=..., out=<optimized out>, error_message=0x7ffd1855cd60, normalized_value=<optimized out>) at ./src/common/options.cc:147
#12 0x00007fd6569e66c2 in md_config_t::_set_val (this=0x5600deb4b510, values=..., observers=..., raw_val=..., opt=..., level=0, error_message=0x7ffd1855cd60) at ./src/common/config.cc:1395
#13 0x00007fd6569e6b37 in md_config_t::set_val_default (this=<optimized out>, values=..., tracker=..., name=..., val="61") at ./src/common/config.cc:273
#14 0x00007fd6569f6b04 in md_config_t::md_config_t (this=this@entry=0x5600deb4b510, values=..., tracker=..., is_daemon=<optimized out>, is_daemon@entry=false) at /usr/include/c++/11/string_view:137
#15 0x00007fd656997d7c in ceph::common::ConfigProxy::ConfigProxy (is_daemon=<optimized out>, this=0x5600deb47c78) at /usr/include/c++/11/bits/stl_tree.h:211
#16 ceph::common::CephContext::CephContext (this=0x5600deb47c70, module_type_=8, options=...) at ./src/common/ceph_context.cc:703
#17 0x00007fd656998e63 in ceph::common::CephContext::CephContext (this=this@entry=0x5600deb47c70, module_type_=<optimized out>, code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, init_flags_=init_flags_@entry=0)
    at /usr/include/c++/11/bits/std_function.h:239
#18 0x00007fd6569dcd45 in common_preinit (iparams=..., code_env=code_env@entry=CODE_ENVIRONMENT_LIBRARY, flags=flags@entry=0) at ./src/common/common_init.cc:38
#19 0x00007fd657688ad9 in ceph_create (cmount=0x7ffd185670e0, id=0x0) at ./src/libcephfs.cc:421
#20 0x00005600dd1c891a in update_root_mode () at ./src/test/libcephfs/reclaim.cc:151
#21 main (argc=<optimized out>, argv=0x7ffd18567288) at ./src/test/libcephfs/reclaim.cc:151

A similar issue was reported and solved in https://tracker.ceph.com/issues/57050. A note from Casey:

What i recall is that it was specific to WITH_STATIC_LIBSTDCXX which we only use for ubuntu focal.
Linking the standard library multiple times is problematic. I guess std::locale crashes there because it depends on global state, but different shared libraries would have different globals.
For rgw, I reverted a commit that moved rgw's global_init stuff into a shared library.

Perhaps something similar needs to happen for cephfs. Looking back at reported instances, all have happened on ubuntu focal.

Here's a job where the test passed on ubuntu. Could provide some clues to what changed:
https://pulpito.ceph.com/yuriw-2023-03-24_16:24:00-smoke-quincy-release-distro-default-smithi/7219355/

(sha1=23eb3b2f0fc65087846571af4e15146a980fc03d)

Possibly more helpful, here's the last instance of it passing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-14_05:00:05-smoke-main-distro-default-smithi/6929877/

And the first instance of it failing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-17_05:00:02-smoke-main-distro-default-smithi/6933658/

Could anything here have caused it?

$ git log --pretty=oneline --no-merges f9bd8922d75a454b4a0ace5a85637213e5ee5700..23eace17e0dc2a432b8532c635906af0ce79391e
2de0574382e2c1c63f20745d6870ac7f82b27b9f qa/tasks: rename persistent write log cache trash task
fddb7dd2b038d52489ec74aefa03cce5dc435e77 (origin/ci-fix-mib, ci/ci-fix-mib) ceph.spec.in: fix path for mib file and properly mark in %files
e589d0d35c1a1b2dc5fbee8bb8aae45ec11a9881 crimson/os/seastore: simplify backref cache
29c44a3bd6919f7fc23083d75d22e84c8b51e1c1 (zdover23/wip-doc-2022-07-15-start-hw-recs-cleanup-1) doc/start: update hardware recs
66bea86ab447b2db8a948c7913dc6c7a4a995c31 (origin/snapshot_key_conversion, ci/snapshot_key_conversion) osd/SnapMapper: fix pacific legacy key conversion and introduce test
d4a39cd046b93cb7bb5b7ce0311139d9f6552802 doc/cephadm: add note about OSDs being recreated to OSD removal section
80964fb5204ab18208d1033c927a620448802451 mgr/dashboard: fix rbdconfiguration init type
965005e0789e566ccadce7a326b0e197ab8d7f5f mgr/dashboard: prevent alert redirect
6f5ac1274bce02b1582be92248af7bce892a28b8 cephadm/box: Choose between docker or podman with --engine
ee9195d09efa6773a1abe2390de9d29543228c81 crimson/os/seastore: fix bug of Transaction::is_retired
0eab8de3c017d8318bd6c846991bb3f7c51fa97d qa/tasks: add thrash test for persistent write log cache
08c980ede37f1d2eba08b43ce624994b939a3ca3 mgr/dashboard: fix api tests + lint
23615b3e1b0d479595929d6483856afce67502d2 mgr/dashboard: cleanups
5a0ead8cc88fd677d882391bddcff5b474a63a9b mgr/dashboard: RbdConfiguration.list reuse image context.
c01ce41266aeec229d60c302b4da8ed6fac6f2d0 mgr/dashboard: change sort char to '+' and page icons
90869173ee4cbe2d9e24e354608995e04793bc7d mgr/dashboard: fix linting errors
3c308804d5a7a5e96842e3234edbf6ad42509e93 mgr/dashboard: add rbd list sorting support
5386dd1cb3195e7bc2dd80dafa1f0d29e8e0eb9e mgr/dashboard: update rbd list api version to 2
c832464998ed3ed34af3bda37056dfdc1729f4a6 mgr/dashboard: debounce reload data with serverSide
68ef0b83e9b0ce3ccf854eff9cbf599a263b3e7e mgr/dashboard: add rbd list search and disable sorting
08b6f22807c14967fb49495ca9f18124d495b19b mgr/dashboard: fix rbd list tests
a27eea3e8c834eb6e148aeef72a5e8522d81ef97 mgr/dashboard: fix frontend unit tests
4060061014506ed1edd32936f54ed46d489c0f11 mgr/dashboard: if null tablecontext then create empty one
27f40514ebc87a05144989be9aa1abb700f49566 mgr/dashboard: more linting
78523ec030621be548c4e3ecb8802d0b5c7bd8e8 mgr/dashboard: fix tests and iscsi rbd list
dbf2552ca57d90f6551553d25ca493375eb1e700 mgr/dashboard: bind serverSide to externalPaging
efd9a8ba5c9194e89b95aae021f62f3e164f3581 mgr/dashboard: rbd pagination poc
ed2c0247f9d4e81283e4837c90792a717ca54bc8 examples/boto3: Updating Python examples to work with Python 3
0e303cffdfb342289211f059fde25f679ae37ab9 cephadm: update LATEST_STABLE_RELEASE
eb94cda7bce38d11b9000fe93635f33bf2b3ff12 cmake: add ndctl and daxctl to build pmdk
b6b24852a6d1c6d8e785e15661f3bf29d662417f cmake: add findndctl and finddaxctl function
7fd52c0669e93fb6491ef11043d9c958c2d81b76 .github/labeler.yml: add core label to some tools
d44c0f696bac30efb79a9a99a897b4c9e994409d tools/crimson/perf_crimson_msgr: correct default addr to msgr v2 version
4cdcf7f1559e0e1453f85253fe5c4bc5ef64a93d tools/crimson/perf_crimson_msgr: init ConfigPorxy when pref_crimson_msgr start running
a581509381ba84b49c906a1fe440ca3ddcab418c rbd-mirror: remove bogus completed_non_primary_snapshots_exist check
13aa47ae2ee89babca7571165ac88eaa665d2fc0 test/rbd-mirror: add last_copied_object_number == 0 coverage
d4e6763a19fb91cd0b25c835ed9f9ccbfe70c07f cephadm/box: Rootless podman box implementation
deb7412d6f4de155bc6843f51c792b7501d2ff97 ceph.spec.in: review changes for mibs package
3a7c8c602282c8d9e95b793985237f7b3df50db7 package SNMP MIB file as an rpm
52dc04ca8a5de0626b334c7544f806b7b7dbd79a mgr/dashboard: rbd striping setting pre-population and pop-over
ae8a0dcebd21ede42aad656ba0937058de58b0bc src/stop.sh: add `sudo` to `cephadm ls` command
37aafb9170f677766996dbec584917cc6b8509b9 src/stop.sh: fixup ceph-mds daemon count

Laura Flores wrote:

Possibly more helpful, here's the last instance of it passing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-14_05:00:05-smoke-main-distro-default-smithi/6929877/

And the first instance of it failing on ubuntu 20.04 main:
https://pulpito.ceph.com/teuthology-2022-07-17_05:00:02-smoke-main-distro-default-smithi/6933658/

Could anything here have caused it?

I can't zero-in on a particular commit that could have triggered this.

The difference I can see in src/test/libcephfs/CMakeLists.txt is

for, say, test.cc:

  target_link_libraries(ceph_test_libcephfs
    ceph-common
    cephfs
    ${UNITTEST_LIBS}
    ${EXTRALIBS}
    ${CMAKE_DL_LIBS}
    )

But, for reclaim.cc:

    target_link_libraries(ceph_test_libcephfs_reclaim
      cephfs
      ${UNITTEST_LIBS}
      ${EXTRALIBS}
      ${CMAKE_DL_LIBS}
      )

Where an explicit linkage to ceph-common is missing. The crash backtrace has `common_preinit` as its origination which is a part of ceph-common, so, I think this crash has to do with another version of the lib that's getting linked somehow. Pretty weird.

Reef: https://pulpito.ceph.com/yuriw-2023-10-16_14:43:00-fs-wip-yuri4-testing-2023-10-11-0735-reef-distro-default-smithi/7429492/