Bug #50089: mon/MonMap.h: FAILED ceph_assert(m < ranks.size()) when reducing number of monitors in the cluster - RADOS - Ceph

Actions

Copy link

#1

Updated by Neha Ojha about 3 years ago

Related to Bug #50088: rados: qa: suites do not test mon removal added

Actions

Copy link

#2

Updated by Neha Ojha about 3 years ago

showed up in a pacific->master upgrade test

2021-04-19T19:33:42.749 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-3406-gb9575dc7/rpm/el8/BUILD/ceph-17.0.0-3406-gb9575dc7/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7fc266907700 time 2021-04-19T19:33:42.396505+0000
2021-04-19T19:33:42.749 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-3406-gb9575dc7/rpm/el8/BUILD/ceph-17.0.0-3406-gb9575dc7/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())
2021-04-19T19:33:42.749 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  ceph version 17.0.0-3406-gb9575dc7 (b9575dc757ca28607c59f4051113e4a25ed8728b) quincy (dev)
2021-04-19T19:33:42.749 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7fc270c8a518]
2021-04-19T19:33:42.750 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  2: /usr/lib64/ceph/libceph-common.so.2(+0x27c720) [0x7fc270c8a720]
2021-04-19T19:33:42.750 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  3: (Elector::send_peer_ping(int, utime_t const*)+0x448) [0x55c28753dea8]
2021-04-19T19:33:42.750 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  4: (Elector::ping_check(int)+0x30f) [0x55c28753e7cf]
2021-04-19T19:33:42.751 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  5: (Context::complete(int)+0xd) [0x55c2874bdbed]
2021-04-19T19:33:42.751 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  6: (SafeTimer::timer_thread()+0x1c0) [0x7fc270d901a0]
2021-04-19T19:33:42.751 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  7: (SafeTimerThread::entry()+0x11) [0x7fc270d92d41]
2021-04-19T19:33:42.751 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  8: (Thread::_entry_func(void*)+0xd) [0x7fc270d81e1d]
2021-04-19T19:33:42.752 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  9: /lib64/libpthread.so.0(+0x814a) [0x7fc26e77414a]
2021-04-19T19:33:42.752 INFO:journalctl@ceph.mon.a.smithi082.stdout:Apr 19 19:33:42 smithi082 conmon[74090]:  10: clone()

rados/upgrade/pacific-x/parallel/{0-start 1-tasks distro1$/{rhel_8.3_kubic_stable} mon_election/classic upgrade-sequence workload/{ec-rados-default rados_api rados_loadgenbig rbd_import_export test_rbd_api test_rbd_python}}

/a/nojha-2021-04-19_18:28:32-rados-master-distro-basic-smithi/6059600

Actions

Copy link

#3

Updated by Tejas C almost 3 years ago

I see a similar crash on quincy, suspect its seen when I try to add mons from 1 to 3 .

/]# ceph crash info 2021-06-28T06:22:31.271856Z_784b13d6-24fd-43ad-9ff2-79a1317cd3d0 {
"assert_condition": "m < ranks.size()",
"assert_file": "/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-5278-g79eb0c85/rpm/el8/BUILD/ceph-17.0.0-5278-g79eb0c85/src/mon/MonMap.h",
"assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const",
"assert_line": 404,
"assert_msg": "/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-5278-g79eb0c85/rpm/el8/BUILD/ceph-17.0.0-5278-g79eb0c85/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f7c985aa700 time 2021-06-28T06:22:31.267650+0000\n/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-5278-g79eb0c85/rpm/el8/BUILD/ceph-17.0.0-5278-g79eb0c85/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())\n",
"assert_thread_name": "safe_timer",
"backtrace": [
"/lib64/libpthread.so.0(+0x12b20) [0x7f7ca01f0b20]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a0) [0x7f7ca26fcc4a]",
"/usr/lib64/ceph/libceph-common.so.2(+0x27ce04) [0x7f7ca26fce04]",
"(Elector::send_peer_ping(int, utime_t const*)+0x448) [0x56180a614608]",
"(Elector::ping_check(int)+0x30f) [0x56180a614f2f]",
"(Context::complete(int)+0xd) [0x56180a59497d]",
"(SafeTimer::timer_thread()+0x1c0) [0x7f7ca2801c90]",
"(SafeTimerThread::entry()+0x11) [0x7f7ca2804831]",
"(Thread::_entry_func(void*)+0xd) [0x7f7ca27f390d]",
"/lib64/libpthread.so.0(+0x814a) [0x7f7ca01e614a]",
"clone()"
],
"ceph_version": "17.0.0-5278-g79eb0c85",
"crash_id": "2021-06-28T06:22:31.271856Z_784b13d6-24fd-43ad-9ff2-79a1317cd3d0",
"entity_name": "mon.clara001",
"os_id": "centos",
"os_name": "CentOS Linux",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mon",
"stack_sig": "ec1f126f91754dcb8260f74bec942786e56cfe2e728d1c50c45bc9a62fd40586",
"timestamp": "2021-06-28T06:22:31.271856Z",
"utsname_hostname": "clara001",
"utsname_machine": "x86_64",
"utsname_release": "4.18.0-305.el8.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Thu Apr 29 08:54:30 EDT 2021"
}

Actions

Copy link

#4

Updated by Neha Ojha almost 3 years ago

Priority changed from High to Urgent

Actions

Copy link

#5

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52183: crash: const entity_addrvec_t& MonMap::get_addrs(unsigned int) const: assert(m < ranks.size()) added

Actions

Copy link

#6

Updated by Neha Ojha over 2 years ago

Has duplicate Bug #52170: crash: const entity_addrvec_t& MonMap::get_addrs(unsigned int) const: assert(m < ranks.size()) added

Actions

Copy link

#7

Updated by Neha Ojha over 2 years ago

Backport changed from pacific to pacific,quincy

Actions

Copy link

#8

Updated by Neha Ojha over 2 years ago

Assignee changed from Greg Farnum to Kamoltat (Junior) Sirivadhna

Actions

Copy link

#9

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

Status changed from New to Fix Under Review

https://github.com/ceph/ceph/pull/44993

Actions

Copy link

#10

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

Pull request ID set to 44993

Actions

Copy link

#11

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

Update:

We ran and analyzed a total of 3 runs:

Local vstart stopping the monitors before removing the monitors out of clusters.
Local vstart removing the monitors out of clusters before stopping the monitors.
Teuthology `ceph orch apply mon 3` to reduce monitors.

Teuthology run, putting 5 monitors in 5 different hosts and reducing them to 3 monitors using the command `ceph orch apply mon 3` recreated the bug. Local vstart run with 5 monitors in 1 node, reducing the number of monitors down by manually stopping them, then removing them did not recreate the bug. However, a local vstart run, reducing the monitors down by manually removing them first before stopping does recreate the bug with similar behavior in the elector code, we can observe by comparing mon.a (leader) logs of both teuthology run and local vstart remove before stop.

By comparing my two attempts in using the local vstart cluster to reproduce the bug we can conclude that removing the monitor from the cluster before stopping the monitor (shutting it down) actually recreated the bug. According to https://docs.ceph.com/en/latest/rados/operations/add-or-rm-mons/#removing-a-monitor-manual The correct manual process of removing a monitor should be in this order:

1. stopping the monitor (shutting it down)
2. removing the monitor from the cluster

This leads to us questioning the mechanism behind the command: `ceph orch apply mon 3` (the command that was used to reduce the monitor from 5 down to 3). Is it possible that reducing the monitors through ceph orch command removes the monitors before shutting it down? We shall confirm this and fix it if need be.

In any case, ceph should not crash even if the user removes the monitor before shutting it down. Therefore, we shall proceed with creating a sanity check for when the rank of the quorum gets reduced before the monitor becomes unreachable through pinging.

Here is the PR I am currently working on with this sanity check: https://github.com/ceph/ceph/pull/44993

For full details regarding the logs I have collected and my full analysis: https://docs.google.com/document/d/1za8dl4lu2wygKfrQAP0NEFAAy6DJ45KcjmGNQTz5Nmc/edit?usp=sharing

Actions

Copy link

#12

Updated by Telemetry Bot about 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v16.0.0, v16.2.0, v16.2.1, v16.2.4, v16.2.5, v16.2.6, v16.2.7 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=96b49c839d59492286f04a76ececd021835a660aabcfedc92ead1b3b31aa9978

Assert condition: m < ranks.size()
Assert function: const entity_addrvec_t& MonMap::get_addrs(unsigned int) const

Sanitized backtrace:

    /lib64/libpthread.so.0(
    /usr/lib64/ceph/libceph-common.so.2(
    Elector::send_peer_ping(int, utime_t const*)
    Elector::begin_peer_ping(int)
    Elector::handle_ping(boost::intrusive_ptr<MonOpRequest>)
    Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)
    Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)
    Monitor::_ms_dispatch(Message*)
    Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)
    DispatchQueue::entry()
    DispatchQueue::DispatchThread::entry()
    /lib64/libpthread.so.0(
    clone()

Crash dump sample:

{
    "assert_condition": "m < ranks.size()",
    "assert_file": "mon/MonMap.h",
    "assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const",
    "assert_line": 404,
    "assert_msg": "mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f00f8918700 time 2022-03-11T05:03:35.740526+0000\nmon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())",
    "assert_thread_name": "ms_dispatch",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7f0103f95ce0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f0106259ba3]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x276d6c) [0x7f0106259d6c]",
        "(Elector::send_peer_ping(int, utime_t const*)+0x448) [0x5580d016bf18]",
        "(Elector::begin_peer_ping(int)+0x1eb) [0x5580d016c14b]",
        "(Elector::handle_ping(boost::intrusive_ptr<MonOpRequest>)+0x27b) [0x5580d016d11b]",
        "(Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xb3) [0x5580d016dd03]",
        "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xb0b) [0x5580d00e8ddb]",
        "(Monitor::_ms_dispatch(Message*)+0x670) [0x5580d00ea080]",
        "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5580d0118f4c]",
        "(DispatchQueue::entry()+0x126a) [0x7f010649daba]",
        "(DispatchQueue::DispatchThread::entry()+0x11) [0x7f010654f5d1]",
        "/lib64/libpthread.so.0(+0x81cf) [0x7f0103f8b1cf]",
        "clone()" 
    ],
    "ceph_version": "16.2.7",
    "crash_id": "2022-03-11T05:03:35.744091Z_e0bbc4d8-76cc-4763-a41b-dcd8a34cd463",
    "entity_name": "mon.e36bb386e509bfd98b0ff41679d32d33d6a9f6e1",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-mon",
    "stack_sig": "e9e13cf41d815dd96f1d1014f9c144fa8e74c842164e8ed8d4fd4c268491ce16",
    "timestamp": "2022-03-11T05:03:35.744091Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.13.0-35-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022" 
}

Actions

Copy link

#13

Updated by Telemetry Bot about 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v16.2.3 added

Actions

Copy link

#14

Updated by Telemetry Bot about 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)

Actions

Copy link

#15

Updated by Telemetry Bot about 2 years ago

Crash signature (v1) updated (diff)

Actions

Copy link

#16

Updated by Laura Flores about 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)

/a/teuthology-2022-01-09_07:01:02-rados-master-distro-default-smithi/6604561
/a/yuriw-2022-03-10_02:41:10-rados-wip-yuri3-testing-2022-03-09-1350-distro-default-smithi/6729296
/a/yuriw-2022-03-19_14:39:53-rados-quincy-distro-default-smithi/6747175

2022-01-10T19:57:37.509 INFO:tasks.workunit.client.0.smithi038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh:105: TEST_mon_quorum:  ceph quorum_status --format=json
2022-01-10T19:57:37.632 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 19:57:37 socat[40133] E write(6, 0x55c37d3266c0, 153): Broken pipe
2022-01-10T19:57:48.484 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 19:57:48 socat[40189] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused

...

2022-01-10T20:47:03.286 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:03 socat[42705] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:03.287 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:03 socat[42706] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused
2022-01-10T20:47:05.351 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:05 socat[42707] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:06.683 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:06 socat[42708] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused
2022-01-10T20:47:18.300 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:18 socat[42717] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:18.302 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:18 socat[42718] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:18.303 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:18 socat[42719] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused
2022-01-10T20:47:20.367 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:20 socat[42720] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:21.699 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:21 socat[42721] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused
2022-01-10T20:47:33.313 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:33 socat[42722] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:33.318 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:33 socat[42723] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:33.319 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:33 socat[42724] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused
2022-01-10T20:47:35.383 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:35 socat[42725] E connect(5, AF=2 127.0.0.1:7137, 16): Connection refused
2022-01-10T20:47:36.715 INFO:tasks.workunit.client.0.smithi038.stderr:2022/01/10 20:47:36 socat[42726] E connect(5, AF=2 127.0.0.1:7136, 16): Connection refused

...

2022-01-10T20:47:37.606 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh:105: TEST_mon_quorum:  jqinput=
2022-01-10T20:47:37.607 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh:106: TEST_mon_quorum:  jq_success '' '.monmap.mons | length == 3'

...

2022-01-10T20:47:37.733 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown:  rm -rf /tmp/ceph-asok.39613
2022-01-10T20:47:37.734 INFO:tasks.workunit.client.0.smithi038.stdout:ERROR: Failure due to cores found
2022-01-10T20:47:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-01-10T20:47:37.736 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown:  '[' yes = yes ']'
2022-01-10T20:47:37.736 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:201: teardown:  echo 'ERROR: Failure due to cores found'
2022-01-10T20:47:37.737 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:202: teardown:  '[' -n '' ']'
2022-01-10T20:47:37.737 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:205: teardown:  return 1
2022-01-10T20:47:37.737 INFO:tasks.workunit.client.0.smithi038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main:  return 1
2022-01-10T20:47:37.738 INFO:tasks.workunit:Stopping ['mon'] on client.0...

2022-03-10T08:45:17.921+0000 7ff1dcbe9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11051-gb5ca1477/rpm/el8/BUILD/ceph-17.0.0-11051-gb5ca1477/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7ff1dcbe9700 time 2022-03-10T08:45:17.921667+0000
/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-11051-gb5ca1477/rpm/el8/BUILD/ceph-17.0.0-11051-gb5ca1477/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())

 ceph version 17.0.0-11051-gb5ca1477 (b5ca14771f888fa46234c86404e0aee4913031fe) quincy (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7ff1e8ea1204]
 2: /usr/lib64/ceph/libceph-common.so.2(+0x284425) [0x7ff1e8ea1425]
 3: (Elector::send_peer_ping(int, utime_t const*)+0x440) [0x561824c71cc0]
 4: (Elector::begin_peer_ping(int)+0x1eb) [0x561824c71eeb]
 5: (Elector::handle_ping(boost::intrusive_ptr<MonOpRequest>)+0x281) [0x561824c72eb1]
 6: (Elector::dispatch(boost::intrusive_ptr<MonOpRequest>)+0xc5) [0x561824c73a95]
 7: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0xc1d) [0x561824bd393d]
 8: (Monitor::_ms_dispatch(Message*)+0x457) [0x561824bd4ad7]
 9: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x561824c0505c]
 10: (DispatchQueue::entry()+0x14fa) [0x7ff1e912873a]
 11: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff1e91dfa61]
 12: /lib64/libpthread.so.0(+0x817f) [0x7ff1e6e2917f]
 13: clone()

Expected output from a successful run:

/a/yuriw-2022-03-18_00:42:20-rados-wip-yuri6-testing-2022-03-17-1547-distro-default-smithi/6743533

2022-03-18T02:51:06.225 INFO:tasks.workunit.client.0.smithi049.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh:105: TEST_mon_quorum:  ceph quorum_status --format=json
2022-03-18T02:51:09.756 INFO:tasks.workunit.client.0.smithi049.stderr:2022/03/18 02:51:09 socat[41600] E write(6, 0x561c37c116c0, 68): Broken pipe
2022-03-18T02:51:09.983 INFO:tasks.workunit.client.0.smithi049.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon/mon-bind.sh:105: TEST_mon_quorum:  jqinput='
2022-03-18T02:51:09.983 INFO:tasks.workunit.client.0.smithi049.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138303579357183","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"]},"monmap":{"epoch":1,"fsid":"c8f27b4a-0323-4ffc-8154-da3f6a79cb0d","modified":"2022-03-18T02:51:03.949146Z","created":"2022-03-18T02:51:03.949146Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7132","nonce":0}]},"addr":"127.0.0.1:7132/0","public_addr":"127.0.0.1:7132/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7133","nonce":0}]},"addr":"127.0.0.1:7133/0","public_addr":"127.0.0.1:7133/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7134","nonce":0}]},"addr":"127.0.0.1:7134/0","public_addr":"127.0.0.1:7134/0","priority":0,"weight":0,"crush_location":"{}"}]}}'

Actions

Copy link

#17

Updated by Laura Flores about 2 years ago

Has duplicate Bug #54529: mon/mon-bind.sh: Failure due to cores found added

Actions

Copy link

#18

Updated by Radoslaw Zarzynski about 2 years ago

Status changed from Fix Under Review to In Progress

Actions

Copy link

#19

Updated by Neha Ojha about 2 years ago

Pull request ID changed from 44993 to 45299

https://github.com/ceph/ceph/pull/44993 was closed

Actions

Copy link

#20

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

Pull request ID changed from 45299 to 44993

Actions

Copy link

#21

Updated by Telemetry Bot almost 2 years ago

Crash signature (v1) updated (diff)
Affected Versions v16.2.9, v17.2.0 added

Actions

Copy link

#22

Updated by Kamoltat (Junior) Sirivadhna almost 2 years ago

Related to Bug #55695: Shutting down a monitor forces Paxos to restart and sometimes disregard subsequent commands added

Actions

Copy link

#23

Updated by Gaurav Sitlani over 1 year ago

Crash signature (v1) updated (diff)

I am seeing the same crash in version : ceph version 16.2.10 and just noticed that PR linked in this thread is merged. @Kamoltat Sirivadhna Is there any plans to backport and fix this issue in pacific ?

Actions

Copy link

#24

Updated by Gaurav Sitlani over 1 year ago

[ceph: root@X /]# ceph crash ls
ID                                                                ENTITY     NEW  
2022-09-28T09:55:37.128747Z_f81dc440-4b96-436d-a272-02c9d094232d  mon.X   *   
[ceph: root@X /]# ceph crash info 2022-09-28T09:55:37.128747Z_f81dc440-4b96-436d-a272-02c9d094232d
{
    "assert_condition": "m < ranks.size()",
    "assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/mon/MonMap.h",
    "assert_func": "const entity_addrvec_t& MonMap::get_addrs(unsigned int) const",
    "assert_line": 404,
    "assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/mon/MonMap.h: In function 'const entity_addrvec_t& MonMap::get_addrs(unsigned int) const' thread 7f6e23c9b700 time 2022-09-28T09:55:37.125709+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/mon/MonMap.h: 404: FAILED ceph_assert(m < ranks.size())\n",
    "assert_thread_name": "safe_timer",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7f6e2cb20ce0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7f6e2ede6e39]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7f6e2ede7002]",
        "(Elector::send_peer_ping(int, utime_t const*)+0x448) [0x55e32ae395d8]",
        "(Elector::ping_check(int)+0x30f) [0x55e32ae39eff]",
        "(Context::complete(int)+0xd) [0x55e32adb938d]",
        "(CommonSafeTimer<std::mutex>::timer_thread()+0x10f) [0x7f6e2eedcc5f]",
        "(CommonSafeTimerThread<std::mutex>::entry()+0x11) [0x7f6e2eeddff1]",
        "/lib64/libpthread.so.0(+0x81ca) [0x7f6e2cb161ca]",
        "clone()" 
    ],
    "ceph_version": "16.2.10",
    "crash_id": "2022-09-28T09:55:37.128747Z_f81dc440-4b96-436d-a272-02c9d094232d",
    "entity_name": "mon.X",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-mon",
    "stack_sig": "45b9556ff37ea7005e07c92442f15b857a77cd73ba7cb0eca73b9478834f57c8",
    "timestamp": "2022-09-28T09:55:37.128747Z",
    "utsname_hostname": "X",
    "utsname_machine": "x86_64",
    "utsname_release": "4.18.0-408.el8.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP Mon Jul 18 17:42:52 UTC 2022" 
}

Actions

Copy link

#25