Project

General

Profile

Actions

Bug #56851

open

crash: int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&): assert(shard_id == onode_ref->extent_map.shards.size())

Added by Telemetry Bot over 1 year ago. Updated 11 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

64d9343e7aa94f1597bde02219a9c0649e791d7e8da0af9f8385d3824d953bfe
5f10f3dfd8b61eafc341360a1a8c4d65e9b0a73729819bd528783413a4af5e62
d08e335d5d2fdc26cad8ecc40b6e402dac8890cec5858bc53bc48a2624636166


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=825b54b05f03883919a608a975f412f9a93f18f76ab3b4c4d2dc678dccd76a24

Assert condition: shard_id == onode_ref->extent_map.shards.size()
Assert function: int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)

Sanitized backtrace:

    BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)
    BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)
    BlueStore::read_allocation_from_drive_on_startup()
    BlueStore::_init_alloc(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)
    BlueStore::_open_db_and_around(bool, bool)
    BlueStore::_mount()
    OSD::init()

Crash dump sample:
{
    "archived": "2022-07-13 02:28:54.598138",
    "assert_condition": "shard_id == onode_ref->extent_map.shards.size()",
    "assert_file": "os/bluestore/BlueStore.cc",
    "assert_func": "int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)",
    "assert_line": 18907,
    "assert_msg": "os/bluestore/BlueStore.cc: In function 'int BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)' thread 7f150bf17240 time 2022-07-12T21:08:35.737618-0500\nos/bluestore/BlueStore.cc: 18907: FAILED ceph_assert(shard_id == onode_ref->extent_map.shards.size())",
    "assert_thread_name": "ceph-osd",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14140) [0x7f150c56e140]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x5622b65f4abc]",
        "/usr/bin/ceph-osd(+0xc21c00) [0x5622b65f4c00]",
        "(BlueStore::read_allocation_from_onodes(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0xde0) [0x5622b6c24d90]",
        "(BlueStore::reconstruct_allocations(SimpleBitmap*, BlueStore::read_alloc_stats_t&)+0x5b) [0x5622b6c2534b]",
        "(BlueStore::read_allocation_from_drive_on_startup()+0x95) [0x5622b6c40a05]",
        "(BlueStore::_init_alloc(std::map<unsigned long, unsigned long, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, unsigned long> > >*)+0x9bd) [0x5622b6c4160d]",
        "(BlueStore::_open_db_and_around(bool, bool)+0x36e) [0x5622b6c6afee]",
        "(BlueStore::_mount()+0x377) [0x5622b6c6d7e7]",
        "(OSD::init()+0x54b) [0x5622b66e2dfb]",
        "main()",
        "__libc_start_main()",
        "_start()" 
    ],
    "ceph_version": "17.2.1",
    "crash_id": "2022-07-13T02:08:35.754659Z_43ffcfe2-897a-4479-95a3-c6ac5e21d882",
    "entity_name": "osd.04045e9a1554d217ec3cd5450e284534ef018f39",
    "os_id": "11",
    "os_name": "Debian GNU/Linux 11 (bullseye)",
    "os_version": "11 (bullseye)",
    "os_version_id": "11",
    "process_name": "ceph-osd",
    "stack_sig": "64d9343e7aa94f1597bde02219a9c0649e791d7e8da0af9f8385d3824d953bfe",
    "timestamp": "2022-07-13T02:08:35.754659Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.15.35-3-pve",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PVE 5.15.35-6 (Fri, 17 Jun 2022 13:42:35 +0200)" 
}

Actions #1

Updated by Telemetry Bot over 1 year ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v17.2.0, v17.2.1 added
Actions #2

Updated by Sudhin Bengeri over 1 year ago

We are running into the same problem in our ceph cluster, we are running ceph v17.2.3

Actions #3

Updated by Sudhin Bengeri over 1 year ago

Sudhin Bengeri wrote:

We are running into the same problem in our ceph cluster, we are running ceph v17.2.3.

We are interested in knowing if there are any workarounds. Thanks.

Actions #4

Updated by Igor Fedotov over 1 year ago

@Sudhin Bengeri - curious if you can reproduce the issue? If so it would be great to get OSD log with debug-bluestore set to 10 (or even 20)

Actions #5

Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v17.2.3, v17.2.5 added
Actions

Also available in: Atom PDF