Project

General

Profile

Actions
Copy link

Bug #55529

open

ceph-17.2.0/src/os/bluestore/BlueStore.cc: 14136: FAILED ceph_assert(!c)

Added by Vikhyat Umrao almost 2 years ago. Updated over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v17.0.0, Ceph - v17.1.0, Ceph - v17.2.0
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

a3100ffad1aba197ea51d060df8922d930fbc86d2bb53a1f7522707edbd923d2
a5f1b7aa92235fd3b017566aed45b0bc839136b5f80a892ab682b6cd1febea6c
e3a1e5e2ba103b54171ea9cf36e69201e0fedf4fb6297eadb8cfc18004ad1f8d

Crash signature (v2):

2add8faed9d1c1a490c4dc7b53175ea2a9a7a7f24347e9cbaba15c6ac4536be4

Description

2022-05-03T14:08:36.984+0000 7f48b0587700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.0/rpm/el8/BUILD/ceph-17.2.0/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f48b0587700 time 2022-05-03T14:08:36.979396+0000
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.2.0/rpm/el8/BUILD/ceph-17.2.0/src/os/bluestore/BlueStore.cc: 14136: FAILED ceph_assert(!c)

 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x56063eae744e]
 2: /usr/bin/ceph-osd(+0x5d766f) [0x56063eae766f]
 3: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x2caa) [0x56063f19cd4a]
 4: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x316) [0x56063f19d496]
 5: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x85) [0x56063ec5abb5]
 6: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0xff) [0x56063ebeda2f]
 7: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b6) [0x56063ec1f3f6]
 8: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x56063eeb6526]
 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xaf8) [0x56063ec110d8]
 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x56063f3169c4]
 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56063f317d64]
 12: /lib64/libpthread.so.0(+0x81cf) [0x7f48d3fc31cf]
 13: clone()

2022-05-03T14:08:37.041+0000 7f48b0587700 -1 *** Caught signal (Aborted) **
 in thread 7f48b0587700 thread_name:tp_osd_tp

 ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)
 1: /lib64/libpthread.so.0(+0x12ce0) [0x7f48d3fcdce0]
 2: gsignal()
 3: abort()
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1b0) [0x56063eae74ac]
 5: /usr/bin/ceph-osd(+0x5d766f) [0x56063eae766f]
 6: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x2caa) [0x56063f19cd4a]
 7: (BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x316) [0x56063f19d496]
 8: (ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x85) [0x56063ec5abb5]
 9: (OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0xff) [0x56063ebeda2f]
 10: (OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x1b6) [0x56063ec1f3f6]
 11: (ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x56) [0x56063eeb6526]
 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xaf8) [0x56063ec110d8]
 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x56063f3169c4]
 14: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56063f317d64]
 15: /lib64/libpthread.so.0(+0x81cf) [0x7f48d3fc31cf]
 16: clone()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Files

ceph-osd.117.log.gz (706 KB) ceph-osd.117.log.gz Tim Wilkinson, 05/03/2022 05:46 PM
Actions
Copy link
 #1

Updated by Vikhyat Umrao almost 2 years ago

  • Subject changed from 17.2.0/rpm/el8/BUILD/ceph-17.2.0/src/os/bluestore/BlueStore.cc: 14136: FAILED ceph_assert(!c) to ceph-17.2.0/src/os/bluestore/BlueStore.cc: 14136: FAILED ceph_assert(!c)
Actions
Copy link
 #3

Updated by Yaarit Hatuka almost 2 years ago

  • Crash signature (v2) updated (diff)
Actions
Copy link
 #4

Updated by Tim Wilkinson almost 2 years ago

Attaching gzipped OSD log

Actions
Copy link
 #5

Updated by Telemetry Bot almost 2 years ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v17.1.0, v17.2.0 added

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=2add8faed9d1c1a490c4dc7b53175ea2a9a7a7f24347e9cbaba15c6ac4536be4

Assert condition: !c
Assert function: void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)

Sanitized backtrace:

    pthread_kill()
    raise()
    BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)
    BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)
    ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)
    OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)
    OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)
    ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)
    OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)
    ShardedThreadPool::shardedthreadpool_worker(unsigned int)
    ShardedThreadPool::WorkThreadSharded::entry()

Crash dump sample:
{
    "assert_condition": "!c",
    "assert_file": "os/bluestore/BlueStore.cc",
    "assert_func": "void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)",
    "assert_line": 14137,
    "assert_msg": "os/bluestore/BlueStore.cc: In function 'void BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)' thread 7f1a69ffe640 time 2022-05-06T21:33:23.565851+0000\nos/bluestore/BlueStore.cc: 14137: FAILED ceph_assert(!c)",
    "assert_thread_name": "tp_osd_tp",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f1a87593520]",
        "pthread_kill()",
        "raise()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x182) [0x55734107a64f]",
        "./ceph-osd(+0x5857b1) [0x55734107a7b1]",
        "(BlueStore::_txc_add_transaction(BlueStore::TransContext*, ceph::os::Transaction*)+0x1f2c) [0x55734171a73c]",
        "(BlueStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ceph::os::Transaction, std::allocator<ceph::os::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x2e0) [0x55734171ff70]",
        "(ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ceph::os::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0xad) [0x5573411d0ddd]",
        "(OSD::dispatch_context(PeeringCtx&, PG*, std::shared_ptr<OSDMap const>, ThreadPool::TPHandle*)+0x1ae) [0x5573411780fe]",
        "(OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&)+0x2c7) [0x5573411a0bc7]",
        "(ceph::osd::scheduler::PGPeeringItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x5b) [0x55734144504b]",
        "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x993) [0x5573411919d3]",
        "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x414) [0x557341871164]",
        "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5573418736c4]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1a875e5b43]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1a87677a00]" 
    ],
    "ceph_version": "17.0.0-11722-gb40738fffcc",
    "crash_id": "2022-05-06T21:33:23.576278Z_3ba04bcd-e0ac-4144-b734-93e157e09d73",
    "entity_name": "osd.8631f4d2570401a1ae6b191aa56ca5dccf3b18c0",
    "os_id": "22.04",
    "os_name": "Ubuntu 22.04 LTS",
    "os_version": "22.04 LTS (Jammy Jellyfish)",
    "os_version_id": "22.04",
    "process_name": "ceph-osd",
    "stack_sig": "e3a1e5e2ba103b54171ea9cf36e69201e0fedf4fb6297eadb8cfc18004ad1f8d",
    "timestamp": "2022-05-06T21:33:23.576278Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.15.0-25-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#25-Ubuntu SMP Wed Mar 30 15:54:22 UTC 2022" 
}

Actions
Copy link

Also available in: Atom PDF