Project

General

Profile

Actions
Copy link

Bug #52185

open

crash: void BlueStore::_kv_sync_thread(): assert(r == 0)

Added by Telemetry Bot over 2 years ago. Updated 11 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v13.2.10, Ceph - v14.2.1, Ceph - v14.2.10, Ceph - v14.2.11, Ceph - v14.2.12, Ceph - v14.2.13, Ceph - v14.2.15, Ceph - v14.2.16, Ceph - v14.2.2, Ceph - v14.2.20, Ceph - v14.2.21, Ceph - v14.2.22, Ceph - v14.2.4, Ceph - v14.2.5, Ceph - v14.2.6, Ceph - v14.2.7, Ceph - v14.2.8, Ceph - v14.2.9, Ceph - v15.2.1, Ceph - v15.2.11, Ceph - v15.2.4, Ceph - v15.2.5, Ceph - v15.2.6, Ceph - v15.2.7, Ceph - v15.2.8, Ceph - v16.2.0, Ceph - v16.2.1, Ceph - v16.2.4, Ceph - v16.2.5, Ceph - v16.2.7, Ceph - v16.2.9, Ceph - v17.2.0, Ceph - v17.2.5
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

0ef35ef76edbb32fb8a499c4fc655af492e6d10b845c6108a8fdc33485b44219
1282129a5e42af4d3fb6c4dc39dc0c5722c429c08f2c2e1cf2c8f99ad060de9b
17f75c46c481ff0ff945ce6d6f66c6c2709796cd57b7af809c819ac4d982f6f7
4fed4c22e9c5322bd98e328420dbe8ab8b5641881536d73f6b5ce0edc95cecb2
fe0f7e2f25de6da54c6a170cea468423caa300c196b047d834f0f45a1fb198c4
0268c7682a679fed05f298415fb6d0e9e47502853159d3a1716eb4ba1dca6198
04d2642b0dbecf8cbbb64432c9c71af1185ab3fee9781171d7751aa582185b98
069598c09cefaac15d64a3fe9af513330b301d7631d4f4db19e47957ae67f585
0f25717a4a6d70ab2d072c2c1bb2f3edd980df7264bfcf932e08dd791c21203f
1512cc22ed4b21e058bf19f8087a14f7a24d1aa31ffaa2d65b9f1d8ded90c817
16d7d67fe1131713038052ac9b956eb20637eed59f21e18098aece42dbde7cfe
214982510876023972f6eeb354e83a55bacec80455900dd0612e8315bec7caf4
27f0034eee0212554f64e70a460d003879ddf87a32d6e378d2e63ea005586072
2a502dff71dca2714cf9f6b6037e1a72e827414b6b554e4c9b4ef3a190fdf77c
2ae9022a6e82708f77190038d8f3b33f6f267dceca0fb2475623e1414caae17e
2f9280a0b87a65a6390c8f115e0cf08d405e76f4bfa4346dfd21885b481643be
31e20570f455f27a7b38a567d7bc2b0bf99b2fbc818116670b03899e423ff097
37bbbb629d70509df7b024d211245d4ce4106b255df23135d02c152050b6c271
37cb4b871d69c8d7f5d3032505f7a35acf81f64679667a339c1c3206ee268fc8
476fbd22e3b50c36a164d7149c3ec2f4db0f4986f9fddf64612589f82e5eeda6
55cff0c728a052d30ecd0862bc151b0504676e4f6490ba1cb80dc9f7e1f9eb97
56fe5c449ebd1465ed164db25aaf399d84d309471c2241d1892cd7db9e3befbe
580ed2e7d5c315bfe10047bc496f6f5249c0704bff13a092af7441ef97e31253
5a9b913fa12b1ee561f9d8512e5f5d91a547fe6c0c2805021159da1f00c872a8
6c5e5bc740ec75bba3bec5f9ac4a3f296bba9e87f7023ffe4817ee2544a7ec68
6d43ad08acaf7a0b718c7ef81a659af17356b56dbd1863114db1f9ec2e217140
6fc0fecad7cb3b63dc991289f78b5639480d2a14f422549c0b0d0cbab95449dc
7f98f7edeedac47b79cb6bd2a3b2b468b8ec71d52a1d1d59664b4076494b6bea
838a1b0e028ec63a33ebb449827e94b593d7a9d8ea3c98a6a405e06a22a24d9f
8794b0bbd1437cfef4553321c138a642e2e0827fd61428c3ae96313664998b2b
8816161b82d4112ed1e339b29e5704783984f7c93eaadc12d8ba202f78154617
88a064b0f6173d87c1ecc248c708f100bd772519a65486eecaef21b843dfc20b
8d87cdbae5fc34186e2c751fbcf10e7a42ef8ff073246a4f422a54352c994ac6
8f66fd171611632b5289fe6f56c8f5f617c47dc29572dd891b0262bf1988eb9d
944fbabdcb374ea52738fbcd3d76f94d67305ddd17ddee6b0f1a03c2055a14be
977b93f0b614bd169081f14bf897dde3895b5d36d697dda33eeecc22bcff4904
9f5af7feca6b01c2fba868b9f539dcfd76e9f302fae787e16a03a7967db954a7
a1642883cbcce6eb1a4c14c5114e7ceadac030d0f64c576f71385d8b977d4860
a6240d6e06bfd594cf679dd1faa41491e152236271aa1d398149d36ca786b4ef
b6ba525aac520510870540bee6686592a5ed0dd45ed534a9318fc027ef7735e1
bf5c51250eecf2bfee05bf43e080259097f738cba84459f1496a4112f23df281
c473bbb5f5851143b340f2fe03f369b9168080ce3db3732bb75ac3b97244b763
ccc308da41c4214a0e0461072b23197eb4d142c88ea59000f46b076c33db36d0
d17f301428a87f22be2331f4f870248d786d9134dbc426ae1d9560b9a9d4f287
d2529dc3a30c983027f277e0b63d73b90386257b55aa052e7cacb75fceb3bbd7
db5611543e607aa80abe8a32cc24a6b741304e87e0d9b921a1a476de9c014a5d
e00fac306c98348480b5f3990b52ee91e5c5e31b27d2e5ec90ce74390936d81e
e406a77c6be202533a26ea782dc7751a8f7f539cc509a31a8ae913b47dabcc33
e9973850ceb7df2987b7d18d74d8aad51f329efe07e17961cfa95dddb956342a
f19e15bc0760c293bbf1763f5ed7e33076d609f62cb357edcfc63586c776448a
f74375ce75636b5988341e142b59a7068102a07c971e1d4979dfabe54d0ffe70
fe507eebefbaa2b87ef937e9e70ced82d65fcda8c6683af9f744aa644e7f192c

Crash signature (v2):

52189c8511a286c4650cbb5fb2d4df5069d9c6a52664a4ce26484baeaf9c9e6c

Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=0ef35ef76edbb32fb8a499c4fc655af492e6d10b845c6108a8fdc33485b44219

Assert condition: r == 0
Assert function: void BlueStore::_kv_sync_thread()

Sanitized backtrace:

    /lib64/libpthread.so.0(
    ceph-osd(
    BlueStore::_kv_sync_thread()
    BlueStore::KVSyncThread::entry()
    /lib64/libpthread.so.0(
    clone()

Crash dump sample:
{
    "assert_condition": "r == 0",
    "assert_file": "os/bluestore/BlueStore.cc",
    "assert_func": "void BlueStore::_kv_sync_thread()",
    "assert_line": 12135,
    "assert_msg": "os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7fbcc4293700 time 2021-08-08T01:15:34.846066+0000\nos/bluestore/BlueStore.cc: 12135: FAILED ceph_assert(r == 0)",
    "assert_thread_name": "bstore_kv_sync",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12b20) [0x7fbcd8d37b20]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x555816287f0b]",
        "ceph-osd(+0x56a0d4) [0x5558162880d4]",
        "(BlueStore::_kv_sync_thread()+0x3101) [0x5558168e9f31]",
        "(BlueStore::KVSyncThread::entry()+0x11) [0x555816910df1]",
        "/lib64/libpthread.so.0(+0x814a) [0x7fbcd8d2d14a]",
        "clone()" 
    ],
    "ceph_version": "16.2.5",
    "crash_id": "2021-08-08T01:15:34.861690Z_1c821a6f-7ad8-4564-bf6b-d65cc046bd40",
    "entity_name": "osd.a6f4d9de23744c4f2fbd5ffa8b55427a42cb1d69",
    "os_id": "centos",
    "os_name": "CentOS Linux",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-osd",
    "stack_sig": "fe0f7e2f25de6da54c6a170cea468423caa300c196b047d834f0f45a1fb198c4",
    "timestamp": "2021-08-08T01:15:34.861690Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.4.0-80-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021" 
}

Actions
Copy link
 #1

Updated by Telemetry Bot over 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.0, v16.2.1, v16.2.4, v16.2.5 added
Actions
Copy link
 #2

Updated by Janne Johansson about 2 years ago

Telemetry Bot wrote:

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=0ef35ef76edbb32fb8a499c4fc655af492e6d10b845c6108a8fdc33485b44219

Assert condition: r == 0
Assert function: void BlueStore::_kv_sync_thread()

Sanitized backtrace:
[...]
Crash dump sample:
[...]

I get this on old luminous too, both while running and from ceph-bluestore-tool fsck/repairs:

SingleDelete(Prefix = L Key = 0x00000000096e51a6)
SingleDelete(Prefix = L Key = 0x00000000096e51a7)
0> 2022-02-02 10:56:57.166035 7f15ea67b700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/BlueStore.cc: In function 'void BlueStore::_kv_sync_thread()' thread 7f15ea67b700 time 2022-02-02 10:56:57.164412
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.13/rpm/el7/BUILD/ceph-12.2.13/src/os/bluestore/BlueStore.cc: 9147: FAILED assert(r == 0)

Actions
Copy link
 #3

Updated by Telemetry Bot about 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
Actions
Copy link
 #4

Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v13.2.10, v14.2.1, v14.2.10, v14.2.11, v14.2.12, v14.2.13, v14.2.15, v14.2.16, v14.2.2, v14.2.20, v14.2.21, v14.2.22, v14.2.4, v14.2.5, v14.2.6, v14.2.7, v14.2.8, v14.2.9, v15.2.1, v15.2.11, v15.2.4, v15.2.5, v15.2.6, v15.2.7, v15.2.8, v16.2.7, v16.2.9, v17.2.0, v17.2.5 added
Actions
Copy link

Also available in: Atom PDF