Project

General

Profile

Actions
Copy link

Bug #64009

closed

Crimson: PGShardMapping::maybe_create_pg() assert failure

Added by Yingxin Cheng 4 months ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Yingxin Cheng
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
56332
Crash signature (v1):
Crash signature (v2):

Description


ERROR 2024-01-10 12:23:02,728 [shard 2] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-587-ga223060f/rpm/el9/BUILD/ceph-19.0.0-587-ga223060f/src/crimson/osd/pg_map.h:73 : In function 'crimson::osd::PGShardMapping::maybe_create_pg(spg_t, crimson::core_id_t)::<lambda(auto:122&)>::<lambda(auto:125&)> [with auto:125 = crimson::osd::PGShardMapping]', ceph_assert(%s)
inserted
ERROR 2024-01-10 12:23:02,728 [shard 2] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-587-ga223060f/rpm/el9/BUILD/ceph-19.0.0-587-ga223060f/src/crimson/osd/pg_map.h:73 : In function 'crimson::osd::PGShardMapping::maybe_create_pg(spg_t, crimson::core_id_t)::<lambda(auto:122&)>::<lambda(auto:125&)> [with auto:125 = crimson::osd::PGShardMapping]', ceph_assert(%s)
inserted
INFO  2024-01-10 12:23:02,728 [shard 0] osd - osd.1: committed_osd_maps: broadcasting osdmaps up to 588 epoch to pgs
Aborting on shard 2.

.. 

crimson/osd/pg_map.h:73 : In function 'crimson::osd::PGShardMapping::maybe_create_pg(spg_t, crimson::core_id_t)::<lambda(auto:122&)>::<lambda(auto:125&)> [with auto:125 = crimson::osd::PGShardMapping]', ceph_assert(%s) inserted
 0# 0x00007F8D47AA154C in /lib64/libc.so.6
 1# raise in /lib64/libc.so.6
 2# abort in /lib64/libc.so.6
 3# ceph::__ceph_assert_fail(char const*, char const*, int, char const*) in ceph-osd
 4# 0x000056348593BBB3 in ceph-osd
 5# 0x0000563485A8695E in ceph-osd
 6# _ZNSt17_Function_handlerIFN7seastar6futureIvEERN7crimson3osd14PGShardMappingEEZNS0_7shardedIS5_E13invoke_on_allIZNS9_16invoke_on_othersIZZNS5_15maybe_create_pgE5spg_tjENKUlRT_E_clIS5_EEDaSE_EUlSE_E0_JEEES2_NS0_21smp_submit_to_optionsESD_DpT0_EUlS6_E_JEEES2_SI_SD_SK_EUlS6_E_E9_M_invokeERKSt9_Any_dataS6_ in ceph-osd
 7# 0x0000563485F02AFA in ceph-osd
 8# 0x00005634943C6FDE in ceph-osd
 9# 0x000056349442A227 in ceph-osd
10# 0x00005634946D0342 in ceph-osd
11# 0x00005634946D439D in ceph-osd
12# 0x00005634946D4919 in ceph-osd
13# 0x000056349413FE5C in ceph-osd
14# 0x00007F8D47A9F802 in /lib64/libc.so.6
15# 0x00007F8D47A3F450 in /lib64/libc.so.6

https://pulpito.ceph.com/yingxin-2024-01-10_02:12:17-crimson-rados-wip-yingxin-crimson-alienstore-submit-concurrently-2-distro-default-smithi/7511015/

Related issues 1 (0 open — 1 closed)

Has duplicate crimson - Bug #64934: crimson: failed assert in get_or_create_mappingDuplicate

Actions
Actions
Copy link
 #1

Updated by Matan Breizman 2 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Matan Breizman 2 months ago

https://pulpito.ceph.com/matan-2024-02-21_12:07:57-crimson-rados-wip-matanb-crimson-alien-buf-v3-testing-distro-crimson-smithi/7569493/

osd 2 (func name changed to get_or_create_pg_mapping)

ERROR 2024-02-21 12:31:30,944 [shard 0:main] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1583-g7b226c5f/rpm/el9/BUILD/ceph-19.0.0-1583-g7b226c5f/src/crimson/osd/pg_map.cc:31 : In function 'crimson::osd::PGShardMapping::get_or_create_pg_mapping(spg_t, crimson::core_id_t)::<lambda(auto:191&)> [with auto:191 = crimson::osd::PGShardMapping]', ceph_assert(%s)
inserted
Aborting on shard 0.
Backtrace:
Reactor stalled for 154 ms on shard 0. Backtrace: 0x45d5d 0x2f29f528 0x2ef9f726 0x2efa4cfc 0x2efa5080 0x2efa51ca 0x2efa5620 0x54daf 0x118a06 0x118829 0x264c2f68 0x264c5579 0x264c89f0 0x264c95df 0x264c9c56 0x264c9fcb 0x264ca4ac 0x264caa50 0x54daf 0xa154b 0x54d05 0x287f2 0x264dfe77 0x233d8130 0x2348c689 0x2f1ddf61 0x2f22b466 0x2f416157 0x2f417fac 0x2ed2dce8 0x2ed2fbf1 0x1fd76d53 0x3feaf 0x3ff5f 0x1f935fc4
kernel callstack: 0xffffffffffffff80 0xffffffffba9dd5a3 0xffffffffba9dde74 0xffffffffba998e0e 0xffffffffba705ce8 0xffffffffba705ea1 0xffffffffba7072af 0xffffffffba707f8b 0xffffffffba708440 0xffffffffbb27e45c 0xffffffffbb4000ea
Reactor stalled for 311 ms on shard 0. Backtrace: 0x45d5d 0x2f29f528 0x2ef9f726 0x2efa4cfc 0x2efa5080 0x2efa51ca 0x2efa5620 0x54daf 0x195bd9 0x264c22b1 0x264c5579 0x264c89f0 0x264c95df 0x264c9c56 0x264c9fcb 0x264ca4ac 0x264caa50 0x54daf 0xa154b 0x54d05 0x287f2 0x264dfe77 0x233d8130 0x2348c689 0x2f1ddf61 0x2f22b466 0x2f416157 0x2f417fac 0x2ed2dce8 0x2ed2fbf1 0x1fd76d53 0x3feaf 0x3ff5f 0x1f935fc4
kernel callstack:
Reactor stalled for 552 ms on shard 0. Backtrace: 0x45d5d 0x2f29f528 0x2ef9f726 0x2efa4cfc 0x2efa5080 0x2efa51ca 0x2efa5620 0x54daf 0x118a06 0x118829 0x264c2f68 0x264c5579 0x264c89f0 0x264c95df 0x264c9c56 0x264c9fcb 0x264ca4ac 0x264caa50 0x54daf 0xa154b 0x54d05 0x287f2 0x264dfe77 0x233d8130 0x2348c689 0x2f1ddf61 0x2f22b466 0x2f416157 0x2f417fac 0x2ed2dce8 0x2ed2fbf1 0x1fd76d53 0x3feaf 0x3ff5f 0x1f935fc4
kernel callstack: 0xffffffffffffff80 0xffffffffba97a57a 0xffffffffba97ab81 0xffffffffba981576 0xffffffffba981916 0xffffffffba705d04 0xffffffffba705ea1 0xffffffffba7072af 0xffffffffba707f8b 0xffffffffba708440 0xffffffffbb27e45c 0xffffffffbb4000ea
 0# 0x00007F00EF6A154C in /lib64/libc.so.6
 1# raise in /lib64/libc.so.6
 2# abort in /lib64/libc.so.6
 3# ceph::__ceph_assert_fail(char const*, char const*, int, char const*) in ceph-osd

See comment from origin PR:
https://github.com/ceph/ceph/pull/51147#discussion_r1247658911

Actions
Copy link
 #3

Updated by Nitzan Mordechai 2 months ago

  • Assignee set to Nitzan Mordechai
Actions
Copy link
 #5

Updated by Yingxin Cheng about 2 months ago

  • Related to Bug #64934: crimson: failed assert in get_or_create_mapping added
Actions
Copy link
 #7

Updated by Matan Breizman about 1 month ago

  • Related to deleted (Bug #64934: crimson: failed assert in get_or_create_mapping)
Actions
Copy link
 #8

Updated by Matan Breizman about 1 month ago

  • Has duplicate Bug #64934: crimson: failed assert in get_or_create_mapping added
Actions
Copy link
 #9

Updated by Matan Breizman about 1 month ago

  • Status changed from New to Resolved
  • Assignee changed from Nitzan Mordechai to Yingxin Cheng
  • Pull request ID set to 56332
Actions
Copy link

Also available in: Atom PDF