Project

General

Profile

Actions
Copy link

Bug #55408

open

libceph: corrupt inc osdmap (-12) epoch 409760 off 60 (ffffacad17925058 of ffffacad1792501c-ffffacad179edf02)

Added by Dan Moraru about 2 years ago. Updated about 2 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
Ilya Dryomov
Category:
libceph
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v15.2.15
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

A Scientific Linux 7.9 system running the latest kernel (3.10.0-1160.62.1.el7.x86_64) logged a "corrupt inc osdmap" message and became wedged after reading data from a Ceph 15.2.15 cluster at a few GBytes/s for ~18 hours. An initial umount was hung un-killable and a subsequent umount -f no longer thinks the filesystem is mounted. However, even though there is no entry in /etc/mtab the libceph and ceph kernel modules are busy and a subsequent attempt to mount hangs (but is killable),

[root@ldas-pcdev12 ~]# umount /ceph/mirror
^C^C^C^C

[root@ldas-pcdev12 ~]# umount -f /ceph/mirror
umount: /ceph/mirror: not mounted

There was plenty of cached memory (>300GB) at the time that cephfs hung.

Additional "libceph: osdc handle_map corrupt msg" and "libceph: corrupt inc osdmap" messages were logged to console.

What went wrong and how would one recover without rebooting?

[root@ldas-pcdev12 ~]# less /var/log/messages
...
Apr 20 15:00:57 ldas-pcdev12 kernel: kworker/103:4: page allocation failure: order:7, mode:0x4010
Apr 20 15:00:57 ldas-pcdev12 kernel: CPU: 103 PID: 1502626 Comm: kworker/103:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Apr 20 15:00:57 ldas-pcdev12 kernel: Hardware name: Dell Inc. PowerEdge XE8545/099K88, BIOS 2.6.6 01/13/2022
Apr 20 15:00:57 ldas-pcdev12 kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: Call Trace:
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa3865a9>] dump_stack+0x19/0x1b
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc4bd0>] warn_alloc_failed+0x110/0x180
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc976f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e193a8>] alloc_pages_current+0x98/0x110
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9de5fc8>] kmalloc_order+0x18/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e24d76>] kmalloc_order_trace+0x26/0xa0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28cb0>] ? __kmalloc+0x1c0/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28d01>] __kmalloc+0x211/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc09064b9>] ? crush_decode+0x879/0x15a0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0904e47>] osdmap_set_crush.isra.16+0x47/0xc0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0908046>] osdmap_apply_incremental+0x216/0x960 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08fdba3>] handle_one_map+0x83/0x250 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0902342>] ceph_osdc_handle_map+0x232/0x8c0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08efff3>] ? read_partial_message+0x1a3/0x900 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f6580>] dispatch+0x350/0x780 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f0ff4>] try_read+0x544/0x1300 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cd76cf>] ? ttwu_do_activate+0x6f/0x80
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce497c>] ? update_curr+0x14c/0x1e0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce10ae>] ? account_entity_dequeue+0xae/0xd0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce4e6c>] ? dequeue_entity+0x11c/0x5c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f1fb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbdfdf>] process_one_work+0x17f/0x440
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbf0f6>] worker_thread+0x126/0x3c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5fb1>] kthread+0xd1/0xe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa399de4>] ret_from_fork_nospec_begin+0xe/0x21
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: Mem-Info:
Apr 20 15:00:57 ldas-pcdev12 kernel: active_anon:33795845 inactive_anon:430684 isolated_anon:0#012 active_file:7465053 inactive_file:85364311 isolated_file:0#012 unevictable:0 dirty:15 writeback:0 unstable:4#012 slab_reclaimable:983293 slab_unreclaimable:633542#012 mapped:312546 shmem:478476 pagetables:211363 bounce:0#012 free:377304 free_pcp:30 free_cma:0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 1464 63924 63924
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32 free:250656kB min:636kB low:792kB high:952kB active_anon:1140668kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1697828kB managed:1499356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:61036kB slab_unreclaimable:33568kB kernel_stack:8336kB pagetables:672kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 62460 62460
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal free:44720kB min:27296kB low:34120kB high:40944kB active_anon:62696204kB inactive_anon:35312kB active_file:4kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65009664kB managed:63959824kB mlocked:0kB dirty:0kB writeback:0kB mapped:74176kB shmem:35420kB slab_reclaimable:29628kB slab_unreclaimable:258528kB kernel_stack:14096kB pagetables:155936kB unstable:0kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal free:322240kB min:28192kB low:35240kB high:42288kB active_anon:12697084kB inactive_anon:151340kB active_file:3648948kB inactive_file:47467320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:8kB writeback:0kB mapped:107200kB shmem:155888kB slab_reclaimable:251080kB slab_unreclaimable:271324kB kernel_stack:10096kB pagetables:134368kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal free:142964kB min:28192kB low:35240kB high:42288kB active_anon:7444452kB inactive_anon:70848kB active_file:3322484kB inactive_file:53133816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:4kB writeback:0kB mapped:97148kB shmem:85068kB slab_reclaimable:610960kB slab_unreclaimable:291144kB kernel_stack:14464kB pagetables:86812kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal free:171384kB min:28184kB low:35228kB high:42276kB active_anon:19556076kB inactive_anon:85964kB active_file:2715820kB inactive_file:42018844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67095552kB managed:66041820kB mlocked:0kB dirty:8kB writeback:0kB mapped:139764kB shmem:86052kB slab_reclaimable:314464kB slab_unreclaimable:267544kB kernel_stack:8160kB pagetables:167968kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal free:170340kB min:28192kB low:35240kB high:42288kB active_anon:2551888kB inactive_anon:102048kB active_file:5649480kB inactive_file:55836912kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:0kB writeback:0kB mapped:41688kB shmem:108872kB slab_reclaimable:703860kB slab_unreclaimable:281712kB kernel_stack:9168kB pagetables:38136kB unstable:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal free:165092kB min:28184kB low:35228kB high:42276kB active_anon:14285360kB inactive_anon:238276kB active_file:4591316kB inactive_file:44858964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66038068kB mlocked:0kB dirty:0kB writeback:0kB mapped:207824kB shmem:345372kB slab_reclaimable:411464kB slab_unreclaimable:301376kB kernel_stack:9440kB pagetables:104888kB unstable:0kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal free:100444kB min:28192kB low:35240kB high:42288kB active_anon:8336840kB inactive_anon:693288kB active_file:4288076kB inactive_file:46886244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:28kB writeback:0kB mapped:487028kB shmem:734192kB slab_reclaimable:699012kB slab_unreclaimable:478720kB kernel_stack:28064kB pagetables:93096kB unstable:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal free:125472kB min:28192kB low:35240kB high:42288kB active_anon:6474864kB inactive_anon:345660kB active_file:5644084kB inactive_file:51255132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66053972kB mlocked:0kB dirty:12kB writeback:0kB mapped:95356kB shmem:363040kB slab_reclaimable:851668kB slab_unreclaimable:350252kB kernel_stack:47360kB pagetables:63576kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32: 2916*4kB (UEM) 2910*8kB (UEM) 1668*16kB (UEM) 1571*32kB (UEM) 888*64kB (UEM) 286*128kB (UEM) 163*256kB (UEM) 3*512kB (U) 0*1024kB 1*2048kB (M) 0*4096kB = 250656kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal: 10977*4kB (UEM) 72*8kB (UEM) 23*16kB (UEM) 4*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal: 67757*4kB (UEM) 6255*8kB (UEM) 186*16kB (UEM) 7*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 324268kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal: 22013*4kB (UEM) 3611*8kB (UEM) 1354*16kB (UEM) 156*32kB (EM) 17*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144684kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal: 12783*4kB (UEM) 12273*8kB (UEM) 627*16kB (UEM) 315*32kB (UM) 39*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172820kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal: 9177*4kB (UEM) 3351*8kB (UEM) 2027*16kB (UEM) 2397*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172716kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal: 21289*4kB (UEM) 6875*8kB (UM) 1285*16kB (UEM) 152*32kB (UM) 15*64kB (M) 5*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 167436kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal: 17140*4kB (UEM) 3783*8kB (UEM) 177*16kB (UEM) 22*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 102360kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal: 10541*4kB (UEM) 2353*8kB (UEM) 1257*16kB (UEM) 1208*32kB (UEM) 119*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 127756kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 93307951 total pagecache pages
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages in swap cache
Apr 20 15:00:57 ldas-pcdev12 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 20 15:00:57 ldas-pcdev12 kernel: Free swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Total swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 134116519 pages RAM
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages HighMem/MovableOnly
Apr 20 15:00:57 ldas-pcdev12 kernel: 2160175 pages reserved
Apr 20 15:00:57 ldas-pcdev12 kernel: libceph: corrupt inc osdmap (12) epoch 409760 off 60 (ffffacad1785b058 of ffffacad1785b01c-ffffacad17923f02)
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 712 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1121 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 565 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1540 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 542 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 545 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 541 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3477 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1040 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1544 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 5457 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3503 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1033 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2314 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 733 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 551 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1525 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2033 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 122 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1000 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1055 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1048 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1041 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 548 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1529 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 20909 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 56 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 162 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 122 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 117 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 150 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 181 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 101 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 275 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 140 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 177 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 123 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 118 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 117 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 134 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 161 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 148 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 255 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 90 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 111 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 105 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 120 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 116 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 126 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 175 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 143 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 143 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 126 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 68 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 65 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 80 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 60 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 61 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 63 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 203 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 173 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 140 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 110 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 103 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 161 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 107 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 145 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2095 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1588 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 923 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 229 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3497 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1526 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2506 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2516 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1029 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1104 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 483 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2076 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 483 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3969 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1018 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 956 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 700 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 139 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 476 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1518 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3963 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 817 kernel messages
Apr 20 15:01:01 ldas-pcdev12 systemd: Started Session 1516 of user root.
Apr 20 15:01:03 ldas-pcdev12 kernel: kworker/226:0: page allocation failure: order:7, mode:0x4010
Apr 20 15:01:03 ldas-pcdev12 kernel: CPU: 226 PID: 1502115 Comm: kworker/226:0 Kdump: loaded Tainted: P OE ----------- 3.10.0-1160.62.1.el7.x86_64 #1
Apr 20 15:01:03 ldas-pcdev12 kernel: Hardware name: Dell Inc. PowerEdge XE8545/099K88, BIOS 2.6.6 01/13/2022
Apr 20 15:01:03 ldas-pcdev12 kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: Call Trace:
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa3865a9>] dump_stack+0x19/0x1b
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9dc4bd0>] warn_alloc_failed+0x110/0x180
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9dc976f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e193a8>] alloc_pages_current+0x98/0x110
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9de5fc8>] kmalloc_order+0x18/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e24d76>] kmalloc_order_trace+0x26/0xa0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e28d01>] __kmalloc+0x211/0x230
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0904e47>] osdmap_set_crush.isra.16+0x47/0xc0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0908046>] osdmap_apply_incremental+0x216/0x960 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08fdba3>] handle_one_map+0x83/0x250 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0902342>] ceph_osdc_handle_map+0x232/0x8c0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0902cd1>] dispatch+0x301/0xca0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa23955a>] ? kernel_recvmsg+0x3a/0x50
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08f0ff4>] try_read+0x544/0x1300 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9c7dfe9>] ? switch_mm_irqs_off+0x109/0x290
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9c7dff5>] ? switch_mm_irqs_off+0x115/0x290
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce497c>] ? update_curr+0x14c/0x1e0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce10ae>] ? account_entity_dequeue+0xae/0xd0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce4e6c>] ? dequeue_entity+0x11c/0x5c0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce2429>] ? pick_next_entity+0xa9/0x190
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08f1fb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbdfdf>] process_one_work+0x17f/0x440
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbf0f6>] worker_thread+0x126/0x3c0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5fb1>] kthread+0xd1/0xe0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa399de4>] ret_from_fork_nospec_begin+0xe/0x21
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: Mem-Info:
Apr 20 15:01:03 ldas-pcdev12 kernel: active_anon:33814505 inactive_anon:432738 isolated_anon:1#012 active_file:7466785 inactive_file:85366143 isolated_file:0#012 unevictable:0 dirty:58 writeback:0 unstable:1#012 slab_reclaimable:985209 slab_unreclaimable:633519#012 mapped:315533 shmem:480536 pagetables:211369 bounce:0#012 free:350586 free_pcp:72 free_cma:0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 1464 63924 63924
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA32 free:250656kB min:636kB low:792kB high:952kB active_anon:1140668kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1697828kB managed:1499356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:61036kB slab_unreclaimable:33568kB kernel_stack:8336kB pagetables:672kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 62460 62460
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 Normal free:34092kB min:27296kB low:34120kB high:40944kB active_anon:62707528kB inactive_anon:35312kB active_file:4kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65009664kB managed:63959824kB mlocked:0kB dirty:0kB writeback:0kB mapped:74176kB shmem:35420kB slab_reclaimable:29628kB slab_unreclaimable:258528kB kernel_stack:14096kB pagetables:155948kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 Normal free:124032kB min:28192kB low:35240kB high:42288kB active_anon:12892684kB inactive_anon:149352kB active_file:3650184kB inactive_file:47467708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:12kB writeback:0kB mapped:107440kB shmem:153900kB slab_reclaimable:251080kB slab_unreclaimable:271324kB kernel_stack:10096kB pagetables:133972kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 Normal free:144320kB min:28192kB low:35240kB high:42288kB active_anon:7444616kB inactive_anon:69000kB active_file:3322488kB inactive_file:53133808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:4kB writeback:0kB mapped:98260kB shmem:83244kB slab_reclaimable:611056kB slab_unreclaimable:291144kB kernel_stack:14480kB pagetables:86680kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 Normal free:180268kB min:28184kB low:35228kB high:42276kB active_anon:19546592kB inactive_anon:87240kB active_file:2715916kB inactive_file:42018840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67095552kB managed:66041820kB mlocked:0kB dirty:132kB writeback:0kB mapped:140352kB shmem:87348kB slab_reclaimable:314464kB slab_unreclaimable:267532kB kernel_stack:8160kB pagetables:167956kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 Normal free:159832kB min:28192kB low:35240kB high:42288kB active_anon:2554960kB inactive_anon:103228kB active_file:5652436kB inactive_file:55840020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:12kB writeback:0kB mapped:42908kB shmem:110052kB slab_reclaimable:703804kB slab_unreclaimable:281524kB kernel_stack:9168kB pagetables:38092kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 Normal free:213796kB min:28184kB low:35228kB high:42276kB active_anon:14234524kB inactive_anon:238344kB active_file:4593372kB inactive_file:44860844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66038068kB mlocked:0kB dirty:0kB writeback:0kB mapped:209136kB shmem:345440kB slab_reclaimable:411592kB slab_unreclaimable:301376kB kernel_stack:9456kB pagetables:105220kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 Normal free:143236kB min:28192kB low:35240kB high:42288kB active_anon:8281604kB inactive_anon:702560kB active_file:4288124kB inactive_file:46886792kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:56kB writeback:0kB mapped:492100kB shmem:743444kB slab_reclaimable:699028kB slab_unreclaimable:478848kB kernel_stack:28128kB pagetables:93372kB unstable:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 Normal free:136084kB min:28192kB low:35240kB high:42288kB active_anon:6454900kB inactive_anon:345916kB active_file:5644616kB inactive_file:51256548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66053972kB mlocked:0kB dirty:16kB writeback:0kB mapped:97760kB shmem:363296kB slab_reclaimable:859148kB slab_unreclaimable:350232kB kernel_stack:47360kB pagetables:63564kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA32: 2916*4kB (UEM) 2910*8kB (UEM) 1668*16kB (UEM) 1571*32kB (UEM) 888*64kB (UEM) 286*128kB (UEM) 163*256kB (UEM) 3*512kB (U) 0*1024kB 1*2048kB (M) 0*4096kB = 250656kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 Normal: 8534*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34136kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 Normal: 17895*4kB (UEM) 6272*8kB (UEM) 236*16kB (UEM) 7*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125756kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 Normal: 22098*4kB (UEM) 3703*8kB (UEM) 1346*16kB (UM) 155*32kB (UM) 16*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145536kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 Normal: 14589*4kB (UEM) 12423*8kB (UEM) 631*16kB (UM) 317*32kB (UM) 42*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181564kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 Normal: 6226*4kB (UEM) 3287*8kB (UEM) 1905*16kB (UEM) 2500*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 161744kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 Normal: 29959*4kB (UEM) 8416*8kB (UM) 1363*16kB (UEM) 153*32kB (UM) 17*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 215852kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 Normal: 13032*4kB (UEM) 10021*8kB (UEM) 684*16kB (UM) 55*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145000kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 Normal: 7130*4kB (UEM) 2210*8kB (UEM) 3046*16kB (UEM) 1119*32kB (UEM) 112*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 137912kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: 93313554 total pagecache pages
Apr 20 15:01:03 ldas-pcdev12 kernel: 0 pages in swap cache
Apr 20 15:01:03 ldas-pcdev12 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 20 15:01:03 ldas-pcdev12 kernel: Free swap = 0kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Total swap = 0kB
Apr 20 15:01:03 ldas-pcdev12 kernel: 134116519 pages RAM
Apr 20 15:01:03 ldas-pcdev12 kernel: 0 pages HighMem/MovableOnly
Apr 20 15:01:03 ldas-pcdev12 kernel: 2160175 pages reserved
Apr 20 15:01:03 ldas-pcdev12 kernel: libceph: corrupt inc osdmap (-12) epoch 409760 off 60 (ffffacad17925058 of ffffacad1792501c-ffffacad179edf02)
Apr 20 15:01:03 ldas-pcdev12 journal: Missed 318 kernel messages
Apr 20 15:01:03 ldas-pcdev12 journal: Missed 461 kernel messages

Related issues 1 (0 open — 1 closed)

Related to Linux kernel client - Bug #40481: osdmap->osd_addr allocation is susceptible to memory fragmentationResolvedIlya Dryomov06/21/2019

Actions
Actions
Copy link
 #1

Updated by Venky Shankar about 2 years ago

  • Project changed from CephFS to Linux kernel client
  • Assignee set to Ilya Dryomov
Actions
Copy link
 #2

Updated by Ilya Dryomov about 2 years ago

  • Category set to libceph
  • Status changed from New to Need More Info

Dan Moraru wrote:

A Scientific Linux 7.9 system running the latest kernel (3.10.0-1160.62.1.el7.x86_64) logged a "corrupt inc osdmap" message and became wedged after reading data from a Ceph 15.2.15 cluster at a few GBytes/s for ~18 hours.

Hi Dan,

This is certainly not related to the amount of data read. The kernel client received a CRUSH map update which couldn't be processed because it failed to allocate a sufficient chunk of physically contiguous memory.

It looks like this kernel is missing the fixes for https://tracker.ceph.com/issues/40481 which removed the "physically contiguous" requirement there. With those fixes, the allocation would have succeeded.

An initial umount was hung un-killable and a subsequent umount -f no longer thinks the filesystem is mounted. However, even though there is no entry in /etc/mtab the libceph and ceph kernel modules are busy and a subsequent attempt to mount hangs (but is killable),

[root@ldas-pcdev12 ~]# umount /ceph/mirror
^C^C^C^C

[root@ldas-pcdev12 ~]# umount -f /ceph/mirror
umount: /ceph/mirror: not mounted

There was plenty of cached memory (>300GB) at the time that cephfs hung.

Additional "libceph: osdc handle_map corrupt msg" and "libceph: corrupt inc osdmap" messages were logged to console.

What went wrong and how would one recover without rebooting?

There is no good way to recover from this. Rebooting is the best option since you don't want a wedged mount with a partially updated map hanging around.

...
Apr 20 15:00:57 ldas-pcdev12 kernel: kworker/103:4: page allocation failure: order:7, mode:0x4010
Apr 20 15:00:57 ldas-pcdev12 kernel: CPU: 103 PID: 1502626 Comm: kworker/103:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Apr 20 15:00:57 ldas-pcdev12 kernel: Hardware name: Dell Inc. PowerEdge XE8545/099K88, BIOS 2.6.6 01/13/2022
Apr 20 15:00:57 ldas-pcdev12 kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: Call Trace:
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa3865a9>] dump_stack+0x19/0x1b
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc4bd0>] warn_alloc_failed+0x110/0x180
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc976f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e193a8>] alloc_pages_current+0x98/0x110
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9de5fc8>] kmalloc_order+0x18/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e24d76>] kmalloc_order_trace+0x26/0xa0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28cb0>] ? __kmalloc+0x1c0/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28d01>] __kmalloc+0x211/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc09064b9>] ? crush_decode+0x879/0x15a0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0904e47>] osdmap_set_crush.isra.16+0x47/0xc0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0908046>] osdmap_apply_incremental+0x216/0x960 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08fdba3>] handle_one_map+0x83/0x250 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0902342>] ceph_osdc_handle_map+0x232/0x8c0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08efff3>] ? read_partial_message+0x1a3/0x900 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f6580>] dispatch+0x350/0x780 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f0ff4>] try_read+0x544/0x1300 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cd76cf>] ? ttwu_do_activate+0x6f/0x80
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce497c>] ? update_curr+0x14c/0x1e0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce10ae>] ? account_entity_dequeue+0xae/0xd0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce4e6c>] ? dequeue_entity+0x11c/0x5c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f1fb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbdfdf>] process_one_work+0x17f/0x440
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbf0f6>] worker_thread+0x126/0x3c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5fb1>] kthread+0xd1/0xe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa399de4>] ret_from_fork_nospec_begin+0xe/0x21
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: Mem-Info:
Apr 20 15:00:57 ldas-pcdev12 kernel: active_anon:33795845 inactive_anon:430684 isolated_anon:0#012 active_file:7465053 inactive_file:85364311 isolated_file:0#012 unevictable:0 dirty:15 writeback:0 unstable:4#012 slab_reclaimable:983293 slab_unreclaimable:633542#012 mapped:312546 shmem:478476 pagetables:211363 bounce:0#012 free:377304 free_pcp:30 free_cma:0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 1464 63924 63924
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32 free:250656kB min:636kB low:792kB high:952kB active_anon:1140668kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1697828kB managed:1499356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:61036kB slab_unreclaimable:33568kB kernel_stack:8336kB pagetables:672kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 62460 62460
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal free:44720kB min:27296kB low:34120kB high:40944kB active_anon:62696204kB inactive_anon:35312kB active_file:4kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65009664kB managed:63959824kB mlocked:0kB dirty:0kB writeback:0kB mapped:74176kB shmem:35420kB slab_reclaimable:29628kB slab_unreclaimable:258528kB kernel_stack:14096kB pagetables:155936kB unstable:0kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal free:322240kB min:28192kB low:35240kB high:42288kB active_anon:12697084kB inactive_anon:151340kB active_file:3648948kB inactive_file:47467320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:8kB writeback:0kB mapped:107200kB shmem:155888kB slab_reclaimable:251080kB slab_unreclaimable:271324kB kernel_stack:10096kB pagetables:134368kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal free:142964kB min:28192kB low:35240kB high:42288kB active_anon:7444452kB inactive_anon:70848kB active_file:3322484kB inactive_file:53133816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:4kB writeback:0kB mapped:97148kB shmem:85068kB slab_reclaimable:610960kB slab_unreclaimable:291144kB kernel_stack:14464kB pagetables:86812kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal free:171384kB min:28184kB low:35228kB high:42276kB active_anon:19556076kB inactive_anon:85964kB active_file:2715820kB inactive_file:42018844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67095552kB managed:66041820kB mlocked:0kB dirty:8kB writeback:0kB mapped:139764kB shmem:86052kB slab_reclaimable:314464kB slab_unreclaimable:267544kB kernel_stack:8160kB pagetables:167968kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal free:170340kB min:28192kB low:35240kB high:42288kB active_anon:2551888kB inactive_anon:102048kB active_file:5649480kB inactive_file:55836912kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:0kB writeback:0kB mapped:41688kB shmem:108872kB slab_reclaimable:703860kB slab_unreclaimable:281712kB kernel_stack:9168kB pagetables:38136kB unstable:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal free:165092kB min:28184kB low:35228kB high:42276kB active_anon:14285360kB inactive_anon:238276kB active_file:4591316kB inactive_file:44858964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66038068kB mlocked:0kB dirty:0kB writeback:0kB mapped:207824kB shmem:345372kB slab_reclaimable:411464kB slab_unreclaimable:301376kB kernel_stack:9440kB pagetables:104888kB unstable:0kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal free:100444kB min:28192kB low:35240kB high:42288kB active_anon:8336840kB inactive_anon:693288kB active_file:4288076kB inactive_file:46886244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:28kB writeback:0kB mapped:487028kB shmem:734192kB slab_reclaimable:699012kB slab_unreclaimable:478720kB kernel_stack:28064kB pagetables:93096kB unstable:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal free:125472kB min:28192kB low:35240kB high:42288kB active_anon:6474864kB inactive_anon:345660kB active_file:5644084kB inactive_file:51255132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66053972kB mlocked:0kB dirty:12kB writeback:0kB mapped:95356kB shmem:363040kB slab_reclaimable:851668kB slab_unreclaimable:350252kB kernel_stack:47360kB pagetables:63576kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32: 2916*4kB (UEM) 2910*8kB (UEM) 1668*16kB (UEM) 1571*32kB (UEM) 888*64kB (UEM) 286*128kB (UEM) 163*256kB (UEM) 3*512kB (U) 0*1024kB 1*2048kB (M) 0*4096kB = 250656kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal: 10977*4kB (UEM) 72*8kB (UEM) 23*16kB (UEM) 4*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal: 67757*4kB (UEM) 6255*8kB (UEM) 186*16kB (UEM) 7*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 324268kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal: 22013*4kB (UEM) 3611*8kB (UEM) 1354*16kB (UEM) 156*32kB (EM) 17*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144684kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal: 12783*4kB (UEM) 12273*8kB (UEM) 627*16kB (UEM) 315*32kB (UM) 39*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172820kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal: 9177*4kB (UEM) 3351*8kB (UEM) 2027*16kB (UEM) 2397*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172716kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal: 21289*4kB (UEM) 6875*8kB (UM) 1285*16kB (UEM) 152*32kB (UM) 15*64kB (M) 5*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 167436kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal: 17140*4kB (UEM) 3783*8kB (UEM) 177*16kB (UEM) 22*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 102360kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal: 10541*4kB (UEM) 2353*8kB (UEM) 1257*16kB (UEM) 1208*32kB (UEM) 119*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 127756kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 93307951 total pagecache pages
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages in swap cache
Apr 20 15:00:57 ldas-pcdev12 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 20 15:00:57 ldas-pcdev12 kernel: Free swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Total swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 134116519 pages RAM
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages HighMem/MovableOnly
Apr 20 15:00:57 ldas-pcdev12 kernel: 2160175 pages reserved

Any idea why the CRUSH map got updated? Was there a maintenance operation being performed on the cluster at that time?

How many OSDs are there in the cluster?

If you attach "ceph osd tree" output and your CRUSH map ("ceph osd getcrushmap -o /path/to/crushmap.bin"), I can double check the CRUSH workspace size calculation to make sure that it requested the correct amount of memory.

Actions
Copy link
 #3

Updated by Ilya Dryomov about 2 years ago

  • Related to Bug #40481: osdmap->osd_addr allocation is susceptible to memory fragmentation added
Actions
Copy link

Also available in: Atom PDF