Ceph » Linux kernel client

A Scientific Linux 7.9 system running the latest kernel (3.10.0-1160.62.1.el7.x86_64) logged a "corrupt inc osdmap" message and became wedged after reading data from a Ceph 15.2.15 cluster at a few GBytes/s for ~18 hours. An initial umount was hung un-killable and a subsequent umount -f no longer thinks the filesystem is mounted. However, even though there is no entry in /etc/mtab the libceph and ceph kernel modules are busy and a subsequent attempt to mount hangs (but is killable),

[root@ldas-pcdev12 ~]# umount /ceph/mirror
^C^C^C^C

[root@ldas-pcdev12 ~]# umount -f /ceph/mirror
umount: /ceph/mirror: not mounted

There was plenty of cached memory (>300GB) at the time that cephfs hung.

Additional "libceph: osdc handle_map corrupt msg" and "libceph: corrupt inc osdmap" messages were logged to console.

What went wrong and how would one recover without rebooting?

[root@ldas-pcdev12 ~]# less /var/log/messages
...
Apr 20 15:00:57 ldas-pcdev12 kernel: kworker/103:4: page allocation failure: order:7, mode:0x4010
Apr 20 15:00:57 ldas-pcdev12 kernel: CPU: 103 PID: 1502626 Comm: kworker/103:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Apr 20 15:00:57 ldas-pcdev12 kernel: Hardware name: Dell Inc. PowerEdge XE8545/099K88, BIOS 2.6.6 01/13/2022
Apr 20 15:00:57 ldas-pcdev12 kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: Call Trace:
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa3865a9>] dump_stack+0x19/0x1b
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc4bd0>] warn_alloc_failed+0x110/0x180
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9dc976f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e193a8>] alloc_pages_current+0x98/0x110
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9de5fc8>] kmalloc_order+0x18/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e24d76>] kmalloc_order_trace+0x26/0xa0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28cb0>] ? __kmalloc+0x1c0/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9e28d01>] __kmalloc+0x211/0x230
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc09064b9>] ? crush_decode+0x879/0x15a0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0904e47>] osdmap_set_crush.isra.16+0x47/0xc0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0908046>] osdmap_apply_incremental+0x216/0x960 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08fdba3>] handle_one_map+0x83/0x250 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc0902342>] ceph_osdc_handle_map+0x232/0x8c0 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08efff3>] ? read_partial_message+0x1a3/0x900 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f6580>] dispatch+0x350/0x780 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f0ff4>] try_read+0x544/0x1300 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cd76cf>] ? ttwu_do_activate+0x6f/0x80
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce497c>] ? update_curr+0x14c/0x1e0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce10ae>] ? account_entity_dequeue+0xae/0xd0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9ce4e6c>] ? dequeue_entity+0x11c/0x5c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffc08f1fb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbdfdf>] process_one_work+0x17f/0x440
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbf0f6>] worker_thread+0x126/0x3c0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cbefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5fb1>] kthread+0xd1/0xe0
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffaa399de4>] ret_from_fork_nospec_begin+0xe/0x21
Apr 20 15:00:57 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:00:57 ldas-pcdev12 kernel: Mem-Info:
Apr 20 15:00:57 ldas-pcdev12 kernel: active_anon:33795845 inactive_anon:430684 isolated_anon:0#012 active_file:7465053 inactive_file:85364311 isolated_file:0#012 unevictable:0 dirty:15 writeback:0 unstable:4#012 slab_reclaimable:983293 slab_unreclaimable:633542#012 mapped:312546 shmem:478476 pagetables:211363 bounce:0#012 free:377304 free_pcp:30 free_cma:0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 1464 63924 63924
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32 free:250656kB min:636kB low:792kB high:952kB active_anon:1140668kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1697828kB managed:1499356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:61036kB slab_unreclaimable:33568kB kernel_stack:8336kB pagetables:672kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 62460 62460
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal free:44720kB min:27296kB low:34120kB high:40944kB active_anon:62696204kB inactive_anon:35312kB active_file:4kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65009664kB managed:63959824kB mlocked:0kB dirty:0kB writeback:0kB mapped:74176kB shmem:35420kB slab_reclaimable:29628kB slab_unreclaimable:258528kB kernel_stack:14096kB pagetables:155936kB unstable:0kB bounce:0kB free_pcp:44kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal free:322240kB min:28192kB low:35240kB high:42288kB active_anon:12697084kB inactive_anon:151340kB active_file:3648948kB inactive_file:47467320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:8kB writeback:0kB mapped:107200kB shmem:155888kB slab_reclaimable:251080kB slab_unreclaimable:271324kB kernel_stack:10096kB pagetables:134368kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal free:142964kB min:28192kB low:35240kB high:42288kB active_anon:7444452kB inactive_anon:70848kB active_file:3322484kB inactive_file:53133816kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:4kB writeback:0kB mapped:97148kB shmem:85068kB slab_reclaimable:610960kB slab_unreclaimable:291144kB kernel_stack:14464kB pagetables:86812kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal free:171384kB min:28184kB low:35228kB high:42276kB active_anon:19556076kB inactive_anon:85964kB active_file:2715820kB inactive_file:42018844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67095552kB managed:66041820kB mlocked:0kB dirty:8kB writeback:0kB mapped:139764kB shmem:86052kB slab_reclaimable:314464kB slab_unreclaimable:267544kB kernel_stack:8160kB pagetables:167968kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal free:170340kB min:28192kB low:35240kB high:42288kB active_anon:2551888kB inactive_anon:102048kB active_file:5649480kB inactive_file:55836912kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:0kB writeback:0kB mapped:41688kB shmem:108872kB slab_reclaimable:703860kB slab_unreclaimable:281712kB kernel_stack:9168kB pagetables:38136kB unstable:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal free:165092kB min:28184kB low:35228kB high:42276kB active_anon:14285360kB inactive_anon:238276kB active_file:4591316kB inactive_file:44858964kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66038068kB mlocked:0kB dirty:0kB writeback:0kB mapped:207824kB shmem:345372kB slab_reclaimable:411464kB slab_unreclaimable:301376kB kernel_stack:9440kB pagetables:104888kB unstable:0kB bounce:0kB free_pcp:116kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal free:100444kB min:28192kB low:35240kB high:42288kB active_anon:8336840kB inactive_anon:693288kB active_file:4288076kB inactive_file:46886244kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:28kB writeback:0kB mapped:487028kB shmem:734192kB slab_reclaimable:699012kB slab_unreclaimable:478720kB kernel_stack:28064kB pagetables:93096kB unstable:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal free:125472kB min:28192kB low:35240kB high:42288kB active_anon:6474864kB inactive_anon:345660kB active_file:5644084kB inactive_file:51255132kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66053972kB mlocked:0kB dirty:12kB writeback:0kB mapped:95356kB shmem:363040kB slab_reclaimable:851668kB slab_unreclaimable:350252kB kernel_stack:47360kB pagetables:63576kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:00:57 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 DMA32: 2916*4kB (UEM) 2910*8kB (UEM) 1668*16kB (UEM) 1571*32kB (UEM) 888*64kB (UEM) 286*128kB (UEM) 163*256kB (UEM) 3*512kB (U) 0*1024kB 1*2048kB (M) 0*4096kB = 250656kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 Normal: 10977*4kB (UEM) 72*8kB (UEM) 23*16kB (UEM) 4*32kB (UEM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44980kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 Normal: 67757*4kB (UEM) 6255*8kB (UEM) 186*16kB (UEM) 7*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 324268kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 Normal: 22013*4kB (UEM) 3611*8kB (UEM) 1354*16kB (UEM) 156*32kB (EM) 17*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 144684kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 Normal: 12783*4kB (UEM) 12273*8kB (UEM) 627*16kB (UEM) 315*32kB (UM) 39*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172820kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 Normal: 9177*4kB (UEM) 3351*8kB (UEM) 2027*16kB (UEM) 2397*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 172716kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 Normal: 21289*4kB (UEM) 6875*8kB (UM) 1285*16kB (UEM) 152*32kB (UM) 15*64kB (M) 5*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 167436kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 Normal: 17140*4kB (UEM) 3783*8kB (UEM) 177*16kB (UEM) 22*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 102360kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 Normal: 10541*4kB (UEM) 2353*8kB (UEM) 1257*16kB (UEM) 1208*32kB (UEM) 119*64kB (UEM) 3*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 127756kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 93307951 total pagecache pages
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages in swap cache
Apr 20 15:00:57 ldas-pcdev12 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 20 15:00:57 ldas-pcdev12 kernel: Free swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: Total swap = 0kB
Apr 20 15:00:57 ldas-pcdev12 kernel: 134116519 pages RAM
Apr 20 15:00:57 ldas-pcdev12 kernel: 0 pages HighMem/MovableOnly
Apr 20 15:00:57 ldas-pcdev12 kernel: 2160175 pages reserved
Apr 20 15:00:57 ldas-pcdev12 kernel: libceph: corrupt inc osdmap (12) epoch 409760 off 60 (ffffacad1785b058 of ffffacad1785b01c-ffffacad17923f02)
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 712 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1121 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 565 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1540 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 542 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 545 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 541 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3477 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1040 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1544 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 5457 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3503 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1033 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2314 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 733 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 551 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1525 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2033 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 122 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1000 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1055 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1048 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1041 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 548 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1529 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 20909 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 56 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 162 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 122 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 117 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 150 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 181 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 101 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 275 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 140 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 177 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 123 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 118 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 117 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 134 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 161 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 148 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 255 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 90 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 174 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 111 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 105 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 120 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 116 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 126 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 175 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 143 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 143 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 126 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 68 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 65 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 80 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 60 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 61 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 63 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 203 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 173 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 140 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 110 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 103 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 161 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 107 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 145 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2095 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1588 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 923 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 229 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3497 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1526 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2506 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2516 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1029 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1104 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 483 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 2076 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 483 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3969 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1018 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 956 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 700 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 139 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 476 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 1518 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 3963 kernel messages
Apr 20 15:00:57 ldas-pcdev12 journal: Missed 817 kernel messages
Apr 20 15:01:01 ldas-pcdev12 systemd: Started Session 1516 of user root.
Apr 20 15:01:03 ldas-pcdev12 kernel: kworker/226:0: page allocation failure: order:7, mode:0x4010
Apr 20 15:01:03 ldas-pcdev12 kernel: CPU: 226 PID: 1502115 Comm: kworker/226:0 Kdump: loaded Tainted: P OE ----------- 3.10.0-1160.62.1.el7.x86_64 #1
Apr 20 15:01:03 ldas-pcdev12 kernel: Hardware name: Dell Inc. PowerEdge XE8545/099K88, BIOS 2.6.6 01/13/2022
Apr 20 15:01:03 ldas-pcdev12 kernel: Workqueue: ceph-msgr ceph_con_workfn [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: Call Trace:
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa3865a9>] dump_stack+0x19/0x1b
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9dc4bd0>] warn_alloc_failed+0x110/0x180
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9dc976f>] __alloc_pages_nodemask+0x9df/0xbe0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e193a8>] alloc_pages_current+0x98/0x110
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9de5fc8>] kmalloc_order+0x18/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e24d76>] kmalloc_order_trace+0x26/0xa0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9e28d01>] __kmalloc+0x211/0x230
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0904e47>] osdmap_set_crush.isra.16+0x47/0xc0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0908046>] osdmap_apply_incremental+0x216/0x960 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08fdba3>] handle_one_map+0x83/0x250 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0902342>] ceph_osdc_handle_map+0x232/0x8c0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc0902cd1>] dispatch+0x301/0xca0 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa23955a>] ? kernel_recvmsg+0x3a/0x50
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08f0ff4>] try_read+0x544/0x1300 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9c7dfe9>] ? switch_mm_irqs_off+0x109/0x290
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9c7dff5>] ? switch_mm_irqs_off+0x115/0x290
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce497c>] ? update_curr+0x14c/0x1e0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce10ae>] ? account_entity_dequeue+0xae/0xd0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce4e6c>] ? dequeue_entity+0x11c/0x5c0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9ce2429>] ? pick_next_entity+0xa9/0x190
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffc08f1fb4>] ceph_con_workfn+0xe4/0x1530 [libceph]
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbdfdf>] process_one_work+0x17f/0x440
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbf0f6>] worker_thread+0x126/0x3c0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cbefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5fb1>] kthread+0xd1/0xe0
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffaa399de4>] ret_from_fork_nospec_begin+0xe/0x21
Apr 20 15:01:03 ldas-pcdev12 kernel: [<ffffffffa9cc5ee0>] ? insert_kthread_work+0x40/0x40
Apr 20 15:01:03 ldas-pcdev12 kernel: Mem-Info:
Apr 20 15:01:03 ldas-pcdev12 kernel: active_anon:33814505 inactive_anon:432738 isolated_anon:1#012 active_file:7466785 inactive_file:85366143 isolated_file:0#012 unevictable:0 dirty:58 writeback:0 unstable:1#012 slab_reclaimable:985209 slab_unreclaimable:633519#012 mapped:315533 shmem:480536 pagetables:211369 bounce:0#012 free:350586 free_pcp:72 free_cma:0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA free:15904kB min:4kB low:4kB high:4kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 1464 63924 63924
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA32 free:250656kB min:636kB low:792kB high:952kB active_anon:1140668kB inactive_anon:0kB active_file:0kB inactive_file:4kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1697828kB managed:1499356kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:61036kB slab_unreclaimable:33568kB kernel_stack:8336kB pagetables:672kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 62460 62460
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 Normal free:34092kB min:27296kB low:34120kB high:40944kB active_anon:62707528kB inactive_anon:35312kB active_file:4kB inactive_file:8kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:65009664kB managed:63959824kB mlocked:0kB dirty:0kB writeback:0kB mapped:74176kB shmem:35420kB slab_reclaimable:29628kB slab_unreclaimable:258528kB kernel_stack:14096kB pagetables:155948kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 Normal free:124032kB min:28192kB low:35240kB high:42288kB active_anon:12892684kB inactive_anon:149352kB active_file:3650184kB inactive_file:47467708kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:12kB writeback:0kB mapped:107440kB shmem:153900kB slab_reclaimable:251080kB slab_unreclaimable:271324kB kernel_stack:10096kB pagetables:133972kB unstable:0kB bounce:0kB free_pcp:112kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 Normal free:144320kB min:28192kB low:35240kB high:42288kB active_anon:7444616kB inactive_anon:69000kB active_file:3322488kB inactive_file:53133808kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:4kB writeback:0kB mapped:98260kB shmem:83244kB slab_reclaimable:611056kB slab_unreclaimable:291144kB kernel_stack:14480kB pagetables:86680kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 Normal free:180268kB min:28184kB low:35228kB high:42276kB active_anon:19546592kB inactive_anon:87240kB active_file:2715916kB inactive_file:42018840kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67095552kB managed:66041820kB mlocked:0kB dirty:132kB writeback:0kB mapped:140352kB shmem:87348kB slab_reclaimable:314464kB slab_unreclaimable:267532kB kernel_stack:8160kB pagetables:167956kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 Normal free:159832kB min:28192kB low:35240kB high:42288kB active_anon:2554960kB inactive_anon:103228kB active_file:5652436kB inactive_file:55840020kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:12kB writeback:0kB mapped:42908kB shmem:110052kB slab_reclaimable:703804kB slab_unreclaimable:281524kB kernel_stack:9168kB pagetables:38092kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 Normal free:213796kB min:28184kB low:35228kB high:42276kB active_anon:14234524kB inactive_anon:238344kB active_file:4593372kB inactive_file:44860844kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66038068kB mlocked:0kB dirty:0kB writeback:0kB mapped:209136kB shmem:345440kB slab_reclaimable:411592kB slab_unreclaimable:301376kB kernel_stack:9456kB pagetables:105220kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 Normal free:143236kB min:28192kB low:35240kB high:42288kB active_anon:8281604kB inactive_anon:702560kB active_file:4288124kB inactive_file:46886792kB unevictable:0kB isolated(anon):4kB isolated(file):0kB present:67107840kB managed:66054108kB mlocked:0kB dirty:56kB writeback:0kB mapped:492100kB shmem:743444kB slab_reclaimable:699028kB slab_unreclaimable:478848kB kernel_stack:28128kB pagetables:93372kB unstable:4kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 Normal free:136084kB min:28192kB low:35240kB high:42288kB active_anon:6454900kB inactive_anon:345916kB active_file:5644616kB inactive_file:51256548kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:67107840kB managed:66053972kB mlocked:0kB dirty:16kB writeback:0kB mapped:97760kB shmem:363296kB slab_reclaimable:859148kB slab_unreclaimable:350232kB kernel_stack:47360kB pagetables:63564kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr 20 15:01:03 ldas-pcdev12 kernel: lowmem_reserve[]: 0 0 0 0
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA: 2*4kB (U) 1*8kB (U) 1*16kB (U) 2*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 DMA32: 2916*4kB (UEM) 2910*8kB (UEM) 1668*16kB (UEM) 1571*32kB (UEM) 888*64kB (UEM) 286*128kB (UEM) 163*256kB (UEM) 3*512kB (U) 0*1024kB 1*2048kB (M) 0*4096kB = 250656kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 Normal: 8534*4kB (U) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 34136kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 Normal: 17895*4kB (UEM) 6272*8kB (UEM) 236*16kB (UEM) 7*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 125756kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 Normal: 22098*4kB (UEM) 3703*8kB (UEM) 1346*16kB (UM) 155*32kB (UM) 16*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145536kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 Normal: 14589*4kB (UEM) 12423*8kB (UEM) 631*16kB (UM) 317*32kB (UM) 42*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 181564kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 Normal: 6226*4kB (UEM) 3287*8kB (UEM) 1905*16kB (UEM) 2500*32kB (UEM) 1*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 161744kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 Normal: 29959*4kB (UEM) 8416*8kB (UM) 1363*16kB (UEM) 153*32kB (UM) 17*64kB (M) 7*128kB (M) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 215852kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 Normal: 13032*4kB (UEM) 10021*8kB (UEM) 684*16kB (UM) 55*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 145000kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 Normal: 7130*4kB (UEM) 2210*8kB (UEM) 3046*16kB (UEM) 1119*32kB (UEM) 112*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 137912kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 4 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 5 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 6 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Node 7 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr 20 15:01:03 ldas-pcdev12 kernel: 93313554 total pagecache pages
Apr 20 15:01:03 ldas-pcdev12 kernel: 0 pages in swap cache
Apr 20 15:01:03 ldas-pcdev12 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 20 15:01:03 ldas-pcdev12 kernel: Free swap = 0kB
Apr 20 15:01:03 ldas-pcdev12 kernel: Total swap = 0kB
Apr 20 15:01:03 ldas-pcdev12 kernel: 134116519 pages RAM
Apr 20 15:01:03 ldas-pcdev12 kernel: 0 pages HighMem/MovableOnly
Apr 20 15:01:03 ldas-pcdev12 kernel: 2160175 pages reserved
Apr 20 15:01:03 ldas-pcdev12 kernel: libceph: corrupt inc osdmap (-12) epoch 409760 off 60 (ffffacad17925058 of ffffacad1792501c-ffffacad179edf02)
Apr 20 15:01:03 ldas-pcdev12 journal: Missed 318 kernel messages
Apr 20 15:01:03 ldas-pcdev12 journal: Missed 461 kernel messages