Project

General

Profile

Actions
Copy link

Bug #8000

closed

SLAB: Unable to allocate memory on node 0

Added by Dmitry Smirnov about 10 years ago. Updated over 8 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ilya Dryomov
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I'm getting the following kernel errors with ext4 on rbd:

Apr  6 11:43:38 debworm kernel: [103549.359234] jbd2/rbd1-8: page allocation failure: order:1, mode:0x204020
Apr  6 11:43:38 debworm kernel: [103549.359239] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P           O 3.13-1-amd64 #1 Debian 3.13.7-1
Apr  6 11:43:38 debworm kernel: [103549.359240] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009
Apr  6 11:43:38 debworm kernel: [103549.359242]  ffff880096bf78c8 ffffffff814a1327 0000000000204020 ffffffff8111b503
Apr  6 11:43:38 debworm kernel: [103549.359245]  0000000000000000 0000000000000000 ffff88022fff6b00 000000012fff5000
Apr  6 11:43:38 debworm kernel: [103549.359247]  ffff88022fff8fb0 0000000200000046 ffff88022fff6b08 0000000000000001
Apr  6 11:43:38 debworm kernel: [103549.359250] Call Trace:
Apr  6 11:43:38 debworm kernel: [103549.359257]  [<ffffffff814a1327>] ? dump_stack+0x41/0x51
Apr  6 11:43:38 debworm kernel: [103549.359260]  [<ffffffff8111b503>] ? warn_alloc_failed+0xe3/0x130
Apr  6 11:43:38 debworm kernel: [103549.359262]  [<ffffffff8111f5ea>] ? __alloc_pages_nodemask+0x81a/0x9d0
Apr  6 11:43:38 debworm kernel: [103549.359265]  [<ffffffff811607ef>] ? kmem_getpages+0x5f/0x140
Apr  6 11:43:38 debworm kernel: [103549.359268]  [<ffffffff81162007>] ? fallback_alloc+0x1c7/0x200
Apr  6 11:43:38 debworm kernel: [103549.359270]  [<ffffffff811625ad>] ? kmem_cache_alloc+0x11d/0x430
Apr  6 11:43:38 debworm kernel: [103549.359281]  [<ffffffffa108cc0e>] ? ceph_osdc_alloc_request+0x4e/0x2d0 [libceph]
Apr  6 11:43:38 debworm kernel: [103549.359284]  [<ffffffffa11571de>] ? rbd_osd_req_create+0x5e/0x1c0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359287]  [<ffffffffa1159803>] ? rbd_img_request_fill+0x273/0x8b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359290]  [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.359293]  [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40
Apr  6 11:43:38 debworm kernel: [103549.359295]  [<ffffffff81248690>] ? queue_unplugged+0x20/0x90
Apr  6 11:43:38 debworm kernel: [103549.359298]  [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240
Apr  6 11:43:38 debworm kernel: [103549.359300]  [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30
Apr  6 11:43:38 debworm kernel: [103549.359318]  [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359321]  [<ffffffff81019895>] ? sched_clock+0x5/0x10
Apr  6 11:43:38 debworm kernel: [103549.359324]  [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80
Apr  6 11:43:38 debworm kernel: [103549.359330]  [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359333]  [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0
Apr  6 11:43:38 debworm kernel: [103549.359338]  [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.359341]  [<ffffffff8107bb41>] ? kthread+0xc1/0xe0
Apr  6 11:43:38 debworm kernel: [103549.359343]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.359346]  [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0
Apr  6 11:43:38 debworm kernel: [103549.359348]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.359349] Mem-Info:
Apr  6 11:43:38 debworm kernel: [103549.359350] Node 0 DMA per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359352] CPU    0: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359353] CPU    1: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359355] CPU    2: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359356] CPU    3: hi:    0, btch:   1 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359357] Node 0 DMA32 per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359359] CPU    0: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359360] CPU    1: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359361] CPU    2: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359363] CPU    3: hi:  186, btch:  31 usd:   0
Apr  6 11:43:38 debworm kernel: [103549.359364] Node 0 Normal per-cpu:
Apr  6 11:43:38 debworm kernel: [103549.359365] CPU    0: hi:  186, btch:  31 usd:   8
Apr  6 11:43:38 debworm kernel: [103549.359367] CPU    1: hi:  186, btch:  31 usd: 213
Apr  6 11:43:38 debworm kernel: [103549.359368] CPU    2: hi:  186, btch:  31 usd: 198
Apr  6 11:43:38 debworm kernel: [103549.359369] CPU    3: hi:  186, btch:  31 usd: 185
Apr  6 11:43:38 debworm kernel: [103549.359372] active_anon:869609 inactive_anon:216123 isolated_anon:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  active_file:380431 inactive_file:414833 isolated_file:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  unevictable:0 dirty:13334 writeback:0 unstable:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  free:96767 slab_reclaimable:25267 slab_unreclaimable:13377
Apr  6 11:43:38 debworm kernel: [103549.359372]  mapped:50681 shmem:177434 pagetables:14845 bounce:0
Apr  6 11:43:38 debworm kernel: [103549.359372]  free_cma:0
Apr  6 11:43:38 debworm kernel: [103549.359375] Node 0 DMA free:15904kB min:252kB low:312kB high:376kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15988kB managed:15904kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr  6 11:43:38 debworm kernel: [103549.359380] lowmem_reserve[]: 0 3236 7984 7984
Apr  6 11:43:38 debworm kernel: [103549.359382] Node 0 DMA32 free:135116kB min:53040kB low:66300kB high:79560kB active_anon:1271424kB inactive_anon:317936kB active_file:819608kB inactive_file:732796kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3390144kB managed:3316452kB mlocked:0kB dirty:392kB writeback:0kB mapped:98084kB shmem:287868kB slab_reclaimable:664kB slab_unreclaimable:100kB kernel_stack:2264kB pagetables:23540kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr  6 11:43:38 debworm kernel: [103549.359386] lowmem_reserve[]: 0 0 4748 4748
Apr  6 11:43:38 debworm kernel: [103549.359389] Node 0 Normal free:236048kB min:77772kB low:97212kB high:116656kB active_anon:2207012kB inactive_anon:546556kB active_file:702116kB inactive_file:926536kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:4980736kB managed:4862776kB mlocked:4kB dirty:52944kB writeback:0kB mapped:104640kB shmem:421868kB slab_reclaimable:100404kB slab_unreclaimable:53408kB kernel_stack:4048kB pagetables:35840kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Apr  6 11:43:38 debworm kernel: [103549.359393] lowmem_reserve[]: 0 0 0 0
Apr  6 11:43:38 debworm kernel: [103549.359395] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15904kB
Apr  6 11:43:38 debworm kernel: [103549.359405] Node 0 DMA32: 32135*4kB (UM) 310*8kB (UM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 135116kB
Apr  6 11:43:38 debworm kernel: [103549.359412] Node 0 Normal: 57959*4kB (UE) 6*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 235980kB
Apr  6 11:43:38 debworm kernel: [103549.359420] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Apr  6 11:43:38 debworm kernel: [103549.359421] 973008 total pagecache pages
Apr  6 11:43:38 debworm kernel: [103549.359423] 287 pages in swap cache
Apr  6 11:43:38 debworm kernel: [103549.359424] Swap cache stats: add 26110, delete 25823, find 332/417
Apr  6 11:43:38 debworm kernel: [103549.359425] Free swap  = 12474668kB
Apr  6 11:43:38 debworm kernel: [103549.359426] Total swap = 12574716kB
Apr  6 11:43:38 debworm kernel: [103549.359428] SLAB: Unable to allocate memory on node 0 (gfp=0x4020)
Apr  6 11:43:38 debworm kernel: [103549.359429]   cache: ceph_osd_request, object size: 864, order: 1
Apr  6 11:43:38 debworm kernel: [103549.359431]   node 0: slabs: 2/2, objs: 18/18, free: 0
Apr  6 11:43:38 debworm kernel: [103549.359457] ------------[ cut here ]------------
Apr  6 11:43:38 debworm kernel: [103549.359505] invalid opcode: 0000 [#1] SMP 
Apr  6 11:43:38 debworm kernel: [103549.359522] Modules linked in: nls_utf8 nls_cp437 usb_storage ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs rbd ip6table_filter ip6_tables iptable_filter ip_tables x_tables cpufreq_powersave snd_hrtimer cpufreq_stats cpufreq_conservative cpufreq_userspace pci_stub vboxpci(O) vboxnetadp(O) vboxnetflt(O) vboxdrv(O) binfmt_misc cbc ceph libceph rpcsec_gss_krb5 nfsv4 dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc bonding xts gf128mul fuse it87 hwmon_vid parport_pc ppdev lp parport loop dm_crypt dm_mod fglrx(PO) snd_hda_codec_hdmi snd_usb_audio snd_hda_codec_realtek snd_usbmidi_lib snd_hda_intel snd_hda_codec snd_hwdep snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev media iTCO_wdt iTCO_vendor_support snd_timer snd acpi_cpufreq pcspkr radeon soundcore ttm drm_kms_helper coretemp drm kvm_intel kvm psmouse serio_raw evdev i2c_i801 button processor i2c_algo_bit lpc_ich i2c_core mfd_core thermal_sys ext4 crc16 mbcache jbd2 btrfs xor raid6_pq crc32c libcrc32c raid1 md_mod sd_mod crct10dif_generic crc_t10dif crct10dif_common hid_generic usbhid hid sg sr_mod cdrom ata_generic firewire_ohci firewire_core crc_itu_t pata_it8213 ahci libahci libata scsi_mod ehci_pci uhci_hcd ehci_hcd r8169 mii usbcore usb_common
Apr  6 11:43:38 debworm kernel: [103549.360352] CPU: 1 PID: 5608 Comm: jbd2/rbd1-8 Tainted: P           O 3.13-1-amd64 #1 Debian 3.13.7-1
Apr  6 11:43:38 debworm kernel: [103549.360391] Hardware name: Gigabyte Technology Co., Ltd. EP45-DS4P/EP45-DS4P, BIOS F9A 04/21/2009
Apr  6 11:43:38 debworm kernel: [103549.360429] task: ffff8801d3133800 ti: ffff880096bf6000 task.ti: ffff880096bf6000
Apr  6 11:43:38 debworm kernel: [103549.360458] RIP: 0010:[<ffffffffa1158662>]  [<ffffffffa1158662>] rbd_obj_request_destroy+0x142/0x150 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360502] RSP: 0018:ffff880096bf7b18  EFLAGS: 00010286
Apr  6 11:43:38 debworm kernel: [103549.360520] RAX: 000000000000006e RBX: ffff88021239d680 RCX: 0000000000000006
Apr  6 11:43:38 debworm kernel: [103549.360546] RDX: 0000000000000007 RSI: 0000000000000046 RDI: 0000000000000246
Apr  6 11:43:38 debworm kernel: [103549.360573] RBP: ffff8801f3fd4f38 R08: 000000000000000a R09: 000000000000a34e
Apr  6 11:43:38 debworm kernel: [103549.360601] R10: 0000000000000000 R11: ffff880096bf787e R12: ffff88021239d5d8
Apr  6 11:43:38 debworm kernel: [103549.360628] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88021239d5d8
Apr  6 11:43:38 debworm kernel: [103549.360660] FS:  0000000000000000(0000) GS:ffff88022fc80000(0000) knlGS:0000000000000000
Apr  6 11:43:38 debworm kernel: [103549.360688] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Apr  6 11:43:38 debworm kernel: [103549.360708] CR2: 00007fa667dc4000 CR3: 00000000a2883000 CR4: 00000000000007e0
Apr  6 11:43:38 debworm kernel: [103549.360732] Stack:
Apr  6 11:43:38 debworm kernel: [103549.360740]  0000000000000000 ffff8801f3fd4f38 000000180c140000 ffffffffa1159bb1
Apr  6 11:43:38 debworm kernel: [103549.360772]  ffff880223321b80 0000000000002000 ffff88022646e9c8 0000100000000000
Apr  6 11:43:38 debworm kernel: [103549.360803]  0000000000140000 000000180c140000 0000000000000246 0000000100000246
Apr  6 11:43:38 debworm kernel: [103549.360835] Call Trace:
Apr  6 11:43:38 debworm kernel: [103549.360847]  [<ffffffffa1159bb1>] ? rbd_img_request_fill+0x621/0x8b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360872]  [<ffffffffa115b47d>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Apr  6 11:43:38 debworm kernel: [103549.360894]  [<ffffffff812485ea>] ? __blk_run_queue+0x2a/0x40
Apr  6 11:43:38 debworm kernel: [103549.360915]  [<ffffffff81248690>] ? queue_unplugged+0x20/0x90
Apr  6 11:43:38 debworm kernel: [103549.360938]  [<ffffffff8124b95f>] ? blk_flush_plug_list+0x1ff/0x240
Apr  6 11:43:38 debworm kernel: [103549.360960]  [<ffffffff8124bcfb>] ? blk_finish_plug+0xb/0x30
Apr  6 11:43:38 debworm kernel: [103549.360987]  [<ffffffffa0299a07>] ? jbd2_journal_commit_transaction+0xd67/0x17d0 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361016]  [<ffffffff81019895>] ? sched_clock+0x5/0x10
Apr  6 11:43:38 debworm kernel: [103549.361035]  [<ffffffff8108b8d5>] ? sched_clock_local+0x15/0x80
Apr  6 11:43:38 debworm kernel: [103549.361060]  [<ffffffffa029e0af>] ? kjournald2+0xaf/0x240 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361081]  [<ffffffff81096f40>] ? prepare_to_wait_event+0xf0/0xf0
Apr  6 11:43:38 debworm kernel: [103549.361107]  [<ffffffffa029e000>] ? commit_timeout+0x10/0x10 [jbd2]
Apr  6 11:43:38 debworm kernel: [103549.361129]  [<ffffffff8107bb41>] ? kthread+0xc1/0xe0
Apr  6 11:43:38 debworm kernel: [103549.361148]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.361173]  [<ffffffff814ae44c>] ? ret_from_fork+0x7c/0xb0
Apr  6 11:43:38 debworm kernel: [103549.361193]  [<ffffffff8107ba80>] ? kthread_create_on_node+0x180/0x180
Apr  6 11:43:38 debworm kernel: [103549.361214] Code: a1 31 c0 e8 14 69 34 e0 0f 0b 48 c7 c1 38 e8 15 a1 ba 4b 07 00 00 48 c7 c6 e0 f8 15 a1 48 c7 c7 d0 e6 15 a1 31 c0 e8 f1 68 34 e0 <0f> 0b 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 85 ff 74 23 8b 87 
Apr  6 11:43:38 debworm kernel: [103549.361423]  RSP <ffff880096bf7b18>
Apr  6 11:43:38 debworm kernel: [103549.368551] ---[ end trace 76340b50a326be77 ]---

This is a catastrophic error with reboot as the only mean to recover.

Actions
Copy link
 #1

Updated by Ian Colle about 10 years ago

  • Assignee set to Ilya Dryomov
Actions
Copy link
 #2

Updated by Ilya Dryomov almost 10 years ago

  • Priority changed from High to Normal
Actions
Copy link
 #3

Updated by Dmitry Smirnov almost 10 years ago

  • Subject changed from 0.78: rbd/linux-3.13.7: page allocation failure to SLAB: Unable to allocate memory on node 0

With rbd client on kernel (3.14.9 and earlier) I'm still getting those errors on one machine every day (~once a day on average):

Jul  8 13:35:58 debworm kernel: [81089.041192] warn_alloc_failed: 21 callbacks suppressed
Jul  8 13:35:58 debworm kernel: [81089.041194] jbd2/rbd0-8: page allocation failure: order:1, mode:0x204020
Jul  8 13:35:58 debworm kernel: [81089.041197] CPU: 1 PID: 4705 Comm: jbd2/rbd0-8 Tainted: PF          O 3.14-1-amd64 #1 Debian 3.14.9-1
Jul  8 13:35:58 debworm kernel: [81089.041198] Hardware name: Gigabyte Technology Co., Ltd. P55-UD6/P55-UD6, BIOS F11d 11/09/2010
Jul  8 13:35:58 debworm kernel: [81089.041199]  ffff8802ee721810 ffffffff814ba29c 0000000000204020 ffffffff81124bc7
Jul  8 13:35:58 debworm kernel: [81089.041201]  0000000000000000 ffff88031fff2b00 ffffffff00000002 000000011fff1000
Jul  8 13:35:58 debworm kernel: [81089.041203]  ffff88031fff4fb0 0000000000000046 ffff88031fff2b08 0000000000000002
Jul  8 13:35:58 debworm kernel: [81089.041205] Call Trace:
Jul  8 13:35:58 debworm kernel: [81089.041215]  [<ffffffff814ba29c>] ? dump_stack+0x41/0x51
Jul  8 13:35:58 debworm kernel: [81089.041219]  [<ffffffff81124bc7>] ? warn_alloc_failed+0xd7/0x130
Jul  8 13:35:58 debworm kernel: [81089.041222]  [<ffffffff81128c6d>] ? __alloc_pages_nodemask+0x85d/0xab0
Jul  8 13:35:58 debworm kernel: [81089.041225]  [<ffffffff8116b1ff>] ? kmem_getpages+0x5f/0x140
Jul  8 13:35:58 debworm kernel: [81089.041227]  [<ffffffff8116ca37>] ? fallback_alloc+0x1c7/0x200
Jul  8 13:35:58 debworm kernel: [81089.041230]  [<ffffffff8116cfdd>] ? kmem_cache_alloc+0x11d/0x430
Jul  8 13:35:58 debworm kernel: [81089.041235]  [<ffffffffa1036fae>] ? ceph_osdc_alloc_request+0x4e/0x2f0 [libceph]
Jul  8 13:35:58 debworm kernel: [81089.041238]  [<ffffffffa105bf57>] ? rbd_osd_req_create.isra.24+0x67/0x170 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041240]  [<ffffffffa105c988>] ? rbd_img_request_fill+0x288/0x830 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041242]  [<ffffffffa105f4fd>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041245]  [<ffffffff81252ffa>] ? __blk_run_queue+0x2a/0x40
Jul  8 13:35:58 debworm kernel: [81089.041247]  [<ffffffff81257ec4>] ? blk_queue_bio+0x274/0x350
Jul  8 13:35:58 debworm kernel: [81089.041249]  [<ffffffff81252e68>] ? generic_make_request+0xa8/0xf0
Jul  8 13:35:58 debworm kernel: [81089.041251]  [<ffffffff81252f0c>] ? submit_bio+0x5c/0x120
Jul  8 13:35:58 debworm kernel: [81089.041254]  [<ffffffff811b92a8>] ? bio_alloc_bioset+0x198/0x290
Jul  8 13:35:58 debworm kernel: [81089.041256]  [<ffffffff811b4ad5>] ? _submit_bh+0x125/0x1f0
Jul  8 13:35:58 debworm kernel: [81089.041272]  [<ffffffffa01b5bed>] ? journal_submit_commit_record.isra.12+0x1ad/0x1d0 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041277]  [<ffffffffa01b6d62>] ? jbd2_journal_commit_transaction+0x1112/0x17d0 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041282]  [<ffffffffa01ba62f>] ? kjournald2+0xaf/0x240 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041286]  [<ffffffff8109e2d0>] ? prepare_to_wait_event+0xf0/0xf0
Jul  8 13:35:58 debworm kernel: [81089.041290]  [<ffffffffa01ba580>] ? commit_timeout+0x10/0x10 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041293]  [<ffffffff8107f858>] ? kthread+0xb8/0xd0
Jul  8 13:35:58 debworm kernel: [81089.041295]  [<ffffffff8107f7a0>] ? kthread_create_on_node+0x180/0x180
Jul  8 13:35:58 debworm kernel: [81089.041297]  [<ffffffff814c7acc>] ? ret_from_fork+0x7c/0xb0
Jul  8 13:35:58 debworm kernel: [81089.041298]  [<ffffffff8107f7a0>] ? kthread_create_on_node+0x180/0x180
Jul  8 13:35:58 debworm kernel: [81089.041300] Mem-Info:
Jul  8 13:35:58 debworm kernel: [81089.041300] Node 0 DMA per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041302] CPU    0: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041303] CPU    1: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041304] CPU    2: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041305] CPU    3: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041306] CPU    4: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041307] CPU    5: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041308] CPU    6: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041309] CPU    7: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041310] Node 0 DMA32 per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041312] CPU    0: hi:  186, btch:  31 usd: 181
Jul  8 13:35:58 debworm kernel: [81089.041313] CPU    1: hi:  186, btch:  31 usd:  87
Jul  8 13:35:58 debworm kernel: [81089.041314] CPU    2: hi:  186, btch:  31 usd: 155
Jul  8 13:35:58 debworm kernel: [81089.041315] CPU    3: hi:  186, btch:  31 usd: 156
Jul  8 13:35:58 debworm kernel: [81089.041316] CPU    4: hi:  186, btch:  31 usd:  52
Jul  8 13:35:58 debworm kernel: [81089.041317] CPU    5: hi:  186, btch:  31 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041318] CPU    6: hi:  186, btch:  31 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041319] CPU    7: hi:  186, btch:  31 usd:  40
Jul  8 13:35:58 debworm kernel: [81089.041320] Node 0 Normal per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041321] CPU    0: hi:  186, btch:  31 usd: 179
Jul  8 13:35:58 debworm kernel: [81089.041322] CPU    1: hi:  186, btch:  31 usd: 104
Jul  8 13:35:58 debworm kernel: [81089.041323] CPU    2: hi:  186, btch:  31 usd: 186
Jul  8 13:35:58 debworm kernel: [81089.041324] CPU    3: hi:  186, btch:  31 usd: 102
Jul  8 13:35:58 debworm kernel: [81089.041325] CPU    4: hi:  186, btch:  31 usd: 180
Jul  8 13:35:58 debworm kernel: [81089.041326] CPU    5: hi:  186, btch:  31 usd: 160
Jul  8 13:35:58 debworm kernel: [81089.041327] CPU    6: hi:  186, btch:  31 usd: 166
Jul  8 13:35:58 debworm kernel: [81089.041328] CPU    7: hi:  186, btch:  31 usd: 122
Jul  8 13:35:58 debworm kernel: [81089.041331] active_anon:824542 inactive_anon:167858 isolated_anon:0
Jul  8 13:35:58 debworm kernel: [81089.041331]  active_file:880735 inactive_file:906639 isolated_file:0
Jul  8 13:35:58 debworm kernel: [81089.041331]  unevictable:0 dirty:2229 writeback:1061 unstable:0
Jul  8 13:35:58 debworm kernel: [81089.041331]  free:119894 slab_reclaimable:38032 slab_unreclaimable:19721
Jul  8 13:35:58 debworm kernel: [81089.041331]  mapped:105749 shmem:112698 pagetables:16071 bounce:0
Jul  8 13:35:58 debworm kernel: [81089.041331]  free_cma:0
Jul  8 13:35:58 debworm kernel: [81089.041333] Node 0 DMA free:15900kB min:336kB low:420kB high:504kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul  8 13:35:58 debworm kernel: [81089.041337] lowmem_reserve[]: 0 3485 12017 12017
Jul  8 13:35:58 debworm kernel: [81089.041339] Node 0 DMA32 free:183992kB min:75960kB low:94948kB high:113940kB active_anon:664892kB inactive_anon:189888kB active_file:1180440kB inactive_file:1233732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3645120kB managed:3571264kB mlocked:0kB dirty:1884kB writeback:752kB mapped:71396kB shmem:83208kB slab_reclaimable:34772kB slab_unreclaimable:16332kB kernel_stack:2304kB pagetables:14368kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul  8 13:35:58 debworm kernel: [81089.041343] lowmem_reserve[]: 0 0 8532 8532
Jul  8 13:35:58 debworm kernel: [81089.041344] Node 0 Normal free:279684kB min:185840kB low:232300kB high:278760kB active_anon:2633276kB inactive_anon:481544kB active_file:2342500kB inactive_file:2392824kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8912896kB managed:8737096kB mlocked:0kB dirty:7032kB writeback:3492kB mapped:351600kB shmem:367584kB slab_reclaimable:117356kB slab_unreclaimable:62552kB kernel_stack:5952kB pagetables:49916kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul  8 13:35:58 debworm kernel: [81089.041348] lowmem_reserve[]: 0 0 0 0
Jul  8 13:35:58 debworm kernel: [81089.041350] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Jul  8 13:35:58 debworm kernel: [81089.041358] Node 0 DMA32: 42462*4kB (UEM) 1337*8kB (UM) 1*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 184656kB
Jul  8 13:35:58 debworm kernel: [81089.041364] Node 0 Normal: 66155*4kB (UEM) 1279*8kB (UEM) 26*16kB (EM) 1*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 279396kB
Jul  8 13:35:58 debworm kernel: [81089.041371] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul  8 13:35:58 debworm kernel: [81089.041372] 1900134 total pagecache pages
Jul  8 13:35:58 debworm kernel: [81089.041373] 52 pages in swap cache
Jul  8 13:35:58 debworm kernel: [81089.041374] Swap cache stats: add 4491, delete 4439, find 187/226
Jul  8 13:35:58 debworm kernel: [81089.041375] Free swap  = 11025828kB
Jul  8 13:35:58 debworm kernel: [81089.041376] Total swap = 11041784kB
Jul  8 13:35:58 debworm kernel: [81089.041377] 3143500 pages RAM
Jul  8 13:35:58 debworm kernel: [81089.041377] 0 pages HighMem/MovableOnly
Jul  8 13:35:58 debworm kernel: [81089.041378] 43950 pages reserved
Jul  8 13:35:58 debworm kernel: [81089.041379] 0 pages hwpoisoned
Jul  8 13:35:58 debworm kernel: [81089.041380] kmem_getpages: 21 callbacks suppressed
Jul  8 13:35:58 debworm kernel: [81089.041381] SLAB: Unable to allocate memory on node 0 (gfp=0x4020)
Jul  8 13:35:58 debworm kernel: [81089.041382]   cache: ceph_osd_request, object size: 1120, order: 1
Jul  8 13:35:58 debworm kernel: [81089.041383]   node 0: slabs: 2/2, objs: 14/14, free: 0
Jul  8 13:35:58 debworm kernel: [81089.041386] rbd: rbd0: write 1000 at 180f345000 result -12
Jul  8 13:35:58 debworm kernel: [81089.041386] 
Jul  8 13:35:58 debworm kernel: [81089.041388] blk_update_request: 21 callbacks suppressed
Jul  8 13:35:58 debworm kernel: [81089.041403] jbd2/rbd0-8: page allocation failure: order:1, mode:0x204020
Jul  8 13:35:58 debworm kernel: [81089.041404] CPU: 1 PID: 4705 Comm: jbd2/rbd0-8 Tainted: PF          O 3.14-1-amd64 #1 Debian 3.14.9-1
Jul  8 13:35:58 debworm kernel: [81089.041405] Hardware name: Gigabyte Technology Co., Ltd. P55-UD6/P55-UD6, BIOS F11d 11/09/2010
Jul  8 13:35:58 debworm kernel: [81089.041406]  ffff8802ee721840 ffffffff814ba29c 0000000000204020 ffffffff81124bc7
Jul  8 13:35:58 debworm kernel: [81089.041408]  0000000000000000 ffff88031fff2b00 ffffffff00000002 000000011fff1000
Jul  8 13:35:58 debworm kernel: [81089.041410]  ffff88031fff4fb0 0000000000000046 ffff88031fff2b08 0000000000000002
Jul  8 13:35:58 debworm kernel: [81089.041411] Call Trace:
Jul  8 13:35:58 debworm kernel: [81089.041413]  [<ffffffff814ba29c>] ? dump_stack+0x41/0x51
Jul  8 13:35:58 debworm kernel: [81089.041415]  [<ffffffff81124bc7>] ? warn_alloc_failed+0xd7/0x130
Jul  8 13:35:58 debworm kernel: [81089.041417]  [<ffffffff81128c6d>] ? __alloc_pages_nodemask+0x85d/0xab0
Jul  8 13:35:58 debworm kernel: [81089.041419]  [<ffffffff8116b1ff>] ? kmem_getpages+0x5f/0x140
Jul  8 13:35:58 debworm kernel: [81089.041421]  [<ffffffff8116ca37>] ? fallback_alloc+0x1c7/0x200
Jul  8 13:35:58 debworm kernel: [81089.041422]  [<ffffffff8116cfdd>] ? kmem_cache_alloc+0x11d/0x430
Jul  8 13:35:58 debworm kernel: [81089.041429]  [<ffffffffa1036fae>] ? ceph_osdc_alloc_request+0x4e/0x2f0 [libceph]
Jul  8 13:35:58 debworm kernel: [81089.041431]  [<ffffffffa105bf57>] ? rbd_osd_req_create.isra.24+0x67/0x170 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041433]  [<ffffffffa105c988>] ? rbd_img_request_fill+0x288/0x830 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041435]  [<ffffffffa105f4fd>] ? rbd_request_fn+0x1dd/0x2b0 [rbd]
Jul  8 13:35:58 debworm kernel: [81089.041437]  [<ffffffff81252ffa>] ? __blk_run_queue+0x2a/0x40
Jul  8 13:35:58 debworm kernel: [81089.041439]  [<ffffffff81257ec4>] ? blk_queue_bio+0x274/0x350
Jul  8 13:35:58 debworm kernel: [81089.041441]  [<ffffffff81252e68>] ? generic_make_request+0xa8/0xf0
Jul  8 13:35:58 debworm kernel: [81089.041443]  [<ffffffff81252f0c>] ? submit_bio+0x5c/0x120
Jul  8 13:35:58 debworm kernel: [81089.041445]  [<ffffffff811b92a8>] ? bio_alloc_bioset+0x198/0x290
Jul  8 13:35:58 debworm kernel: [81089.041447]  [<ffffffff811b4ad5>] ? _submit_bh+0x125/0x1f0
Jul  8 13:35:58 debworm kernel: [81089.041452]  [<ffffffffa01bce6a>] ? jbd2_write_superblock+0x6a/0x140 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041456]  [<ffffffffa01b6ffb>] ? jbd2_journal_commit_transaction+0x13ab/0x17d0 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041461]  [<ffffffffa01ba62f>] ? kjournald2+0xaf/0x240 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041463]  [<ffffffff8109e2d0>] ? prepare_to_wait_event+0xf0/0xf0
Jul  8 13:35:58 debworm kernel: [81089.041467]  [<ffffffffa01ba580>] ? commit_timeout+0x10/0x10 [jbd2]
Jul  8 13:35:58 debworm kernel: [81089.041469]  [<ffffffff8107f858>] ? kthread+0xb8/0xd0
Jul  8 13:35:58 debworm kernel: [81089.041471]  [<ffffffff8107f7a0>] ? kthread_create_on_node+0x180/0x180
Jul  8 13:35:58 debworm kernel: [81089.041472]  [<ffffffff814c7acc>] ? ret_from_fork+0x7c/0xb0
Jul  8 13:35:58 debworm kernel: [81089.041474]  [<ffffffff8107f7a0>] ? kthread_create_on_node+0x180/0x180
Jul  8 13:35:58 debworm kernel: [81089.041475] Mem-Info:
Jul  8 13:35:58 debworm kernel: [81089.041476] Node 0 DMA per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041477] CPU    0: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041478] CPU    1: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041479] CPU    2: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041480] CPU    3: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041481] CPU    4: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041482] CPU    5: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041483] CPU    6: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041484] CPU    7: hi:    0, btch:   1 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041485] Node 0 DMA32 per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041486] CPU    0: hi:  186, btch:  31 usd: 181
Jul  8 13:35:58 debworm kernel: [81089.041487] CPU    1: hi:  186, btch:  31 usd:  87
Jul  8 13:35:58 debworm kernel: [81089.041488] CPU    2: hi:  186, btch:  31 usd: 155
Jul  8 13:35:58 debworm kernel: [81089.041489] CPU    3: hi:  186, btch:  31 usd: 156
Jul  8 13:35:58 debworm kernel: [81089.041490] CPU    4: hi:  186, btch:  31 usd:  52
Jul  8 13:35:58 debworm kernel: [81089.041491] CPU    5: hi:  186, btch:  31 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041492] CPU    6: hi:  186, btch:  31 usd:   0
Jul  8 13:35:58 debworm kernel: [81089.041493] CPU    7: hi:  186, btch:  31 usd:  40
Jul  8 13:35:58 debworm kernel: [81089.041494] Node 0 Normal per-cpu:
Jul  8 13:35:58 debworm kernel: [81089.041495] CPU    0: hi:  186, btch:  31 usd: 179
Jul  8 13:35:58 debworm kernel: [81089.041496] CPU    1: hi:  186, btch:  31 usd: 104
Jul  8 13:35:58 debworm kernel: [81089.041497] CPU    2: hi:  186, btch:  31 usd: 163
Jul  8 13:35:58 debworm kernel: [81089.041498] CPU    3: hi:  186, btch:  31 usd: 102
Jul  8 13:35:58 debworm kernel: [81089.041499] CPU    4: hi:  186, btch:  31 usd: 180
Jul  8 13:35:58 debworm kernel: [81089.041500] CPU    5: hi:  186, btch:  31 usd: 160
Jul  8 13:35:58 debworm kernel: [81089.041501] CPU    6: hi:  186, btch:  31 usd: 166
Jul  8 13:35:58 debworm kernel: [81089.041502] CPU    7: hi:  186, btch:  31 usd: 122
Jul  8 13:35:58 debworm kernel: [81089.041504] active_anon:824542 inactive_anon:167858 isolated_anon:0
Jul  8 13:35:58 debworm kernel: [81089.041504]  active_file:880735 inactive_file:906543 isolated_file:0
Jul  8 13:35:58 debworm kernel: [81089.041504]  unevictable:0 dirty:2229 writeback:1061 unstable:0
Jul  8 13:35:58 debworm kernel: [81089.041504]  free:119959 slab_reclaimable:38032 slab_unreclaimable:19721
Jul  8 13:35:58 debworm kernel: [81089.041504]  mapped:105749 shmem:112698 pagetables:16071 bounce:0
Jul  8 13:35:58 debworm kernel: [81089.041504]  free_cma:0
Jul  8 13:35:58 debworm kernel: [81089.041507] Node 0 DMA free:15900kB min:336kB low:420kB high:504kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Jul  8 13:35:58 debworm kernel: [81089.041510] lowmem_reserve[]: 0 3485 12017 12017
Jul  8 13:35:58 debworm kernel: [81089.041512] Node 0 DMA32 free:183992kB min:75960kB low:94948kB high:113940kB active_anon:664892kB inactive_anon:189888kB active_file:1180440kB inactive_file:1233732kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3645120kB managed:3571264kB mlocked:0kB dirty:1884kB writeback:752kB mapped:71396kB shmem:83208kB slab_reclaimable:34772kB slab_unreclaimable:16332kB kernel_stack:2304kB pagetables:14368kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Jul  8 13:35:58 debworm kernel: [81089.041516] lowmem_reserve[]: 0 0 8532 8532
Jul  8 13:35:58 debworm kernel: [81089.041517] Node 0 Normal free:279944kB min:185840kB low:232300kB high:278760kB active_anon:2633276kB inactive_anon:481544kB active_file:2342500kB inactive_file:2392440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:8912896kB managed:8737096kB mlocked:0kB dirty:7032kB writeback:3492kB mapped:351600kB shmem:367584kB slab_reclaimable:117356kB slab_unreclaimable:62552kB kernel_stack:5952kB pagetables:49916kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:32 all_unreclaimable? no
Jul  8 13:35:58 debworm kernel: [81089.041521] lowmem_reserve[]: 0 0 0 0
Jul  8 13:35:58 debworm kernel: [81089.041523] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 15900kB
Jul  8 13:35:58 debworm kernel: [81089.041530] Node 0 DMA32: 42462*4kB (UEM) 1337*8kB (UM) 1*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 184656kB
Jul  8 13:35:58 debworm kernel: [81089.041536] Node 0 Normal: 66279*4kB (UEM) 1279*8kB (UEM) 26*16kB (EM) 1*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (R) = 279892kB
Jul  8 13:35:58 debworm kernel: [81089.041543] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jul  8 13:35:58 debworm kernel: [81089.041544] 1900037 total pagecache pages
Jul  8 13:35:58 debworm kernel: [81089.041545] 52 pages in swap cache
Jul  8 13:35:58 debworm kernel: [81089.041546] Swap cache stats: add 4491, delete 4439, find 187/226
Jul  8 13:35:58 debworm kernel: [81089.041547] Free swap  = 11025828kB
Jul  8 13:35:58 debworm kernel: [81089.041548] Total swap = 11041784kB
Jul  8 13:35:58 debworm kernel: [81089.041548] 3143500 pages RAM
Jul  8 13:35:58 debworm kernel: [81089.041549] 0 pages HighMem/MovableOnly
Jul  8 13:35:58 debworm kernel: [81089.041550] 43950 pages reserved
Jul  8 13:35:58 debworm kernel: [81089.041550] 0 pages hwpoisoned
Jul  8 13:35:58 debworm kernel: [81089.041551] SLAB: Unable to allocate memory on node 0 (gfp=0x4020)
Jul  8 13:35:58 debworm kernel: [81089.041552]   cache: ceph_osd_request, object size: 1120, order: 1
Jul  8 13:35:58 debworm kernel: [81089.041553]   node 0: slabs: 2/2, objs: 14/14, free: 0
Jul  8 13:35:58 debworm kernel: [81089.041555] rbd: rbd0: write 1000 at 1808000000 result -12
Jul  8 13:35:58 debworm kernel: [81089.041555] 
Jul  8 13:35:58 debworm kernel: [81089.041559] lost page write due to I/O error on rbd0
Jul  8 14:09:04 debworm kernel: [83075.585796] EXT4-fs (rbd0): mounted filesystem with ordered data mode. Opts: (null)

I can't explain why only one machine is affected.
There is /home on ext4 on rbd0 and user works with Icedove, libreoffice Chromium etc.
Earlier I suspected hardware but replacing workstation did not help.
Cluster version 0.80.1.

Actions
Copy link
 #4

Updated by Sage Weil almost 10 years ago

Are there a lot of snapshots on the RBD image(s) mapped by this host?

Actions
Copy link
 #5

Updated by Dmitry Smirnov almost 10 years ago

Sage Weil wrote:

Are there a lot of snapshots on the RBD image(s) mapped by this host?

None. RBD image format v1; file system ext4...

Actions
Copy link
 #6

Updated by Dmitry Smirnov almost 10 years ago

It may be more likely to happen during deep-scrub...
Any ideas?

Actions
Copy link
 #7

Updated by Dmitry Smirnov almost 10 years ago

I'm getting confident that this kernel bug always hit during deep-scrub.
I reproduced it several times just by starting deep-scrub on all OSDs: "ceph osd deep-scrub \*".
Upgrading to kernel 3.15.5 did not help, same problem...

Actions
Copy link
 #8

Updated by Dmitry Smirnov over 9 years ago

This problem remains to be very painful... Average frequency is one crash per day. Less than 24 hours ago I had two crashes from just rsync'ing files from ext4@rbd over the wireless connection. Nothing else were using RBD file system at a time... There are no sysctl customisations although I tried to increase `vm.min_free_kbytes` from default `67584` to `262144` without any impact on the problem. Machine is equipped with 12 GiB RAM; memory utilisation never goes above 50% and usually stay around ~35%. Swap is barely used (44 of 10782 MB)... All cluster components are 0.80.4.
I'm still looking for anything to try in order fix it...

Actions
Copy link
 #9

Updated by Ilya Dryomov over 9 years ago

What do you mean by "I can't explain why only one machine is affected" above? Do you have other similar boxes/setups that don't exhibit this behavior?

Actions
Copy link
 #10

Updated by Dmitry Smirnov over 9 years ago

Ilya Dryomov wrote:

What do you mean by "I can't explain why only one machine is affected" above? Do you have other similar boxes/setups that don't exhibit this behavior?

Exactly. I've seen at least one such error on another machine but is was never as bad as one crash per day (on average) that one particular workstation exhibit for months...

Actions
Copy link
 #11

Updated by Dmitry Smirnov over 9 years ago

Finally I've isolated the issue.
Something was wrong with a particular RBD image (format 1) that was created on Ceph not later than 80.0 (most likely 0.78 or 0.79).
Because only one machine was exhibiting such crash I initially suspected hardware but changing it did not help.
Problems stopped only when I converted a particular RBD image to format 2.
I have to say that changing RBD image format was not what's fixed it -- on the same (replicated) pool I have other images (format 1) that are used by other machines in the similar fashion without such problem.
Whatever was causing it was persistent in one particular RBD image and survived complete rebuild of all OSDs in the cluster.
I presume such issues should not exist yet I've experienced around 100(!) kernel crashes so I'm too familiar with it so say with confidence that somehow some defect in the RBD image was causing the problem so only rebuild of the RBD image helped.
I used the following (sample) command to rebuild the image:

rbd export rbd/myrbd - | rbd import --image-format 2 - rbd/myrbd_v2

Needless to say that (deep-)scrub detected no errors or inconsistencies on pool where affected image was based.

Actions
Copy link
 #12

Updated by Dmitry Smirnov over 9 years ago

Unfortunately converting RBD to image format 2 did not fix it. User returned after being away for a week and her system is crashing just as before. However I have a feeling that I'm getting closer to identifying a pattern of failure. On affected system (only) user have Libreoffice configured with "Always create backup copy" (in Tools --> Options --> Load/Save -- General) and crashes are happening while Libreoffice is in use (which is pretty much always). Typically up to ten Libreoffice applications are opened at a time. In "/home/user/.config/libreoffice/4/user/backup" folder about 28000 files accumulated, roughly taking 1100M. As mentioned earlier "/home" is on RBD, formatted as ext4.

Actions
Copy link
 #13

Updated by Dmitry Smirnov over 9 years ago

  • Status changed from New to Closed

No particular access pattern seems to provoke this issue and frankly I have no clue what's causing it apart from "deep-scrub" which appears to be a necessary condition for manifestation of this problem.

After re-examining hardware settings I lowered RAM frequency due to possibility that BIOS chosen over optimistic timings. This had dramatic effect on occurrence of this issue -- there were only one incident in several weeks. This is to say that RAM was extensively tested with `memtest`, `memtester`, `mprime` and no errors were found. Somehow only deep-scrub stressed CPU/RAM/NICs and IO enough to cause this issue. Memory settings were within specification (no overclocking) but perhaps a particular combination of RAM chips wasn't working too reliable...

I'm pretty sure I've seen no more than few isolated cases of this problem on another machine but it was long time ago and I don't have a reproducible case for it -- therefore I'm closing this issue. Thanks.

Actions
Copy link
 #14

Updated by Ilya Dryomov over 9 years ago

RAM frequency, interesting. Something to keep in mind..

Actions
Copy link
 #15

Updated by Jérôme Poulin over 8 years ago

I seem to have the exact same problem on kernel 4.2.3 so I backtracked to a working kernel which was 3.16.

This bug caused 2 other kernel panics before.

Nov  6 09:46:43 CASRV0104 kernel: [692354.650311] kworker/3:1: page allocation failure: order:1, mode:0x204020
Nov  6 09:46:43 CASRV0104 kernel: [692354.650316] CPU: 3 PID: 11596 Comm: kworker/3:1 Not tainted 4.2.0-0.bpo.1-amd64 #1 Debian 4.2.3-2~bpo8+1
Nov  6 09:46:43 CASRV0104 kernel: [692354.650318] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.5.2 10/15/2010
Nov  6 09:46:43 CASRV0104 kernel: [692354.650325] Workqueue: rbd rbd_queue_workfn [rbd]
Nov  6 09:46:43 CASRV0104 kernel: [692354.650327]  0000000000000000 0000000000000001 ffffffff8154dbb1 0000000000204020
Nov  6 09:46:43 CASRV0104 kernel: [692354.650329]  ffffffff8115508f ffff88043fff0b00 0000000000000000 0000000000000001
Nov  6 09:46:43 CASRV0104 kernel: [692354.650331]  0000000100000000 0000000000000000 ffff88043fff2fa8 0000000000000046
Nov  6 09:46:43 CASRV0104 kernel: [692354.650334] Call Trace:
Nov  6 09:46:43 CASRV0104 kernel: [692354.650340]  [<ffffffff8154dbb1>] ? dump_stack+0x40/0x50
Nov  6 09:46:43 CASRV0104 kernel: [692354.650344]  [<ffffffff8115508f>] ? warn_alloc_failed+0xcf/0x130
Nov  6 09:46:43 CASRV0104 kernel: [692354.650347]  [<ffffffff811587e0>] ? __alloc_pages_nodemask+0x2d0/0xa00
Nov  6 09:46:43 CASRV0104 kernel: [692354.650351]  [<ffffffff811a0691>] ? kmem_getpages+0x61/0x110
Nov  6 09:46:43 CASRV0104 kernel: [692354.650353]  [<ffffffff811a235c>] ? fallback_alloc+0x14c/0x200
Nov  6 09:46:43 CASRV0104 kernel: [692354.650356]  [<ffffffff811a3ec4>] ? kmem_cache_alloc+0x1f4/0x440
Nov  6 09:46:43 CASRV0104 kernel: [692354.650367]  [<ffffffffa07c3e95>] ? ceph_osdc_alloc_request+0x55/0x250 [libceph]
Nov  6 09:46:43 CASRV0104 kernel: [692354.650371]  [<ffffffffa0809935>] ? rbd_osd_req_create.isra.26+0x55/0x1b0 [rbd]
Nov  6 09:46:43 CASRV0104 kernel: [692354.650374]  [<ffffffffa080c20f>] ? rbd_img_request_fill+0x22f/0x850 [rbd]
Nov  6 09:46:43 CASRV0104 kernel: [692354.650378]  [<ffffffffa080d90f>] ? rbd_queue_workfn+0x2bf/0x3c0 [rbd]
Nov  6 09:46:43 CASRV0104 kernel: [692354.650382]  [<ffffffff8108671a>] ? process_one_work+0x14a/0x3d0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650385]  [<ffffffff81087105>] ? worker_thread+0x65/0x470
Nov  6 09:46:43 CASRV0104 kernel: [692354.650387]  [<ffffffff810870a0>] ? rescuer_thread+0x2f0/0x2f0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650389]  [<ffffffff8108c583>] ? kthread+0xd3/0xf0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650391]  [<ffffffff8108c4b0>] ? kthread_create_on_node+0x170/0x170
Nov  6 09:46:43 CASRV0104 kernel: [692354.650394]  [<ffffffff8155381f>] ? ret_from_fork+0x3f/0x70
Nov  6 09:46:43 CASRV0104 kernel: [692354.650396]  [<ffffffff8108c4b0>] ? kthread_create_on_node+0x170/0x170
Nov  6 09:46:43 CASRV0104 kernel: [692354.650397] Mem-Info:
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401] active_anon:1750510 inactive_anon:237494 isolated_anon:0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401]  active_file:915398 inactive_file:938093 isolated_file:0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401]  unevictable:1 dirty:31831 writeback:256 unstable:0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401]  slab_reclaimable:174720 slab_unreclaimable:17676
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401]  mapped:17254 shmem:26883 pagetables:10815 bounce:0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650401]  free:22847 free_pcp:1043 free_cma:0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650405] Node 0 DMA free:15900kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov  6 09:46:43 CASRV0104 kernel: [692354.650409] lowmem_reserve[]: 0 2969 16020 16020
Nov  6 09:46:43 CASRV0104 kernel: [692354.650412] Node 0 DMA32 free:56132kB min:2988kB low:3732kB high:4480kB active_anon:1145880kB inactive_anon:310196kB active_file:648040kB inactive_file:734252kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3119716kB managed:3043968kB mlocked:0kB dirty:21844kB writeback:440kB mapped:8536kB shmem:2084kB slab_reclaimable:97944kB slab_unreclaimable:16276kB kernel_stack:2400kB pagetables:9744kB unstable:0kB bounce:0kB free_pcp:2264kB local_pcp:364kB free_cma:0kB writeback_tmp:0kB pages_scanned:76 all_unreclaimable? no
Nov  6 09:46:43 CASRV0104 kernel: [692354.650417] lowmem_reserve[]: 0 0 13050 13050
Nov  6 09:46:43 CASRV0104 kernel: [692354.650419] Node 0 Normal free:19356kB min:13148kB low:16432kB high:19720kB active_anon:5856160kB inactive_anon:639780kB active_file:3013552kB inactive_file:3017944kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13364192kB mlocked:4kB dirty:105480kB writeback:584kB mapped:60480kB shmem:105448kB slab_reclaimable:600936kB slab_unreclaimable:54428kB kernel_stack:5008kB pagetables:33516kB unstable:0kB bounce:0kB free_pcp:1880kB local_pcp:668kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov  6 09:46:43 CASRV0104 kernel: [692354.650424] lowmem_reserve[]: 0 0 0 0
Nov  6 09:46:43 CASRV0104 kernel: [692354.650426] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650436] Node 0 DMA32: 13976*4kB (UEM) 44*8kB (UEM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 56256kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650443] Node 0 Normal: 4624*4kB (UEM) 83*8kB (UEM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19160kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650450] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650451] 1889223 total pagecache pages
Nov  6 09:46:43 CASRV0104 kernel: [692354.650453] 8854 pages in swap cache
Nov  6 09:46:43 CASRV0104 kernel: [692354.650454] Swap cache stats: add 998038, delete 989184, find 4654933/4901647
Nov  6 09:46:43 CASRV0104 kernel: [692354.650455] Free swap  = 43508kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650456] Total swap = 974844kB
Nov  6 09:46:43 CASRV0104 kernel: [692354.650457] 4191797 pages RAM
Nov  6 09:46:43 CASRV0104 kernel: [692354.650458] 0 pages HighMem/MovableOnly
Nov  6 09:46:43 CASRV0104 kernel: [692354.650459] 85782 pages reserved
Nov  6 09:46:43 CASRV0104 kernel: [692354.650459] 0 pages hwpoisoned
Nov  6 09:46:43 CASRV0104 kernel: [692354.650483] rbd: rbd0: write 40000 at 41e5268000 result -12
Nov  6 09:46:43 CASRV0104 kernel: [692354.650485] blk_update_request: I/O error, dev rbd0, sector 552768320

Nov  6 09:47:14 CASRV0104 kernel: [692385.095115] vmalloc: allocation failure: 18446744072439583573 bytes
Nov  6 09:47:14 CASRV0104 kernel: [692385.095118] kworker/2:1: page allocation failure: order:0, mode:0x22
Nov  6 09:47:14 CASRV0104 kernel: [692385.095121] CPU: 2 PID: 22092 Comm: kworker/2:1 Not tainted 4.2.0-0.bpo.1-amd64 #1 Debian 4.2.3-2~bpo8+1
Nov  6 09:47:14 CASRV0104 kernel: [692385.095123] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.5.2 10/15/2010
Nov  6 09:47:14 CASRV0104 kernel: [692385.095129] Workqueue: rbd rbd_queue_workfn [rbd]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095132]  0000000000000000 0000000000000000 ffffffff8154dbb1 0000000000000022
Nov  6 09:47:14 CASRV0104 kernel: [692385.095134]  ffffffff8115508f 0000000000000000 ffffffff818000f8 ffff88001194bb00
Nov  6 09:47:14 CASRV0104 kernel: [692385.095136]  0000000000000018 ffff88001194bb78 ffff88001194bb18 0000000000000000
Nov  6 09:47:14 CASRV0104 kernel: [692385.095138] Call Trace:
Nov  6 09:47:14 CASRV0104 kernel: [692385.095144]  [<ffffffff8154dbb1>] ? dump_stack+0x40/0x50
Nov  6 09:47:14 CASRV0104 kernel: [692385.095148]  [<ffffffff8115508f>] ? warn_alloc_failed+0xcf/0x130
Nov  6 09:47:14 CASRV0104 kernel: [692385.095151]  [<ffffffff8118acdb>] ? __vmalloc_node_range+0x22b/0x2b0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095153]  [<ffffffff8118ada7>] ? __vmalloc+0x47/0x50
Nov  6 09:47:14 CASRV0104 kernel: [692385.095161]  [<ffffffffa07ba147>] ? ceph_kvmalloc+0x47/0x50 [libceph]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095165]  [<ffffffffa07ba147>] ? ceph_kvmalloc+0x47/0x50 [libceph]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095171]  [<ffffffffa07bca6c>] ? ceph_msg_new+0xbc/0x170 [libceph]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095178]  [<ffffffffa07c4062>] ? ceph_osdc_alloc_request+0x222/0x250 [libceph]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095181]  [<ffffffffa0809935>] ? rbd_osd_req_create.isra.26+0x55/0x1b0 [rbd]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095185]  [<ffffffffa080c20f>] ? rbd_img_request_fill+0x22f/0x850 [rbd]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095189]  [<ffffffffa080d90f>] ? rbd_queue_workfn+0x2bf/0x3c0 [rbd]
Nov  6 09:47:14 CASRV0104 kernel: [692385.095192]  [<ffffffff8108671a>] ? process_one_work+0x14a/0x3d0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095194]  [<ffffffff81087105>] ? worker_thread+0x65/0x470
Nov  6 09:47:14 CASRV0104 kernel: [692385.095196]  [<ffffffff8154f38f>] ? __schedule+0x28f/0x8b0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095198]  [<ffffffff810870a0>] ? rescuer_thread+0x2f0/0x2f0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095200]  [<ffffffff8108c583>] ? kthread+0xd3/0xf0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095202]  [<ffffffff8108c4b0>] ? kthread_create_on_node+0x170/0x170
Nov  6 09:47:14 CASRV0104 kernel: [692385.095205]  [<ffffffff8155381f>] ? ret_from_fork+0x3f/0x70
Nov  6 09:47:14 CASRV0104 kernel: [692385.095206]  [<ffffffff8108c4b0>] ? kthread_create_on_node+0x170/0x170
Nov  6 09:47:14 CASRV0104 kernel: [692385.095208] Mem-Info:
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212] active_anon:1751330 inactive_anon:237499 isolated_anon:0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212]  active_file:916518 inactive_file:936558 isolated_file:0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212]  unevictable:1 dirty:33367 writeback:219 unstable:0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212]  slab_reclaimable:174302 slab_unreclaimable:17722
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212]  mapped:17266 shmem:26885 pagetables:11083 bounce:0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095212]  free:22950 free_pcp:870 free_cma:0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095215] Node 0 DMA free:15900kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Nov  6 09:47:14 CASRV0104 kernel: [692385.095220] lowmem_reserve[]: 0 2969 16020 16020
Nov  6 09:47:14 CASRV0104 kernel: [692385.095222] Node 0 DMA32 free:55796kB min:2988kB low:3732kB high:4480kB active_anon:1145880kB inactive_anon:310196kB active_file:651900kB inactive_file:732276kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3119716kB managed:3043968kB mlocked:0kB dirty:27320kB writeback:0kB mapped:8536kB shmem:2084kB slab_reclaimable:97936kB slab_unreclaimable:16320kB kernel_stack:2208kB pagetables:9744kB unstable:0kB bounce:0kB free_pcp:1960kB local_pcp:376kB free_cma:0kB writeback_tmp:0kB pages_scanned:760 all_unreclaimable? no
Nov  6 09:47:14 CASRV0104 kernel: [692385.095227] lowmem_reserve[]: 0 0 13050 13050
Nov  6 09:47:14 CASRV0104 kernel: [692385.095229] Node 0 Normal free:20104kB min:13148kB low:16432kB high:19720kB active_anon:5859440kB inactive_anon:639800kB active_file:3014172kB inactive_file:3013956kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:13631488kB managed:13364192kB mlocked:4kB dirty:106148kB writeback:876kB mapped:60528kB shmem:105456kB slab_reclaimable:599272kB slab_unreclaimable:54568kB kernel_stack:5008kB pagetables:34588kB unstable:0kB bounce:0kB free_pcp:1520kB local_pcp:188kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Nov  6 09:47:14 CASRV0104 kernel: [692385.095234] lowmem_reserve[]: 0 0 0 0
Nov  6 09:47:14 CASRV0104 kernel: [692385.095236] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095246] Node 0 DMA32: 13919*4kB (UEM) 13*8kB (UEM) 1*16kB (E) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55796kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095253] Node 0 Normal: 5046*4kB (UE) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20184kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095260] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095261] 1888830 total pagecache pages
Nov  6 09:47:14 CASRV0104 kernel: [692385.095263] 8856 pages in swap cache
Nov  6 09:47:14 CASRV0104 kernel: [692385.095264] Swap cache stats: add 998051, delete 989195, find 4655306/4902030
Nov  6 09:47:14 CASRV0104 kernel: [692385.095265] Free swap  = 43552kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095266] Total swap = 974844kB
Nov  6 09:47:14 CASRV0104 kernel: [692385.095267] 4191797 pages RAM
Nov  6 09:47:14 CASRV0104 kernel: [692385.095268] 0 pages HighMem/MovableOnly
Nov  6 09:47:14 CASRV0104 kernel: [692385.095268] 85782 pages reserved
Nov  6 09:47:14 CASRV0104 kernel: [692385.095269] 0 pages hwpoisoned
Nov  6 09:47:14 CASRV0104 kernel: [692385.095273] rbd: rbd0: write 40000 at 425f19f000 result -12
Nov  6 09:47:14 CASRV0104 kernel: [692385.095275] blk_update_request: I/O error, dev rbd0, sector 556764408
Nov  6 09:47:14 CASRV0104 kernel: [692385.173843] vmalloc: allocation failure: 18446744072439583573 bytes
Nov  6 09:47:14 CASRV0104 kernel: [692385.173847] kworker/2:1: page allocation failure: order:0, mode:0x22
Actions
Copy link
 #16

Updated by Jérôme Poulin over 8 years ago

I created issue #13712 since this report is closed.

Actions
Copy link

Also available in: Atom PDF