https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2011-04-03T21:07:15Z
Ceph
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3302
2011-04-03T21:07:15Z
Greg Farnum
gfarnum@redhat.com
<ul></ul><p>Well the first line there is "can't allocate 512 bytes"...looks like you ran out of memory. Was there memory pressure on this machine for some reason?</p>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3598
2011-04-22T12:17:31Z
Sage Weil
sage@newdream.net
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>Linux kernel client</i></li><li><strong>Category</strong> changed from <i>24</i> to <i>libceph</i></li><li><strong>Target version</strong> deleted (<del><i>v0.25.3</i></del>)</li></ul>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3599
2011-04-22T12:17:44Z
Sage Weil
sage@newdream.net
<ul><li><strong>Target version</strong> set to <i>v2.6.39</i></li></ul><p>probably a memory leak?</p>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3643
2011-04-28T21:13:44Z
Sage Weil
sage@newdream.net
<ul><li><strong>translation missing: en.field_story_points</strong> set to <i>3</i></li><li><strong>translation missing: en.field_position</strong> set to <i>1</i></li><li><strong>translation missing: en.field_position</strong> changed from <i>1</i> to <i>562</i></li></ul>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3652
2011-04-29T07:22:59Z
DongJin Lee
dongjin.lee@auckland.ac.nz
<ul></ul><pre>
[ 9326.513556] libceph: client4117 fsid 4f209795-4899-8134-6829-f952b295d00e
[ 9326.513682] libceph: mon0 192.168.1.4:6789 session established
[ 9905.382664] libceph: msg_new can't allocate 4096 bytes
[ 9905.382690] general protection fault: 0000 [#1] SMP
[ 9905.382786] last sysfs file: /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq
[ 9905.382903] CPU 1
[ 9905.382939] Modules linked in: ceph fuse w83793 w83627hf hwmon_vid coretemp edd bonding cpufreq_conservative cpufreq_userspace cpufreq_powersave acpi_cpufreq mperf loop dm_mod sr_mod cdrom sg iTCO_wdt shpchp pci_hotplug igb ioatdma mptctl iTCO_vendor_support ghes i5400_edac edac_core dca i2c_i801 i5k_amb pcspkr container hed button ext4 jbd2 crc16 uhci_hcd radeon ttm drm_kms_helper mptsas mptscsih ehci_hcd drm i2c_algo_bit mptbase scsi_transport_sas usbcore fan thermal processor thermal_sys
[ 9905.384009]
[ 9905.384009] Pid: 12563, comm: flush-ceph-14 Not tainted 2.6.38.1-2-default #1 Supermicro X7DW3/X7DWN+
[ 9905.384009] RIP: 0010:[<ffffffff812557c2>] [<ffffffff812557c2>] kref_put+0x22/0x70
[ 9905.384009] RSP: 0018:ffff88011d981770 EFLAGS: 00010202
[ 9905.384009] RAX: ffff8800cb597a38 RBX: 8000000000000000 RCX: 0000000000001c3f
[ 9905.384009] RDX: ffffffff81a15501 RSI: ffffffff8149ba10 RDI: 8000000000000000
[ 9905.384009] RBP: ffffffff8149ba10 R08: 000000000001322d R09: 000000000000000a
[ 9905.384009] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000011200
[ 9905.384009] R13: 0000000000001000 R14: 0000000000000000 R15: ffff8800cb597a48
[ 9905.384009] FS: 0000000000000000(0000) GS:ffff8800cfd00000(0000) knlGS:0000000000000000
[ 9905.384009] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 9905.384009] CR2: 00007fdd88d59000 CR3: 000000012540e000 CR4: 00000000000006e0
[ 9905.384009] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 9905.384009] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 9905.384009] Process flush-ceph-14 (pid: 12563, threadinfo ffff88011d980000, task ffff8801206e6280)
[ 9905.384009] Stack:
[ 9905.384009] ffffffff00000010 ffff8800cb597a48 ffff8800cb5979c0 ffffffff8149b7e0
[ 9905.384009] ffff88012ffedc00 ffff8800cb597a48 ffffffff8149b7a0 ffffffff812557d3
[ 9905.384009] 0000000000000000 ffff8800cb5979c0 0000000000001000 ffffffff81496114
[ 9905.384009] Call Trace:
[ 9905.384009] [<ffffffff8149b7e0>] ceph_msg_last_put+0x40/0x100
[ 9905.384009] [<ffffffff812557d3>] kref_put+0x33/0x70
[ 9905.384009] [<ffffffff81496114>] ceph_msg_new+0x124/0x250
[ 9905.384009] [<ffffffff8149b8bd>] alloc_fn+0x1d/0x50
[ 9905.384009] [<ffffffff810f5169>] mempool_alloc+0x59/0x130
[ 9905.384009] [<ffffffff814a1579>] ceph_osdc_alloc_request+0x1c9/0x2e0
[ 9905.384009] [<ffffffff814a1779>] ceph_osdc_new_request+0xe9/0x210
[ 9905.384009] [<ffffffffa047ba72>] ceph_writepages_start+0x742/0x11d0 [ceph]
[ 9905.384009] [<ffffffff81172950>] writeback_single_inode+0x90/0x230
[ 9905.384009] [<ffffffff81172d15>] generic_writeback_sb_inodes+0xd5/0x160
[ 9905.384009] [<ffffffff81173050>] writeback_inodes_wb+0x1b0/0x1c0
[ 9905.384009] [<ffffffff811732d5>] wb_writeback+0x275/0x350
[ 9905.384009] [<ffffffff81173e81>] wb_do_writeback+0xa1/0x220
[ 9905.384009] [<ffffffff8117409a>] bdi_writeback_thread+0x9a/0x270
[ 9905.384009] [<ffffffff81078516>] kthread+0x96/0xa0
[ 9905.384009] [<ffffffff81003c44>] kernel_thread_helper+0x4/0x10
[ 9905.384009] Code: 2e 0f 1f 84 00 00 00 00 00 48 83 ec 18 48 85 f6 48 89 5c 24 08 48 89 6c 24 10 48 89 fb 48 89 f5 74 41 48 81 fe d0 ba 13 81 74 25 <f0> ff 0b 0f 94 c2 31 c0 84 d2 74 0a 48 89 df ff d5 b8 01 00 00
[ 9905.384009] RIP [<ffffffff812557c2>] kref_put+0x22/0x70
[ 9905.384009] RSP <ffff88011d981770>
[ 9905.413309] ---[ end trace b020ccf21254f339 ]---
</pre>
<p>A bit different error, any ideas? this time total files are 10GB, about 200 files. the fault happened while reading those files, I think it could be due to the heat/ cpu scaling_factor?</p>
<p>thanks</p>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3681
2011-05-02T10:06:08Z
Sage Weil
sage@newdream.net
<ul><li><strong>translation missing: en.field_position</strong> deleted (<del><i>562</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>7</i></li></ul>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3685
2011-05-02T10:16:52Z
Fyodor Ustinov
ufmufm@gmail.com
<ul></ul><p>The same.</p>
<p>[ 1877.996453] libceph: msg_new can't allocate 512 bytes<br />[ 1877.996607] libceph: msg_new can't create type 0 front 512<br />[ 1877.996654] libceph: msgpool osd_op_reply alloc failed<br />[ 1877.996710] libceph: msg_new can't allocate 512 bytes<br />[ 1877.996766] general protection fault: 0000 [<a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: gpf in tcp_sendpage (Closed)" href="https://tracker.ceph.com/issues/1">#1</a>] SMP <br />[ 1877.996811] last sysfs file: /sys/devices/virtual/bdi/ceph-1/uevent<br />[ 1877.996863] CPU 3 <br />[ 1877.996880] Modules linked in: ceph libceph libcrc32c ppdev vmw_balloon psmouse serio_raw i2c_piix4 parport_pc shpchp lp parport mptspi mptscsih e1000 vmw_pvscsi floppy mptbase vmxnet3<br />[ 1877.998339] <br />[ 1877.998354] Pid: 892, comm: flush-ceph-1 Not tainted 2.6.38-8-server #42-Ubuntu VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform<br />[ 1877.998476] RIP: 0010:[<ffffffff812dd325>] [<ffffffff812dd325>] kref_put+0x25/0x70<br />[ 1877.998555] RSP: 0018:ffff8800b14ed6e0 EFLAGS: 00010206<br />[ 1877.998600] RAX: ffff8800b169d4f8 RBX: 0001000001e4ce01 RCX: 00000000ffffffff<br />[ 1877.998674] RDX: 0000000000000001 RSI: ffffffffa00ebaa0 RDI: 0001000001e4ce01<br />[ 1877.998734] RBP: ffff8800b14ed6f0 R08: 0000000000000000 R09: ffffffff816423e0<br />[ 1877.998793] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffffa00ebaa0<br />[ 1877.999006] R13: 0000000000011200 R14: 0000000000000200 R15: 0000000000000000<br />[ 1877.999132] FS: 0000000000000000(0000) GS:ffff8800bb580000(0000) knlGS:0000000000000000<br />[ 1877.999338] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b<br />[ 1877.999437] CR2: 0000000003fd7000 CR3: 00000000b2f31000 CR4: 00000000000006e0<br />[ 1877.999579] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br />[ 1877.999715] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br />[ 1877.999827] Process flush-ceph-1 (pid: 892, threadinfo ffff8800b14ec000, task ffff8800b0eddb80)<br />[ 1878.000001] Stack:<br />[ 1878.000069] ffff8800b169d508 ffff8800b169d480 ffff8800b14ed710 ffffffffa00eb88a<br />[ 1878.000238] ffff8800b169d508 ffffffffa00eb850 ffff8800b14ed730 ffffffff812dd337<br />[ 1878.000409] ffff8800b169d480 0000000000000200 ffff8800b14ed780 ffffffffa00ea9bf<br />[ 1878.000578] Call Trace:<br />[ 1878.000658] [<ffffffffa00eb88a>] ceph_msg_last_put+0x3a/0xd0 [libceph]<br />[ 1878.000768] [<ffffffffa00eb850>] ? ceph_msg_last_put+0x0/0xd0 [libceph]<br />[ 1878.000877] [<ffffffff812dd337>] kref_put+0x37/0x70<br />[ 1878.000971] [<ffffffffa00ea9bf>] ceph_msg_new+0x20f/0x230 [libceph]<br />[ 1878.001087] [<ffffffffa00eb945>] alloc_fn+0x25/0x50 [libceph]<br />[ 1878.001189] [<ffffffff8110de03>] mempool_alloc+0x53/0x130<br />[ 1878.001286] [<ffffffff8110de62>] ? mempool_alloc+0xb2/0x130<br />[ 1878.001387] [<ffffffff812e41c9>] ? vsnprintf+0x479/0x620<br />[ 1878.001486] [<ffffffffa00eba35>] ceph_msgpool_get+0x25/0x60 [libceph]<br />[ 1878.001601] [<ffffffffa00f0036>] ceph_osdc_alloc_request+0x266/0x310 [libceph]<br />[ 1878.001768] [<ffffffffa00f01a7>] ceph_osdc_new_request+0xc7/0x1d0 [libceph]<br />[ 1878.001881] [<ffffffffa00ee422>] ? __map_osds+0xd2/0x3a0 [libceph]<br />[ 1878.001987] [<ffffffff8110b9d0>] ? find_get_pages_tag+0x40/0x120<br />[ 1878.002090] [<ffffffff815d779e>] ? _raw_spin_lock+0xe/0x20<br />[ 1878.002196] [<ffffffffa011cc41>] ceph_writepages_start+0x681/0x970 [ceph]<br />[ 1878.002310] [<ffffffff8105e463>] ? balance_tasks+0x103/0x1b0<br />[ 1878.002412] [<ffffffff815d779e>] ? _raw_spin_lock+0xe/0x20<br />[ 1878.002513] [<ffffffff811167a1>] do_writepages+0x21/0x40<br />[ 1878.002612] [<ffffffff8118b7af>] writeback_single_inode+0x9f/0x240<br />[ 1878.002719] [<ffffffff8118bb8b>] writeback_sb_inodes+0xcb/0x160<br />[ 1878.002821] [<ffffffff8118bddb>] writeback_inodes_wb+0x10b/0x1c0<br />[ 1878.002923] [<ffffffff8118c20e>] wb_writeback+0x37e/0x490<br />[ 1878.003019] [<ffffffff815d794f>] ? _raw_spin_lock_irqsave+0x2f/0x40<br />[ 1878.003124] [<ffffffff81074ceb>] ? lock_timer_base.clone.20+0x3b/0x70<br />[ 1878.003229] [<ffffffff8118c541>] wb_do_writeback+0x221/0x230<br />[ 1878.003335] [<ffffffff8118c5d2>] bdi_writeback_thread+0x82/0x260<br />[ 1878.003437] [<ffffffff8118c550>] ? bdi_writeback_thread+0x0/0x260<br />[ 1878.003541] [<ffffffff810871f6>] kthread+0x96/0xa0<br />[ 1878.003634] [<ffffffff8100cde4>] kernel_thread_helper+0x4/0x10<br />[ 1878.003735] [<ffffffff81087160>] ? kthread+0x0/0xa0<br />[ 1878.003826] [<ffffffff8100cde0>] ? kernel_thread_helper+0x0/0x10<br />[ 1878.003927] Code: e8 eb a7 0f 1f 00 55 48 89 e5 48 83 ec 10 48 85 f6 48 89 1c 24 4c 89 64 24 08 48 89 fb 49 89 f4 74 3e 48 81 fe a0 59 15 81 74 22 <f0> ff 0b 0f 94 c2 31 c0 84 d2 74 0b 48 89 df 41 ff d4 b8 01 00 <br />[ 1878.004372] RIP [<ffffffff812dd325>] kref_put+0x25/0x70<br />[ 1878.004488] RSP <ffff8800b14ed6e0><br />[ 1878.004832] ---[ end trace 9d4e026fc60681d0 ]---</p>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3714
2011-05-03T09:15:51Z
Anonymous
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Resolved</i></li></ul><p>Fixed in ceph-client.git master</p>
<pre>
commit 56f63aeb6360fb3ba9584bd5b094d55283a9e332
Author: Henry C Chang <henry.cy.chang@gmail.com>
Date: 2011-05-03 02:29:56 +0000
libceph: fix ceph_msg_new error path
If memory allocation failed, calling ceph_msg_put() will cause GPF
since some of ceph_msg variables are not initialized first.
Fix Bug #970.
Signed-off-by: Henry C Chang <henry_c_chang@tcloudcomputing.com>
Signed-off-by: Tommi Virtanen <tommi.virtanen@dreamhost.com>
</pre>
Linux kernel client - Bug #970: Kernel crash (cause?: lots of small files)
https://tracker.ceph.com/issues/970?journal_id=3717
2011-05-03T10:45:29Z
Anonymous
<ul></ul><p>Make that ca20892db7567c40e8ed0668f46cf0d085d7db6d in for-linus instead.</p>