https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2011-06-10T11:10:54ZCeph CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=42852011-06-10T11:10:54ZBrian Chrismanbrchrisman@gmail.com
<ul></ul><p>I've verified this problem continues with 2.6.39 kernel with today's (5/26/11) build from ceph-client git repo (master) and today's ceph repo master branch.<br />What am I looking for in the logs here?</p>
<p>[root@buildvm-e2bf11 ~]# du -sh /mnt/buildshare ; sleep 600 ; du -sh /mnt/buildshare<br />166M /mnt/buildshare<br />167M /mnt/buildshare<br />[root@buildvm-e2bf11 ~]#</p> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=43492011-06-15T16:16:48ZBrian Chrismanbrchrisman@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/download/243/CephBDIstats.pdf">CephBDIstats.pdf</a> added</li></ul><p>Here's a 20-second interval time evolution of /sys/kernel/debug/ceph/*/bdi/stats<br />I can go ahead and graph these..</p> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=43502011-06-15T16:32:09ZBrian Chrismanbrchrisman@gmail.com
<ul><li><strong>File</strong> <a href="/attachments/download/244/Screen_shot_2011-06-15_at_4.29.37_PM.png">Screen_shot_2011-06-15_at_4.29.37_PM.png</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/244/Screen_shot_2011-06-15_at_4.29.37_PM.png">View</a> added</li><li><strong>File</strong> <a href="/attachments/download/245/Screen_shot_2011-06-15_at_4.29.01_PM.png">Screen_shot_2011-06-15_at_4.29.01_PM.png</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/245/Screen_shot_2011-06-15_at_4.29.01_PM.png">View</a> added</li><li><strong>File</strong> <a href="/attachments/download/246/Screen_shot_2011-06-15_at_4.27.38_PM.png">Screen_shot_2011-06-15_at_4.27.38_PM.png</a> <a class="icon-only icon-magnifier" title="View" href="/attachments/246/Screen_shot_2011-06-15_at_4.27.38_PM.png">View</a> added</li></ul><p>Here's the time sequence with each x-axis increment being 20 seconds.<br />Plotted separately to avoid scaling axes and all that crap.<br />Google charts doesn't plot much better.</p> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=43682011-06-16T14:11:45ZBrian Chrismanbrchrisman@gmail.com
<ul></ul><p>I think I've figured out the more general symptom.<br />File creation via NFS-ontop-of-ceph takes 1-3 seconds per file.<br />My copy appeared to be working because it initially copies some very large files.<br />I interrupted the copy and manually performed a few more copies, each taking 5-10s even though they were 100 byte or 500 byte files.</p>
<p>Performing a 'touch' of a new file on an NFS mounted ceph filesystem takes 1-3+ seconds.</p>
<p>Here's the end of a strace for a touch... what's not clear here is that the 2-3 second pause shows up in the middle of the utimensat() call after the open and close.<br />I exported an ext3 filesystem from the same server and the same touch takes < 0.1s.</p>
<pre>
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=99158752, ...}) = 0
mmap(NULL, 99158752, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fbd89d58000
close(3) = 0
open("/mnt/buildshare/file7", O_WRONLY|O_CREAT|O_NOCTTY|O_NONBLOCK, 0666) = 3
dup2(3, 0) = 0
close(3) = 0
utimensat(0, NULL, NULL, 0) = 0
close(0) = 0
close(1) = 0
close(2) = 0
exit_group(0) = ?
</pre> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=52762011-08-08T09:28:18ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> set to <i>v0.34</i></li><li><strong>translation missing: en.field_position</strong> set to <i>790</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=53262011-08-08T11:45:03ZSage Weilsage@newdream.net
<ul><li><strong>translation missing: en.field_position</strong> deleted (<del><i>798</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>1</i></li><li><strong>translation missing: en.field_position</strong> changed from <i>1</i> to <i>805</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=53312011-08-08T11:45:38ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>4</i></li><li><strong>translation missing: en.field_position</strong> deleted (<del><i>807</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>808</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=55262011-08-20T11:08:38ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.34</i> to <i>v0.35</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=60072011-09-01T13:07:00ZSage Weilsage@newdream.net
<ul><li><strong>Assignee</strong> set to <i>Brian Chrisman</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=61002011-09-06T21:49:15ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.35</i> to <i>v0.36</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=61442011-09-06T21:53:22ZSage Weilsage@newdream.net
<ul><li><strong>translation missing: en.field_position</strong> deleted (<del><i>847</i></del>)</li><li><strong>translation missing: en.field_position</strong> set to <i>1</i></li><li><strong>translation missing: en.field_position</strong> changed from <i>1</i> to <i>884</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=64162011-09-25T14:14:57ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.36</i> to <i>v0.37</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=65802011-10-09T20:39:53ZSage Weilsage@newdream.net
<ul><li><strong>Target version</strong> changed from <i>v0.37</i> to <i>v0.38</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=67772011-10-25T16:48:07ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>4</i> to <i>Need More Info</i></li><li><strong>Target version</strong> deleted (<del><i>v0.38</i></del>)</li></ul><p>Need to reproduce this on the current trunk and fully characterize what is going on.</p>
<p>- the the nfs server in sync or async mode<br />- which ceph operation(s) are slow</p>
<p>I suspect the ceph flushes aren't being prioritized or something.</p> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=89552012-03-05T10:47:13ZSage Weilsage@newdream.net
<ul><li><strong>Project</strong> changed from <i>Ceph</i> to <i>Linux kernel client</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=152492013-01-02T13:25:30ZSage Weilsage@newdream.net
<ul><li><strong>Assignee</strong> deleted (<del><i>Brian Chrisman</i></del>)</li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=187682013-03-13T14:23:39ZIan Colleicolle@redhat.com
<ul><li><strong>Project</strong> changed from <i>Linux kernel client</i> to <i>CephFS</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=750842016-07-20T06:11:00ZGreg Farnumgfarnum@redhat.com
<ul><li><strong>Category</strong> set to <i>NFS (Linux Kernel)</i></li></ul> CephFS - Bug #1114: NFS export extreme slowdown https://tracker.ceph.com/issues/1114?journal_id=1312332019-03-09T00:19:57ZPatrick Donnellypdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>Need More Info</i> to <i>Rejected</i></li></ul><p>NFS over the kernel client is not a recommended configuration.</p>