Project

General

Profile

Actions
Copy link

Bug #15502

closed

files read or written with cephfs (fuse or kernel) on client drop all their page cache pages on close

Added by Barry Marson about 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
High
Assignee:
Zheng Yan
Category:
Performance/Resource Usage
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Testing cephfs file system I/O with early jewel bits (ceph-10.0.4-1.el7cp.x86_64) on:

RHEL72 client mounting a cephfs (fuse or kernel) file systems on a RHEL72 4 node server cluster shows that every time we close a file on the client, all of the page cache pages associated with that file get dropped.

A simple test case is attached ...

Below is some sample output:

procs -----------memory---------- ---swap-- -----io---- system- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 106428 48332732 0 234132 0 0 30 106 1 1 1 0 98 0 0
0 0 106428 48332312 0 234212 0 0 0 0 82 134 0 0 100 0 0
0 0 106428 48332572 0 234212 0 0 0 0 121 124 0 0 100 0 0
0 0 106428 48332572 0 234212 0 0 0 0 83 122 0 0 100 0 0
0 0 106428 48332572 0 234212 0 0 0 0 143 173 0 0 100 0 0
Dropping Cache on purpose
0 0 106428 48332572 0 234212 0 0 32 10 91 204 0 0 100 0 0
Sleeping for 5 seconds
0 0 106428 48383008 0 185880 0 0 1240 51 261 368 0 0 100 0 0
0 0 106428 48383624 0 185104 0 0 0 0 53 91 0 0 100 0 0
0 0 106428 48383624 0 185104 0 0 0 0 124 141 0 0 100 0 0
0 0 106428 48383632 0 185104 0 0 0 0 40 63 0 0 100 0 0
0 0 106428 48383632 0 185104 0 0 0 0 92 69 0 0 100 0 0
Create a 10GB file with : dd if=/dev/zero of=/sasdata/ddfile.log bs=128k count=102400 conv=fsync
0 0 106428 48382888 0 185104 0 0 128 0 111 233 0 0 100 0 0
1 0 106428 46386068 0 2182464 0 0 0 0 1055 184 0 4 96 0 0
1 0 106428 44044976 0 4524148 0 0 0 0 1072 92 0 4 96 0 0
2 0 106428 41883136 0 6684856 0 0 0 0 30939 5315 0 6 94 0 0
3 0 106428 39729552 0 8838996 0 0 0 0 29685 5536 0 7 93 0 0
1 0 106428 37863336 0 10705224 0 0 0 0 30309 5337 0 7 93 0 0
0 1 106428 37855084 0 10714604 0 0 0 0 18560 6098 0 2 94 4 0
0 1 106428 37855332 0 10714356 0 0 0 0 18490 5969 0 2 94 4 0
2 1 106428 37856164 0 10713548 0 0 0 0 17895 6318 0 2 94 4 0
1 1 106428 37857660 0 10712308 0 0 0 0 11395 5344 0 2 94 4 0
1 1 106428 37859740 0 10710196 0 0 0 0 8840 4560 0 2 94 4 0
0 1 106428 37862488 0 10707604 0 0 0 0 8160 3117 0 1 94 4 0
0 1 106428 37867152 0 10702940 0 0 0 0 1317 631 0 0 96 4 0
0 1 106428 37873184 0 10696908 0 0 0 1 924 848 0 0 96 4 0
0 1 106428 37880596 0 10689620 0 0 0 0 504 1157 0 0 96 4 0
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB) copied, 14.9595 s, 718 MB/s
Note page cache usage now ...
sleeping 5 seconds
0 0 106428 48383240 0 185932 0 0 96 0 1301 916 0 3 96 0 0 <======= FILES CACHE DROPPED
0 0 106428 48383428 0 185772 0 0 0 0 54 97 0 0 100 0 0
0 0 106428 48383676 0 185772 0 0 0 0 120 122 0 0 100 0 0
0 0 106428 48383800 0 185772 0 0 0 0 43 63 0 0 100 0 0
0 0 106428 48383924 0 185772 0 0 0 0 94 68 0 0 100 0 0
Read the 10GB file with -> dd if=/sasdata/ddfile.log of=/dev/null bs=16k
0 1 106428 48309964 0 260880 0 0 0 0 1442 2055 0 0 99 0 0
0 1 106428 47605496 0 968972 0 0 0 0 11158 17486 0 3 94 3 0
0 1 106428 46897808 0 1676576 0 0 0 0 11903 18367 0 2 94 3 0
0 1 106428 46172880 0 2400252 0 0 0 0 11966 18425 0 2 94 3 0
0 1 106428 45458936 0 3115524 0 0 0 0 12346 19059 0 3 94 3 0
0 1 106428 44729800 0 3845044 0 0 0 16 11846 18127 0 3 94 3 0
2 1 106428 44006744 0 4567652 0 0 0 0 12050 17993 0 3 94 3 0
1 1 106428 43284408 0 5291036 0 0 0 0 11900 18420 0 2 94 3 0
0 1 106428 42558892 0 6016704 0 0 0 0 12650 18308 0 2 94 3 0
0 1 106428 41848088 0 6726356 0 0 0 0 12678 18419 0 3 94 3 0
1 1 106428 41125968 0 7449704 0 0 0 4 12493 18905 0 3 94 3 0
1 1 106428 40416464 0 8159692 0 0 0 0 12446 18430 0 2 94 3 0
0 1 106428 39701936 0 8873680 0 0 0 0 12691 18567 0 2 94 3 0
0 1 106428 38995404 0 9580452 0 0 0 0 13141 18703 0 3 94 3 0
1 1 106428 38242380 0 10335368 0 0 0 0 12206 18146 0 3 94 3 0
1 0 106428 44433100 0 4142496 0 0 0 0 7367 9652 0 3 95 2 0
163840+0 records in
163840+0 records out
10737418240 bytes (11 GB) copied, 15.4091 s, 697 MB/s
Note page cache usage now ...
sleeping 5 seconds
0 0 106428 48389876 0 186476 0 0 0 0 521 358 0 1 99 0 0 <======= FILES CACHE DROPPED
0 0 106428 48389980 0 186476 0 0 0 0 111 162 0 0 100 0 0
0 0 106428 48389980 0 186476 0 0 0 0 67 65 0 0 100 0 0
0 0 106428 48389980 0 186476 0 0 0 0 69 68 0 0 100 0 0
0 0 106428 48389980 0 186476 0 0 0 0 80 81 0 0 100 0 0
cephdropcache.sh: line 29: 22294 Terminated vmstat 1

Files

cephdropcache.sh (570 Bytes) cephdropcache.sh Script demonstrates files being dropped from cache on close Barry Marson, 04/14/2016 05:34 PM
Actions
Copy link

Also available in: Atom PDF