Actions
Bug #40951
closed(think this is bug) Why CephFS metadata pool usage is only growing?
Status:
Rejected
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Can't find a clue: singe MDS fs without actually usage, but metadata is only growing (screenshot attached). MDS with default settings, `mds_cache_memory_limit` is only changed to "25769803776". May be meta (log?) trimmed later and just not yet reach some threshold?
fs_meta pool is filestore OSD.
# ceph fs status cephfs - 9 clients ====== +------+--------+-----------+---------------+-------+-------+ | Rank | State | MDS | Activity | dns | inos | +------+--------+-----------+---------------+-------+-------+ | 0 | active | ceph-mds0 | Reqs: 0 /s | 154 | 38 | +------+--------+-----------+---------------+-------+-------+ +--------------+----------+-------+-------+ | Pool | type | used | avail | +--------------+----------+-------+-------+ | fs_meta | metadata | 122M | 10.9T | | fs_data | data | 23.0G | 67.8T | | fs_data_nvme | data | 0 | 10.9T | +--------------+----------+-------+-------+ +-------------+ | Standby MDS | +-------------+ +-------------+ MDS version: ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
MDS permissions:
[mds.ceph-mds0]
key = <snip></snap>
caps mds = "allow *"
caps mgr = "profile mds"
caps mon = "profile mds"
caps osd = "allow *"
kernel client permissions:
# kernel client 3.10.0-957.12.2.el7.x86_64
client.mds_ovirt_iso
key: <snip></snap>
caps: [mds] allow rw path=/iso
caps: [mon] allow r
caps: [osd] allow rwx pool=fs_data, allow rwx pool=fs_data_nvme
# samba vfs luminous 12.2.12 client
client.mds_samba
key: <snip></snap>
caps: [mds] allow rwp
caps: [mon] allow r
caps: [osd] allow rwx pool=fs_data, allow rw pool=fs_data_nvme
fs dump:
dumped fsmap epoch 17
e17
enable_multiple, ever_enabled_multiple: 0,0
compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
legacy client fscid: 1
Filesystem 'cephfs' (1)
fs_name cephfs
epoch 17
flags c
created 2019-07-09 15:38:01.614140
modified 2019-07-24 11:47:34.048711
tableserver 0
root 0
session_timeout 60
session_autoclose 300
max_file_size 1099511627776
last_failure 0
last_failure_osd_epoch 56984
compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2}
max_mds 1
in 0
up {0=377152890}
failed
damaged
stopped
data_pools [21,23]
metadata_pool 22
inline_data disabled
balancer
standby_count_wanted 0
377152890: 172.16.16.5:6800/2469778625 'ceph-mds0' mds.0.14 up:active seq 5
fs data pool on this week:
fs meta pool on this week:
Files
Updated by Konstantin Shalygin over 4 years ago
- File another_cluster.jpg another_cluster.jpg added
Found another cluster with this Ceph version. Data usage 10x more, but meta is not much.
Metadata pool in this cluster is Bluestore ODS.
Updated by Patrick Donnelly over 4 years ago
Did you just upgrade your cluster to Nautilus? The data usage stats changed recently so that omaps in the metadata pool are properly accounted for but it happens over time with scrub. That would explain the increase with no activity.
Updated by Patrick Donnelly over 4 years ago
I see now that you're using 12.2.12? That wouldn't be the cause then I guess. The PR which backports the relevant change has not been merged yet. (#40638)
Updated by Konstantin Shalygin over 4 years ago
Patrick, this is definitely Luminous 12.2.12. My actual question is why metadata usage (bytes) is growing and new objects is created without write new objects to data pools?
Also due to BZ1558836 : Red Hat still not released selinux-policy with fix for this BZ, I set SELinux policy on clients to permissive - AFAIK this cephfs calls is not blocked (audit only notify about this).
[Thu Jul 25 12:33:59 2019] libceph: mon0 172.16.16.2:6789 session established
[Thu Jul 25 12:33:59 2019] libceph: client377786016 fsid 5532c4fd-60db-43ff-af9a-c4eb8523382b
[Thu Jul 25 14:44:43 2019] type=1400 audit(1564040474.722:843): avc: denied { read } for pid=90932 comm="qemu-kvm" name="CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:44:43 2019] type=1400 audit(1564040474.723:844): avc: denied { open } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:44:43 2019] type=1400 audit(1564040474.725:845): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:44:43 2019] type=1400 audit(1564040474.726:846): avc: denied { lock } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:44:59 2019] type=1400 audit(1564040490.630:847): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:45:14 2019] type=1400 audit(1564040505.615:848): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:45:29 2019] type=1400 audit(1564040520.629:849): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:45:44 2019] type=1400 audit(1564040535.621:850): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:45:59 2019] type=1400 audit(1564040550.614:851): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:46:14 2019] type=1400 audit(1564040565.624:852): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:46:29 2019] type=1400 audit(1564040580.623:853): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:46:44 2019] type=1400 audit(1564040595.624:854): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:46:59 2019] type=1400 audit(1564040610.619:855): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:47:14 2019] type=1400 audit(1564040625.622:856): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:47:29 2019] type=1400 audit(1564040640.616:857): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:47:44 2019] type=1400 audit(1564040655.622:858): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:47:59 2019] type=1400 audit(1564040670.616:859): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:48:14 2019] type=1400 audit(1564040685.624:860): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:48:29 2019] type=1400 audit(1564040700.620:861): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:48:59 2019] type=1400 audit(1564040730.622:862): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:49:14 2019] type=1400 audit(1564040745.630:863): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:49:29 2019] type=1400 audit(1564040760.623:864): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:49:44 2019] type=1400 audit(1564040775.622:865): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:49:59 2019] type=1400 audit(1564040790.626:866): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:50:14 2019] type=1400 audit(1564040805.624:867): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:50:29 2019] type=1400 audit(1564040820.625:868): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:50:44 2019] type=1400 audit(1564040835.617:869): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:50:59 2019] type=1400 audit(1564040850.629:870): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:51:14 2019] type=1400 audit(1564040865.622:871): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:51:29 2019] type=1400 audit(1564040880.621:872): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:51:44 2019] type=1400 audit(1564040895.625:873): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:51:59 2019] type=1400 audit(1564040910.615:874): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:52:14 2019] type=1400 audit(1564040925.626:875): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:52:29 2019] type=1400 audit(1564040940.623:876): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:52:44 2019] type=1400 audit(1564040955.635:877): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:52:59 2019] type=1400 audit(1564040970.618:878): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:53:14 2019] type=1400 audit(1564040985.620:879): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:53:29 2019] type=1400 audit(1564041000.633:880): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:53:44 2019] type=1400 audit(1564041015.623:881): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:53:59 2019] type=1400 audit(1564041030.629:882): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:54:29 2019] type=1400 audit(1564041060.624:883): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:54:44 2019] type=1400 audit(1564041075.621:884): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:54:59 2019] type=1400 audit(1564041090.618:885): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:55:14 2019] type=1400 audit(1564041105.625:886): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:55:29 2019] type=1400 audit(1564041120.634:887): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:55:44 2019] type=1400 audit(1564041135.632:888): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:55:59 2019] type=1400 audit(1564041150.639:889): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:56:14 2019] type=1400 audit(1564041165.641:890): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:56:29 2019] type=1400 audit(1564041180.647:891): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:56:44 2019] type=1400 audit(1564041195.643:892): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:56:59 2019] type=1400 audit(1564041210.649:893): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:57:14 2019] type=1400 audit(1564041225.653:894): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:57:29 2019] type=1400 audit(1564041240.654:895): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:57:44 2019] type=1400 audit(1564041255.656:896): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:57:59 2019] type=1400 audit(1564041270.653:897): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:58:14 2019] type=1400 audit(1564041285.647:898): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:58:29 2019] type=1400 audit(1564041300.654:899): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:58:44 2019] type=1400 audit(1564041315.647:900): avc: denied { getattr } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
[Thu Jul 25 14:58:48 2019] type=1400 audit(1564041318.777:901): avc: denied { lock } for pid=90932 comm="qemu-kvm" path="/rhev/data-center/mnt/172.16.16.2,172.16.16.3,172.16.16.4:_iso/249a241a-711b-4f35-a922-291f8a1634c8/images/11111111-1111-1111-1111-111111111111/CentOS-7-x86_64-Minimal-1810.iso" dev="ceph" ino=1099511628840 scontext=system_u:system_r:svirt_t:s0:c12,c544 tcontext=system_u:object_r:cephfs_t:s0 tclass=file permissive=1
Actions
#5
Updated by Konstantin Shalygin over 4 years ago
- File fs_data_26.07.19.png fs_data_26.07.19.png added
- File fs_meta_26.07.19.png fs_meta_26.07.19.png added
+10MBytes for last ~24H.
Actual pool's usage:
fs_data:
fs_meta:
Actions
#6
Updated by Konstantin Shalygin over 4 years ago
- File fs_data_26.07.19_test.png fs_data_26.07.19_test.png added
- File fs_meta_26.07.19_test.png fs_meta_26.07.19_test.png added
- File cachedump cachedump added
I was abstract of kernel client and make some tests with Samba VFS - this is Luminous client.
First, I just copied exists content, then delete it. Data pool reclaim space - works as expected. Then I perform:
(desktop gvfs -> samba vfs):
[k0ste@WorkStation folder]$ for e in seq 1 1000 ; do mkdir "dir${e}" ; done
[k0ste@WorkStation folder]$ for e in seq 1 1000 ; do touch "dir${e}\file${e}" ; done
This give me ~ 133.9MB -> 173MB -> 182.3MB metadata and not reduced. But some objects was deleted.
~ 50MB for thousand empty dir and file. cache dump is attached, something interesting?
fs_data:
fs_meta:
Updated by Konstantin Shalygin over 4 years ago
- File cache_drop.png cache_drop.png added
After another bunch of simulations I was call `cache drop` to MDS via admin socket. Pool usage 198.8MB -> 2.8MB.
As I understand some cache saved in RADOS for consistency between MDS'es? How much it can grow? Any options to control this? Documentation, etc...
# ceph daemon mds.ceph-mds0 cache status
{
"pool": {
"items": 17351,
"bytes": 430328
}
}
# ceph daemon mds.ceph-mds0 cache drop
{
"client_recall": {
"return_code": 0,
"message": "(0) Success",
"recalled": 0
},
"flush_journal": {
"return_code": 0,
"message": ""
},
"trim_cache": {
"trimmed": 2
},
"cache": {
"pool": {
"items": 2716,
"bytes": 154972
}
},
"duration": 0.005907
}
Updated by Patrick Donnelly over 4 years ago
The MDS tracks opened files and capabilities in the MDS journal. That would explain the space usage in the metadata pool. After you dropped cache, the MDS can trim a lot of that space down.
If that explains the issue for you, please close this ticket.
Updated by Konstantin Shalygin over 4 years ago
- File fs_meta_2019-07-29.png fs_meta_2019-07-29.png added
The MDS tracks opened files and capabilities in the MDS journal. That would explain the space usage in the metadata pool.
1. Mount with kernel client;
2. Create /newdir;
3. Create 20000 dirs int the /newdir;
4. Touch 20000 files inside this dirs;
5. Delete /newdir;
6. sync;
7. Umount.
In theory - no data = no cache. Actually looks like MDS journal left something behind. But why?
I'm just try to understand this may be go near with mds_cache_memory_limit? And how predict data usage.
Updated by Patrick Donnelly over 4 years ago
Konstantin Shalygin wrote:
The MDS tracks opened files and capabilities in the MDS journal. That would explain the space usage in the metadata pool.
1. Mount with kernel client;
2. Create /newdir;
3. Create 20000 dirs int the /newdir;
4. Touch 20000 files inside this dirs;
5. Delete /newdir;
6. sync;
7. Umount.In theory - no data = no cache. Actually looks like MDS journal left something behind. But why?
I'm just try to understand this may be go near with mds_cache_memory_limit? And how predict data usage.
The files may be deleted but they still are in the MDS journal. The files won't be removed from the journal until trimmed. This won't happen if there is no other activity on the MDS.
Updated by Konstantin Shalygin over 4 years ago
Bingo - mds_log_max_segments.
In Luminous desc for this option is empty:
{
"option": {
"name": "mds_log_max_segments",
"type": "uint64_t",
"level": "advanced",
"desc": "",
"long_desc": "",
"default": 128,
"daemon_default": "",
"tags": [],
"services": [
"mds"
],
"see_also": [],
"min": "",
"max": ""
}
}
Even in master doc declared default value to 30, actually default value is 128. This is threshold that I looking form.
Actions