Ceph » CephFS

Patrick Donnelly wrote:

Were there any cluster health warnings? Some of the surround MDS logs would also be helpful.

Hi Pertick
On load we see
1 MDSs behind on trimming

Or/and:
1 filesystem is degraded

Also I want to mention that we enlarged mds_cache_memory_limit to 8G (on one of the setup we now changed it to 16GB)
and changed mds_min_caps_per_client to 150

In addition we now have setup which its mds keep restarting and I didn't manage to recover it at all even after trying redhat recovery procedure: http://people.redhat.com/bhubbard/nature/default/cephfs/disaster-recovery-experts/

Hi Yael,

Do you have the mds side logs ? Thanks.

Xiubo Li wrote:

Hi Yael,

Do you have the mds side logs ? Thanks.

Hi Xiubo
We reinstalled the setup and unfortunately I don't find logs of the failure now.
Thanks

Yael Azulay wrote:

Xiubo Li wrote:

Hi Yael,

Do you have the mds side logs ? Thanks.

Hi Xiubo
We reinstalled the setup and unfortunately I don't find logs of the failure now.
Thanks

That's okay, thanks.

I have checked the MDS related code, I just suspect that the cephfs_metadata pool was full of the journals, because since you have enlarged the mds_cache_memory_limit from 1G to 8G, so only the MDCache has been exceed the too full threshold value (8G * 95%) will the journal flush function, which will try to trim the journal logs, will be triggered. In your case I am afraid it was never triggered, because the too full threshold value of the MDCache had never reached, so the journals were piling up to 1.5TB.

Hi Yael,

BTW, did you see any memory killer logs in the /var/log/message* log files ? It should be OS's the memories were used up or it had reached up limitation of the mds process.

I found some memories are not counted into the MDCache memories, such as the inode_map map in the MDCache class.

Hi Yael,

BTW, have you ever changed the "mds_log_segment_size" and "mds_log_events_per_segment" options ?

If not, usually if the mds could trim the mdlog without stuck, the mdlog will occupy around 128 * 4MB == 512MB disk space.

@xiubo Li
Thanks much, Li, for your analysis
I didnt change mds_log_segment_size and mds_log_segment_size.
To what value should i change them given I have the mds_cache_memory_limit is 8G?

At the time this segmentation happened mds_min_caps_per_client was enlarged to 150

(Currently we have also enlarged other parameters like mds_cache_trim_threshold, mds_recall_global_max_decay_threshold, mds_recall_max_caps,
mds_recall_max_decay_rate)

- Not sure why my journal was never flushed? what should trigger journal flush?

- Did you mean there is a bug that inode_map is not counted in MDCache - which case this?

@xiubo Li
Hi Li
Thanks again
- What are the recommended values for mds_log_segment_size and mds_log_segment_size.?
- Dont understand: is the journal pool supposed to be emptied from time to time?
- What is the fix you did to the crash? In which version is it included?

Yael Azulay wrote:

@xiubo Li
Hi Li
Thanks again
- What are the recommended values for mds_log_segment_size and mds_log_segment_size.?

There has no other recommended values for this too options except the default ones, this should depend on your capacity of your disk or pool you use, mostly the default values should be okay.

- Dont understand: is the journal pool supposed to be emptied from time to time?

The journals won't be emptied, but it will be trimmed once the usage reached the upper limitation. For each MDS the threshold should be be 8G * 95% for you.

- What is the fix you did to the crash? In which version is it included?

The fixing is about not to take the inode numbers twice, the first one will be when replaying the journal logs, but it didn't erase this in the ino table, so when there has a new client request allocating new inode number, it may take the used ino, which will trigger a crash. This should be from Nautilus or earlier versions.

I will check whether will the frequently crashes of the MDSes make the journal logs to be cleaned or not.

ceph df 

RAW STORAGE:
    CLASS     SIZE        AVAIL       USED        RAW USED     %RAW USED
    ssd       8.7 TiB     5.0 TiB     3.8 TiB      3.8 TiB         43.29
    TOTAL     8.7 TiB     5.0 TiB     3.8 TiB      3.8 TiB         43.29

POOLS:
    POOL                          ID     STORED      OBJECTS     USED        %USED     MAX AVAIL
    cephfs_data                    1     246 GiB     591.31k     499 GiB     13.36       1.6 TiB
    cephfs_metadata                2     1.5 TiB     561.84k     3.0 TiB     48.69       1.6 TiB
    default.rgw.meta               3         0 B           0         0 B         0       1.6 TiB
    .rgw.root                      4     3.5 KiB           8     256 KiB         0       1.6 TiB
    default.rgw.buckets.index      5         0 B           0         0 B         0       1.6 TiB
    default.rgw.control            6         0 B           8         0 B         0       1.6 TiB
    default.rgw.buckets.data       7         0 B           0         0 B         0       1.6 TiB
    default.rgw.log                8         0 B         207         0 B         0       1.6 TiB
    volumes                        9     141 GiB      57.69k     282 GiB      8.01       1.6 TiB
    backups                       10         0 B           0         0 B         0       1.6 TiB
    metrics                       11         0 B           0         0 B         0       1.6 TiB

For this issue, it's the same with https://tracker.ceph.com/issues/40002, which was caused by the submit_mutex unfair mutex problem. It's a similar issue with https://tracker.ceph.com/issues/51722.

Hi xiubo Li
Thanks for information
The problem happen to us on nautilus 14.2.7

Are the fixes should be included on this version?