Ceph » CephFS

I reproduced this locally.

It was caused by stuck log flush

[root@zhyan-alpha build]# bin/ceph daemon mds.a dump_ops_in_flight
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
    "ops": [
        {
            "description": "client_request(client.4155:8 setattr size=557842432 #0x10000000001 2017-12-21 10:17:59.001781 caller_uid=0, caller_gid=0{0,})",
            "initiated_at": "2017-12-21 10:17:59.001834",
            "age": 371.770464,
            "duration": 371.770485,
            "type_data": {
                "flag_point": "failed to xlock, waiting",
                "reqid": "client.4155:8",
                "op_type": "client_request",
                "client_info": {
                    "client": "client.4155",
                    "tid": 8
                },
                "events": [
                    {
                        "time": "2017-12-21 10:17:59.001834",
                        "event": "initiated" 
                    },
                    {
                        "time": "2017-12-21 10:17:59.001921",
                        "event": "failed to xlock, waiting" 
                    },
                    {
                        "time": "2017-12-21 10:17:59.002146",
                        "event": "failed to xlock, waiting" 
                    }
                ]
            }
        }
    ],
    "num_ops": 1
}

[root@zhyan-alpha build]# bin/ceph daemon mds.a objecter_requests
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
    "ops": [
        {
            "tid": 67,
            "pg": "7.6e5f474",
            "osd": 0,
            "object_id": "200.00000001",
            "object_locator": "@7",
            "target_object_id": "200.00000001",
            "target_object_locator": "@7",
            "paused": 0,
            "used_replica": 0,
            "precalc_pgid": 0,
            "last_sent": "262226s",
            "attempts": 1,
            "snapid": "head",
            "snap_context": "0=[]",
            "mtime": "2017-12-21 10:17:59.0.001474s",
            "osd_ops": [
                "write 75939~4420 [fadvise_dontneed]" 
            ]
        },
        {
            "tid": 68,
            "pg": "7.844f3494",
            "osd": 0,
            "object_id": "200.00000000",
            "object_locator": "@7",
            "target_object_id": "200.00000000",
            "target_object_locator": "@7",
            "paused": 0,
            "used_replica": 0,
            "precalc_pgid": 0,
            "last_sent": "262226s",
            "attempts": 1,
            "snapid": "head",
            "snap_context": "0=[]",
            "mtime": "2017-12-21 10:17:59.0.001545s",
            "osd_ops": [
                "writefull 0~90 [fadvise_dontneed]" 
            ]
        },
        {
            "tid": 69,
            "pg": "7.6e5f474",
            "osd": 0,
            "object_id": "200.00000001",
            "object_locator": "@7",
            "target_object_id": "200.00000001",
            "target_object_locator": "@7",
            "paused": 0,
            "used_replica": 0,
            "precalc_pgid": 0,
            "last_sent": "262226s",
            "attempts": 1,
            "snapid": "head",
            "snap_context": "0=[]",
            "mtime": "2017-12-21 10:17:59.0.002128s",
            "osd_ops": [
                "write 80359~1935 [fadvise_dontneed]" 
            ]
        }
    ],
    "linger_ops": [],
    "pool_ops": [],
    "pool_stat_ops": [],
    "statfs_ops": [],
    "command_ops": []
}

cap flush process held wrlock

[inode 0x10000000001 [2,head] /large_file_b auth v54 pv58 ap=4+0 dirtyparent s=439676928 n(v0 b439676928 1=1+0) (iauth excl) (ifile lock->excl w=2) (ixattr excl) (iversion lock w=1 last_client=4155) cr={4155=0-880803840@1} caps={4155=pAsxLsXsxFcb/pAsxXsxFsxcrwb@19},l=4155 | ptrwaiter=2 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 waiter=1 authpin=1 0x55555eff4700]

I only reproduced this with vstart cluster's default conf

        osd failsafe full ratio = .99
        mon osd nearfull ratio = .99
        mon osd backfillfull ratio = .99
        mon osd full ratio = .99

can't reproduce this with full test's config

        mon osd nearfull ratio: 0.6
        mon osd backfillfull ratio: 0.6
        mon osd full ratio: 0.7
        osd failsafe full ratio: 1.0

Presumably that would be because with the vstart config the MDS writes cannot actually be written whereas with the test configs (0.7 full ratio), the writes can still proceed because the MDS is treated specially. Right?

right. full test should have no problem