https://tracker.ceph.com/
https://tracker.ceph.com/favicon.ico
2017-12-19T21:55:32Z
Ceph
CephFS - Bug #22487: mds: setattr blocked when metadata pool is full
https://tracker.ceph.com/issues/22487?journal_id=103935
2017-12-19T21:55:32Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-1 priority-4 priority-default" href="/issues/22488">Bug #22488</a>: mds: unlink blocks on large file when metadata pool is full</i> added</li></ul>
CephFS - Bug #22487: mds: setattr blocked when metadata pool is full
https://tracker.ceph.com/issues/22487?journal_id=104031
2017-12-21T14:35:04Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>I reproduced this locally.</p>
<p>It was caused by stuck log flush</p>
<pre>
[root@zhyan-alpha build]# bin/ceph daemon mds.a dump_ops_in_flight
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
"ops": [
{
"description": "client_request(client.4155:8 setattr size=557842432 #0x10000000001 2017-12-21 10:17:59.001781 caller_uid=0, caller_gid=0{0,})",
"initiated_at": "2017-12-21 10:17:59.001834",
"age": 371.770464,
"duration": 371.770485,
"type_data": {
"flag_point": "failed to xlock, waiting",
"reqid": "client.4155:8",
"op_type": "client_request",
"client_info": {
"client": "client.4155",
"tid": 8
},
"events": [
{
"time": "2017-12-21 10:17:59.001834",
"event": "initiated"
},
{
"time": "2017-12-21 10:17:59.001921",
"event": "failed to xlock, waiting"
},
{
"time": "2017-12-21 10:17:59.002146",
"event": "failed to xlock, waiting"
}
]
}
}
],
"num_ops": 1
}
</pre>
<pre>
[root@zhyan-alpha build]# bin/ceph daemon mds.a objecter_requests
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
"ops": [
{
"tid": 67,
"pg": "7.6e5f474",
"osd": 0,
"object_id": "200.00000001",
"object_locator": "@7",
"target_object_id": "200.00000001",
"target_object_locator": "@7",
"paused": 0,
"used_replica": 0,
"precalc_pgid": 0,
"last_sent": "262226s",
"attempts": 1,
"snapid": "head",
"snap_context": "0=[]",
"mtime": "2017-12-21 10:17:59.0.001474s",
"osd_ops": [
"write 75939~4420 [fadvise_dontneed]"
]
},
{
"tid": 68,
"pg": "7.844f3494",
"osd": 0,
"object_id": "200.00000000",
"object_locator": "@7",
"target_object_id": "200.00000000",
"target_object_locator": "@7",
"paused": 0,
"used_replica": 0,
"precalc_pgid": 0,
"last_sent": "262226s",
"attempts": 1,
"snapid": "head",
"snap_context": "0=[]",
"mtime": "2017-12-21 10:17:59.0.001545s",
"osd_ops": [
"writefull 0~90 [fadvise_dontneed]"
]
},
{
"tid": 69,
"pg": "7.6e5f474",
"osd": 0,
"object_id": "200.00000001",
"object_locator": "@7",
"target_object_id": "200.00000001",
"target_object_locator": "@7",
"paused": 0,
"used_replica": 0,
"precalc_pgid": 0,
"last_sent": "262226s",
"attempts": 1,
"snapid": "head",
"snap_context": "0=[]",
"mtime": "2017-12-21 10:17:59.0.002128s",
"osd_ops": [
"write 80359~1935 [fadvise_dontneed]"
]
}
],
"linger_ops": [],
"pool_ops": [],
"pool_stat_ops": [],
"statfs_ops": [],
"command_ops": []
}
</pre>
<p>cap flush process held wrlock<br /><pre>
[inode 0x10000000001 [2,head] /large_file_b auth v54 pv58 ap=4+0 dirtyparent s=439676928 n(v0 b439676928 1=1+0) (iauth excl) (ifile lock->excl w=2) (ixattr excl) (iversion lock w=1 last_client=4155) cr={4155=0-880803840@1} caps={4155=pAsxLsXsxFcb/pAsxXsxFsxcrwb@19},l=4155 | ptrwaiter=2 request=1 lock=3 caps=1 dirtyparent=1 dirty=1 waiter=1 authpin=1 0x55555eff4700]
</pre></p>
<p>I only reproduced this with vstart cluster's default conf<br /><pre>
osd failsafe full ratio = .99
mon osd nearfull ratio = .99
mon osd backfillfull ratio = .99
mon osd full ratio = .99
</pre></p>
<p>can't reproduce this with full test's config<br /><pre>
mon osd nearfull ratio: 0.6
mon osd backfillfull ratio: 0.6
mon osd full ratio: 0.7
osd failsafe full ratio: 1.0
</pre></p>
CephFS - Bug #22487: mds: setattr blocked when metadata pool is full
https://tracker.ceph.com/issues/22487?journal_id=104043
2017-12-21T22:27:52Z
Patrick Donnelly
pdonnell@redhat.com
<ul></ul><p>Presumably that would be because with the vstart config the MDS writes cannot actually be written whereas with the test configs (0.7 full ratio), the writes can still proceed because the MDS is treated specially. Right?</p>
CephFS - Bug #22487: mds: setattr blocked when metadata pool is full
https://tracker.ceph.com/issues/22487?journal_id=104045
2017-12-21T23:45:29Z
Zheng Yan
ukernel@gmail.com
<ul></ul><p>right. full test should have no problem</p>
CephFS - Bug #22487: mds: setattr blocked when metadata pool is full
https://tracker.ceph.com/issues/22487?journal_id=107224
2018-02-10T04:31:07Z
Patrick Donnelly
pdonnell@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>Rejected</i></li></ul>