Actions
Bug #22487
closedmds: setattr blocked when metadata pool is full
Status:
Rejected
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Development
Tags:
Backport:
luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
With both of these PRs on mimic-dev1:
https://github.com/ceph/ceph/pull/19588
https://github.com/ceph/ceph/pull/19602
$ python2 ../qa/tasks/vstart_runner.py --interactive tasks.cephfs.test_full.TestClusterFull.test_full_different_file ... 2017-12-19 16:49:11,247.247 INFO:tasks.cephfs.test_full:Writing 66MB should fill this cluster 2017-12-19 16:49:11,247.247 INFO:__main__:run args=['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:11,248.248 INFO:__main__:Running ['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:11,259.259 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmp7IWgkx/mnt.0 2017-12-19 16:49:11,260.260 INFO:__main__:run args=['dd', 'if=/dev/urandom', 'of=large_file_a', 'bs=1M', 'conv=fdatasync', 'count=33', 'seek=0'] 2017-12-19 16:49:11,260.260 INFO:__main__:Running ['dd', 'if=/dev/urandom', 'of=large_file_a', 'bs=1M', 'conv=fdatasync', 'count=33', 'seek=0'] 2017-12-19 16:49:15,319.319 INFO:__main__:run args=['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:15,320.320 INFO:__main__:Running ['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:15,331.331 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmp7IWgkx/mnt.0 2017-12-19 16:49:15,331.331 INFO:__main__:run args=['dd', 'if=/dev/urandom', 'of=large_file_b', 'bs=1M', 'conv=fdatasync', 'count=33', 'seek=0'] 2017-12-19 16:49:15,331.331 INFO:__main__:Running ['dd', 'if=/dev/urandom', 'of=large_file_b', 'bs=1M', 'conv=fdatasync', 'count=33', 'seek=0'] 2017-12-19 16:49:19,304.304 INFO:tasks.cephfs.test_full:Writing file B succeeded (full status will happen soon) 2017-12-19 16:49:19,305.305 INFO:__main__:run args=['./bin/ceph', 'osd', 'dump', '--format=json-pretty'] 2017-12-19 16:49:19,305.305 INFO:__main__:Running ['./bin/ceph', 'osd', 'dump', '--format=json-pretty'] 2017-12-19 16:49:24,644.644 INFO:__main__:run args=['./bin/ceph', 'osd', 'dump', '--format=json-pretty'] 2017-12-19 16:49:24,645.645 INFO:__main__:Running ['./bin/ceph', 'osd', 'dump', '--format=json-pretty'] 2017-12-19 16:49:24,996.996 INFO:__main__:run args=['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:24,996.996 INFO:__main__:Running ['stat', '--file-system', '--printf=%T\n', '--', '/tmp/tmp7IWgkx/mnt.0'] 2017-12-19 16:49:25,008.008 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /tmp/tmp7IWgkx/mnt.0 2017-12-19 16:49:25,008.008 INFO:__main__:run args=['dd', 'if=/dev/urandom', 'of=large_file_b', 'bs=1M', 'conv=fdatasync', 'count=50', 'seek=33'] 2017-12-19 16:49:25,008.008 INFO:__main__:Running ['dd', 'if=/dev/urandom', 'of=large_file_b', 'bs=1M', 'conv=fdatasync', 'count=50', 'seek=33']
$ bin/ceph --admin-daemon /tmp/ceph-asok.pu3EX7/mds.c.asok dump_blocked_ops
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
{
"ops": [
{
"description": "client_request(client.4230:8 setattr size=34603008 #0x10000000001 2017-12-19 16:49:25.020565 caller_uid=1163, caller_gid=1163{27,119,1163,})",
"initiated_at": "2017-12-19 16:49:25.020873",
"age": 133.987241,
"duration": 133.987273,
"type_data": {
"flag_point": "failed to xlock, waiting",
"reqid": "client.4230:8",
"op_type": "client_request",
"client_info": {
"client": "client.4230",
"tid": 8
},
"events": [
{
"time": "2017-12-19 16:49:25.020873",
"event": "initiated"
},
{
"time": "2017-12-19 16:49:25.021836",
"event": "failed to xlock, waiting"
},
{
"time": "2017-12-19 16:49:25.023219",
"event": "failed to xlock, waiting"
}
]
}
}
],
"complaint_time": 30.000000,
"num_blocked_ops": 1
}
Actions