Bug #19866
osd-dup.sh failing with EMFILE on arm64
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
two of three OSDs died; one, at least, died because of
2017-05-04 23:14:12.847887 ffff76164cb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaab29c00244 (16
00.1.0, or op 0, counting from 0)
2017-05-04 23:14:12.847914 ffff76164cb0 0 filestore(td/osd-dup/1) unexpected error code
ulimit -f is set to 1024. The test tries to warn, but nothing comes to the console. Running with ulimit -n set to 2048 is not sufficient to make the test succeed on arm64 (it dies in rados bench because the osds have died). ulimit -n 2048 makes the test succeed on x86_64.
History
#1 Updated by Dan Mick almost 7 years ago
- Subject changed from osd-dup.sh failing with EMFILE on arm64 to osd-dup.sh failing with EMFILE
#2 Updated by Dan Mick almost 7 years ago
- Description updated (diff)
- Status changed from New to Rejected
#3 Updated by Dan Mick almost 7 years ago
- Subject changed from osd-dup.sh failing with EMFILE to osd-dup.sh failing with EMFILE on arm64
- Status changed from Rejected to New
#4 Updated by Dan Mick almost 7 years ago
Confusing myself mightily, but, yes, the osds die early on arm64 for reasons I don't yet understand.
#5 Updated by Dan Mick almost 7 years ago
dmick@omani002:~/src/ceph/build/src/test/osd/td/osd-dup$ ulimit -n 2048 dmick@omani002:~/src/ceph/build/src/test/osd/td/osd-dup$ grep "Too many" *log osd.0.log:2017-05-04 23:52:47.485694 ffff81e9dcb0 -1 filestore(td/osd-dup/0) error (24) Too many open files not handled on operation 0xaaab1eb798c4 (3340.1.0, or op 0, counting from 0) osd.0.log:2017-05-04 23:52:47.485948 ffff7c69dcb0 -1 filestore(td/osd-dup/0) error (24) Too many open files not handled on operation 0xaaab1f540484 (3341.1.0, or op 0, counting from 0) osd.0.log: -321> 2017-05-04 23:52:47.485694 ffff81e9dcb0 -1 filestore(td/osd-dup/0) error (24) Too many open files not handled on operation 0xaaab1eb798c4 (3340.1.0, or op 0, counting from 0) osd.0.log: -280> 2017-05-04 23:52:47.485948 ffff7c69dcb0 -1 filestore(td/osd-dup/0) error (24) Too many open files not handled on operation 0xaaab1f540484 (3341.1.0, or op 0, counting from 0) osd.1.log:2017-05-04 23:52:47.484736 ffffaa9edcb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaaac95538c4 (3325.1.0, or op 0, counting from 0) osd.1.log:2017-05-04 23:52:47.485750 ffffa51edcb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaaac9944b20 (3327.0.0, or op 0, counting from 0) osd.1.log: -3940> 2017-05-04 23:52:47.484736 ffffaa9edcb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaaac95538c4 (3325.1.0, or op 0, counting from 0) osd.1.log: -3807> 2017-05-04 23:52:47.485750 ffffa51edcb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaaac9944b20 (3327.0.0, or op 0, counting from 0) osd.2.log:2017-05-04 23:52:47.487961 ffff74b95cb0 -1 filestore(td/osd-dup/2) error (24) Too many open files not handled on operation 0xaaaac3100b44 (3303.1.0, or op 0, counting from 0) osd.2.log:2017-05-04 23:52:47.490102 ffff6f395cb0 -1 filestore(td/osd-dup/2) error (24) Too many open files not handled on operation 0xaaaac2b44404 (3307.1.0, or op 0, counting from 0) osd.2.log: -1377> 2017-05-04 23:52:47.487961 ffff74b95cb0 -1 filestore(td/osd-dup/2) error (24) Too many open files not handled on operation 0xaaaac3100b44 (3303.1.0, or op 0, counting from 0) osd.2.log: -1200> 2017-05-04 23:52:47.490102 ffff6f395cb0 -1 filestore(td/osd-dup/2) error (24) Too many open files not handled on operation 0xaaaac2b44404 (3307.1.0, or op 0, counting from 0)
#6 Updated by Dan Mick almost 7 years ago
- Status changed from New to Resolved
- Assignee set to Dan Mick
- Source set to Development
#7 Updated by Dan Mick almost 7 years ago
- Category set to OSD