Project

General

Profile

Bug #19866

osd-dup.sh failing with EMFILE on arm64

Added by Dan Mick almost 7 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

two of three OSDs died; one, at least, died because of

2017-05-04 23:14:12.847887 ffff76164cb0 -1 filestore(td/osd-dup/1) error (24) Too many open files not handled on operation 0xaaab29c00244 (16
00.1.0, or op 0, counting from 0)
2017-05-04 23:14:12.847914 ffff76164cb0 0 filestore(td/osd-dup/1) unexpected error code

ulimit -f is set to 1024. The test tries to warn, but nothing comes to the console. Running with ulimit -n set to 2048 is not sufficient to make the test succeed on arm64 (it dies in rados bench because the osds have died). ulimit -n 2048 makes the test succeed on x86_64.

History

#1 Updated by Dan Mick almost 7 years ago

  • Subject changed from osd-dup.sh failing with EMFILE on arm64 to osd-dup.sh failing with EMFILE

#2 Updated by Dan Mick almost 7 years ago

  • Description updated (diff)
  • Status changed from New to Rejected

#3 Updated by Dan Mick almost 7 years ago

  • Subject changed from osd-dup.sh failing with EMFILE to osd-dup.sh failing with EMFILE on arm64
  • Status changed from Rejected to New

#4 Updated by Dan Mick almost 7 years ago

Confusing myself mightily, but, yes, the osds die early on arm64 for reasons I don't yet understand.

#5 Updated by Dan Mick almost 7 years ago

dmick@omani002:~/src/ceph/build/src/test/osd/td/osd-dup$ ulimit -n
2048
dmick@omani002:~/src/ceph/build/src/test/osd/td/osd-dup$ grep "Too many" *log
osd.0.log:2017-05-04 23:52:47.485694 ffff81e9dcb0 -1 filestore(td/osd-dup/0)  error (24) Too many open files not handled on operation 0xaaab1eb798c4 (3340.1.0, or op 0, counting from 0)
osd.0.log:2017-05-04 23:52:47.485948 ffff7c69dcb0 -1 filestore(td/osd-dup/0)  error (24) Too many open files not handled on operation 0xaaab1f540484 (3341.1.0, or op 0, counting from 0)
osd.0.log:  -321> 2017-05-04 23:52:47.485694 ffff81e9dcb0 -1 filestore(td/osd-dup/0)  error (24) Too many open files not handled on operation 0xaaab1eb798c4 (3340.1.0, or op 0, counting from 0)
osd.0.log:  -280> 2017-05-04 23:52:47.485948 ffff7c69dcb0 -1 filestore(td/osd-dup/0)  error (24) Too many open files not handled on operation 0xaaab1f540484 (3341.1.0, or op 0, counting from 0)
osd.1.log:2017-05-04 23:52:47.484736 ffffaa9edcb0 -1 filestore(td/osd-dup/1)  error (24) Too many open files not handled on operation 0xaaaac95538c4 (3325.1.0, or op 0, counting from 0)
osd.1.log:2017-05-04 23:52:47.485750 ffffa51edcb0 -1 filestore(td/osd-dup/1)  error (24) Too many open files not handled on operation 0xaaaac9944b20 (3327.0.0, or op 0, counting from 0)
osd.1.log: -3940> 2017-05-04 23:52:47.484736 ffffaa9edcb0 -1 filestore(td/osd-dup/1)  error (24) Too many open files not handled on operation 0xaaaac95538c4 (3325.1.0, or op 0, counting from 0)
osd.1.log: -3807> 2017-05-04 23:52:47.485750 ffffa51edcb0 -1 filestore(td/osd-dup/1)  error (24) Too many open files not handled on operation 0xaaaac9944b20 (3327.0.0, or op 0, counting from 0)
osd.2.log:2017-05-04 23:52:47.487961 ffff74b95cb0 -1 filestore(td/osd-dup/2)  error (24) Too many open files not handled on operation 0xaaaac3100b44 (3303.1.0, or op 0, counting from 0)
osd.2.log:2017-05-04 23:52:47.490102 ffff6f395cb0 -1 filestore(td/osd-dup/2)  error (24) Too many open files not handled on operation 0xaaaac2b44404 (3307.1.0, or op 0, counting from 0)
osd.2.log: -1377> 2017-05-04 23:52:47.487961 ffff74b95cb0 -1 filestore(td/osd-dup/2)  error (24) Too many open files not handled on operation 0xaaaac3100b44 (3303.1.0, or op 0, counting from 0)
osd.2.log: -1200> 2017-05-04 23:52:47.490102 ffff6f395cb0 -1 filestore(td/osd-dup/2)  error (24) Too many open files not handled on operation 0xaaaac2b44404 (3307.1.0, or op 0, counting from 0)

#6 Updated by Dan Mick almost 7 years ago

  • Status changed from New to Resolved
  • Assignee set to Dan Mick
  • Source set to Development

#7 Updated by Dan Mick almost 7 years ago

  • Category set to OSD

Also available in: Atom PDF