Actions
Bug #11260
closed"Cannot allocate memory" (??) in upgrade:dumpling-firefly-x-hammer-distro-basic-vps run
% Done:
0%
Source:
Q/A
Tags:
Backport:
hammer
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/dumpling-firefly-x
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
All dead jobs in this run http://pulpito.front.sepia.ceph.com/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/
One for example: 823052
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823052/
2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr:terminate called after throwing an instance of 'ceph::buffer::bad_alloc' 2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr: what(): buffer::bad_alloc 2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr:*** Caught signal (Aborted) ** 2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr: in thread 7f7457fcb700 2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr:terminate called after throwing an instance of 'ceph::buffer::bad_alloc' 2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr: what(): buffer::bad_alloc 2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr:*** Caught signal (Aborted) ** 2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr: in thread 7fdafa51c700 2015-03-27T13:54:30.840 INFO:tasks.ceph.osd.2.vpm186.stderr: ceph version 0.93-196-g6994648 (6994648bc443429dc2edfbb38fbaaa9a19e2bdd1) 2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 1: ceph-osd() [0xa47b75] 2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 2: (()+0xf710) [0x7f7487182710] 2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 3: (gsignal()+0x35) [0x7f7485e4c635] 2015-03-27T13:54:30.842 INFO:tasks.ceph.osd.2.vpm186.stderr: 4: (abort()+0x175) [0x7f7485e4de15] 2015-03-27T13:54:30.842 INFO:tasks.ceph.osd.2.vpm186.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f7486706a5d] 2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 6: (()+0xbcbe6) [0x7f7486704be6] 2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 7: (()+0xbcc13) [0x7f7486704c13] 2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 8: (()+0xbcd0e) [0x7f7486704d0e] 2015-03-27T13:54:30.844 INFO:tasks.ceph.osd.2.vpm186.stderr: 9: (ceph::buffer::create_aligned(unsigned int, unsigned int)+0x153) [0xb98f33] 2015-03-27T13:54:30.844 INFO:tasks.ceph.osd.2.vpm186.stderr: 10: (Pipe::read_message(Message**, AuthSessionHandler*)+0x219a) [0xc1506a] 2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 11: (Pipe::reader()+0xa59) [0xc16479] 2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 12: (Pipe::Reader::entry()+0xd) [0xc1984d] 2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 13: (()+0x79d1) [0x7f748717a9d1] 2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 14: (clone()+0x6d) [0x7f7485f0286d] 2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr:2015-03-27 16:54:30.835470 7f7457fcb700 -1 *** Caught signal (Aborted) ** 2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: in thread 7f7457fcb700 2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: 2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: ceph version 0.93-196-g6994648 (6994648bc443429dc2edfbb38fbaaa9a19e2bdd1) 2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: 1: ceph-osd() [0xa47b75] 2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 2: (()+0xf710) [0x7f7487182710] 2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 3: (gsignal()+0x35) [0x7f7485e4c635] 2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 4: (abort()+0x175) [0x7f7485e4de15] 2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f7486706a5d] 2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 6: (()+0xbcbe6) [0x7f7486704be6] 2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 7: (()+0xbcc13) [0x7f7486704c13] 2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 8: (()+0xbcd0e) [0x7f7486704d0e] 2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 9: (ceph::buffer::create_aligned(unsigned int, unsigned int)+0x153) [0xb98f33] 2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 10: (Pipe::read_message(Message**, AuthSessionHandler*)+0x219a) [0xc1506a] 2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 11: (Pipe::reader()+0xa59) [0xc16479] 2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 12: (Pipe::Reader::entry()+0xd) [0xc1984d] 2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 13: (()+0x79d1) [0x7f748717a9d1] 2015-03-27T13:54:30.850 INFO:tasks.ceph.osd.2.vpm186.stderr: 14: (clone()+0x6d) [0x7f7485f0286d] 2015-03-27T13:54:30.850 INFO:tasks.ceph.osd.2.vpm186.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Also:
teuthology@teuthology:~$ grep -r "Cannot allocate memory" /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/*/*.log /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823054/teuthology.log:2015-03-27T14:16:20.120 INFO:teuthology.orchestra.run.vpm016.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823055/teuthology.log:2015-03-27T13:47:45.096 INFO:teuthology.orchestra.run.vpm070.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823057/teuthology.log:2015-03-27T21:30:08.889 INFO:teuthology.orchestra.run.vpm124.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823058/teuthology.log:2015-03-27T17:40:16.992 INFO:teuthology.orchestra.run.vpm060.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory
Sage, looks like vps memory problem, but need your confirmation.
Actions