Ceph

All dead jobs in this run http://pulpito.front.sepia.ceph.com/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/

One for example: 823052
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823052/

2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr:terminate called after throwing an instance of 'ceph::buffer::bad_alloc'
2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr:  what():  buffer::bad_alloc
2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr:*** Caught signal (Aborted) **
2015-03-27T13:54:30.776 INFO:tasks.ceph.osd.2.vpm186.stderr: in thread 7f7457fcb700
2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr:terminate called after throwing an instance of 'ceph::buffer::bad_alloc'
2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr:  what():  buffer::bad_alloc
2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr:*** Caught signal (Aborted) **
2015-03-27T13:54:30.777 INFO:tasks.ceph.osd.3.vpm186.stderr: in thread 7fdafa51c700
2015-03-27T13:54:30.840 INFO:tasks.ceph.osd.2.vpm186.stderr: ceph version 0.93-196-g6994648 (6994648bc443429dc2edfbb38fbaaa9a19e2bdd1)
2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 1: ceph-osd() [0xa47b75]
2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 2: (()+0xf710) [0x7f7487182710]
2015-03-27T13:54:30.841 INFO:tasks.ceph.osd.2.vpm186.stderr: 3: (gsignal()+0x35) [0x7f7485e4c635]
2015-03-27T13:54:30.842 INFO:tasks.ceph.osd.2.vpm186.stderr: 4: (abort()+0x175) [0x7f7485e4de15]
2015-03-27T13:54:30.842 INFO:tasks.ceph.osd.2.vpm186.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f7486706a5d]
2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 6: (()+0xbcbe6) [0x7f7486704be6]
2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 7: (()+0xbcc13) [0x7f7486704c13]
2015-03-27T13:54:30.843 INFO:tasks.ceph.osd.2.vpm186.stderr: 8: (()+0xbcd0e) [0x7f7486704d0e]
2015-03-27T13:54:30.844 INFO:tasks.ceph.osd.2.vpm186.stderr: 9: (ceph::buffer::create_aligned(unsigned int, unsigned int)+0x153) [0xb98f33]
2015-03-27T13:54:30.844 INFO:tasks.ceph.osd.2.vpm186.stderr: 10: (Pipe::read_message(Message**, AuthSessionHandler*)+0x219a) [0xc1506a]
2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 11: (Pipe::reader()+0xa59) [0xc16479]
2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 12: (Pipe::Reader::entry()+0xd) [0xc1984d]
2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 13: (()+0x79d1) [0x7f748717a9d1]
2015-03-27T13:54:30.845 INFO:tasks.ceph.osd.2.vpm186.stderr: 14: (clone()+0x6d) [0x7f7485f0286d]
2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr:2015-03-27 16:54:30.835470 7f7457fcb700 -1 *** Caught signal (Aborted) **
2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: in thread 7f7457fcb700
2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr:
2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: ceph version 0.93-196-g6994648 (6994648bc443429dc2edfbb38fbaaa9a19e2bdd1)
2015-03-27T13:54:30.846 INFO:tasks.ceph.osd.2.vpm186.stderr: 1: ceph-osd() [0xa47b75]
2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 2: (()+0xf710) [0x7f7487182710]
2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 3: (gsignal()+0x35) [0x7f7485e4c635]
2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 4: (abort()+0x175) [0x7f7485e4de15]
2015-03-27T13:54:30.847 INFO:tasks.ceph.osd.2.vpm186.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x12d) [0x7f7486706a5d]
2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 6: (()+0xbcbe6) [0x7f7486704be6]
2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 7: (()+0xbcc13) [0x7f7486704c13]
2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 8: (()+0xbcd0e) [0x7f7486704d0e]
2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 9: (ceph::buffer::create_aligned(unsigned int, unsigned int)+0x153) [0xb98f33]
2015-03-27T13:54:30.848 INFO:tasks.ceph.osd.2.vpm186.stderr: 10: (Pipe::read_message(Message**, AuthSessionHandler*)+0x219a) [0xc1506a]
2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 11: (Pipe::reader()+0xa59) [0xc16479]
2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 12: (Pipe::Reader::entry()+0xd) [0xc1984d]
2015-03-27T13:54:30.849 INFO:tasks.ceph.osd.2.vpm186.stderr: 13: (()+0x79d1) [0x7f748717a9d1]
2015-03-27T13:54:30.850 INFO:tasks.ceph.osd.2.vpm186.stderr: 14: (clone()+0x6d) [0x7f7485f0286d]
2015-03-27T13:54:30.850 INFO:tasks.ceph.osd.2.vpm186.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Also:

teuthology@teuthology:~$ grep -r "Cannot allocate memory" /a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/*/*.log
/a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823054/teuthology.log:2015-03-27T14:16:20.120 INFO:teuthology.orchestra.run.vpm016.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory
/a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823055/teuthology.log:2015-03-27T13:47:45.096 INFO:teuthology.orchestra.run.vpm070.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory
/a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823057/teuthology.log:2015-03-27T21:30:08.889 INFO:teuthology.orchestra.run.vpm124.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory
/a/teuthology-2015-03-26_17:15:01-upgrade:dumpling-firefly-x-hammer-distro-basic-vps/823058/teuthology.log:2015-03-27T17:40:16.992 INFO:teuthology.orchestra.run.vpm060.stderr:OSError: librados.so.2: cannot map zero-fill pages: Cannot allocate memory

Sage, looks like vps memory problem, but need your confirmation.