Project

General

Profile

Actions
Copy link

Bug #20360

open

rados/verify valgrind tests: osds fail to start (xenial valgrind)

Added by Sage Weil almost 7 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2017-06-20T18:16:03.097 INFO:tasks.ceph.osd.0:Started
...
2017-06-20T18:16:42.924 INFO:tasks.ceph.osd.0.smithi202.stderr:daemon-helper: command crashed with signal 9

no core

osd log show successful mkfs finishing at 

2017-06-20 18:15:26.222581 7fb0ad506c40 -1 created object store /var/lib/ceph/osd/ceph-0 for osd.0 fsid 6d5f37ab-680e-4909-877f-f53d1f982003
2017-06-20 18:15:26.222629 7fb0ad506c40 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory
2017-06-20 18:15:26.222812 7fb0ad506c40 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring

and nothing after that.

always xenial? these 3 cases so far are:

/a/sage-2017-06-20_06:19:52-rados-wip-sage-testing-distro-basic-smithi/1304118
/a/sage-2017-06-20_16:21:45-rados-wip-sage-testing2-distro-basic-smithi/1305550
/a/sage-2017-06-20_16:21:45-rados-wip-sage-testing2-distro-basic-smithi/1305676

Related issues 1 (0 open — 1 closed)

Related to Ceph - Bug #18126: Illegal instruction from Messenger::create -> std::random_device::_M_getvalResolvedDavid Galloway

Actions
Actions
Copy link
 #1

Updated by Sage Weil almost 7 years ago

related? also started seeing these:

2017-06-20T20:50:06.635 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem <<< SHOW_SEGMENTS: out_of_memory (88 segments)
2017-06-20T20:50:06.635 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem 3 segment names in 3 slots
2017-06-20T20:50:06.635 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem freelist is empty
2017-06-20T20:50:06.635 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem (0,4,3) /usr/lib/valgrind/memcheck-amd64-linux
2017-06-20T20:50:06.636 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem (1,47,3) /usr/bin/ceph-osd
2017-06-20T20:50:06.636 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem (2,69,3) /lib/x86_64-linux-gnu/ld-2.23.so
2017-06-20T20:50:06.636 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   0: RSVN 0000000000-0000107fff 1081344 ----- SmFixed
2017-06-20T20:50:06.636 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   1: file 0000108000-0001292fff     17m r-x-- d=0x801 i=2113798 o=0       (1,47)
2017-06-20T20:50:06.636 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   2: RSVN 0001293000-0001491fff 2093056 ----- SmFixed
2017-06-20T20:50:06.637 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   3: file 0001492000-00014cafff  233472 rw--- d=0x801 i=2113798 o=18391040 (1,47)
2017-06-20T20:50:06.637 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   4: anon 00014cb000-00096f5fff    130m rw---
2017-06-20T20:50:06.637 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   5: file 00096f6000-000971bfff  155648 r-x-- d=0x801 i=54526117 o=0       (2,69)
2017-06-20T20:50:06.637 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   6:      000971c000-000991afff 2093056
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   7: file 000991b000-000991cfff    8192 rw--- d=0x801 i=54526117 o=151552  (2,69)
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   8: anon 000991d000-000991dfff    4096 rw---
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem   9: anon 000991e000-000991efff    4096 rwx--
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  10: RSVN 000991f000-000a11dfff 8384512 ----- SmLower
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  11:      000a11e000-0037ffffff    734m
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  12: FILE 0038000000-003822dfff 2285568 r-x-- d=0x801 i=2231828 o=0       (0,4)
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  13:      003822e000-003842cfff 2093056
2017-06-20T20:50:06.638 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  14: FILE 003842d000-0038432fff   24576 rw--- d=0x801 i=2231828 o=2281472 (0,4)
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  15: ANON 0038433000-0039600fff     17m rw---
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  16:      0039601000-0801ffffff  31881m
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  17: RSVN 0802000000-0802000fff    4096 ----- SmFixed
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  18: ANON 0802001000-0803b73fff     27m rwx--
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  19:      0803b74000-0803c32fff  782336
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  20: ANON 0803c33000-0804133fff 5246976 rwx--
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  21:      0804134000-0804233fff 1048576
2017-06-20T20:50:06.639 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  22: ANON 0804234000-0804551fff 3268608 rwx--
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  23:      0804552000-0804560fff   61440
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  24: ANON 0804561000-0804d12fff 8069120 rwx--
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  25:      0804d13000-0804d21fff   61440
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  26: ANON 0804d22000-08054d3fff 8069120 rwx--
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  27:      08054d4000-08054e2fff   61440
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  28: ANON 08054e3000-0805c94fff 8069120 rwx--
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  29:      0805c95000-0805ca3fff   61440
2017-06-20T20:50:06.640 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  30: ANON 0805ca4000-08069f3fff     13m rwx--
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  31:      08069f4000-0806a02fff   61440
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  32: ANON 0806a03000-08072befff 9158656 rwx--
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  33:      08072bf000-0807336fff  491520
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  34: ANON 0807337000-0811e52fff    171m rwx--
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  35:      0811e53000-0811f1dfff  831488
2017-06-20T20:50:06.641 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  36: ANON 0811f1e000-08140d1fff     33m rwx--
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  37:      08140d2000-08141d1fff 1048576
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  38: ANON 08141d2000-08142dbfff 1089536 rwx--
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  39:      08142dc000-08142f4fff  102400
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  40: ANON 08142f5000-0814cbafff      9m rwx--
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  41:      0814cbb000-0814d37fff  512000
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  42: ANON 0814d38000-0849ef3fff    849m rwx--
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  43:      0849ef4000-084a29cfff 3837952
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  44: ANON 084a29d000-084c664fff     35m rwx--
2017-06-20T20:50:06.642 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  45:      084c665000-084c730fff  835584
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  46: ANON 084c731000-08a1470fff   1357m rwx--
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  47:      08a1471000-08a147ffff   61440
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  48: ANON 08a1480000-08a21cffff     13m rwx--
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  49:      08a21d0000-08a21defff   61440
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  50: ANON 08a21df000-08a2f2efff     13m rwx--
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  51:      08a2f2f000-08a2f3dfff   61440
2017-06-20T20:50:06.643 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  52: ANON 08a2f3e000-08a3903fff      9m rwx--
2017-06-20T20:50:06.644 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  53:      08a3904000-08a3923fff  131072
2017-06-20T20:50:06.644 INFO:tasks.ceph.osd.3.smithi202.stderr:--22062:0: aspacem  54: ANON 08a3924000-08eef3ffff   1206m rwx--

smithi202 is also xenial.

Actions
Copy link
 #3

Updated by Sage Weil almost 7 years ago

  • Status changed from 12 to 7
Actions
Copy link
 #4

Updated by Sage Weil almost 7 years ago

  • Subject changed from rados/verify valgrind tests: osds fail to start to rados/verify valgrind tests: osds fail to start (xenial valgrind)
  • Status changed from 7 to 12
  • Priority changed from Immediate to High

ok, valgrind is now restricted to centos again.

Actions
Copy link
 #5

Updated by Sage Weil almost 7 years ago

1:3.12.0-1.1ubuntu1
xenial on smithi107
/a/sage-2017-06-26_14:37:54-rados-wip-sage-testing2-distro-basic-smithi/1327163

Actions
Copy link
 #6

Updated by Josh Durgin almost 7 years ago

  • Status changed from 12 to 15
  • Priority changed from High to Normal
Actions
Copy link
 #7

Updated by Sage Weil almost 7 years ago

  • Related to Bug #18126: Illegal instruction from Messenger::create -> std::random_device::_M_getval added
Actions
Copy link
 #8

Updated by Sage Weil almost 7 years ago

  • Status changed from 15 to 12
Actions
Copy link
 #9

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link

Also available in: Atom PDF