Project

General

Profile

Bug #12383

"Segmentation fault rbd()"

Added by Yuri Weinstein over 6 years ago. Updated over 2 years ago.

Status:
Won't Fix
Priority:
Low
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
upgrade/hammer
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-07-16_16:51:31-upgrade:hammer-hammer-distro-basic-vps/
Job: [975862]
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-07-16_16:51:31-upgrade:hammer-hammer-distro-basic-vps/975862/teuthology.log

Importing image: 50% complete...
Importing image: 100% complete...
Importing image: 100% complete...done.
2015-07-16T17:21:49.028 INFO:tasks.workunit.client.0.vpm053.stderr:+ grep -i 2048k
2015-07-16T17:21:49.029 INFO:tasks.workunit.client.0.vpm053.stderr:+ grep sparse1
2015-07-16T17:21:49.031 INFO:tasks.workunit.client.0.vpm053.stderr:+ rbd ls -l
2015-07-16T17:21:49.092 INFO:tasks.workunit.client.0.vpm053.stderr:*** Caught signal (Segmentation fault) **
2015-07-16T17:21:49.092 INFO:tasks.workunit.client.0.vpm053.stderr: in thread 7fea3f2e3780
2015-07-16T17:21:49.174 INFO:teuthology.orchestra.run.vpm053.stdout:Preparing to replace librbd1 0.94.1-1precise (using .../librbd1_0.94.2-103-g5e399b0-1precise_amd64.deb) ...
2015-07-16T17:21:49.178 INFO:teuthology.orchestra.run.vpm053.stdout:Unpacking replacement librbd1 ...
2015-07-16T17:21:49.222 INFO:tasks.workunit.client.0.vpm053.stderr: ceph version 0.94.2-103-g5e399b0 (5e399b035d7cf861cf66a8ead00b388c4857cbb6)
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 1: rbd() [0x599e05]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 2: (()+0xfcb0) [0x7fea39b48cb0]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 3: (()+0x170fcb) [0x7fea3a0dffcb]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 4: (()+0x13cef8) [0x7fea3a0abef8]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 5: (()+0x6ce3f) [0x7fea39fdbe3f]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 6: (main()+0x2b8f) [0x4bbeaf]
2015-07-16T17:21:49.223 INFO:tasks.workunit.client.0.vpm053.stderr: 7: (__libc_start_main()+0xed) [0x7fea3864e76d]
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr: 8: rbd() [0x4c19b9]
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr:2015-07-17 00:21:49.208439 7fea3f2e3780 -1 *** Caught signal (Segmentation fault) **
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr: in thread 7fea3f2e3780
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr:
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr: ceph version 0.94.2-103-g5e399b0 (5e399b035d7cf861cf66a8ead00b388c4857cbb6)
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr: 1: rbd() [0x599e05]
2015-07-16T17:21:49.224 INFO:tasks.workunit.client.0.vpm053.stderr: 2: (()+0xfcb0) [0x7fea39b48cb0]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 3: (()+0x170fcb) [0x7fea3a0dffcb]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 4: (()+0x13cef8) [0x7fea3a0abef8]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 5: (()+0x6ce3f) [0x7fea39fdbe3f]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 6: (main()+0x2b8f) [0x4bbeaf]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 7: (__libc_start_main()+0xed) [0x7fea3864e76d]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: 8: rbd() [0x4c19b9]
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-07-16T17:21:49.225 INFO:tasks.workunit.client.0.vpm053.stderr:
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:--- begin dump of recent events ---
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -22> 2015-07-17 00:21:49.062404 7fea3f2e3780  5 asok(0x44ca6a0) register_command perfcounters_dump hook 0x44cc870
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -21> 2015-07-17 00:21:49.062431 7fea3f2e3780  5 asok(0x44ca6a0) register_command 1 hook 0x44cc870
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -20> 2015-07-17 00:21:49.062438 7fea3f2e3780  5 asok(0x44ca6a0) register_command perf dump hook 0x44cc870
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -19> 2015-07-17 00:21:49.062444 7fea3f2e3780  5 asok(0x44ca6a0) register_command perfcounters_schema hook 0x44cc870
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -18> 2015-07-17 00:21:49.062448 7fea3f2e3780  5 asok(0x44ca6a0) register_command 2 hook 0x44cc870
2015-07-16T17:21:49.226 INFO:tasks.workunit.client.0.vpm053.stderr:   -17> 2015-07-17 00:21:49.062450 7fea3f2e3780  5 asok(0x44ca6a0) register_command perf schema hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -16> 2015-07-17 00:21:49.062453 7fea3f2e3780  5 asok(0x44ca6a0) register_command perf reset hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -15> 2015-07-17 00:21:49.062455 7fea3f2e3780  5 asok(0x44ca6a0) register_command config show hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -14> 2015-07-17 00:21:49.062457 7fea3f2e3780  5 asok(0x44ca6a0) register_command config set hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -13> 2015-07-17 00:21:49.062462 7fea3f2e3780  5 asok(0x44ca6a0) register_command config get hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -12> 2015-07-17 00:21:49.062464 7fea3f2e3780  5 asok(0x44ca6a0) register_command config diff hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -11> 2015-07-17 00:21:49.062466 7fea3f2e3780  5 asok(0x44ca6a0) register_command log flush hook 0x44cc870
2015-07-16T17:21:49.227 INFO:tasks.workunit.client.0.vpm053.stderr:   -10> 2015-07-17 00:21:49.062468 7fea3f2e3780  5 asok(0x44ca6a0) register_command log dump hook 0x44cc870
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -9> 2015-07-17 00:21:49.062470 7fea3f2e3780  5 asok(0x44ca6a0) register_command log reopen hook 0x44cc870
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -8> 2015-07-17 00:21:49.078770 7fea3f2e3780  5 asok(0x44ca6a0) init /var/run/ceph/ceph-client.admin.10795.asok
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -7> 2015-07-17 00:21:49.078802 7fea3f2e3780  5 asok(0x44ca6a0) bind_and_listen /var/run/ceph/ceph-client.admin.10795.asok
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -6> 2015-07-17 00:21:49.078907 7fea3f2e3780  5 asok(0x44ca6a0) register_command 0 hook 0x44e8ac0
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -5> 2015-07-17 00:21:49.078925 7fea3f2e3780  5 asok(0x44ca6a0) register_command version hook 0x44e8ac0
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -4> 2015-07-17 00:21:49.078934 7fea3f2e3780  5 asok(0x44ca6a0) register_command git_version hook 0x44e8ac0
2015-07-16T17:21:49.228 INFO:tasks.workunit.client.0.vpm053.stderr:    -3> 2015-07-17 00:21:49.078942 7fea3f2e3780  5 asok(0x44ca6a0) register_command help hook 0x44cfef0
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr:    -2> 2015-07-17 00:21:49.078950 7fea3f2e3780  5 asok(0x44ca6a0) register_command get_command_descriptions hook 0x44e9690
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr:    -1> 2015-07-17 00:21:49.079288 7fea3648a700  5 asok(0x44ca6a0) entry start
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr:     0> 2015-07-17 00:21:49.208439 7fea3f2e3780 -1 *** Caught signal (Segmentation fault) **
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr: in thread 7fea3f2e3780
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr:
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr: ceph version 0.94.2-103-g5e399b0 (5e399b035d7cf861cf66a8ead00b388c4857cbb6)
2015-07-16T17:21:49.229 INFO:tasks.workunit.client.0.vpm053.stderr: 1: rbd() [0x599e05]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 2: (()+0xfcb0) [0x7fea39b48cb0]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 3: (()+0x170fcb) [0x7fea3a0dffcb]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 4: (()+0x13cef8) [0x7fea3a0abef8]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 5: (()+0x6ce3f) [0x7fea39fdbe3f]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 6: (main()+0x2b8f) [0x4bbeaf]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 7: (__libc_start_main()+0xed) [0x7fea3864e76d]
2015-07-16T17:21:49.230 INFO:tasks.workunit.client.0.vpm053.stderr: 8: rbd() [0x4c19b9]
2015-07-16T17:21:49.231 INFO:tasks.workunit.client.0.vpm053.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

History

#1 Updated by Yuri Weinstein over 6 years ago

  • Subject changed from "Segmentation fault" to "Segmentation fault rbd()"

#3 Updated by Jason Dillaman over 6 years ago

Crash occurs within 'ceph::crypto::init', most likely due to ABI difference in librados2 and some internal representation used by librbd1.

#0  ceph::crypto::init (cct=0x26a66a0) at common/ceph_crypto.cc:48
#1  0x00007ffff2da7ef8 in common_init_finish (cct=0x26a66a0, flags=0) at common/common_init.cc:118
#2  0x00007ffff2cd7e3f in librados::RadosClient::connect (this=0x26c26d0) at librados/RadosClient.cc:191
#3  0x00000000004bbeaf in main (argc=<optimized out>, argv=<optimized out>) at rbd.cc:3016

#4 Updated by Jason Dillaman over 6 years ago

Crash occurs trying to acceess md_config_t::nss_db_path, which is corrupt due to the addition of 'client_check_pool_perm' in 0.94.2 which throws off the memory layout of the 0.94.1 librados2 library that was still installed. The rbd CLI tool includes the symbol information for md_config_t, which results in the temporary conflict until librados2 is upgraded.

#5 Updated by Jason Dillaman over 6 years ago

Long term: the rbd CLI should only be able to use public API methods to avoid scenarios like this.

#6 Updated by Jason Dillaman about 6 years ago

  • Status changed from New to Fix Under Review

#7 Updated by Jason Dillaman about 6 years ago

#9 Updated by Jason Dillaman almost 6 years ago

  • Status changed from Fix Under Review to In Progress

#10 Updated by Josh Durgin almost 6 years ago

  • Priority changed from Normal to Low

Packaging should make this unhittable

#11 Updated by Jason Dillaman almost 5 years ago

  • Status changed from In Progress to 12

#12 Updated by Jason Dillaman over 2 years ago

  • Status changed from 12 to Won't Fix

Also available in: Atom PDF