Project

General

Profile

Actions
Copy link

Bug #27215

open

radosgw:Segmentation fault lead to rgw hangup

Added by 伟杰 谭 over 5 years ago. Updated over 5 years ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v12.2.7
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

My environment:
[root@gz-ceph-52-234 ceph]# ceph --version
ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)
[root@gz-ceph-52-234 ceph]# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)

My radosgw was not working in 1:59:17 am,and i inspected my machine but just found a few message:

in /var/log/message
Aug 24 01:59:17 gz-ceph-52-234 radosgw: ** Caught signal (Segmentation fault) *
Aug 24 01:59:17 gz-ceph-52-234 radosgw: in thread 7f82ecd40700 thread_name:msgr-worker-0

in ceph-client.rgw.gz-ceph-52-234.log,log lost:
2018-08-24 01:59:17.824924 7f8277a50700 1 civetweb: 0x7f8301688000: 172.25.52.188 - - [24/Aug/2018:01:59:17 +0800] "PUT /2018-08-24-01/5b4eb9f3b4779b9c8392ae9f/15350471575440.44546559477692993.jpg HTTP/1.1" 200 0 - aws-sdk-nodejs/2.262.1 linux/v8.11.3 promise
2018-08-24 01:59:17.839995 7f8268231700 1 civetweb: 0x7f8301762000: 172.25.52.188 - - [24/Aug/2018:01:59:17 +0800] "PUT /2018-08-24-01/5aae03eb8f3a668795f530c8/15350471548240.9385758989251696.jpg HTTP/1.1" 200 0 - aws-sdk-nodejs/2.262.1 linux/v8.11.3 promise
2018-08-24 03:06:02.073940 7f8291a84700 -1 received signal: Hangup from PID: 2532943 task name: killall -q -1 ceph-mon ceph-mgr ceph-mds ceph-osd ceph-fuse radosgw UID: 0

Actions
Copy link
 #1

Updated by Kefu Chai over 5 years ago

  • Project changed from RADOS to rgw
Actions
Copy link
 #2

Updated by Abhishek Lekshmanan over 5 years ago

Can you paste the backtrace in the logs to determine possible causes

Actions
Copy link
 #3

Updated by Abhishek Lekshmanan over 5 years ago

  • Status changed from New to Need More Info
Actions
Copy link
 #4

Updated by 伟杰 谭 over 5 years ago

today i found my cluster's radosgw blocked again.
Missing separate debuginfos, use: debuginfo-install ceph-radosgw-12.2.7-0.el7.x86_64
(gdb) bt
#0 0x00007f4b2b9ba7fd in read () from /lib64/libpthread.so.0
#1 0x00007f4b2335070c in safe_read () from /usr/lib64/ceph/libceph-common.so.0
#2 0x00007f4b23350759 in safe_read_exact () from /usr/lib64/ceph/libceph-common.so.0
#3 0x00007f4b2cfc9f06 in main ()

there is no more info.Then i try to figure out more detail:
perf top -p 397100

Actions
Copy link

Also available in: Atom PDF