Project

General

Profile

Actions
Copy link

Bug #16749

closed

Ceph filestore statfs assert and osd core dump

Added by Clive Xu almost 8 years ago. Updated almost 7 years ago.

Status:
Won't Fix
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
v10.2.0
ceph-qa-suite:
ceph-deploy
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[root@web-ui ~]# ceph -v
ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)

[root@web-ui audit]# ceph -s
cluster 4b5c8c0a-ff60-454b-a1b4-9747aa737d20
health HEALTH_ERR
683 pgs are stuck inactive for more than 300 seconds
683 pgs stale
683 pgs stuck stale
3 requests are blocked > 32 sec
2/3 in osds are down
monmap e1: 1 mons at {web-ui=192.169.39.211:6789/0}
election epoch 21, quorum 0 web-ui
osdmap e724: 5 osds: 1 up, 3 in
flags sortbitwise
pgmap v82428: 832 pgs, 7 pools, 21400 MB data, 2966 objects
16140 MB used, 2680 GB / 2695 GB avail
683 stale+active+clean
149 active+clean
[root@web-ui audit]# ps -ef|grep ceph-osd
root 32733 11593 0 19:35 pts/13 00:00:00 grep --color=auto ceph-osd

Jul 15 16:58:18 web-ui ceph-osd: 0> 2016-07-15 16:58:18.542750 7f6d9af92700 -1 os/filestore/FileStore.cc: In function 'virtual int FileStore::statfs(statfs*)' thread 7f6d9af92700 time 2016-07-15 16:58:18.541330
Jul 15 16:58:18 web-ui ceph-osd: os/filestore/FileStore.cc: 706: FAILED assert(r != -2)
Jul 15 16:58:18 web-ui ceph-osd: ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Jul 15 16:58:18 web-ui ceph-osd: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x7f6dbf616b95]
Jul 15 16:58:18 web-ui ceph-osd: 2: (FileStore::statfs(statfs*)+0x5b) [0x7f6dbf2b445b]
Jul 15 16:58:18 web-ui ceph-osd: 3: (OSDService::update_osd_stat(std::vector<int, std::allocator<int> >&)+0xbd) [0x7f6dbef6643d]
Jul 15 16:58:18 web-ui ceph-osd: 4: (OSD::heartbeat()+0x210) [0x7f6dbef84110]
Jul 15 16:58:18 web-ui ceph-osd: 5: (OSD::heartbeat_entry()+0x95) [0x7f6dbef84f25]
Jul 15 16:58:18 web-ui ceph-osd: 6: (OSD::T_Heartbeat::entry()+0xd) [0x7f6dbeff072d]
Jul 15 16:58:18 web-ui ceph-osd: 7: (()+0x7dc5) [0x7f6dbd548dc5]
Jul 15 16:58:18 web-ui ceph-osd: 8: (clone()+0x6d) [0x7f6dbbbd421d]
Jul 15 16:58:18 web-ui ceph-osd: ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Jul 15 16:58:18 web-ui ceph-osd: 1: (()+0x9119aa) [0x7f6dbf5169aa]
Jul 15 16:58:18 web-ui ceph-osd: 2: (()+0xf100) [0x7f6dbd550100]
Jul 15 16:58:18 web-ui ceph-osd: 3: (gsignal()+0x37) [0x7f6dbbb135f7]
Jul 15 16:58:18 web-ui ceph-osd: 4: (abort()+0x148) [0x7f6dbbb14ce8]
Jul 15 16:58:18 web-ui ceph-osd: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f6dbf616d77]
Jul 15 16:58:18 web-ui ceph-osd: 6: (FileStore::statfs(statfs*)+0x5b) [0x7f6dbf2b445b]
Jul 15 16:58:18 web-ui ceph-osd: 7: (OSDService::update_osd_stat(std::vector<int, std::allocator<int> >&)+0xbd) [0x7f6dbef6643d]
Jul 15 16:58:18 web-ui ceph-osd: 8: (OSD::heartbeat()+0x210) [0x7f6dbef84110]
Jul 15 16:58:18 web-ui ceph-osd: 9: (OSD::heartbeat_entry()+0x95) [0x7f6dbef84f25]
Jul 15 16:58:18 web-ui ceph-osd: 10: (OSD::T_Heartbeat::entry()+0xd) [0x7f6dbeff072d]
Jul 15 16:58:18 web-ui ceph-osd: 11: (()+0x7dc5) [0x7f6dbd548dc5]
Jul 15 16:58:18 web-ui ceph-osd: 4: (abort()+0x148) [0x7f6dbbb14ce8]
Jul 15 16:58:18 web-ui ceph-osd: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f6dbf616d77]
Jul 15 16:58:18 web-ui ceph-osd: 6: (FileStore::statfs(statfs*)+0x5b) [0x7f6dbf2b445b]
Jul 15 16:58:18 web-ui ceph-osd: 7: (OSDService::update_osd_stat(std::vector<int, std::allocator<int> >&)+0xbd) [0x7f6dbef6643d]
Jul 15 16:58:18 web-ui ceph-osd: 8: (OSD::heartbeat()+0x210) [0x7f6dbef84110]
Jul 15 16:58:18 web-ui ceph-osd: 9: (OSD::heartbeat_entry()+0x95) [0x7f6dbef84f25]
Jul 15 16:58:18 web-ui ceph-osd: 10: (OSD::T_Heartbeat::entry()+0xd) [0x7f6dbeff072d]
Jul 15 16:58:18 web-ui ceph-osd: 11: (()+0x7dc5) [0x7f6dbd548dc5]
Jul 15 16:58:18 web-ui ceph-osd: 12: (clone()+0x6d) [0x7f6dbbbd421d]
Jul 15 16:58:18 web-ui ceph-osd: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f6dbf616d77]
Jul 15 16:58:18 web-ui ceph-osd: ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Jul 15 16:58:18 web-ui ceph-osd: 1: (()+0x9119aa) [0x7f9abf80e9aa]
Jul 15 16:58:18 web-ui ceph-osd: 2: (()+0xf100) [0x7f9abd848100]
Jul 15 16:58:18 web-ui ceph-osd: 3: (gsignal()+0x37) [0x7f9abbe0b5f7]
Jul 15 16:58:18 web-ui ceph-osd: 4: (abort()+0x148) [0x7f9abbe0cce8]
Jul 15 16:58:18 web-ui ceph-osd: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f9abf90ed77]
Jul 15 16:58:18 web-ui ceph-osd: 6: (FileStore::statfs(statfs*)+0x5b) [0x7f9abf5ac45b]
Jul 15 16:58:18 web-ui ceph-osd: ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Jul 15 16:58:18 web-ui ceph-osd: 1: (()+0x9119aa) [0x7f9abf80e9aa]
Jul 15 16:58:18 web-ui ceph-osd: 4: (abort()+0x148) [0x7f9abbe0cce8]
Jul 15 16:58:18 web-ui ceph-osd: 3: (gsignal()+0x37) [0x7f6dbbb135f7]
Jul 15 16:58:18 web-ui ceph-osd: 10: (OSD::T_Heartbeat::entry()+0xd) [0x7f9abf2e872d]
Jul 15 16:58:18 web-ui ceph-osd: 11: (()+0x7dc5) [0x7f9abd840dc5]
Jul 15 16:58:18 web-ui ceph-osd: 12: (clone()+0x6d) [0x7f9abbecc21d]
Jul 15 16:58:18 web-ui ceph-osd: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 15 16:58:18 web-ui ceph-osd: 0> 2016-07-15 16:58:18.095003 7f9a9b28a700 -1 ** Caught signal (Aborted) *
Jul 15 16:58:18 web-ui ceph-osd: in thread 7f9a9b28a700 thread_name:osd_srv_heartbt
Jul 15 16:58:18 web-ui ceph-osd: ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)
Jul 15 16:58:18 web-ui ceph-osd: 1: (()+0x9119aa) [0x7f9abf80e9aa]
Jul 15 16:58:18 web-ui ceph-osd: 2: (()+0xf100) [0x7f9abd848100]
Jul 15 16:58:18 web-ui ceph-osd: 3: (gsignal()+0x37) [0x7f9abbe0b5f7]
Jul 15 16:58:18 web-ui ceph-osd: 4: (abort()+0x148) [0x7f9abbe0cce8]
Jul 15 16:58:18 web-ui ceph-osd: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x267) [0x7f9abf90ed77]
Jul 15 16:58:18 web-ui systemd: holdoff time over, scheduling restart.
Jul 15 16:58:18 web-ui systemd: failed to run 'start-pre' task: No such file or directory
Jul 15 16:58:18 web-ui systemd: Failed to start Ceph object storage daemon.
Jul 15 16:58:18 web-ui systemd: Unit entered failed state.
Jul 15 16:58:18 web-ui systemd: failed.
Jul 15 16:58:18 web-ui systemd: Starting Ceph object storage daemon...
Jul 15 16:58:18 web-ui ceph-osd: 6: (FileStore::statfs(statfs*)+0x5b) [0x7f9abf5ac45b]
Jul 15 16:58:18 web-ui ceph-osd: 7: (OSDService::update_osd_stat(std::vector<int, std::allocator<int> >&)+0xbd) [0x7f9abf25e43d]
Jul 15 16:58:18 web-ui ceph-osd: 8: (OSD::heartbeat()+0x210) [0x7f9abf27c110]
Jul 15 16:58:18 web-ui ceph-osd: 9: (OSD::heartbeat_entry()+0x95) [0x7f9abf27cf25]
Jul 15 16:58:18 web-ui ceph-osd: 10: (OSD::T_Heartbeat::entry()+0xd) [0x7f9abf2e872d]
Jul 15 16:58:18 web-ui ceph-osd: 11: (()+0x7dc5) [0x7f9abd840dc5]
Jul 15 16:58:18 web-ui ceph-osd: 12: (clone()+0x6d) [0x7f9abbecc21d]
Jul 15 16:58:18 web-ui ceph-osd: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Jul 15 16:58:18 web-ui kernel: libceph: osd4 192.169.39.211:6812 socket closed (con state OPEN)
Jul 15 16:58:18 web-ui systemd: : main process exited, code=killed, status=6/ABRT
Jul 15 16:58:18 web-ui systemd: Unit entered failed state.
Jul 15 16:58:18 web-ui systemd: failed.
Jul 15 16:58:18 web-ui kubelet: E0715 16:58:18.271551 7222 fs.go:211] Stat fs failed. Error: no such file or directory
Jul 15 16:58:18 web-ui kubelet: E0715 16:58:18.271575 7222 fs.go:211] Stat fs failed. Error: no such file or directory
Jul 15 16:58:18 web-ui kubelet: E0715 16:58:18.271582 7222 fs.go:211] Stat fs failed. Error: no such file or directory
Jul 15 16:58:18 web-ui kubelet: E0715 16:58:18.322560 7222 fs.go:211] Stat fs failed. Error: no such file or directory
Jul 15 16:58:18 web-ui kubelet: E0715 16:58:18.322593 7222 fs.go:211] Stat fs failed. Error: no such file or directory
Jul 15 16:58:18 web-ui systemd: holdoff time over, scheduling restart.
Jul 15 16:58:18 web-ui systemd: holdoff time over, scheduling restart.
Jul 15 16:58:18 web-ui systemd: failed to run 'start-pre' task: No such file or directory
Jul 15 16:58:18 web-ui systemd: Failed to start Ceph object storage daemon.
Jul 15 16:58:18 web-ui systemd: Unit entered failed state.
Jul 15 16:58:18 web-ui systemd: failed.

Actions
Copy link
 #1

Updated by Josh Durgin almost 7 years ago

  • Status changed from New to Won't Fix

This looks like a configuration issue where the statfs call or access to the disk wasn't allowed for your container.

Actions
Copy link

Also available in: Atom PDF