Project

General

Profile

Bug #10632

ceph-objectstore-tool: crash in FileJournal.cc

Added by Sage Weil about 9 years ago. Updated about 9 years ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2015-01-23T15:16:12.671 INFO:teuthology.orchestra.run.plana22:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op remove --pgid 1.15s1'
2015-01-23T15:16:12.887 INFO:tasks.radosbench.radosbench.0.plana22.stdout:   444      16      3621      3605   32.4722         0         -   1.93311
2015-01-23T15:16:12.929 INFO:teuthology.orchestra.run.plana22.stderr:os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f97ee9fb780 time 2015-01-23 15:16:12.925387
2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr:os/FileJournal.cc: 1769: FAILED assert(0)
2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: ceph version 0.91-716-gb782030 (b782030e872254f2175196f966863a26b20f7296)
2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xaf042f]
2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x877) [0x9ed977]
2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 3: (JournalingObjectStore::journal_replay(unsigned long)+0x224) [0x95f064]
2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 4: (FileStore::mount()+0x35b1) [0x934321]
2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 5: (main()+0x11b0) [0x62c680]
2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 6: (__libc_start_main()+0xed) [0x7f97e9e4d76d]
2015-01-23T15:16:12.932 INFO:teuthology.orchestra.run.plana22.stderr: 7: ceph-objectstore-tool() [0x635379]
2015-01-23T15:16:12.932 INFO:teuthology.orchestra.run.plana22.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr:*** Caught signal (Aborted) **
2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr: in thread 7f97ee9fb780
2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: ceph version 0.91-716-gb782030 (b782030e872254f2175196f966863a26b20f7296)
2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 1: ceph-objectstore-tool() [0xa0e0c5]
2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 2: (()+0xfcb0) [0x7f97edd0fcb0]
2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 3: (gsignal()+0x35) [0x7f97e9e620d5]
2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 4: (abort()+0x17b) [0x7f97e9e6583b]
2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97ea7b469d]
2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 6: (()+0xb5846) [0x7f97ea7b2846]
2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 7: (()+0xb5873) [0x7f97ea7b2873]
2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 8: (()+0xb596e) [0x7f97ea7b296e]
2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xaf0609]
2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 10: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x877) [0x9ed977]
2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 11: (JournalingObjectStore::journal_replay(unsigned long)+0x224) [0x95f064]
2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 12: (FileStore::mount()+0x35b1) [0x934321]
2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 13: (main()+0x11b0) [0x62c680]
2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 14: (__libc_start_main()+0xed) [0x7f97e9e4d76d]
2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 15: ceph-objectstore-tool() [0x635379]

ubuntu@teuthology:/a/samuelj-2015-01-23_06:57:45-rados-master-wip-testing-vanilla-fixes-basic-multi/719387

Related issues

Related to Ceph - Bug #6003: journal Unable to read past sequence 406 ... Resolved 08/15/2013

History

#1 Updated by Samuel Just about 9 years ago

2015-02-05T01:10:12.541 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 5, 4, 1]
2015-02-05T01:10:12.541 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-02-05T01:10:12.542 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 3, 2, 5, 4, 1]
2015-02-05T01:10:18.542 INFO:tasks.ceph.osd.4:Stopped
2015-02-05T01:10:18.542 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd
2015-02-05T01:10:18.543 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
2015-02-05T01:10:18.823 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:24.568 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 0.5 --file /home/ubuntu/cephtest/data/exp.0.5.4'
2015-02-05T01:10:24.649 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:25.036 INFO:teuthology.orchestra.run.plana46.stderr:Exporting 0.5
2015-02-05T01:10:25.037 INFO:teuthology.orchestra.run.plana46.stderr:Export successful
2015-02-05T01:10:25.089 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op remove --pgid 0.5'
2015-02-05T01:10:25.190 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:25.554 INFO:teuthology.orchestra.run.plana46.stderr:os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7fa1e3729900 time 2015-02-05 01:10:25.560051
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr:os/FileJournal.cc: 1769: FAILED assert(0)
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: ceph version 0.92-818-g8530739 (8530739e54fd2546e530d8cefd41cdc4cdb587ea)
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbcedeb]
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xae3) [0xa9e493]
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 3: (JournalingObjectStore::journal_replay(unsigned long)+0x191) [0x9b6c91]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 4: (FileStore::mount()+0x3bb6) [0x987c16]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 5: (main()+0x15d9) [0x657da9]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 6: (_libc_start_main()+0xf5) [0x7fa1ddfe4ec5]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 7: ceph-objectstore-tool() [0x6742c7]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr:*** Caught signal (Aborted) *
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr: in thread 7fa1e3729900
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: ceph version 0.92-818-g8530739 (8530739e54fd2546e530d8cefd41cdc4cdb587ea)
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 1: ceph-objectstore-tool() [0xaedcfa]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 2: (()+0x10340) [0x7fa1e21d0340]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 3: (gsignal()+0x39) [0x7fa1ddff9bb9]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 4: (abort()+0x148) [0x7fa1ddffcfc8]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 5: (
_gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa1de9056b5]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 6: (()+0x5e836) [0x7fa1de903836]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 7: (()+0x5e863) [0x7fa1de903863]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 8: (()+0x5eaa2) [0x7fa1de903aa2]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 9: (ceph::__ceph_assert_fail(char const
, char const*, int, char const*)+0x278) [0xbcefd8]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 10: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xae3) [0xa9e493]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 11: (JournalingObjectStore::journal_replay(unsigned long)+0x191) [0x9b6c91]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 12: (FileStore::mount()+0x3bb6) [0x987c16]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 13: (main()+0x15d9) [0x657da9]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 14: (__libc_start_main()+0xf5) [0x7fa1ddfe4ec5]
2015-02-05T01:10:25.565 INFO:teuthology.orchestra.run.plana46.stderr: 15: ceph-objectstore-tool() [0x6742c7]
2015-02-05T01:10:25.607 INFO:tasks.rados.rados.0.burnupi16.stdout:1611: finishing write tid 1 to burnupi1623274-22
2015-02-05T01:10:25.727 INFO:tasks.rados.rados.0.burnupi16.stdout:1611: finishing write tid 2 to burnupi1623274-22
2015-02-05T01:10:28.579 INFO:tasks.ceph.osd.2.burnupi16.stderr:2015-02-05 01:10:28.588517 7fc8cf850700 -1 osd.2 144 heartbeat_check: no reply from osd.4 since back 2015-02-05 01:10:08.363677 front 2015-02-05 01:10:08.363677 (cutoff 2015-02-05 01:10:08.588514)

An interesting twist here, just before, the tool had successfully mounted the filestore to perform the export.

ubuntu@teuthology:/a/samuelj-2015-02-04_22:13:07-rados-wip-sam-working-wip-testing-vanilla-fixes-basic-multi/740076

#2 Updated by David Zafman about 9 years ago

  • Assignee set to David Zafman

#3 Updated by Samuel Just about 9 years ago

  • Status changed from New to Duplicate
  • Assignee deleted (David Zafman)

Figured it out, and it's a dup of 6003.

Also available in: Atom PDF