Bug #10632
ceph-objectstore-tool: crash in FileJournal.cc
0%
Description
2015-01-23T15:16:12.671 INFO:teuthology.orchestra.run.plana22:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op remove --pgid 1.15s1' 2015-01-23T15:16:12.887 INFO:tasks.radosbench.radosbench.0.plana22.stdout: 444 16 3621 3605 32.4722 0 - 1.93311 2015-01-23T15:16:12.929 INFO:teuthology.orchestra.run.plana22.stderr:os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7f97ee9fb780 time 2015-01-23 15:16:12.925387 2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr:os/FileJournal.cc: 1769: FAILED assert(0) 2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: ceph version 0.91-716-gb782030 (b782030e872254f2175196f966863a26b20f7296) 2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x7f) [0xaf042f] 2015-01-23T15:16:12.930 INFO:teuthology.orchestra.run.plana22.stderr: 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x877) [0x9ed977] 2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 3: (JournalingObjectStore::journal_replay(unsigned long)+0x224) [0x95f064] 2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 4: (FileStore::mount()+0x35b1) [0x934321] 2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 5: (main()+0x11b0) [0x62c680] 2015-01-23T15:16:12.931 INFO:teuthology.orchestra.run.plana22.stderr: 6: (__libc_start_main()+0xed) [0x7f97e9e4d76d] 2015-01-23T15:16:12.932 INFO:teuthology.orchestra.run.plana22.stderr: 7: ceph-objectstore-tool() [0x635379] 2015-01-23T15:16:12.932 INFO:teuthology.orchestra.run.plana22.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. 2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion' 2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr:*** Caught signal (Aborted) ** 2015-01-23T15:16:12.933 INFO:teuthology.orchestra.run.plana22.stderr: in thread 7f97ee9fb780 2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: ceph version 0.91-716-gb782030 (b782030e872254f2175196f966863a26b20f7296) 2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 1: ceph-objectstore-tool() [0xa0e0c5] 2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 2: (()+0xfcb0) [0x7f97edd0fcb0] 2015-01-23T15:16:12.935 INFO:teuthology.orchestra.run.plana22.stderr: 3: (gsignal()+0x35) [0x7f97e9e620d5] 2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 4: (abort()+0x17b) [0x7f97e9e6583b] 2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 5: (__gnu_cxx::__verbose_terminate_handler()+0x11d) [0x7f97ea7b469d] 2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 6: (()+0xb5846) [0x7f97ea7b2846] 2015-01-23T15:16:12.936 INFO:teuthology.orchestra.run.plana22.stderr: 7: (()+0xb5873) [0x7f97ea7b2873] 2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 8: (()+0xb596e) [0x7f97ea7b296e] 2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 9: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x259) [0xaf0609] 2015-01-23T15:16:12.937 INFO:teuthology.orchestra.run.plana22.stderr: 10: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0x877) [0x9ed977] 2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 11: (JournalingObjectStore::journal_replay(unsigned long)+0x224) [0x95f064] 2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 12: (FileStore::mount()+0x35b1) [0x934321] 2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 13: (main()+0x11b0) [0x62c680] 2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 14: (__libc_start_main()+0xed) [0x7f97e9e4d76d] 2015-01-23T15:16:12.938 INFO:teuthology.orchestra.run.plana22.stderr: 15: ceph-objectstore-tool() [0x635379]ubuntu@teuthology:/a/samuelj-2015-01-23_06:57:45-rados-master-wip-testing-vanilla-fixes-basic-multi/719387
Related issues
History
#1 Updated by Samuel Just about 9 years ago
2015-02-05T01:10:12.541 INFO:tasks.thrashosds.thrasher:in_osds: [2, 3, 4, 5, 1, 0] out_osds: [] dead_osds: [] live_osds: [0, 3, 2, 5, 4, 1]
2015-02-05T01:10:12.541 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0
2015-02-05T01:10:12.542 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 3, 2, 5, 4, 1]
2015-02-05T01:10:18.542 INFO:tasks.ceph.osd.4:Stopped
2015-02-05T01:10:18.542 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd
2015-02-05T01:10:18.543 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
2015-02-05T01:10:18.823 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:24.568 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op export --pgid 0.5 --file /home/ubuntu/cephtest/data/exp.0.5.4'
2015-02-05T01:10:24.649 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:25.036 INFO:teuthology.orchestra.run.plana46.stderr:Exporting 0.5
2015-02-05T01:10:25.037 INFO:teuthology.orchestra.run.plana46.stderr:Export successful
2015-02-05T01:10:25.089 INFO:teuthology.orchestra.run.plana46:Running: 'sudo ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op remove --pgid 0.5'
2015-02-05T01:10:25.190 INFO:teuthology.orchestra.run.plana46.stderr:warning: line 25: 'osd_pool_default_size' in section 'global' redefined
2015-02-05T01:10:25.554 INFO:teuthology.orchestra.run.plana46.stderr:os/FileJournal.cc: In function 'bool FileJournal::read_entry(ceph::bufferlist&, uint64_t&, bool*)' thread 7fa1e3729900 time 2015-02-05 01:10:25.560051
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr:os/FileJournal.cc: 1769: FAILED assert(0)
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: ceph version 0.92-818-g8530739 (8530739e54fd2546e530d8cefd41cdc4cdb587ea)
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0xbcedeb]
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 2: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xae3) [0xa9e493]
2015-02-05T01:10:25.555 INFO:teuthology.orchestra.run.plana46.stderr: 3: (JournalingObjectStore::journal_replay(unsigned long)+0x191) [0x9b6c91]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 4: (FileStore::mount()+0x3bb6) [0x987c16]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 5: (main()+0x15d9) [0x657da9]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 6: (_libc_start_main()+0xf5) [0x7fa1ddfe4ec5]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: 7: ceph-objectstore-tool() [0x6742c7]
2015-02-05T01:10:25.556 INFO:teuthology.orchestra.run.plana46.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr:terminate called after throwing an instance of 'ceph::FailedAssertion'
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr:*** Caught signal (Aborted) *
2015-02-05T01:10:25.558 INFO:teuthology.orchestra.run.plana46.stderr: in thread 7fa1e3729900
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: ceph version 0.92-818-g8530739 (8530739e54fd2546e530d8cefd41cdc4cdb587ea)
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 1: ceph-objectstore-tool() [0xaedcfa]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 2: (()+0x10340) [0x7fa1e21d0340]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 3: (gsignal()+0x39) [0x7fa1ddff9bb9]
2015-02-05T01:10:25.562 INFO:teuthology.orchestra.run.plana46.stderr: 4: (abort()+0x148) [0x7fa1ddffcfc8]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 5: (_gnu_cxx::__verbose_terminate_handler()+0x155) [0x7fa1de9056b5]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 6: (()+0x5e836) [0x7fa1de903836]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 7: (()+0x5e863) [0x7fa1de903863]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 8: (()+0x5eaa2) [0x7fa1de903aa2]
2015-02-05T01:10:25.563 INFO:teuthology.orchestra.run.plana46.stderr: 9: (ceph::__ceph_assert_fail(char const, char const*, int, char const*)+0x278) [0xbcefd8]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 10: (FileJournal::read_entry(ceph::buffer::list&, unsigned long&, bool*)+0xae3) [0xa9e493]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 11: (JournalingObjectStore::journal_replay(unsigned long)+0x191) [0x9b6c91]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 12: (FileStore::mount()+0x3bb6) [0x987c16]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 13: (main()+0x15d9) [0x657da9]
2015-02-05T01:10:25.564 INFO:teuthology.orchestra.run.plana46.stderr: 14: (__libc_start_main()+0xf5) [0x7fa1ddfe4ec5]
2015-02-05T01:10:25.565 INFO:teuthology.orchestra.run.plana46.stderr: 15: ceph-objectstore-tool() [0x6742c7]
2015-02-05T01:10:25.607 INFO:tasks.rados.rados.0.burnupi16.stdout:1611: finishing write tid 1 to burnupi1623274-22
2015-02-05T01:10:25.727 INFO:tasks.rados.rados.0.burnupi16.stdout:1611: finishing write tid 2 to burnupi1623274-22
2015-02-05T01:10:28.579 INFO:tasks.ceph.osd.2.burnupi16.stderr:2015-02-05 01:10:28.588517 7fc8cf850700 -1 osd.2 144 heartbeat_check: no reply from osd.4 since back 2015-02-05 01:10:08.363677 front 2015-02-05 01:10:08.363677 (cutoff 2015-02-05 01:10:08.588514)
An interesting twist here, just before, the tool had successfully mounted the filestore to perform the export.
ubuntu@teuthology:/a/samuelj-2015-02-04_22:13:07-rados-wip-sam-working-wip-testing-vanilla-fixes-basic-multi/740076
#2 Updated by David Zafman about 9 years ago
- Assignee set to David Zafman
#3 Updated by Samuel Just about 9 years ago
- Status changed from New to Duplicate
- Assignee deleted (
David Zafman)
Figured it out, and it's a dup of 6003.