Project

General

Profile

Actions
Copy link

Bug #38023

closed

segv on FileJournal::prepare_entry in bufferlist

Added by Sage Weil about 5 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
High
Assignee:
Radoslaw Zarzynski
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  -314> 2019-01-23 03:56:31.338 7f4159f6e700  5 filestore(/var/lib/ceph/osd/ceph-2) queue_transactions(2291): osr 0x55d66cbbda40 osr(5.11s1_head)
  -313> 2019-01-23 03:56:31.338 7f4159f6e700 10 journal prepare_entry [Transaction(0x55d66fc6be40)]


(gdb) bt
#0  0x00007f418376559b in raise () from /lib64/libpthread.so.0
#1  0x000055d65ff12855 in reraise_fatal (signum=11) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/global/signal_handler.cc:81
#2  handle_fatal_signal (signum=11) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/global/signal_handler.cc:298
#3  <signal handler called>
#4  0x000055d6605cec7b in crc32_iscsi_00 ()
#5  0x0000000000000ffb in ?? ()
#6  0x6d68636e65623ad5 in ?? ()
#7  0x000055d6605cec1b in ceph_crc32c_intel_fast (crc=<optimized out>, buffer=0x6d68636e65623ad5 <Address 0x6d68636e65623ad5 out of bounds>, len=<optimized out>)
    at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/common/crc32c_intel_fast.c:28
#8  0x000055d66010fdb4 in ceph_crc32c (length=<optimized out>, data=<optimized out>, crc=2301157715) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/include/crc32c.h:50
#9  ceph::buffer::list::crc32c (this=this@entry=0x7f4159f6afe0, crc=2301157715, crc@entry=0) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/common/buffer.cc:1944
#10 0x000055d65fe8cc90 in FileJournal::prepare_entry (this=0x55d66b2f4c00, tls=std::vector of length 1, capacity 1 = {...}, tbl=0x7f4159f6b170)
    at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/os/filestore/FileJournal.cc:1563
#11 0x000055d65fcf0f66 in FileStore::queue_transactions (this=0x55d66b230000, ch=..., tls=std::vector of length 0, capacity 0, osd_op=..., handle=0x0)
    at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/os/filestore/FileStore.cc:2304
#12 0x000055d65f9cbe02 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) (this=0x55d66b230000, 
    ch=..., t=<optimized out>, op=..., handle=0x0) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/os/ObjectStore.h:1448
#13 0x000055d65fb4d48f in non-virtual thunk to PrimaryLogPG::queue_transaction(ObjectStore::Transaction&&, boost::intrusive_ptr<OpRequest>) ()
#14 0x000055d65fc6ceb1 in ECBackend::dispatch_recovery_messages (this=this@entry=0x55d672b3ac00, m=..., priority=priority@entry=127) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/ECBackend.cc:547
#15 0x000055d65fc7f254 in ECBackend::_handle_message (this=0x55d672b3ac00, _op=...) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/ECBackend.cc:828
#16 0x000055d65fb5d1e7 in PGBackend::handle_message (this=<optimized out>, op=...) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/PGBackend.cc:114
#17 0x000055d65fb09c25 in PrimaryLogPG::do_request (this=0x55d67658e000, op=..., handle=...) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/PrimaryLogPG.cc:1848
#18 0x000055d65f950009 in OSD::dequeue_op (this=this@entry=0x55d66b3d2000, pg=..., op=..., handle=...) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/OSD.cc:9629
#19 0x000055d65fbdfe12 in PGOpItem::run (this=<optimized out>, osd=0x55d66b3d2000, sdata=<optimized out>, pg=..., handle=...) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/OpQueueItem.cc:24
#20 0x000055d65f96cabc in run (handle=..., pg=..., sdata=<optimized out>, osd=<optimized out>, this=0x7f4159f6b8b0) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/OpQueueItem.h:134
#21 OSD::ShardedOpWQ::_process (this=0x55d66b3d3000, thread_index=<optimized out>, hb=<optimized out>) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/osd/OSD.cc:10804
#22 0x000055d65ff67ce3 in ShardedThreadPool::shardedthreadpool_worker (this=0x55d66b3d29f8, thread_index=<optimized out>) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/common/WorkQueue.cc:311
#23 0x000055d65ff6ad80 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /usr/src/debug/ceph-14.0.1-2862-gd4c4082/src/common/WorkQueue.h:699
#24 0x00007f418375de25 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f4182626bad in clone () from /lib64/libc.so.6

/a/nojha-2019-01-23_02:37:14-rados:thrash-erasure-code-master-distro-basic-smithi/3494088

Related issues 2 (0 open — 2 closed)

Related to RADOS - Bug #37910: segv during crc of incoming message frontResolvedRadoslaw Zarzynski01/14/2019

Actions
Related to RADOS - Bug #38024: segv, heap corruption in ec encode_and_writeResolvedRadoslaw Zarzynski01/23/2019

Actions
Actions
Copy link
 #1

Updated by Sage Weil about 5 years ago

  • Related to Bug #37910: segv during crc of incoming message front added
Actions
Copy link
 #2

Updated by Sage Weil about 5 years ago

  • Related to Bug #38024: segv, heap corruption in ec encode_and_write added
Actions
Copy link
 #3

Updated by Kefu Chai about 5 years ago 

 ceph version 14.0.1-3228-g47d4a86 (47d4a86eadbe0b7681b131479eb8718cb66d1523) nautilus (dev)
 1: (()+0x11390) [0x7fc155b59390]
 2: (ceph::buffer::list::crc32c(unsigned int) const+0x6b) [0x1178f7b]
 3: (FileJournal::prepare_entry(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, ceph::buffer::list*)+0x35e) [0xecd64e]
 4: (FileStore::queue_transactions(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intr
usive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x6b3) [0xd1b763]
 5: (non-virtual thunk to PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x54)
[0xb7a6d4]
 6: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, ever
sion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0x6b0) [0xc75fd0]
 7: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0xe96) [0xad3526]
 8: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x108e) [0xb3707e]
 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x36ae) [0xb3b05e]
 10: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xd08) [0xb3d048]
 11: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x1bf) [0x968cbf]
 12: (PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x62) [0xc11c82]
 13: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xbed) [0x986a0d]
 14: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x4ac) [0xfb0c8c]
 15: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0xfb3e40]
 16: (()+0x76ba) [0x7fc155b4f6ba]

/a/kchai-2019-02-03_02:07:02-rados-wip-kefu2-testing-2019-02-03-0001-distro-basic-smithi/3544036

Actions
Copy link
 #4

Updated by Greg Farnum over 4 years ago

  • Status changed from 12 to Closed

Seems to have been resolved alongside those related tickets?

Actions
Copy link

Also available in: Atom PDF