Project

General

Profile

Bug #21154

use-after-free from RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR

Added by Sage Weil over 6 years ago. Updated about 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
luminous mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  <kind>InvalidWrite</kind>
  <what>Invalid write of size 4</what>
  <stack>
    <frame>
      <ip>0x4F1160D</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
    </frame>
    <frame>
      <ip>0x4F1E1C2</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
    </frame>
    <frame>
      <ip>0x4F24659</ip>
      <obj>/usr/lib64/librados.so.2.0.0</obj>
    </frame>
    <frame>
      <ip>0x5FD1F5D</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.0</obj>
      <fn>DispatchQueue::fast_dispatch(Message*)</fn>
    </frame>
    <frame>
      <ip>0x60D6497</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.0</obj>
      <fn>AsyncConnection::process()</fn>
    </frame>
    <frame>
      <ip>0x60E67A8</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.0</obj>
      <fn>EventCenter::process_events(int, std::chrono::duration&lt;unsigned long, std::ratio&lt;1l, 1000000000l&gt; &gt;*)</fn>
    </frame>
    <frame>
      <ip>0x60E9EDD</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.0</obj>
    </frame>
    <frame>
      <ip>0x106B622F</ip>
      <obj>/usr/lib64/libstdc++.so.6.0.19</obj>
    </frame>
    <frame>
      <ip>0x5A89DC4</ip>
      <obj>/usr/lib64/libpthread-2.17.so</obj>
      <fn>start_thread</fn>
    </frame>
    <frame>
      <ip>0x10F1973C</ip>
      <obj>/usr/lib64/libc-2.17.so</obj>
      <fn>clone</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x33c9b3d4 is 1,412 bytes inside a block of size 1,496 free'd</auxwhat>
  <stack>
    <frame>
      <ip>0x4C2918D</ip>
      <obj>/usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete(void*)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.11.0/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>576</line>
    </frame>
    <frame>
      <ip>0x3A9443</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR()</fn>
    </frame>
    <frame>
      <ip>0x32C959</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RefCountedObject::put() const</fn>
    </frame>
    <frame>
      <ip>0x39CE08</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWCoroutinesStack::~RGWCoroutinesStack()</fn>
    </frame>
    <frame>
      <ip>0x39CEB8</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWCoroutinesStack::~RGWCoroutinesStack()</fn>
    </frame>
    <frame>
      <ip>0x32C959</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RefCountedObject::put() const</fn>
    </frame>
    <frame>

/a/sage-2017-08-26_20:43:41-rgw-luminous-distro-basic-smithi/1568205

Related issues

Copied to rgw - Backport #36538: luminous: use-after-free from RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR Rejected
Copied to rgw - Backport #36539: mimic: use-after-free from RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR Resolved

History

#1 Updated by Casey Bodley over 6 years ago

  • Assignee set to Casey Bodley
  • Priority changed from Urgent to High

investigated but couldn't find any obvious errors in ref counting. moving to high since it isn't a blocker for luminous

#2 Updated by Casey Bodley over 6 years ago

  • Status changed from New to 12

#3 Updated by Matt Benjamin over 6 years ago

  • Priority changed from High to Normal

#4 Updated by Casey Bodley over 6 years ago

it looks like this is happening because the parent coroutine RGWDataSyncShardCR is passing pointers to its members (entries and error_entries) to child coroutine RGWRadosGetOmapKeysCR, without any guarantee that the parent outlives the child

#5 Updated by Yehuda Sadeh over 6 years ago

  • Status changed from 12 to In Progress

#6 Updated by Casey Bodley over 6 years ago

  • Status changed from In Progress to Fix Under Review
  • Backport set to jewel luminous

#7 Updated by Casey Bodley over 5 years ago

  • Status changed from Fix Under Review to 7

#8 Updated by Casey Bodley over 5 years ago

  • Status changed from 7 to Pending Backport
  • Backport changed from jewel luminous to luminous mimic

backports need fix from http://tracker.ceph.com/issues/36537 as well

#9 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36538: luminous: use-after-free from RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR added

#10 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #36539: mimic: use-after-free from RGWRadosGetOmapKeysCR::~RGWRadosGetOmapKeysCR added

#11 Updated by Nathan Cutler about 3 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF