Project

General

Profile

Bug #6240

rgw: invalid read on addr in msgr via objecter

Added by Sage Weil over 10 years ago. Updated over 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

  <unique>0x3</unique>
  <tid>1</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x76A0A0E</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>sprintf</fn>
      <dir>/build/buildd/eglibc-2.15/stdio-common</dir>
      <file>sprintf.c</file>
      <line>38</line>
    </frame>
    <frame>
      <ip>0x774F463</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>inet_ntop</fn>
      <dir>/build/buildd/eglibc-2.15/resolv</dir>
      <file>inet_ntop.c</file>
      <line>99</line>
    </frame>
    <frame>
      <ip>0x775FE2F</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>getnameinfo</fn>
      <dir>/build/buildd/eglibc-2.15/inet</dir>
      <file>getnameinfo.c</file>
      <line>357</line>
    </frame>
    <frame>
      <ip>0x5144E6E</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>operator&lt;&lt;(std::ostream&amp;, sockaddr_storage const&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/msg</dir>
      <file>msg_types.cc</file>
      <line>149</line>
    </frame>
    <frame>
      <ip>0x513A523</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./msg</dir>
      <file>msg_types.h</file>
      <line>352</line>
    </frame>
    <frame>
      <ip>0x514027F</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>SimpleMessenger::_send_message(Message*, Connection*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/msg</dir>
      <file>SimpleMessenger.cc</file>
      <line>127</line>
    </frame>
    <frame>
      <ip>0x4FC4116</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>Objecter::send_op(Objecter::Op*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./msg</dir>
      <file>Messenger.h</file>
      <line>411</line>
    </frame>
    <frame>
      <ip>0x4FC8695</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>Objecter::_op_submit(Objecter::Op*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/osdc</dir>
      <file>Objecter.cc</file>
      <line>1272</line>
    </frame>
    <frame>
      <ip>0x4FB31B2</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>librados::IoCtxImpl::operate_read(object_t const&amp;, ObjectOperation*, ceph::buffer::list*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./osdc</dir>
      <file>Objecter.h</file>
      <line>1307</line>
    </frame>
    <frame>
      <ip>0x4F90305</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>librados::IoCtx::operate(std::string const&amp;, librados::ObjectReadOperation*, ceph::buffer::list*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/librados</dir>
      <file>librados.cc</file>
      <line>903</line>
    </frame>
    <frame>
      <ip>0x4F7211</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRados::obj_stat(void*, rgw_obj&amp;, unsigned long*, long*, unsigned long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*, ceph::buffer::list*, RGWObjVersionTracker*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>4522</line>
    </frame>
    <frame>
      <ip>0x51356D</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWCache&lt;RGWRados&gt;::obj_stat(void*, rgw_obj&amp;, unsigned long*, long*, unsigned long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*, ceph::buffer::list*, RGWObjVersionTracker*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/rgw</dir>
      <file>rgw_cache.h</file>
      <line>487</line>
    </frame>
  </stack>
  <auxwhat>Address 0x7feffdcd8 is on thread 1's stack</auxwhat>
</error>

runs were

http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21761/
http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21758/
http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21759/

rgw.fail.log.gz (5.35 MB) Sage Weil, 09/09/2013 09:52 AM

rgw.ok.log.gz (5.28 MB) Sage Weil, 09/09/2013 09:52 AM

History

#1 Updated by Sage Weil over 10 years ago

  • Project changed from Ceph to rgw

#2 Updated by Ian Colle over 10 years ago

  • Assignee set to Yehuda Sadeh

#3 Updated by Sage Weil over 10 years ago

full trace:<error>
  <unique>0x3</unique>
  <tid>1</tid>
  <kind>InvalidRead</kind>
  <what>Invalid read of size 8</what>
  <stack>
    <frame>
      <ip>0x76A0A0E</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>sprintf</fn>
      <dir>/build/buildd/eglibc-2.15/stdio-common</dir>
      <file>sprintf.c</file>
      <line>38</line>
    </frame>
    <frame>
      <ip>0x774F463</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>inet_ntop</fn>
      <dir>/build/buildd/eglibc-2.15/resolv</dir>
      <file>inet_ntop.c</file>
      <line>99</line>
    </frame>
    <frame>
      <ip>0x775FE2F</ip>
      <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj>
      <fn>getnameinfo</fn>
      <dir>/build/buildd/eglibc-2.15/inet</dir>
      <file>getnameinfo.c</file>
      <line>357</line>
    </frame>
    <frame>
      <ip>0x5144E6E</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>operator&lt;&lt;(std::ostream&amp;, sockaddr_storage const&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/msg</dir>
      <file>msg_types.cc</file>
      <line>149</line>
    </frame>
    <frame>
      <ip>0x513A523</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./msg</dir>
      <file>msg_types.h</file>
      <line>352</line>
    </frame>
    <frame>
      <ip>0x514027F</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>SimpleMessenger::_send_message(Message*, Connection*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/msg</dir>
      <file>SimpleMessenger.cc</file>
      <line>127</line>
    </frame>
    <frame>
      <ip>0x4FC4116</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>Objecter::send_op(Objecter::Op*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./msg</dir>
      <file>Messenger.h</file>
      <line>411</line>
    </frame>
    <frame>
      <ip>0x4FC8695</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>Objecter::_op_submit(Objecter::Op*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/osdc</dir>
      <file>Objecter.cc</file>
      <line>1272</line>
    </frame>
    <frame>
      <ip>0x4FB31B2</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>librados::IoCtxImpl::operate_read(object_t const&amp;, ObjectOperation*, ceph::buffer::list*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./osdc</dir>
      <file>Objecter.h</file>
      <line>1307</line>
    </frame>
    <frame>
      <ip>0x4F90305</ip>
      <obj>/usr/lib/librados.so.2.0.0</obj>
      <fn>librados::IoCtx::operate(std::string const&amp;, librados::ObjectReadOperation*, ceph::buffer::list*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/librados</dir>
      <file>librados.cc</file>
      <line>903</line>
    </frame>
    <frame>
      <ip>0x4F74B1</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRados::obj_stat(void*, rgw_obj&amp;, unsigned long*, long*, unsigned long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*, ceph::buffer::list*, RGWObjVersionTracker*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>4529</line>
    </frame>
    <frame>
      <ip>0x51358D</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWCache&lt;RGWRados&gt;::obj_stat(void*, rgw_obj&amp;, unsigned long*, long*, unsigned long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*, ceph::buffer::list*, RGWObjVersionTracker*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_cache.h</file>
      <line>487</line>
    </frame>
    <frame>
      <ip>0x4F8AAA</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRados::get_obj_state(RGWRadosCtx*, rgw_obj&amp;, RGWObjState**, RGWObjVersionTracker*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>3257</line>
    </frame>
    <frame>
      <ip>0x4FD2B7</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRados::prepare_get_obj(void*, rgw_obj&amp;, long*, long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*, long const*, long const*, long*, char const*, char const*, unsigned long*, unsigned long*, RGWObjVersionTracker*, void**, rgw_err*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>3611</line>
    </frame>
    <frame>
      <ip>0x4DEECF</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>rgw_get_system_obj(RGWRados*, void*, rgw_bucket&amp;, std::string const&amp;, ceph::buffer::list&amp;, RGWObjVersionTracker*, long*, std::map&lt;std::string, ceph::buffer::list, std::less&lt;std::string&gt;, std::allocator&lt;std::pair&lt;std::string const, ceph::buffer::list&gt; &gt; &gt;*)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_tools.cc</file>
      <line>49</line>
    </frame>
    <frame>
      <ip>0x4E7628</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRegion::read_default(RGWDefaultRegionInfo&amp;)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>105</line>
    </frame>
    <frame>
      <ip>0x4F1822</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRegion::init(CephContext*, RGWRados*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>158</line>
    </frame>
    <frame>
      <ip>0x4F3D21</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWRados::init_complete()</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.cc</file>
      <line>885</line>
    </frame>
    <frame>
      <ip>0x4ED7F9</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>RGWStoreManager::init_storage_provider(CephContext*, bool)</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.h</file>
      <line>922</line>
    </frame>
    <frame>
      <ip>0x45F750</ip>
      <obj>/usr/bin/radosgw</obj>
      <fn>main</fn>
      <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir>
      <file>rgw_rados.h</file>
      <line>1453</line>
    </frame>
  </stack>
  <auxwhat>Address 0x7feffdd08 is on thread 1's stack</auxwhat>
</error>

#4 Updated by Sage Weil over 10 years ago

the first difference i see between a passing and failing run is that the passing run gets ENOENT when reading default.region but the failing one gets a success. from the teuth log, it looks like that's because the region is created via radosgw-admin user create shortly after radosgw has started.

#5 Updated by Yehuda Sadeh over 10 years ago

What I see in the failed case is this:

2013-09-09 08:59:46.873170 4041280 -1 error storing region info: (17) File exists

iirc, that means that there's a racing default region creation. It seems that we run multiple gateways on that test.

#6 Updated by Sage Weil over 10 years ago

#7 Updated by Sage Weil over 10 years ago

  • Status changed from New to 7

valgrind shows

Address 0x7feffdd78 is on thread 1's stack

and we verified this is deeper in the stack than the operator<< that is outputting the entity_addr_t. something stupid with the precise glibc.

#8 Updated by Sage Weil over 10 years ago

  • Status changed from 7 to Resolved

pushed valgrind suppressions to all relevant teuthology.git branches

Also available in: Atom PDF