Bug #6240
rgw: invalid read on addr in msgr via objecter
0%
Description
<unique>0x3</unique> <tid>1</tid> <kind>InvalidRead</kind> <what>Invalid read of size 8</what> <stack> <frame> <ip>0x76A0A0E</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>sprintf</fn> <dir>/build/buildd/eglibc-2.15/stdio-common</dir> <file>sprintf.c</file> <line>38</line> </frame> <frame> <ip>0x774F463</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>inet_ntop</fn> <dir>/build/buildd/eglibc-2.15/resolv</dir> <file>inet_ntop.c</file> <line>99</line> </frame> <frame> <ip>0x775FE2F</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>getnameinfo</fn> <dir>/build/buildd/eglibc-2.15/inet</dir> <file>getnameinfo.c</file> <line>357</line> </frame> <frame> <ip>0x5144E6E</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>operator<<(std::ostream&, sockaddr_storage const&)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/msg</dir> <file>msg_types.cc</file> <line>149</line> </frame> <frame> <ip>0x513A523</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./msg</dir> <file>msg_types.h</file> <line>352</line> </frame> <frame> <ip>0x514027F</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>SimpleMessenger::_send_message(Message*, Connection*, bool)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/msg</dir> <file>SimpleMessenger.cc</file> <line>127</line> </frame> <frame> <ip>0x4FC4116</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>Objecter::send_op(Objecter::Op*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./msg</dir> <file>Messenger.h</file> <line>411</line> </frame> <frame> <ip>0x4FC8695</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>Objecter::_op_submit(Objecter::Op*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/osdc</dir> <file>Objecter.cc</file> <line>1272</line> </frame> <frame> <ip>0x4FB31B2</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>librados::IoCtxImpl::operate_read(object_t const&, ObjectOperation*, ceph::buffer::list*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/./osdc</dir> <file>Objecter.h</file> <line>1307</line> </frame> <frame> <ip>0x4F90305</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>librados::IoCtx::operate(std::string const&, librados::ObjectReadOperation*, ceph::buffer::list*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/librados</dir> <file>librados.cc</file> <line>903</line> </frame> <frame> <ip>0x4F7211</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRados::obj_stat(void*, rgw_obj&, unsigned long*, long*, unsigned long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, ceph::buffer::list*, RGWObjVersionTracker*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/rgw</dir> <file>rgw_rados.cc</file> <line>4522</line> </frame> <frame> <ip>0x51356D</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWCache<RGWRados>::obj_stat(void*, rgw_obj&, unsigned long*, long*, unsigned long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, ceph::buffer::list*, RGWObjVersionTracker*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-25-g1a96510/src/rgw</dir> <file>rgw_cache.h</file> <line>487</line> </frame> </stack> <auxwhat>Address 0x7feffdcd8 is on thread 1's stack</auxwhat> </error>
runs were
http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21761/
http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21758/
http://qa-proxy.ceph.com/teuthology/teuthology-2013-09-04_20:01:08-rgw-dumpling-testing-basic-plana/21759/
History
#1 Updated by Sage Weil over 10 years ago
- Project changed from Ceph to rgw
#2 Updated by Ian Colle over 10 years ago
- Assignee set to Yehuda Sadeh
#3 Updated by Sage Weil over 10 years ago
full trace:<error> <unique>0x3</unique> <tid>1</tid> <kind>InvalidRead</kind> <what>Invalid read of size 8</what> <stack> <frame> <ip>0x76A0A0E</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>sprintf</fn> <dir>/build/buildd/eglibc-2.15/stdio-common</dir> <file>sprintf.c</file> <line>38</line> </frame> <frame> <ip>0x774F463</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>inet_ntop</fn> <dir>/build/buildd/eglibc-2.15/resolv</dir> <file>inet_ntop.c</file> <line>99</line> </frame> <frame> <ip>0x775FE2F</ip> <obj>/lib/x86_64-linux-gnu/libc-2.15.so</obj> <fn>getnameinfo</fn> <dir>/build/buildd/eglibc-2.15/inet</dir> <file>getnameinfo.c</file> <line>357</line> </frame> <frame> <ip>0x5144E6E</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>operator<<(std::ostream&, sockaddr_storage const&)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/msg</dir> <file>msg_types.cc</file> <line>149</line> </frame> <frame> <ip>0x513A523</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./msg</dir> <file>msg_types.h</file> <line>352</line> </frame> <frame> <ip>0x514027F</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>SimpleMessenger::_send_message(Message*, Connection*, bool)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/msg</dir> <file>SimpleMessenger.cc</file> <line>127</line> </frame> <frame> <ip>0x4FC4116</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>Objecter::send_op(Objecter::Op*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./msg</dir> <file>Messenger.h</file> <line>411</line> </frame> <frame> <ip>0x4FC8695</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>Objecter::_op_submit(Objecter::Op*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/osdc</dir> <file>Objecter.cc</file> <line>1272</line> </frame> <frame> <ip>0x4FB31B2</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>librados::IoCtxImpl::operate_read(object_t const&, ObjectOperation*, ceph::buffer::list*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/./osdc</dir> <file>Objecter.h</file> <line>1307</line> </frame> <frame> <ip>0x4F90305</ip> <obj>/usr/lib/librados.so.2.0.0</obj> <fn>librados::IoCtx::operate(std::string const&, librados::ObjectReadOperation*, ceph::buffer::list*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/librados</dir> <file>librados.cc</file> <line>903</line> </frame> <frame> <ip>0x4F74B1</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRados::obj_stat(void*, rgw_obj&, unsigned long*, long*, unsigned long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, ceph::buffer::list*, RGWObjVersionTracker*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>4529</line> </frame> <frame> <ip>0x51358D</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWCache<RGWRados>::obj_stat(void*, rgw_obj&, unsigned long*, long*, unsigned long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, ceph::buffer::list*, RGWObjVersionTracker*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_cache.h</file> <line>487</line> </frame> <frame> <ip>0x4F8AAA</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRados::get_obj_state(RGWRadosCtx*, rgw_obj&, RGWObjState**, RGWObjVersionTracker*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>3257</line> </frame> <frame> <ip>0x4FD2B7</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRados::prepare_get_obj(void*, rgw_obj&, long*, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, long const*, long const*, long*, char const*, char const*, unsigned long*, unsigned long*, RGWObjVersionTracker*, void**, rgw_err*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>3611</line> </frame> <frame> <ip>0x4DEECF</ip> <obj>/usr/bin/radosgw</obj> <fn>rgw_get_system_obj(RGWRados*, void*, rgw_bucket&, std::string const&, ceph::buffer::list&, RGWObjVersionTracker*, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_tools.cc</file> <line>49</line> </frame> <frame> <ip>0x4E7628</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRegion::read_default(RGWDefaultRegionInfo&)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>105</line> </frame> <frame> <ip>0x4F1822</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRegion::init(CephContext*, RGWRados*, bool)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>158</line> </frame> <frame> <ip>0x4F3D21</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWRados::init_complete()</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.cc</file> <line>885</line> </frame> <frame> <ip>0x4ED7F9</ip> <obj>/usr/bin/radosgw</obj> <fn>RGWStoreManager::init_storage_provider(CephContext*, bool)</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.h</file> <line>922</line> </frame> <frame> <ip>0x45F750</ip> <obj>/usr/bin/radosgw</obj> <fn>main</fn> <dir>/srv/autobuild-ceph/gitbuilder.git/build/out~/ceph-0.67.2-27-g4261eb5/src/rgw</dir> <file>rgw_rados.h</file> <line>1453</line> </frame> </stack> <auxwhat>Address 0x7feffdd08 is on thread 1's stack</auxwhat> </error>
#4 Updated by Sage Weil over 10 years ago
the first difference i see between a passing and failing run is that the passing run gets ENOENT when reading default.region but the failing one gets a success. from the teuth log, it looks like that's because the region is created via radosgw-admin user create shortly after radosgw has started.
#5 Updated by Yehuda Sadeh over 10 years ago
What I see in the failed case is this:
2013-09-09 08:59:46.873170 4041280 -1 error storing region info: (17) File exists
iirc, that means that there's a racing default region creation. It seems that we run multiple gateways on that test.
#6 Updated by Sage Weil over 10 years ago
- File rgw.fail.log.gz added
- File rgw.ok.log.gz added
#7 Updated by Sage Weil over 10 years ago
- Status changed from New to 7
valgrind shows
Address 0x7feffdd78 is on thread 1's stack
and we verified this is deeper in the stack than the operator<< that is outputting the entity_addr_t. something stupid with the precise glibc.
#8 Updated by Sage Weil over 10 years ago
- Status changed from 7 to Resolved
pushed valgrind suppressions to all relevant teuthology.git branches