Project

General

Profile

Bug #10572

rgw: crash on shutdown, when syncing user stats

Added by Yehuda Sadeh about 9 years ago. Updated almost 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
firefly
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

A crash during shutdown when sync user stats operation is running:

 ceph version 0.90-648-gdbaa142 (dbaa1420c40092e26b26da6b3175d026983e32a3)
 1: radosgw() [0x6880fa]
 2: (()+0x10340) [0x8111340]
 3: (PerfCounters::inc(int, unsigned long)+0x7) [0x71ffb7]
 4: (ObjectCache::get(std::string&, ObjectCacheInfo&, unsigned int, rgw_cache_entry_info*)+0x5d8) [0x62d988]
 5: (RGWCache<RGWRados>::obj_stat(void*, rgw_obj&, unsigned long*, long*, unsigned long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, ceph::buffer::list*, RGWObjVersionTracker*)+0x283) [0x6025f3]
 6: (RGWRados::get_obj_state(RGWRadosCtx*, rgw_obj&, RGWObjState**, RGWObjVersionTracker*)+0xd2) [0x5da502]
 7: (RGWRados::prepare_get_obj(void*, rgw_obj&, long*, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, long const*, long const*, long*, char const*, char const*, unsigned long*, unsigned long*, RGWObjVersionTracker*, void**, rgw_err*)+0xc4) [0x5df254]
 8: (rgw_get_system_obj(RGWRados*, void*, rgw_bucket&, std::string const&, ceph::buffer::list&, RGWObjVersionTracker*, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, rgw_cache_entry_info*)+0x341) [0x5baee1]
 9: (RGWRados::get_bucket_instance_from_oid(void*, std::string&, RGWBucketInfo&, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*, rgw_cache_entry_info*)+0x123) [0x5c71e3]
 10: (RGWRados::get_bucket_instance_info(void*, rgw_bucket&, RGWBucketInfo&, long*, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >*)+0x67) [0x5c7617]
 11: (RGWRados::open_bucket_index(rgw_bucket&, librados::IoCtx&, std::map<int, std::string, std::less<int>, std::allocator<std::pair<int const, std::string> > >&, int, std::map<int, std::string, std::less<int>, std::allocator<std::pair<int const, std::string> > >*)+0x13e) [0x5d224e]
 12: (int RGWRados::open_bucket_index<rgw_cls_list_ret>(rgw_bucket&, librados::IoCtx&, std::map<int, std::string, std::less<int>, std::allocator<std::pair<int const, std::string> > >&, std::map<int, rgw_cls_list_ret, std::less<int>, std::allocator<std::pair<int const, rgw_cls_list_ret> > >&, int, std::map<int, std::string, std::less<int>, std::allocator<std::pair<int const, std::string> > >*)+0x29) [0x6014a9]
 13: (RGWRados::cls_bucket_head(rgw_bucket&, std::map<std::string, rgw_bucket_dir_header, std::less<std::string>, std::allocator<std::pair<std::string const, rgw_bucket_dir_header> > >&, std::map<int, std::string, std::less<int>, std::allocator<std::pair<int const, std::string> > >*)+0xf3) [0x5ea283]
 14: (RGWRados::cls_user_sync_bucket_stats(rgw_obj&, rgw_bucket&)+0x5e) [0x5eb01e]
 15: (rgw_bucket_sync_user_stats(RGWRados*, std::string const&, rgw_bucket&)+0x58) [0x5a5838]
 16: (RGWUserStatsCache::sync_bucket(std::string const&, rgw_bucket&)+0x23) [0x6501b3]
 17: (RGWUserStatsCache::BucketsSyncThread::entry()+0x171) [0x654de1]
 18: (()+0x8182) [0x8109182]
 19: (clone()+0x6d) [0x99ed38d]

Could be that we're not waiting to or not shutting down the sync thread correctly.

Associated revisions

Revision de2e5fa0 (diff)
Added by Yehuda Sadeh about 9 years ago

rgw: finalize perfcounters after shutting down storage

Fixes: #10572
Backport: giant, firefly
First disable the storage subsystem, then disable perfcounters.

Signed-off-by: Yehuda Sadeh <>

Revision cfab01e3 (diff)
Added by Yehuda Sadeh about 9 years ago

rgw: move perf cleanup before context cleanup

Fixes: #10722
Fixes: #10572
Backport: giant, firefly

This fixes a regression introduced in commit
de2e5fa048639de6c9ee004a93ab295625fa3b94.

Signed-off-by: Yehuda Sadeh <>

History

#2 Updated by Yehuda Sadeh about 9 years ago

  • Status changed from New to Fix Under Review
  • Assignee changed from Yehuda Sadeh to Josh Durgin

#3 Updated by Josh Durgin about 9 years ago

  • Status changed from Fix Under Review to Pending Backport

This is fixed in next in commit:de2e5fa048639de6c9ee004a93ab295625fa3b94

I tried merging next to master, but there are several conflicts in rgw. Could you resolve those Yehuda?

#4 Updated by Yehuda Sadeh about 9 years ago

The fix for this is broken. When backporting, please also cherry-pick the follow up fix.

#5 Updated by Sage Weil almost 9 years ago

  • Backport set to firefly

#6 Updated by Sage Weil almost 9 years ago

  • Status changed from Pending Backport to Resolved

not going to backport this unless we see it on firefly.

Also available in: Atom PDF