Actions
Bug #41370
closed[RGW] RGW in website mode: rgw_rados.h: 2150: FAILED assert(!obj.empty()
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
Hi guys!
Luminous 12.2.12
I have several instances in website mode. See their config:
[client]
rgw crypt require ssl = false
rgw dns name = xxx.xx
rgw enable apis = s3, admin
rgw dynamic resharding = true
rgw enable usage log = false
rgw enable ops log = false
rgw enable gc threads = false
rgw enable lc threads = false
rgw remote addr param = HTTP_X_FORWARDED_FOR
rgw max objs per shard = 256000
rgw num rados handles = 2
rgw thread pool size = 256
[client.xxx.x]
host = xxx
keyring = /var/lib/ceph/radosgw/xxx.x.keyring
rgw dns s3website name = xxx.xx
rgw enable apis = s3website
rgw enable gc threads = true
rgw enable lc threads = true
rgw gc max objs = 24
rgw frontends = civetweb num_threads=256 port=0.0.0.0:7481 enable_keep_alive=yes access_log_file=xxx.log error_log_file=xxx.log
All rgw instances in website mode are crashed a few time a day with this error:
2019-08-21 08:12:58.407798 7f2a10393700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: In function 'void RGWObjectCtxImpl<T, S>::set_atomic(T&) [with T = rgw_obj; S = RGWObjState]' thread 7f2a10393700 time 2019-08-21 08:12:58.406053
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: 2150: FAILED assert(!obj.empty())
ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f2a4301ffd0]
2: (RGWObjectCtxImpl<rgw_obj, RGWObjState>::set_atomic(rgw_obj&)+0x238) [0x7f2a4d28b8b8]
3: (RGWGetObj::verify_permission()+0x107) [0x7f2a4d25f907]
4: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x100) [0x7f2a4d290990]
5: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xb98) [0x7f2a4d2917b8]
6: (RGWCivetWebFrontend::process(mg_connection*)+0x3a2) [0x7f2a4d11a8d2]
7: (()+0x2afa7f) [0x7f2a4d189a7f]
8: (()+0x2b140b) [0x7f2a4d18b40b]
9: (()+0x7dc5) [0x7f2a4ba5bdc5]
10: (clone()+0x6d) [0x7f2a3ffa3ced]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- begin dump of recent events ---
Actions