Ceph » rgw

Hi guys!

Luminous 12.2.12

I have several instances in website mode. See their config:

[client]
    rgw crypt require ssl = false
    rgw dns name = xxx.xx
    rgw enable apis = s3, admin
    rgw dynamic resharding = true
    rgw enable usage log = false
    rgw enable ops log = false
    rgw enable gc threads = false
    rgw enable lc threads = false
    rgw remote addr param = HTTP_X_FORWARDED_FOR
    rgw max objs per shard = 256000
    rgw num rados handles = 2
    rgw thread pool size = 256
[client.xxx.x]
    host = xxx
    keyring = /var/lib/ceph/radosgw/xxx.x.keyring
    rgw dns s3website name = xxx.xx
    rgw enable apis = s3website
    rgw enable gc threads = true
    rgw enable lc threads = true
    rgw gc max objs = 24
    rgw frontends = civetweb num_threads=256 port=0.0.0.0:7481 enable_keep_alive=yes access_log_file=xxx.log error_log_file=xxx.log

All rgw instances in website mode are crashed a few time a day with this error:

2019-08-21 08:12:58.407798 7f2a10393700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: In function 'void RGWObjectCtxImpl<T, S>::set_atomic(T&) [with T = rgw_obj; S = RGWObjState]' thread 7f2a10393700 time 2019-08-21 08:12:58.406053
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: 2150: FAILED assert(!obj.empty())

 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f2a4301ffd0]
 2: (RGWObjectCtxImpl<rgw_obj, RGWObjState>::set_atomic(rgw_obj&)+0x238) [0x7f2a4d28b8b8]
 3: (RGWGetObj::verify_permission()+0x107) [0x7f2a4d25f907]
 4: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x100) [0x7f2a4d290990]
 5: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xb98) [0x7f2a4d2917b8]
 6: (RGWCivetWebFrontend::process(mg_connection*)+0x3a2) [0x7f2a4d11a8d2]
 7: (()+0x2afa7f) [0x7f2a4d189a7f]
 8: (()+0x2b140b) [0x7f2a4d18b40b]
 9: (()+0x7dc5) [0x7f2a4ba5bdc5]
 10: (clone()+0x6d) [0x7f2a3ffa3ced]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---