Project

General

Profile

Bug #41370

[RGW] RGW in website mode: rgw_rados.h: 2150: FAILED assert(!obj.empty()

Added by Aleksandr Rudenko 29 days ago. Updated 17 days ago.

Status:
New
Priority:
Normal
Assignee:
Target version:
Start date:
08/21/2019
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Hi guys!

Luminous 12.2.12

I have several instances in website mode. See their config:

[client]
    rgw crypt require ssl = false
    rgw dns name = xxx.xx
    rgw enable apis = s3, admin
    rgw dynamic resharding = true
    rgw enable usage log = false
    rgw enable ops log = false
    rgw enable gc threads = false
    rgw enable lc threads = false
    rgw remote addr param = HTTP_X_FORWARDED_FOR
    rgw max objs per shard = 256000
    rgw num rados handles = 2
    rgw thread pool size = 256
[client.xxx.x]
    host = xxx
    keyring = /var/lib/ceph/radosgw/xxx.x.keyring
    rgw dns s3website name = xxx.xx
    rgw enable apis = s3website
    rgw enable gc threads = true
    rgw enable lc threads = true
    rgw gc max objs = 24
    rgw frontends = civetweb num_threads=256 port=0.0.0.0:7481 enable_keep_alive=yes access_log_file=xxx.log error_log_file=xxx.log

All rgw instances in website mode are crashed a few time a day with this error:


2019-08-21 08:12:58.407798 7f2a10393700 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: In function 'void RGWObjectCtxImpl<T, S>::set_atomic(T&) [with T = rgw_obj; S = RGWObjState]' thread 7f2a10393700 time 2019-08-21 08:12:58.406053
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.12/rpm/el7/BUILD/ceph-12.2.12/src/rgw/rgw_rados.h: 2150: FAILED assert(!obj.empty())

 ceph version 12.2.12 (1436006594665279fe734b4c15d7e08c13ebd777) luminous (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f2a4301ffd0]
 2: (RGWObjectCtxImpl<rgw_obj, RGWObjState>::set_atomic(rgw_obj&)+0x238) [0x7f2a4d28b8b8]
 3: (RGWGetObj::verify_permission()+0x107) [0x7f2a4d25f907]
 4: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x100) [0x7f2a4d290990]
 5: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xb98) [0x7f2a4d2917b8]
 6: (RGWCivetWebFrontend::process(mg_connection*)+0x3a2) [0x7f2a4d11a8d2]
 7: (()+0x2afa7f) [0x7f2a4d189a7f]
 8: (()+0x2b140b) [0x7f2a4d18b40b]
 9: (()+0x7dc5) [0x7f2a4ba5bdc5]
 10: (clone()+0x6d) [0x7f2a3ffa3ced]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---

History

#1 Updated by Casey Bodley 28 days ago

  • Assignee set to Casey Bodley

#2 Updated by Aleksandr Rudenko 17 days ago

It happens if website mode was enabled but index page was not set.
After this, all requests for index page will crash rgw instance with this error.

Also available in: Atom PDF