Ceph » rgw

can you share teh ceph.conf and s3cmd.cfg

# cat .s3cfg
[default]
access_key = XXXXXXXXXXXX
access_token = 
add_encoding_exts = 
add_headers = 
bucket_location = US
ca_certs_file = 
cache_file = 
check_ssl_certificate = True
check_ssl_hostname = True
#cloudfront_host = cloudfront.amazonaws.com
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date = 
expiry_days = 
expiry_prefix = 
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = rdu.download.ceph.com
host_bucket = %(bucket)s.rdu.download.ceph.com
human_readable_sizes = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
kms_key = 
limitrate = 0
list_md5 = False
log_target_prefix = 
long_listing = False
max_delete = -1
mime_type = 
multipart_chunk_size_mb = 15
multipart_max_chunks = 10000
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 65536
reduced_redundancy = False
requester_pays = False
restore_days = 1
secret_key = XXXXXXXXXX
send_chunk = 65536
server_side_encryption = False
signature_v2 = False
simpledb_host = sdb.amazonaws.com
skip_existing = False
socket_timeout = 300
stats = False
stop_on_error = False
storage_class = 
urlencoding_mode = normal
use_https = True
#use_https = False
use_mime_magic = True
verbosity = WARNING
#website_endpoint = http://%(bucket).rdu-website.download.ceph.com/
#website_endpoint = http://rdu-website.download.ceph.com/
website_error = 
website_index = index.html

# cat /etc/ceph/ceph.conf 

<snip>

[client.radosgw.dl-rgw-pub01]
  log file = /var/log/ceph/$id.log
  rgw dns name = rdu.download.ceph.com
  rgw dns s3website name = rdu-website.download.ceph.com
  rgw thread pool size = 1024
  rgw enable ops log = false
  ms dispatch throttle bytes = 10737418240
  rgw cache lru size = 10000
  admin socket = /var/run/ceph/ceph-$name.asok
  objecter inflight op bytes = 10737418240
  objecter inflight ops = 4096
  rgw frontends = civetweb port=[::]:7480 access_log_file=/var/log/civetweb/access.log error_log_file=/var/log/civetweb/error.log request_timeout_ms=300000
  admin socket = /var/run/ceph/ceph-$name.asok
  rgw cache enabled = 1
  rgw resolve cname = true
  rgw enable usage log = true
  rgw expose bucket = true
  rgw zonegroup = default
  rgw_enable_apis = s3, swift, swift_auth, admin # no Website config
  rgw zone = default
  rgw enable static website = true
  rgw_max_chunk_size = 4194304

Is this still reproducible in the setup, I tried reproducing this locally, but I'm not successful wkith that

I can't reproduce using a different folder from my workstation but I do still get the same error when trying to sync from download.ceph.com to the RGW. Are there more logs or debug output I could gather for you?

Is this reproducible still with latest mimic (13.2.0)?

I updated the RGWs to 13.2.0 and am still seeing the same issue.

Can we have this reproduced at debug rgw=20 debug ms=1.

I was able to log in the node and see the bt, the bt itself arose from FD_SET where a fileno > 1024 was passed which was the configured ulimit for nofile in the system, increasing ulimit should hopefully solve the issue

I was able to log in the node and see the bt, the bt itself arose from FD_SET where a fileno > 1024 was passed which was the configured ulimit for nofile in the system, increasing ulimit should hopefully solve the issue

No dice even with the following set

root@dl-rgw-pub01:~# ps -u ceph
    PID TTY          TIME CMD
2787067 ?        00:02:01 radosgw

root@dl-rgw-pub01:~# grep 'open files' /proc/2787067/limits
Max open files            1048576              1048576              files     

I am seeing exact same bug:

in thread 7f5644c45700 thread_name:civetweb-worker

ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable)
 1: (()+0x281670) [0x55fd3e88f670]
 2: (()+0xf6d0) [0x7f577208c6d0]
 3: (gsignal()+0x37) [0x7f57664f0277]
 4: (abort()+0x148) [0x7f57664f1968]
 5: (()+0x78d37) [0x7f5766532d37]
 6: (__fortify_fail()+0x37) [0x7f57665d26e7]
 7: (()+0x116862) [0x7f57665d0862]
 8: (()+0x118647) [0x7f57665d2647]
 9: (()+0x2655da) [0x55fd3e8735da]
 10: (mg_write()+0xc1) [0x55fd3e87a521]
 11: (RGWCivetWeb::write_data(char const*, unsigned long)+0x40) [0x55fd3e885ae0]
 12: (rgw::io::BufferingFilter<rgw::io::ChunkingFilter<rgw::io::ConLenControllingFilter<RGWCivetWeb*> > >::send_body(char const*, unsigned long)+0xf8) [0x55fd3e812f78]
 13: (rgw::io::AccountingFilter<rgw::io::RestfulClient*>::send_body(char const*, unsigned long)+0x15) [0x55fd3e80fe05]
 14: (dump_body(req_state*, char const*, unsigned long)+0x43) [0x55fd3ea22bd3]
 15: (RGWGetObj_ObjStore_S3::send_response_data(ceph::buffer::list&, long, long)+0x280) [0x55fd3ea7a2a0]
 16: (RGWGetObj::get_data_cb(ceph::buffer::list&, long, long)+0x67) [0x55fd3e94ffb7]
 17: (RGWRados::get_obj_iterate_cb(RGWObjectCtx*, RGWObjState*, RGWBucketInfo const&, rgw_obj const&, rgw_raw_obj const&, long, long, long, bool, void*)+0xea) [0x55fd3e9d4d9a]
 18: (()+0x3c7596) [0x55fd3e9d5596]
 19: (RGWRados::iterate_obj(RGWObjectCtx&, RGWBucketInfo const&, rgw_obj const&, long, long, unsigned long, int ()(RGWBucketInfo const&, rgw_obj const&, rgw_raw_obj const&, long, long, long, bool, RGWObjState, void*), void*)+0x3f9) [0x55fd3e9d7f69]
 20: (RGWRados::Object::Read::iterate(long, long, RGWGetDataCB*)+0x2a1) [0x55fd3e9d83c1]
 21: (RGWGetObj::execute()+0xc5c) [0x55fd3e976ebc]
 22: (rgw_process_authenticated(RGWHandler_REST*, RGWOp*&, RGWRequest*, req_state*, bool)+0x1cd) [0x55fd3e9903ad]
 23: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::string const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xe4c) [0x55fd3e9912fc]
 24: (RGWCivetWebFrontend::process(mg_connection*)+0x37c) [0x55fd3e810f9c]
 25: (()+0x27249e) [0x55fd3e88049e]
 26: (()+0x274188) [0x55fd3e882188]
 27: (()+0x274628) [0x55fd3e882628]
 28: (()+0x7e25) [0x7f5772084e25]
 29: (clone()+0x6d) [0x7f57665b8bad]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Any resolution?

Browsing buckets with large object/file counts seems to trigger this. I updated the NOFILE to 10X and it did not help.

Was able to trigger this, have a fix staged at https://github.com/ceph/civetweb/pull/29

Kevin, For now, if you're blocked, changing the rgw_frontends string to use beast instead of civetweb will get you moving, until this patch is backported to mimic.

Thank you very much for the quick fix. Using beast seems to work for now, and will change back to civetweb once this patch is GA.