Bug #24364
Updated by David Galloway almost 6 years ago
I'm trying to sync download.ceph.com packages to the Sepia Long Running Cluster and it's failing after upgrading to mimic (13.1.0-162-g233ee97-1xenial) I've attached the rgw log. Let me know if you need ssh access to the rgw nodes. The s3cmd output: <pre> WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 6 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 9 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 12 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 15 sec... ERROR: exits with <code>ERROR: S3 Temporary Error: Request failed for: /. Please try again later. </pre> RGW output: later.</code> <pre> -1> 2018-05-31 11:03:22.118 7fe53e23b700 2 req 16:0.087059:s3:GET /:list_bucket:completing 0> 2018-05-31 11:03:22.142 7fe53e23b700 -1 *** Caught signal (Aborted) ** in thread 7fe53e23b700 thread_name:civetweb-worker ceph version 13.1.0-162-g233ee97 (233ee9764e2d1702dde72165d9660fec89ab56fc) mimic (rc) 1: (()+0x291290) [0x5615a7344290] 2: (()+0x11390) [0x7fe571475390] 3: (gsignal()+0x38) [0x7fe566e16428] 4: (abort()+0x16a) [0x7fe566e1802a] 5: (()+0x777ea) [0x7fe566e587ea] 6: (__fortify_fail()+0x5c) [0x7fe566efa15c] 7: (()+0x117160) [0x7fe566ef8160] 8: (()+0x1190a7) [0x7fe566efa0a7] 9: (()+0x27564a) [0x5615a732864a] 10: (mg_write()+0xc1) [0x5615a732f5a1] 11: (RGWCivetWeb::write_data(char const*, unsigned long)+0x40) [0x5615a733ab20] 12: (rgw::io::BufferingFilter<rgw::io::ChunkingFilter<rgw::io::ConLenControllingFilter<RGWCivetWeb*> > >::complete_request()+0x133) [0x5615a72c8a93] 13: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xfc3) [0x5615a7451643] 14: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5615a72c6271] 15: (()+0x28251e) [0x5615a733551e] 16: (()+0x284208) [0x5615a7337208] 17: (()+0x2846a8) [0x5615a73376a8] 18: (()+0x76ba) [0x7fe57146b6ba] 19: (clone()+0x6d) [0x7fe566ee841d] NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. --- logging levels --- 0/ 5 none 0/ 1 lockdep 0/ 1 context 1/ 1 crush 1/ 5 mds 1/ 5 mds_balancer 1/ 5 mds_locker 1/ 5 mds_log 1/ 5 mds_log_expire 1/ 5 mds_migrator 0/ 1 buffer 0/ 1 timer 0/ 1 filer 0/ 1 striper 0/ 1 objecter 0/ 5 rados 0/ 5 rbd 0/ 5 rbd_mirror 0/ 5 rbd_replay 0/ 5 journaler 0/ 5 objectcacher 0/ 5 client 1/ 5 osd 0/ 5 optracker 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/radosgw.dl-rgw-pub01.log --- end dump of recent events --- </pre>