Actions
Bug #24364
closedrgw: s3cmd sync fails
% Done:
0%
Source:
Tags:
Backport:
mimic
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I'm trying to sync download.ceph.com packages to the Sepia Long Running Cluster and it's failing after upgrading to mimic (13.1.0-162-g233ee97-1xenial)
I've attached the rgw log. Let me know if you need ssh access to the rgw nodes.
s3cmd output:
WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 3 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 6 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 9 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 12 sec... WARNING: Retrying failed request: / (IncompleteRead(85432 bytes read, 260881 more expected)) WARNING: Waiting 15 sec... ERROR: S3 Temporary Error: Request failed for: /. Please try again later.
RGW output:
-1> 2018-05-31 11:03:22.118 7fe53e23b700 2 req 16:0.087059:s3:GET /:list_bucket:completing
0> 2018-05-31 11:03:22.142 7fe53e23b700 -1 *** Caught signal (Aborted) **
in thread 7fe53e23b700 thread_name:civetweb-worker
ceph version 13.1.0-162-g233ee97 (233ee9764e2d1702dde72165d9660fec89ab56fc) mimic (rc)
1: (()+0x291290) [0x5615a7344290]
2: (()+0x11390) [0x7fe571475390]
3: (gsignal()+0x38) [0x7fe566e16428]
4: (abort()+0x16a) [0x7fe566e1802a]
5: (()+0x777ea) [0x7fe566e587ea]
6: (__fortify_fail()+0x5c) [0x7fe566efa15c]
7: (()+0x117160) [0x7fe566ef8160]
8: (()+0x1190a7) [0x7fe566efa0a7]
9: (()+0x27564a) [0x5615a732864a]
10: (mg_write()+0xc1) [0x5615a732f5a1]
11: (RGWCivetWeb::write_data(char const*, unsigned long)+0x40) [0x5615a733ab20]
12: (rgw::io::BufferingFilter<rgw::io::ChunkingFilter<rgw::io::ConLenControllingFilter<RGWCivetWeb*> > >::complete_request()+0x133) [0x5615a72c8a93]
13: (process_request(RGWRados*, RGWREST*, RGWRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rgw::auth::StrategyRegistry const&, RGWRestfulIO*, OpsLogSocket*, int*)+0xfc3) [0x5615a7451643]
14: (RGWCivetWebFrontend::process(mg_connection*)+0x381) [0x5615a72c6271]
15: (()+0x28251e) [0x5615a733551e]
16: (()+0x284208) [0x5615a7337208]
17: (()+0x2846a8) [0x5615a73376a8]
18: (()+0x76ba) [0x7fe57146b6ba]
19: (clone()+0x6d) [0x7fe566ee841d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
--- logging levels ---
0/ 5 none
0/ 1 lockdep
0/ 1 context
1/ 1 crush
1/ 5 mds
1/ 5 mds_balancer
1/ 5 mds_locker
1/ 5 mds_log
1/ 5 mds_log_expire
1/ 5 mds_migrator
0/ 1 buffer
0/ 1 timer
0/ 1 filer
0/ 1 striper
0/ 1 objecter
0/ 5 rados
0/ 5 rbd
0/ 5 rbd_mirror
0/ 5 rbd_replay
0/ 5 journaler
0/ 5 objectcacher
0/ 5 client
1/ 5 osd
0/ 5 optracker
0/ 5 objclass
1/ 3 filestore
1/ 3 journal
0/ 5 ms
1/ 5 mon
0/10 monc
1/ 5 paxos
0/ 5 tp
1/ 5 auth
1/ 5 crypto
1/ 1 finisher
1/ 1 reserver
1/ 5 heartbeatmap
1/ 5 perfcounter
1/ 5 rgw
1/ 5 rgw_sync
1/10 civetweb
1/ 5 javaclient
1/ 5 asok
1/ 1 throttle
0/ 0 refs
1/ 5 xio
1/ 5 compressor
1/ 5 bluestore
1/ 5 bluefs
1/ 3 bdev
1/ 5 kstore
4/ 5 rocksdb
4/ 5 leveldb
4/ 5 memdb
1/ 5 kinetic
1/ 5 fuse
1/ 5 mgr
1/ 5 mgrc
1/ 5 dpdk
1/ 5 eventtrace
-2/-2 (syslog threshold)
-1/-1 (stderr threshold)
max_recent 10000
max_new 1000
log_file /var/log/ceph/radosgw.dl-rgw-pub01.log
--- end dump of recent events ---
Files
Actions