2018-07-31 00:18:04.305 7fc229303700 -1 filestore(/var/lib/ceph/osd/ceph-2) _do_copy_range(3941): read error at 1908736~2465792, (5) Input/output error
2018-07-31 00:18:04.309 7fc229303700 -1 /build/ceph-14.0.0-1729-ge9d8990/src/os/filestore/FileStore.cc: In function 'int FileStore::_do_copy_range(int, int, uint64_t, uint64_t, uin
t64_t, bool)' thread 7fc229303700 time 2018-07-31 00:18:04.307478
/build/ceph-14.0.0-1729-ge9d8990/src/os/filestore/FileStore.cc: 3978: FAILED assert(replaying || pos == end)
ceph version 14.0.0-1729-ge9d8990 (e9d899031e672701d4d99ae7281cee06d7a5f0f3) nautilus (dev)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x55ea7125e499]
2: (ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x55ea7125e61b]
3: (FileStore::_do_copy_range(int, int, unsigned long, unsigned long, unsigned long, bool)+0x146a) [0x55ea716adf0a]
4: (GenericFileStoreBackend::clone_range(int, int, unsigned long, unsigned long, unsigned long)+0x6b) [0x55ea71702acb]
5: (FileStore::_clone(coll_t const&, ghobject_t const&, ghobject_t const&, SequencerPosition const&)+0x22c) [0x55ea716d3ccc]
6: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*, char const*)+0x301e) [0x55ea716de68e]
7: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, unsigned long, ThreadPool::TPHandle*, char const*)+0x48) [0x55ea
716e28f8]
8: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x185) [0x55ea716e2ab5]
9: (ThreadPool::worker(ThreadPool::WorkThread*)+0x8eb) [0x55ea7192429b]
10: (ThreadPool::WorkThread::entry()+0x10) [0x55ea71924cd0]
11: (()+0x76db) [0x7fc23ae156db]
12: (clone()+0x3f) [0x7fc23978f88f]
[ 9142.899442] print_req_error: I/O error, dev sdg, sector 373912
[ 9173.440347] sd 0:0:0:6: [sdg] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 9173.440365] sd 0:0:0:6: [sdg] tag#0 Sense Key : Medium Error [current]
[ 9173.440371] sd 0:0:0:6: [sdg] tag#0 Add. Sense: Unrecovered read error
[ 9173.440378] sd 0:0:0:6: [sdg] tag#0 CDB: Read(10) 28 00 00 05 78 98 00 03 08 00
[ 9173.440383] print_req_error: I/O error, dev sdg, sector 358552
[ 9455.251391] sd 0:0:0:6: [sdg] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 9455.251413] sd 0:0:0:6: [sdg] tag#1 Sense Key : Medium Error [current]
[ 9455.251419] sd 0:0:0:6: [sdg] tag#1 Add. Sense: Unrecovered read error
[ 9455.251426] sd 0:0:0:6: [sdg] tag#1 CDB: Read(10) 28 00 00 05 b4 98 00 02 30 00
[ 9455.251431] print_req_error: I/O error, dev sdg, sector 373912
[ 9491.745067] sd 0:0:0:6: [sdg] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[ 9491.745072] sd 0:0:0:6: [sdg] tag#1 Sense Key : Medium Error [current]
[ 9491.745074] sd 0:0:0:6: [sdg] tag#1 Add. Sense: Unrecovered read error
[ 9491.745077] sd 0:0:0:6: [sdg] tag#1 CDB: Read(10) 28 00 00 05 78 98 00 03 08 00
[ 9491.745079] print_req_error: I/O error, dev sdg, sector 358552
/usr/libexec/smart.sh
CRITICAL - 2 of 8 drives need replacing
Drive 4 (sdd) has 2 reallocated sectors
Drive 7 (sdg) has 4015 reallocated sectors