Actions
Backport #21150
closedjewel: tests: btrfs copy_clone returns errno 95 (Operation not supported)
Added by Kefu Chai over 6 years ago. Updated over 6 years ago.
Release:
jewel
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Updated by Kefu Chai over 6 years ago
- Copied from Bug #20911: mira083 sdf flaky added
Updated by David Galloway over 6 years ago
- Assignee set to David Galloway
Is this causing job failures? I'm having trouble finding anything indicating this would be fatal without an actual I/O error.
Updated by Kefu Chai over 6 years ago
- Subject changed from mira054 sdf flaky to mira038/sdh, mira054/sdf flaky
- Priority changed from Normal to Urgent
David, sorry for the latency. yeah, it is causing test failures. the errno is 95 (Operation not supported), it's not EIO but i believe it is due to the underlying "Illegal Request". see the latest comment from me.
on mira038:
[158423.946994] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158423.955155] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158423.961897] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158423.968977] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 00 00 00 00 7f ff ff 00 00 [158423.978317] blk_update_request: 6 callbacks suppressed [158423.983805] blk_update_request: critical target error, dev sdh, sector 0 [158423.990915] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158423.999072] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.005875] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.012946] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff 00 00 [158424.022324] blk_update_request: critical target error, dev sdh, sector 8388607 [158424.029992] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158424.038158] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.044901] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.051981] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff 00 00 [158424.061324] blk_update_request: critical target error, dev sdh, sector 16777214 [158424.069079] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158424.077246] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.083980] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.091242] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff 00 00 [158424.100655] blk_update_request: critical target error, dev sdh, sector 25165821 [158424.108339] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158424.116505] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.123334] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.130503] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff 00 00 [158424.139982] blk_update_request: critical target error, dev sdh, sector 33554428 [158424.147730] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158424.155982] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.162726] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.169891] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff 00 00 [158424.179328] blk_update_request: critical target error, dev sdh, sector 41943035 [158424.187077] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [158424.195243] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current] [158424.202073] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code [158424.209151] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff 00 00 [158424.218501] blk_update_request: critical target error, dev sdh, sector 58720249 ...
a typical failure is http://pulpito.ceph.com/kchai-2017-10-02_07:55:29-rados-wip-kefu-testing-2017-10-02-1347-jewel-distro-basic-mira/1694954/
and the backtrace looks like:
-48> 2017-10-02 09:16:43.299838 7ff799dcd700 10 filestore(/var/lib/ceph/osd/ceph-0) clone_range 7.4_head/#7:2baf52a2:test-rados-api-mira038-31711-4::src:he
ad# -> 7.4_head/#7:2baf52a2:test-rados-api-mira038-31711-4:src:dst:head# 0~128 to 0 = -95
-47> 2017-10-02 09:16:43.299849 7ff799dcd700 0 filestore(/var/lib/ceph/osd/ceph-0) error (95) Operation not supported not handled on operation 0x56553848
b248 (378.0.1, or op 1, counting from 0)
-46> 2017-10-02 09:16:43.299861 7ff799dcd700 0 filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
...
0> 2017-10-02 09:16:43.305278 7ff799dcd700 -1 os/filestore/FileStore.cc: In function 'void FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t
, int, ThreadPool::TPHandle*)' thread 7ff799dcd700 time 2017-10-02 09:16:43.299979
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")
ceph version 10.2.9-389-g3f09962 (3f0996289daaca32abaa042ab6b784bf8b58361a)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x56552c4c0835]
2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xf15) [0x56552c196d75]
3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, unsigned long, ThreadPool::TPHandle*)+0x3b
) [0x56552c19ca9b]
4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x2cd) [0x56552c19cd9d]
5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x56552c4b187e]
6: (ThreadPool::WorkThread::entry()+0x10) [0x56552c4b2760]
7: (()+0x7dc5) [0x7ff7a65a9dc5]
8: (clone()+0x6d) [0x7ff7a4c3473d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
see /a/kchai-2017-10-02_07:55:29-rados-wip-kefu-testing-2017-10-02-1347-jewel-distro-basic-mira/1694954$ zless remote/mira038/log/ceph-osd.0.log.gz
i am marking this "urgent" because it'd be great if we can have less noisy in test result.
Updated by Kefu Chai over 6 years ago
- Project changed from sepia to RADOS
- Subject changed from mira038/sdh, mira054/sdf flaky to btrfs copy_clone returns errno 95 (Operation not supported)
- Category changed from Test Node to Tests
- Assignee changed from David Galloway to Kefu Chai
i suspected that btrfs somehow failed to handle the ioctl(BTRFS_IOC_CLONE_RANGE) call. but i checked linux kernel of 3.10.107, the btrfs does handle it. anyway, will switch to xfs, and try again.
Updated by Kefu Chai over 6 years ago
- Status changed from New to Fix Under Review
Updated by Kefu Chai over 6 years ago
- Status changed from Fix Under Review to Resolved
- Release set to jewel
Updated by Nathan Cutler over 6 years ago
- Tracker changed from Bug to Backport
- Subject changed from btrfs copy_clone returns errno 95 (Operation not supported) to jewel: tests: btrfs copy_clone returns errno 95 (Operation not supported)
- Description updated (diff)
- Target version set to v10.2.11
description¶
[ 868.987891] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 868.996015] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.002576] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.009406] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 00 00 00 00 7f ff ff 00 00 [ 869.018316] blk_update_request: critical target error, dev sdf, sector 0 [ 869.025112] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.032884] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.039274] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.046059] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff 00 00 [ 869.054968] blk_update_request: critical target error, dev sdf, sector 8388607 [ 869.062298] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.070059] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.076448] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.083199] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff 00 00 [ 869.092098] blk_update_request: critical target error, dev sdf, sector 16777214 [ 869.099456] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.107233] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.113656] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.120416] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff 00 00 [ 869.129317] blk_update_request: critical target error, dev sdf, sector 25165821 [ 869.136664] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.144581] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.150978] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.157721] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff 00 00 [ 869.166631] blk_update_request: critical target error, dev sdf, sector 33554428 [ 869.173986] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.181816] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.188203] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.194938] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff 00 00 [ 869.203890] blk_update_request: critical target error, dev sdf, sector 41943035 [ 869.211204] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.218982] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.225379] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.232114] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff 00 00 [ 869.241092] blk_update_request: critical target error, dev sdf, sector 50331642 [ 869.248441] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.256227] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.262815] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.269830] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff 00 00 [ 869.279166] blk_update_request: critical target error, dev sdf, sector 58720249 [ 869.286836] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.294909] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.301527] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.308517] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 03 ff ff f8 00 7f ff ff 00 00 [ 869.317828] blk_update_request: critical target error, dev sdf, sector 67108856 [ 869.325445] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE [ 869.333527] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current] [ 869.340170] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code [ 869.347179] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 04 7f ff f7 00 7f ff ff 00 00 [ 869.356428] blk_update_request: critical target error, dev sdf, sector 75497463 [ 874.587675] XFS (sdf): Mounting V5 Filesystem [ 874.745239] XFS (sdf): Ending clean mount
# /usr/libexec/smart.sh OK - All 8 drives healthy # cli64 disk info # Enc# Slot# ModelName Capacity Usage =============================================================================== 1 01 Slot#1 ST31000524AS 1000.2GB JBOD 2 01 Slot#2 Hitachi HUA722010CLA330 1000.2GB JBOD 3 01 Slot#3 Hitachi HUA722010CLA330 1000.2GB JBOD 4 01 Slot#4 Hitachi HDS721010CLA330 1000.2GB JBOD 5 01 Slot#5 Hitachi HUA722010CLA330 1000.2GB JBOD 6 01 Slot#6 Hitachi HUA722010CLA330 1000.2GB JBOD 7 01 Slot#7 Hitachi HUA722010CLA330 1000.2GB JBOD 8 01 Slot#8 Hitachi HUA722010CLA330 1000.2GB JBOD =============================================================================== GuiErrMsg<0x00>: Success.
Actions