Project

General

Profile

Actions
Copy link

Backport #21150

closed

jewel: tests: btrfs copy_clone returns errno 95 (Operation not supported)

Added by Kefu Chai over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Kefu Chai
Target version:
Ceph - v10.2.11
Release:
jewel
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

https://github.com/ceph/ceph/pull/18165

Related issues 1 (1 open — 0 closed)

Copied from Infrastructure - Bug #20911: mira083 sdf flakyNew08/04/2017

Actions
Actions
Copy link
 #1

Updated by Kefu Chai over 6 years ago

Actions
Copy link
 #2

Updated by Kefu Chai over 6 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by David Galloway over 6 years ago

  • Assignee set to David Galloway

Is this causing job failures? I'm having trouble finding anything indicating this would be fatal without an actual I/O error.

Actions
Copy link
 #4

Updated by Kefu Chai over 6 years ago

  • Subject changed from mira054 sdf flaky to mira038/sdh, mira054/sdf flaky
  • Priority changed from Normal to Urgent

David, sorry for the latency. yeah, it is causing test failures. the errno is 95 (Operation not supported), it's not EIO but i believe it is due to the underlying "Illegal Request". see the latest comment from me.

on mira038:

[158423.946994] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158423.955155] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158423.961897] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158423.968977] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 00 00 00 00 7f ff ff 00 00
[158423.978317] blk_update_request: 6 callbacks suppressed
[158423.983805] blk_update_request: critical target error, dev sdh, sector 0
[158423.990915] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158423.999072] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.005875] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.012946] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff 00 00
[158424.022324] blk_update_request: critical target error, dev sdh, sector 8388607
[158424.029992] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158424.038158] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.044901] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.051981] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff 00 00
[158424.061324] blk_update_request: critical target error, dev sdh, sector 16777214
[158424.069079] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158424.077246] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.083980] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.091242] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff 00 00
[158424.100655] blk_update_request: critical target error, dev sdh, sector 25165821
[158424.108339] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158424.116505] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.123334] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.130503] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff 00 00
[158424.139982] blk_update_request: critical target error, dev sdh, sector 33554428
[158424.147730] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158424.155982] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.162726] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.169891] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff 00 00
[158424.179328] blk_update_request: critical target error, dev sdh, sector 41943035
[158424.187077] sd 0:0:0:7: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[158424.195243] sd 0:0:0:7: [sdh] Sense Key : Illegal Request [current]
[158424.202073] sd 0:0:0:7: [sdh] Add. Sense: Invalid command operation code
[158424.209151] sd 0:0:0:7: [sdh] CDB: Write same(16) 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff 00 00
[158424.218501] blk_update_request: critical target error, dev sdh, sector 58720249
...

a typical failure is http://pulpito.ceph.com/kchai-2017-10-02_07:55:29-rados-wip-kefu-testing-2017-10-02-1347-jewel-distro-basic-mira/1694954/

and the backtrace looks like:

   -48> 2017-10-02 09:16:43.299838 7ff799dcd700 10 filestore(/var/lib/ceph/osd/ceph-0) clone_range 7.4_head/#7:2baf52a2:test-rados-api-mira038-31711-4::src:he
ad# -> 7.4_head/#7:2baf52a2:test-rados-api-mira038-31711-4:src:dst:head# 0~128 to 0 = -95
   -47> 2017-10-02 09:16:43.299849 7ff799dcd700  0 filestore(/var/lib/ceph/osd/ceph-0)  error (95) Operation not supported not handled on operation 0x56553848
b248 (378.0.1, or op 1, counting from 0)
   -46> 2017-10-02 09:16:43.299861 7ff799dcd700  0 filestore(/var/lib/ceph/osd/ceph-0) unexpected error code
...
     0> 2017-10-02 09:16:43.305278 7ff799dcd700 -1 os/filestore/FileStore.cc: In function 'void FileStore::_do_transaction(ObjectStore::Transaction&, uint64_t
, int, ThreadPool::TPHandle*)' thread 7ff799dcd700 time 2017-10-02 09:16:43.299979
os/filestore/FileStore.cc: 2930: FAILED assert(0 == "unexpected error")

 ceph version 10.2.9-389-g3f09962 (3f0996289daaca32abaa042ab6b784bf8b58361a)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0x56552c4c0835]
 2: (FileStore::_do_transaction(ObjectStore::Transaction&, unsigned long, int, ThreadPool::TPHandle*)+0xf15) [0x56552c196d75]
 3: (FileStore::_do_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, unsigned long, ThreadPool::TPHandle*)+0x3b
) [0x56552c19ca9b]
 4: (FileStore::_do_op(FileStore::OpSequencer*, ThreadPool::TPHandle&)+0x2cd) [0x56552c19cd9d]
 5: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa7e) [0x56552c4b187e]
 6: (ThreadPool::WorkThread::entry()+0x10) [0x56552c4b2760]
 7: (()+0x7dc5) [0x7ff7a65a9dc5]
 8: (clone()+0x6d) [0x7ff7a4c3473d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

see /a/kchai-2017-10-02_07:55:29-rados-wip-kefu-testing-2017-10-02-1347-jewel-distro-basic-mira/1694954$ zless remote/mira038/log/ceph-osd.0.log.gz

i am marking this "urgent" because it'd be great if we can have less noisy in test result.

Actions
Copy link
 #5

Updated by Kefu Chai over 6 years ago

  • Project changed from sepia to RADOS
  • Subject changed from mira038/sdh, mira054/sdf flaky to btrfs copy_clone returns errno 95 (Operation not supported)
  • Category changed from Test Node to Tests
  • Assignee changed from David Galloway to Kefu Chai

i suspected that btrfs somehow failed to handle the ioctl(BTRFS_IOC_CLONE_RANGE) call. but i checked linux kernel of 3.10.107, the btrfs does handle it. anyway, will switch to xfs, and try again.

Actions
Copy link
 #6

Updated by Kefu Chai over 6 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #7

Updated by Kefu Chai over 6 years ago

  • Status changed from Fix Under Review to Resolved
  • Release set to jewel
Actions
Copy link
 #8

Updated by Nathan Cutler over 6 years ago

  • Tracker changed from Bug to Backport
  • Subject changed from btrfs copy_clone returns errno 95 (Operation not supported) to jewel: tests: btrfs copy_clone returns errno 95 (Operation not supported)
  • Description updated (diff)
  • Target version set to v10.2.11

description

[  868.987891] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  868.996015] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.002576] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.009406] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 00 00 00 00 7f ff ff 00 00
[  869.018316] blk_update_request: critical target error, dev sdf, sector 0
[  869.025112] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.032884] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.039274] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.046059] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 7f ff ff 00 7f ff ff 00 00
[  869.054968] blk_update_request: critical target error, dev sdf, sector 8388607
[  869.062298] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.070059] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.076448] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.083199] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 00 ff ff fe 00 7f ff ff 00 00
[  869.092098] blk_update_request: critical target error, dev sdf, sector 16777214
[  869.099456] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.107233] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.113656] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.120416] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 01 7f ff fd 00 7f ff ff 00 00
[  869.129317] blk_update_request: critical target error, dev sdf, sector 25165821
[  869.136664] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.144581] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.150978] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.157721] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 01 ff ff fc 00 7f ff ff 00 00
[  869.166631] blk_update_request: critical target error, dev sdf, sector 33554428
[  869.173986] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.181816] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.188203] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.194938] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 02 7f ff fb 00 7f ff ff 00 00
[  869.203890] blk_update_request: critical target error, dev sdf, sector 41943035
[  869.211204] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.218982] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.225379] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.232114] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 02 ff ff fa 00 7f ff ff 00 00
[  869.241092] blk_update_request: critical target error, dev sdf, sector 50331642
[  869.248441] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.256227] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.262815] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.269830] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 03 7f ff f9 00 7f ff ff 00 00
[  869.279166] blk_update_request: critical target error, dev sdf, sector 58720249
[  869.286836] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.294909] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.301527] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.308517] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 03 ff ff f8 00 7f ff ff 00 00
[  869.317828] blk_update_request: critical target error, dev sdf, sector 67108856
[  869.325445] sd 0:0:0:5: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[  869.333527] sd 0:0:0:5: [sdf] Sense Key : Illegal Request [current]
[  869.340170] sd 0:0:0:5: [sdf] Add. Sense: Invalid command operation code
[  869.347179] sd 0:0:0:5: [sdf] CDB: Write same(16) 93 08 00 00 00 00 04 7f ff f7 00 7f ff ff 00 00
[  869.356428] blk_update_request: critical target error, dev sdf, sector 75497463
[  874.587675] XFS (sdf): Mounting V5 Filesystem
[  874.745239] XFS (sdf): Ending clean mount

#  /usr/libexec/smart.sh
OK - All 8 drives healthy

# cli64 disk info
  # Enc# Slot#   ModelName                        Capacity  Usage
===============================================================================
  1  01  Slot#1  ST31000524AS                     1000.2GB  JBOD
  2  01  Slot#2  Hitachi HUA722010CLA330          1000.2GB  JBOD
  3  01  Slot#3  Hitachi HUA722010CLA330          1000.2GB  JBOD
  4  01  Slot#4  Hitachi HDS721010CLA330          1000.2GB  JBOD
  5  01  Slot#5  Hitachi HUA722010CLA330          1000.2GB  JBOD
  6  01  Slot#6  Hitachi HUA722010CLA330          1000.2GB  JBOD
  7  01  Slot#7  Hitachi HUA722010CLA330          1000.2GB  JBOD
  8  01  Slot#8  Hitachi HUA722010CLA330          1000.2GB  JBOD
===============================================================================
GuiErrMsg<0x00>: Success.
Actions
Copy link
 #9

Updated by Nathan Cutler over 6 years ago

  • Description updated (diff)
Actions
Copy link

Also available in: Atom PDF