Project

General

Profile

Actions

Bug #54094

closed

rook/smoke: Failed to write to /dev/nvme-fabrics: Input/output error

Added by Laura Flores about 2 years ago. Updated almost 2 years ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy, pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/benhanokh-2022-01-26_21:12:05-rados-WIP_GBH_NCB_new_alloc_map_A6-distro-basic-smithi/6642176

2022-01-27T03:39:21.853 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: -
2022-01-27T03:39:21.862 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in
2022-01-27T03:39:21.862 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out
2022-01-27T03:39:21.863 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000421415 s, 1.2 MB/s
2022-01-27T03:39:21.867 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3
2022-01-27T03:39:21.874 INFO:teuthology.orchestra.run.smithi084.stdout:  File: /dev/vg_nvme/lv_3 -> ../dm-2
2022-01-27T03:39:21.874 INFO:teuthology.orchestra.run.smithi084.stdout:  Size: 7                Blocks: 0          IO Block: 4096   symbolic link
2022-01-27T03:39:21.875 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d    Inode: 362         Links: 1
2022-01-27T03:39:21.875 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2022-01-27T03:39:21.875 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-01-27 03:39:21.258476520 +0000
2022-01-27T03:39:21.875 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-01-27 03:39:21.258476520 +0000
2022-01-27T03:39:21.875 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-01-27 03:39:21.258476520 +0000
2022-01-27T03:39:21.876 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: -
2022-01-27T03:39:21.881 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in
2022-01-27T03:39:21.882 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out
2022-01-27T03:39:21.882 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000405373 s, 1.3 MB/s
2022-01-27T03:39:21.886 DEBUG:teuthology.orchestra.run.smithi084:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4
2022-01-27T03:39:21.894 INFO:teuthology.orchestra.run.smithi084.stdout:  File: /dev/vg_nvme/lv_4 -> ../dm-3
2022-01-27T03:39:21.894 INFO:teuthology.orchestra.run.smithi084.stdout:  Size: 7                Blocks: 0          IO Block: 4096   symbolic link
2022-01-27T03:39:21.894 INFO:teuthology.orchestra.run.smithi084.stdout:Device: 6h/6d    Inode: 366         Links: 1
2022-01-27T03:39:21.895 INFO:teuthology.orchestra.run.smithi084.stdout:Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
2022-01-27T03:39:21.895 INFO:teuthology.orchestra.run.smithi084.stdout:Access: 2022-01-27 03:39:21.338474321 +0000
2022-01-27T03:39:21.895 INFO:teuthology.orchestra.run.smithi084.stdout:Modify: 2022-01-27 03:39:21.338474321 +0000
2022-01-27T03:39:21.895 INFO:teuthology.orchestra.run.smithi084.stdout:Change: 2022-01-27 03:39:21.338474321 +0000
2022-01-27T03:39:21.895 INFO:teuthology.orchestra.run.smithi084.stdout: Birth: -
2022-01-27T03:39:21.904 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records in
2022-01-27T03:39:21.904 INFO:teuthology.orchestra.run.smithi084.stderr:1+0 records out
2022-01-27T03:39:21.905 INFO:teuthology.orchestra.run.smithi084.stderr:512 bytes copied, 0.000388159 s, 1.3 MB/s
2022-01-27T03:39:21.908 DEBUG:teuthology.orchestra.run.smithi084:> sudo modprobe nvme_loop && sudo mkdir -p /sys/kernel/config/nvmet/hosts/hostnqn && sudo mkdir -p /sys/kernel/config/nvmet/ports/1 && echo loop | sudo tee /sys/kernel/config/nvmet/ports/1/addr_trtype
2022-01-27T03:39:21.993 INFO:teuthology.orchestra.run.smithi084.stdout:loop
2022-01-27T03:39:21.994 INFO:tasks.nvme_loop:Connecting nvme_loop smithi084:/dev/vg_nvme/lv_1...
2022-01-27T03:39:21.995 DEBUG:teuthology.orchestra.run.smithi084:> sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1 && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/attr_allow_any_host && sudo mkdir -p /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1 && echo -n /dev/vg_nvme/lv_1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/device_path && echo 1 | sudo tee /sys/kernel/config/nvmet/subsystems/lv_1/namespaces/1/enable && sudo ln -s /sys/kernel/config/nvmet/subsystems/lv_1 /sys/kernel/config/nvmet/ports/1/subsystems/lv_1 && sudo nvme connect -t loop -n lv_1 -q hostnqn
2022-01-27T03:39:22.016 INFO:teuthology.orchestra.run.smithi084.stdout:1
2022-01-27T03:39:22.037 INFO:teuthology.orchestra.run.smithi084.stdout:/dev/vg_nvme/lv_11
2022-01-27T03:39:22.097 INFO:teuthology.orchestra.run.smithi084.stderr:Failed to write to /dev/nvme-fabrics: Input/output error

Other occurrences:
/a/benhanokh-2022-01-26_21:12:05-rados-WIP_GBH_NCB_new_alloc_map_A6-distro-basic-smithi/6642336
/a/yuriw-2022-01-27_15:09:25-rados-wip-yuri6-testing-2022-01-26-1547-distro-default-smithi/6643853

Actions #1

Updated by Neha Ojha about 2 years ago

/a/yuriw-2022-01-27_15:12:15-rados-wip-yuri7-testing-2022-01-26-1547-distro-default-smithi/6644066

Actions #2

Updated by Guillaume Abrioux about 2 years ago

  • Pull request ID set to 44917
Actions #3

Updated by Guillaume Abrioux about 2 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Guillaume Abrioux
Actions #4

Updated by Laura Flores about 2 years ago

/a/yuriw-2022-02-12_15:59:43-rados-wip-yuri4-testing-2022-02-11-0858-distro-default-smithi/6678504

Actions #5

Updated by Adam King about 2 years ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to quincy, pacific
Actions #8

Updated by Guillaume Abrioux almost 2 years ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF