Project

General

Profile

Actions
Copy link

Bug #15933

closed

bluestore+spdk can not get serial number by ceph user

Added by Clive Xu almost 8 years ago. Updated about 7 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v10.2.0
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

[root@ceph2 0000:82:00.0]# ceph -v
ceph version 10.2.0 (3a9fba20ec743699b69bd0181dd6c54dc01c64b9)

[ceph2][WARNIN] 2016-05-16 03:58:59.481521 7fbc93924b40 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
[ceph2][DEBUG ] EAL: Detected lcore 0 as core 0 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 1 as core 1 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 2 as core 2 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 3 as core 3 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 4 as core 4 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 5 as core 5 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 6 as core 6 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 7 as core 7 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 8 as core 0 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 9 as core 1 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 10 as core 2 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 11 as core 3 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 12 as core 4 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 13 as core 5 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 14 as core 6 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 15 as core 7 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 16 as core 0 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 17 as core 1 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 18 as core 2 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 19 as core 3 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 20 as core 4 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 21 as core 5 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 22 as core 6 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 23 as core 7 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 24 as core 0 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 25 as core 1 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 26 as core 2 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 27 as core 3 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 28 as core 4 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 29 as core 5 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 30 as core 6 on socket 0
[ceph2][DEBUG ] EAL: Detected lcore 31 as core 7 on socket 0
[ceph2][DEBUG ] EAL: Support maximum 128 logical core(s) by configuration.
[ceph2][DEBUG ] EAL: Detected 32 lcore(s)
[ceph2][DEBUG ] EAL: VFIO modules not all loaded, skip VFIO support...
[ceph2][DEBUG ] EAL: Setting up physically contiguous memory...
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc8ae00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x1000000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc89c00000 (size = 0x1000000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc89800000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc89400000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x600000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc88c00000 (size = 0x600000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc88800000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x1000000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc87600000 (size = 0x1000000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc87200000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc86e00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc86a00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x64800000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc22000000 (size = 0x64800000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x9000000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc18e00000 (size = 0x9000000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc18a00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0xc00000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc17c00000 (size = 0xc00000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc17800000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc17400000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x3000000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc14200000 (size = 0x3000000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0xc00000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc13400000 (size = 0xc00000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc13000000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc12c00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x8c00000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc09e00000 (size = 0x8c00000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc09a00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc09600000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0xc00000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc08800000 (size = 0xc00000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc08400000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc08000000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Ask a virtual area of 0x200000 bytes
[ceph2][DEBUG ] EAL: Virtual area found at 0x7fbc07c00000 (size = 0x200000)
[ceph2][DEBUG ] EAL: Requesting 1024 pages of size 2MB from socket 0
[ceph2][DEBUG ] EAL: TSC frequency is ~2500003 KHz
[ceph2][DEBUG ] EAL: Master lcore 0 is ready (tid=8b987700;cpuset=[0])
[ceph2][DEBUG ] EAL: Failed to set thread name for interrupt handling
[ceph2][DEBUG ] EAL: Cannot set name for lcore thread
[ceph2][DEBUG ] EAL: lcore 1 is ready (tid=8c188700;cpuset=[1])
[ceph2][WARNIN] 2016-05-16 03:59:01.251761 7fbc93924b40 -1 bdev() open failed to get nvme deivce with sn 55cd2e414c924e98
[ceph2][WARNIN] 2016-05-16 03:59:01.251782 7fbc93924b40 -1 OSD::mkfs: ObjectStore::mkfs failed with error -1
[ceph2][WARNIN] 2016-05-16 03:59:01.251818 7fbc93924b40 -1 ** ERROR: error creating empty object store in /home/ceph/test_osd: (1) Operation not permitted

in the log file:
2016-05-16 04:02:46.061909 7f01fc789b40 -1 WARNING: the following dangerous and experimental features are enabled: bluestore,rocksdb
2016-05-16 04:02:46.062463 7f01fc789b40 1 bluestore(/home/ceph/test_osd) mkfs path /home/ceph/test_osd
2016-05-16 04:02:46.062590 7f01fc789b40 1 bluestore(/home/ceph/test_osd) _setup_block_symlink_or_file created block file with
2016-05-16 04:02:46.062614 7f01fc789b40 1 bdev create path /home/ceph/test_osd/block type ust-nvme
2016-05-16 04:02:46.062622 7f01fc789b40 1 bdev() open path /home/ceph/test_osd/block
2016-05-16 04:02:48.007014 7f01f47ec700 0 bdev probe_cb found device at name: Unknown bus: <82>:^:^ vendor:0x32902 device:0x2387
2016-05-16 04:02:48.007055 7f01f47ec700 10 bdev probe_cb failed to get serial number from Unknown
2016-05-16 04:02:48.008431 7f01fc789b40 -1 bdev() open failed to get nvme deivce with sn 55cd2e414c924e98
2016-05-16 04:02:48.008449 7f01fc789b40 -1 OSD::mkfs: ObjectStore::mkfs failed with error -1
2016-05-16 04:02:48.008489 7f01fc789b40 -1 [[0;31m ** ERROR: error creating empty object store in /home/ceph/test_osd: (1) Operation not permitted[[0m
Connection closed by foreign host. 52098C

Here maybe the reason for pci_device_cfg_read cann't read pci config correctly:
int pci_device_cfg_read ( struct pci_device * dev,
void * data,
pciaddr_t offset,
pciaddr_t size,
pciaddr_t * bytes_read
)
Read arbitrary bytes from device's PCI config space

Reads data from the device's PCI configuration space. As with the system read command, less data may be returned, without an error, than was requested. This is particuarly the case if a non-root user tries to read beyond the first 64-bytes of configuration space.

Seems we can only use bluestore+spdk by root user, that what should be improved.

Actions
Copy link
 #1

Updated by Haomai Wang almost 8 years ago

@Sage Weil

I'm not sure, do we have any way to make ceph-osd started with root since Jewel?

Actions
Copy link
 #2

Updated by Nathan Cutler almost 8 years ago

Haomai: Yes, with setuser match path - see the Jewel release notes: http://docs.ceph.com/docs/master/release-notes/#upgrading-from-hammer

(I have not tried it)

Actions
Copy link
 #3

Updated by Clive Xu almost 8 years ago

Nathan Cutler wrote:

Haomai: Yes, with setuser match path - see the Jewel release notes: http://docs.ceph.com/docs/master/release-notes/#upgrading-from-hammer

(I have not tried it)

Yes, but setuser_match_path only used for upgrading, for new deployment, we should change files such as ceph-disk and systemd files, which is hard code for ceph user

Actions
Copy link
 #4

Updated by Sage Weil about 7 years ago

  • Status changed from New to Can't reproduce
Actions
Copy link

Also available in: Atom PDF