Bug #5189
closedceph-deploy disk prepare fails silently
0%
Description
$ ceph-deploy disk list ceph01-test
/dev/sda :
/dev/sda1 other, ext2, mounted on /boot
/dev/sda2 other
/dev/sda5 other, LVM2_member
/dev/sdb other, unknown
/dev/sr0 other, unknown
$ ceph-deploy disk prepare ceph01-test:sdb
No output, but ceph01-test:/var/log/ceph/ceph-osd.0.log contains:
2013-05-29 17:56:40.219020 7fe0847ed780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-osd, pid 5022
2013-05-29 17:56:40.228864 7fe0847ed780 1 filestore(/var/lib/ceph/tmp/mnt.E7TPSt) mkfs in /var/lib/ceph/tmp/mnt.E7TPSt
2013-05-29 17:56:40.228910 7fe0847ed780 1 filestore(/var/lib/ceph/tmp/mnt.E7TPSt) mkfs fsid is already set to a7034255-4abc-4330-b1d1-27e62973bcb4
2013-05-29 17:56:40.232552 7fe0847ed780 1 filestore(/var/lib/ceph/tmp/mnt.E7TPSt) leveldb db exists/created
2013-05-29 17:56:40.232687 7fe0847ed780 -1 filestore(/var/lib/ceph/tmp/mnt.E7TPSt) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.E7TPSt/journal: (2) No such file or directory
2013-05-29 17:56:40.232707 7fe0847ed780 -1 OSD::mkfs: FileStore::mkfs failed with error -2
2013-05-29 17:56:40.232745 7fe0847ed780 -1 * ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.E7TPSt: (2) No such file or directory
2013-05-29 17:56:40.538993 7fd260870780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-osd, pid 5066
2013-05-29 17:56:40.542576 7fd260870780 1 filestore(/var/lib/ceph/tmp/mnt.sAgfPQ) mkfs in /var/lib/ceph/tmp/mnt.sAgfPQ
2013-05-29 17:56:40.542625 7fd260870780 1 filestore(/var/lib/ceph/tmp/mnt.sAgfPQ) mkfs fsid is already set to a7034255-4abc-4330-b1d1-27e62973bcb4
2013-05-29 17:56:40.545946 7fd260870780 1 filestore(/var/lib/ceph/tmp/mnt.sAgfPQ) leveldb db exists/created
2013-05-29 17:56:40.546095 7fd260870780 -1 filestore(/var/lib/ceph/tmp/mnt.sAgfPQ) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.sAgfPQ/journal: (2) No such file or directory
2013-05-29 17:56:40.546114 7fd260870780 -1 OSD::mkfs: FileStore::mkfs failed with error -2
2013-05-29 17:56:40.546148 7fd260870780 -1 * ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.sAgfPQ: (2) No such file or directory
Updated by Robert Sander almost 11 years ago
Ceph was installed from the cuttlefish Debian/Ubuntu repo (including ceph-deploy).
ceph-deploy was used to create the cluster and the monitor servers.
I am now stuck when trying to create the first OSD.
Updated by Robert Sander almost 11 years ago
When I add another disk to the test VM (/dev/sdc) and create a partition /dev/sdc1 ceph-deploy succeeds with:
ceph-deploy osd create ceph01-test:sdb:/dev/sdc1
/dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,attr2,delaylog,noquota)
root@ceph01-test:~# l /var/lib/ceph/osd/ceph-0/
total 52
drwxr-xr-x 3 root root 182 May 29 19:01 ./
drwxr-xr-x 3 root root 4096 May 29 19:01 ../rw-r--r- 1 root root 499 May 29 19:01 activate.monmaprw-r--r- 1 root root 3 May 29 19:01 activerw-r--r- 1 root root 37 May 29 19:01 ceph_fsid
drwxr-xr-x 196 root root 8192 May 29 19:01 current/rw-r--r- 1 root root 37 May 29 19:01 fsid
lrwxrwxrwx 1 root root 9 May 29 19:01 journal > /dev/sdc1 1 root root 56 May 29 19:01 keyring
-rw------rw-r--r- 1 root root 21 May 29 19:01 magicrw-r--r- 1 root root 6 May 29 19:01 readyrw-r--r- 1 root root 4 May 29 19:01 store_versionrw-r--r- 1 root root 0 May 29 19:01 sysvinitrw-r--r- 1 root root 2 May 29 19:01 whoami
2013-05-29 19:01:16.538957 7f1e4bf81780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-osd, pid 3207
2013-05-29 19:01:16.591057 7f1e4bf81780 1 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mkfs in /var/lib/ceph/tmp/mnt.SLp5L9
2013-05-29 19:01:16.591149 7f1e4bf81780 1 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mkfs fsid is already set to b850484e-ac3f-47e5-9467-76119861ec97
2013-05-29 19:01:16.596155 7f1e4bf81780 1 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) leveldb db exists/created
2013-05-29 19:01:16.605998 7f1e4bf81780 -1 journal _check_disk_write_cache: fclose error: (0) Success
2013-05-29 19:01:16.606128 7f1e4bf81780 1 journal _open /var/lib/ceph/tmp/mnt.SLp5L9/journal fd 9: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:16.606442 7f1e4bf81780 -1 journal check: ondisk fsid 00000000-0000-0000-0000-000000000000 doesn't match expected b850484e-ac3f-47e5-9467-76119861ec97, invalid (someone else's?) journal
2013-05-29 19:01:16.607654 7f1e4bf81780 -1 journal _check_disk_write_cache: fclose error: (0) Success
2013-05-29 19:01:16.607684 7f1e4bf81780 1 journal _open /var/lib/ceph/tmp/mnt.SLp5L9/journal fd 9: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:16.608053 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mkjournal created journal on /var/lib/ceph/tmp/mnt.SLp5L9/journal
2013-05-29 19:01:16.608083 7f1e4bf81780 1 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mkfs done in /var/lib/ceph/tmp/mnt.SLp5L9
2013-05-29 19:01:16.612142 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount FIEMAP ioctl is supported and appears to work
2013-05-29 19:01:16.612162 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-05-29 19:01:16.612404 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount did NOT detect btrfs
2013-05-29 19:01:16.626275 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount syscall(SYS_syncfs, fd) fully supported
2013-05-29 19:01:16.626373 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount found snaps <>
2013-05-29 19:01:16.635964 7f1e4bf81780 0 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-05-29 19:01:16.637423 7f1e4bf81780 -1 journal _check_disk_write_cache: fclose error: (0) Success
2013-05-29 19:01:16.637463 7f1e4bf81780 1 journal _open /var/lib/ceph/tmp/mnt.SLp5L9/journal fd 15: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:16.639085 7f1e4bf81780 -1 journal _check_disk_write_cache: fclose error: (0) Success
2013-05-29 19:01:16.639119 7f1e4bf81780 1 journal _open /var/lib/ceph/tmp/mnt.SLp5L9/journal fd 15: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:16.647752 7f1e4bf81780 -1 filestore(/var/lib/ceph/tmp/mnt.SLp5L9) could not find 23c2fcde/osd_superblock/0//-1 in index: (2) No such file or directory
2013-05-29 19:01:16.702668 7f1e4bf81780 1 journal close /var/lib/ceph/tmp/mnt.SLp5L9/journal
2013-05-29 19:01:16.703705 7f1e4bf81780 -1 created object store /var/lib/ceph/tmp/mnt.SLp5L9 journal /var/lib/ceph/tmp/mnt.SLp5L9/journal for osd.0 fsid 79629e45-fa39-4aec-a125-d24c65513fb5
2013-05-29 19:01:16.704472 7f1e4bf81780 -1 auth: error reading file: /var/lib/ceph/tmp/mnt.SLp5L9/keyring: can't open /var/lib/ceph/tmp/mnt.SLp5L9/keyring: (2) No such file or directory
2013-05-29 19:01:16.704572 7f1e4bf81780 -1 created new key in keyring /var/lib/ceph/tmp/mnt.SLp5L9/keyring
2013-05-29 19:01:22.635153 7f05e9b93780 0 ceph version 0.61.2 (fea782543a844bb277ae94d3391788b76c5bee60), process ceph-osd, pid 3433
2013-05-29 19:01:22.664389 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount FIEMAP ioctl is supported and appears to work
2013-05-29 19:01:22.664431 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-05-29 19:01:22.664740 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount did NOT detect btrfs
2013-05-29 19:01:22.670025 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount syscall(SYS_syncfs, fd) fully supported
2013-05-29 19:01:22.670137 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount found snaps <>
2013-05-29 19:01:22.673622 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-05-29 19:01:22.675325 7f05e9b93780 -1 journal _check_disk_write_cache: fclose error: (61) No data available
2013-05-29 19:01:22.675373 7f05e9b93780 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:22.677206 7f05e9b93780 -1 journal _check_disk_write_cache: fclose error: (61) No data available
2013-05-29 19:01:22.677248 7f05e9b93780 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 18: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:22.678020 7f05e9b93780 1 journal close /var/lib/ceph/osd/ceph-0/journal
2013-05-29 19:01:22.684128 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount FIEMAP ioctl is supported and appears to work
2013-05-29 19:01:22.684154 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount FIEMAP ioctl is disabled via 'filestore fiemap' config option
2013-05-29 19:01:22.684413 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount did NOT detect btrfs
2013-05-29 19:01:22.706269 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount syscall(SYS_syncfs, fd) fully supported
2013-05-29 19:01:22.706370 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount found snaps <>
2013-05-29 19:01:22.707629 7f05e9b93780 0 filestore(/var/lib/ceph/osd/ceph-0) mount: enabling WRITEAHEAD journal mode: btrfs not detected
2013-05-29 19:01:22.714775 7f05e9b93780 -1 journal _check_disk_write_cache: fclose error: (61) No data available
2013-05-29 19:01:22.714843 7f05e9b93780 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 26: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
2013-05-29 19:01:22.716719 7f05e9b93780 -1 journal _check_disk_write_cache: fclose error: (61) No data available
2013-05-29 19:01:22.716760 7f05e9b93780 1 journal _open /var/lib/ceph/osd/ceph-0/journal fd 26: 1072693248 bytes, block size 4096 bytes, directio = 1, aio = 1
Updated by Sage Weil almost 11 years ago
- Status changed from New to Need More Info
- Assignee changed from Anonymous to Sage Weil
Hi Robert-
Are you still having this problem? Can you share a bit more information about the environment? What distro, what kind of VM, what cat /proc/partitions look like, 'find /dev/disk -ls' output.
Tamil, have we seen this behavior in our own testing?
Updated by Robert Sander almost 11 years ago
Hi Sage,
We are currently testing with some Debian wheezy VMs on a VMware ESXi host.
root@ceph01-test:~# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 7.0 (wheezy)
Release: 7.0
Codename: wheezy
root@ceph01-test:~# cat /proc/partitions
major minor #blocks name
8 0 10485760 sda
8 1 248832 sda1
8 2 1 sda2
8 5 10233856 sda5
8 16 16777216 sdb
8 17 16776175 sdb1
11 0 1048575 sr0
254 0 8388608 dm-0
254 1 471040 dm-1
254 2 1048576 dm-2
8 32 1048576 sdc
8 33 1047552 sdc1
root@ceph01-test:~# find /dev/disk -ls
1379 0 drwxr-xr-x 5 root root 100 May 29 18:53 /dev/disk
3165 0 drwxr-xr-x 2 root root 120 May 29 19:01 /dev/disk/by-uuid
8242 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-uuid/252666f7-a8d0-4c88-b186-89e6c782f97c -> ../../sdb1
4566 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-uuid/1c13ee3c-9f00-44b3-bb99-b93c73d0db54 -> ../../dm-1
3186 0 lrwxrwxrwx 1 root root 10 May 29 18:53 /dev/disk/by-uuid/620ce482-c580-45ba-ba9b-093a99aa9fdc -> ../../dm-0
3166 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-uuid/c5b1b672-1b4c-4141-8248-15247bbebfd5 -> ../../sda1
1384 0 drwxr-xr-x 2 root root 220 May 29 19:01 /dev/disk/by-path
8239 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0-part1 -> ../../sdb1
7992 0 lrwxrwxrwx 1 root root 10 Jun 10 09:03 /dev/disk/by-path/pci-0000:1b:00.0-scsi-0:0:0:0-part1 -> ../../sdc1
7582 0 lrwxrwxrwx 1 root root 9 Jun 3 18:13 /dev/disk/by-path/pci-0000:1b:00.0-scsi-0:0:0:0 -> ../../sdc
3162 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 -> ../../sda1
3155 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part5 -> ../../sda5
3148 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 -> ../../sda2
1391 0 lrwxrwxrwx 1 root root 9 Jun 3 18:13 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sda
3137 0 lrwxrwxrwx 1 root root 9 Jun 3 18:13 /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0 -> ../../sdb
1385 0 lrwxrwxrwx 1 root root 9 May 29 19:01 /dev/disk/by-path/pci-0000:00:07.1-scsi-1:0:0:0 -> ../../sr0
1380 0 drwxr-xr-x 2 root root 180 May 29 18:53 /dev/disk/by-id
3817 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-id/dm-uuid-LVM-xS9SLjlDiKzztIJHGT26xCIG6uWzFkbD5RFHNUoQP2Usp7X1j1etCWo76O99TrhX -> ../../dm-2
3814 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-id/dm-name-system-cephlog -> ../../dm-2
4563 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-id/dm-uuid-LVM-xS9SLjlDiKzztIJHGT26xCIG6uWzFkbD1eDV0QUCCYuxNj7INFYHOkF9drr7dM2I -> ../../dm-1
4560 0 lrwxrwxrwx 1 root root 10 May 29 19:01 /dev/disk/by-id/dm-name-system-swap_1 -> ../../dm-1
3183 0 lrwxrwxrwx 1 root root 10 May 29 18:53 /dev/disk/by-id/dm-uuid-LVM-xS9SLjlDiKzztIJHGT26xCIG6uWzFkbDIb6ddSp6lC2IKV8XFIQGI1mGVPue3y8F -> ../../dm-0
3180 0 lrwxrwxrwx 1 root root 10 May 29 18:53 /dev/disk/by-id/dm-name-system-root -> ../../dm-0
1381 0 lrwxrwxrwx 1 root root 9 May 29 19:01 /dev/disk/by-id/ata-VMware_Virtual_IDE_CDROM_Drive_10000000000000000001 -> ../../sr0Updated by Sage Weil almost 11 years ago
- Status changed from Need More Info to Resolved
this is now working with the fixes from #4984.