Bug #1094
closed"mkcephfs -c /etc/ceph.conf --allhosts --mkbtrfs" finds /tmp/mkcephfs.**** directory no longer exists
Description
I have used ceph0.23 for quite sometime. But now after a fresh install and build of ceph 0.27.1
I see that during "mkcephfs" init /tmp/mkcephfs.**** gets deleted intermittently. There is no apparent error.
Log:
=== osd.50 ===
pushing conf and monmap to bzt10
umount: /dev/sdc3: not mounted
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdc3
nodesize 4096 leafsize 4096 sectorsize 4096 size 281.11GB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
* WARNING: Ceph is still under heavy development, and is only suitable for *
* testing and review. Do not trust it with important data. *
** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
rollback), but btrfs does not support the SNAP_CREATE_V2 ioctl
(added in Linux 2.6.37). Expect slow btrfs sync/commit
performance.
2011-05-17 11:19:39.235658 7fce764ce6f0 created object store /data/osd50 journal /data/osd50/journal for osd50 fsid 41cddb12-6412-8a82-29f6-7d7df26fab21
creating private key for osd.50 keyring /tmp/mkcephfs.tRHuFp9666/keyring.osd.50
creating /tmp/mkcephfs.tRHuFp9666/keyring.osd.50
collecting osd.50 key
=== osd.51 ===
pushing conf and monmap to bzt10
umount: /dev/sdd3: not mounted
WARNING! - Btrfs Btrfs v0.19 IS EXPERIMENTAL
WARNING! - see http://btrfs.wiki.kernel.org before using
fs created label (null) on /dev/sdd3
nodesize 4096 leafsize 4096 sectorsize 4096 size 281.11GB
Btrfs Btrfs v0.19
Scanning for Btrfs filesystems
* WARNING: Ceph is still under heavy development, and is only suitable for *
* testing and review. Do not trust it with important data. *
** WARNING: 'filestore btrfs snap' is enabled (for safe transactions,
rollback), but btrfs does not support the SNAP_CREATE_V2 ioctl
(added in Linux 2.6.37). Expect slow btrfs sync/commit
performance.
2011-05-17 11:22:18.164137 7f1a031586f0 created object store /data/osd51 journal /data/osd51/journal for osd51 fsid 41cddb12-6412-8a82-29f6-7d7df26fab21
creating private key for osd.51 keyring /tmp/mkcephfs.tRHuFp9666/keyring.osd.51
creating /tmp/mkcephfs.tRHuFp9666/keyring.osd.51
collecting osd.51 key
...
...
It has sucessfully created all 54 osds on 9 differnt node prior to failure below:
=== mds.0 ===
pushing conf and monmap to bzt1
creating private key for mds.0 keyring /tmp/mkcephfs.foFYOU2693/keyring.mds.0
creating /tmp/mkcephfs.foFYOU2693/keyring.mds.0
collecting mds.0 key
Building generic osdmap
2011-05-16 17:24:33.513745 7faea71146f0 common_init: unable to open config file.
highest numbered osd in /tmp/mkcephfs.foFYOU2693/conf is osd.
2011-05-16 17:24:33.519108 7f3ffa27c6f0 common_init: unable to open config file.
num osd = 1
/usr/local/bin/osdmaptool: osdmap file '/tmp/mkcephfs.foFYOU2693/osdmap'
/usr/local/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.foFYOU2693/osdmap
bufferlist::write_file(/tmp/mkcephfs.foFYOU2693/osdmap): failed to open file: error 2: No such file or directory
osdmaptool: error writing to '/tmp/mkcephfs.foFYOU2693/osdmap': error 2: No such file or directory
rm: cannot remove `/tmp/mkcephfs.foFYOU2693/*': No such file or directory
It seems that directory and files were created moment ago.. but no longer there.
Sample config file:
------------------
; global
[global]
; enable secure authentication
;auth supported = cephx
max open files = 131072
log file = /scratch/ceph_bup/ceph.log; monitors
; You need at least one. You need at least three if you want to
; tolerate any node failures. Always create an odd number.
[mon]
mon data = /scratch/ceph_bup/mon$id
; some minimal logging (just message traffic) to aid debugging
debug ms = 1
debug mon = 20[mon.0]
host = bzt1
mon addr = 192.168.2.101:6789
; mds
; You need at least one. Define two to get a standby.
[mds]
; where the mds keeps it's secret encryption keys
debug ms = 1
debug mds = 20
[mds.0]
host = bzt1
; osd
; You need at least one. Two if you want data to be replicated.
; Define as many as you like.
[osd]
debug ms = 1
debug osd = 10
debug journal = 20
debug filestore = 20
osd journal = /data/osd$id/journal
osd journal size = 1000 ; journal size, in megabytes
; This is where the btrfs volume will be mounted.
osd data = /data/osd$id
osd.0]
host = bzt2
btrfs devs = /dev/sda3
[osd.1]
host = bzt2
btrfs devs = /dev/sdb3
[osd.2]
host = bzt2
btrfs devs = /dev/sdc3
[osd.3]
host = bzt2
btrfs devs = /dev/sdd3
[osd.4]
host = bzt2
btrfs devs = /dev/sde3
[osd.5]
host = bzt2
btrfs devs = /dev/sdf3
; next host bz3
[osd.6]
host = bzt3
btrfs devs = /dev/sda3
[osd.7]
host = bzt3
btrfs devs = /dev/sdb3
[osd.8]
host = bzt3
btrfs devs = /dev/sdc3
[osd.9]
host = bzt3
btrfs devs = /dev/sdd3
[osd.10]
host = bzt3
btrfs devs = /dev/sde3
[osd.11]
host = bzt3