https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2016-10-21T18:29:02ZCeph Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802082016-10-21T18:29:02ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/80208/diff?detail_id=77226">diff</a>)</li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802092016-10-21T18:30:11ZLoïc Dacharyloic@dachary.org
<ul></ul><p>Could you please explain where exactly the return statement should go ?</p> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802132016-10-21T19:47:35ZJayashree Ajay Candadaijayaajay@indiana.edu
<ul></ul><p>This needs to be added in line 1930.</p> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802142016-10-21T19:48:16ZJayashree Ajay Candadaijayaajay@indiana.edu
<ul></ul><p>I have created the pull request with <br /><a class="external" href="https://github.com/ceph/ceph/pull/11600">https://github.com/ceph/ceph/pull/11600</a></p> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802162016-10-21T21:29:06ZJayashree Ajay Candadaijayaajay@indiana.edu
<ul></ul><p>For the performance improvement I am trying to use 4G of tmpfs as journal space.</p>
<p>On my puppet master in my `site.pp`, I have the following snippet of code for OSD nodes</p>
<pre>
node /osd*/ {
class { 'ceph::repo': }
class { 'ceph':
fsid => $fsid,
mon_initial_members => 'mon1',
mon_host => '10.10.1.1',
}
#prepare and activate present in ceph::osd class. '/dev/sdb' is the file system - mount point.
ceph::osd {
'/dev/sdc':
journal => '/tmp/ramdisk/journal-sdc.journal';
'/dev/sdd':
journal => '/tmp/ramdisk/journal-sdd.journal';
'/dev/sde':
journal => '/tmp/ramdisk/journal-sde.journal';
}
ceph::key {'client.bootstrap-osd':
keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',
secret => $bootstrap_osd_key,
}
ceph::key { 'client.admin':
secret => $admin_key,
}
}
</pre>
<p>On the OSD nodes:</p>
<p>I am creating:</p>
<pre>
# tempfs as CEPH journal
mkdir /tmp/ramdisk
chmod 777 /tmp/ramdisk
# Create a mount point and reserve small amount of RAM(4G) as filesystem.
sudo mount -t tmpfs -o size=4G tmpfs /tmp/ramdisk/
</pre>
<p>Suppose If I create the journal files<br /><pre>
# Create the journal files for each OSD disk
cd /tmp/ramdisk
touch journal-sdc.journal
touch journal-sdd.journal
touch journal-sde.journal
sudo chmod 664 journal-sd*
</pre></p>
<p>and then apply the <br />`sudo puppet agent -t --waitforcert 1 --server <servername>`</p>
<p>Then I get an error<br /><pre>
*Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: ceph-disk: Error: <built-in method capitalize of str object at 0x7fbca1acf180> /tmp/ramdisk/journal-sdc.journal is neither a block device nor regular file*
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
mkdir -p /dev/sdc
if getent passwd ceph >/dev/null 2>&1; then
chown -h ceph:ceph /dev/sdc
fi
fi
ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
mkdir -p /dev/sdc
if getent passwd ceph >/dev/null 2>&1; then
chown -h ceph:ceph /dev/sdc
fi
fi
ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
returned 1 instead of one of [0]
</pre></p>
<p>This is because in `site.pp`, `return` statement is missing in line 1930 <br /><pre>
if stat.S_ISREG(mode):
1926 if getattr(args, name + '_dev'):
1927 raise Error('%s is not a block device' % name.capitalize,
1928 getattr(args, name))
1929 self.type = self.FILE
1930
1931 raise Error('%s %s is neither a block device nor regular file' %
1932 (name.capitalize, getattr(args, name)))
1933
</pre></p>
<p>On a local `main.py` if I inserted return statement in `/usr/lib/python2.7/site-packages/ceph_disk/main.py` on my OSD and reran the command it gives an error:</p>
<pre>
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + test -b /dev/sdc
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: prepare_file: OSD will not be hot-swappable if journal is not the same device as the osd data
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/sbin/ceph-disk", line 9, in <module>
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5012, in run
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: main(sys.argv[1:])
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4965, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: main_catch(args.func, args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4990, in main_catch
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: func(args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1791, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: Prepare.factory(args).prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1779, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: self.prepare_locked()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1811, in prepare_locked
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: self.data.prepare(self.journal)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2464, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: self.prepare_device(*to_prepare_list)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2640, in prepare_device
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: to_prepare.prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1984, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: self.prepare_file()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2005, in prepare_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: self.space_symlink = space_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: UnboundLocalError: local variable 'space_file' referenced before assignment
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
mkdir -p /dev/sdc
if getent passwd ceph >/dev/null 2>&1; then
chown -h ceph:ceph /dev/sdc
fi
fi
ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
mkdir -p /dev/sdc
if getent passwd ceph >/dev/null 2>&1; then
chown -h ceph:ceph /dev/sdc
fi
fi
ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Dependency Exec[ceph-osd-prepare-/dev/sdc] has failures: true
Warning: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Skipping because of failed dependencies
</pre>
<p>On removing the `journal` files in `/tmp/ramdisk` the command goes through, but the journal space does not get used</p>
<pre>
[jayaajay@osd2 tmp]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 16G 3.1G 12G 21% /
devtmpfs 32G 0 32G 0% /dev
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 32G 33M 32G 1% /run
tmpfs 32G 0 32G 0% /sys/fs/cgroup
ops.apt.emulab.net:/proj/idms-PG0 100G 112K 100G 1% /proj/idms-PG0
ops.apt.emulab.net:/share 243G 22G 202G 10% /share
tmpfs 6.3G 0 6.3G 0% /run/user/20001
tmpfs 4.0G 0 4.0G 0% /tmp/ramdisk
[jayaajay@osd2 tmp]$ ls /dev/sd*
/dev/sda /dev/sda2 /dev/sda4 /dev/sdc /dev/sdd /dev/sde
/dev/sda1 /dev/sda3 /dev/sdb /dev/sdc1 /dev/sdd1 /dev/sde1
[jayaajay@osd2 ramdisk]$ ls
journal-sdc.journal journal-sdd.journal journal-sde.journal
[jayaajay@osd2 tmp]$ cat /etc/ceph/ceph.conf
[global]
auth_service_required = cephx
mon_initial_members = mon1
fsid = 066F558C-6789-4A93-AAF1-5AF1BA01A3AD
auth_supported = cephx
auth_cluster_required = cephx
mon_host = 10.10.1.1
auth_client_required = cephx
[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/
total 0
drwxr-xr-x 2 root root 100 Oct 21 14:34 .
drwxr-xr-x 9 root root 180 Oct 21 14:34 ..
lrwxrwxrwx 1 root root 10 Oct 21 14:34 60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Oct 21 14:34 6403c621-d064-4724-8e6f-97101862e75f -> ../../sde1
lrwxrwxrwx 1 root root 10 Oct 21 14:34 d4e539de-ca5a-4ae9-b687-14bdb7516531 -> ../../sdd1
[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76
lrwxrwxrwx 1 root root 10 Oct 21 14:34 /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1
</pre> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802182016-10-21T22:24:35ZJayashree Ajay Candadaijayaajay@indiana.edu
<ul></ul><p>The space_symlink issue seam to have been fixed in <a class="external" href="https://github.com/ceph/ceph/commit/d290454cf5f660c1681eefd70a38e79da683525f">https://github.com/ceph/ceph/commit/d290454cf5f660c1681eefd70a38e79da683525f</a> commit into master. On changing it in jewel release local file `/usr/lib/python2.7/site-packages/ceph_disk/main.py` it does resolve the `UnboundLocalError: local variable 'space_file' referenced before assignment` correctly, But still the journal space is not being used by the OSD.</p> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802262016-10-22T10:33:23ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Backport</strong> set to <i>jewel</i></li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802482016-10-24T09:57:44ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assignee</strong> set to <i>Jayashree Ajay Candadai</i></li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802542016-10-24T10:10:50ZLoïc Dacharyloic@dachary.org
<ul></ul><p><a class="external" href="https://github.com/ceph/ceph/pull/11619">https://github.com/ceph/ceph/pull/11619</a></p> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=802692016-10-24T14:24:45ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Subject</strong> changed from <i>return statement missing in OSD preparespace</i> to <i>ceph-disk: using a regular file as a journal fails</i></li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=803722016-10-26T07:03:16ZKefu Chaitchaikov@gmail.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Pending Backport</i></li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=803752016-10-26T07:08:04ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-6 priority-high2 closed" href="/issues/17707">Backport #17707</a>: jewel: ceph-disk: using a regular file as a journal fails</i> added</li></ul> Ceph - Bug #17662: ceph-disk: using a regular file as a journal failshttps://tracker.ceph.com/issues/17662?journal_id=804422016-10-27T07:48:59ZLoïc Dacharyloic@dachary.org
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>