Bug #17662: ceph-disk: using a regular file as a journal fails - Ceph - Ceph

Actions

Copy link

Bug #17662

closed

ceph-disk: using a regular file as a journal fails

Added by Jayashree Ajay Candadai over 7 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Jayashree Ajay Candadai

Category:

Target version:

% Done:

Source:

other

Tags:

Backport:

jewel

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

ceph-disk

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The return statement is missing when creating the journal space as regular file

1925         if stat.S_ISREG(mode):
1926             if getattr(args, name + '_dev'):
1927                 raise Error('%s is not a block device' % name.capitalize,
1928                             getattr(args, name))
1929             self.type = self.FILE

return is missing in the ceph-disk/main.py

So when we create the journal space we get the error:

*Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: ceph-disk: Error: <built-in method capitalize of str object at 0x7fbca1acf180> /tmp/ramdisk/journal-sdc.journal is neither a block device nor regular file*
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Loïc Dachary over 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Loïc Dachary over 7 years ago

Could you please explain where exactly the return statement should go ?

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

This needs to be added in line 1930.

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

I have created the pull request with
https://github.com/ceph/ceph/pull/11600

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

For the performance improvement I am trying to use 4G of tmpfs as journal space.

On my puppet master in my `site.pp`, I have the following snippet of code for OSD nodes

 
node /osd*/ {
  class { 'ceph::repo': }

  class { 'ceph':
    fsid                => $fsid,
    mon_initial_members => 'mon1',
    mon_host            => '10.10.1.1',
  }

  #prepare and activate present in ceph::osd class. '/dev/sdb' is the file system - mount point.
  ceph::osd { 
    '/dev/sdc':
    journal => '/tmp/ramdisk/journal-sdc.journal';
    '/dev/sdd':
    journal => '/tmp/ramdisk/journal-sdd.journal';
    '/dev/sde':
    journal => '/tmp/ramdisk/journal-sde.journal';
  }

  ceph::key {'client.bootstrap-osd':
    keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',
    secret       => $bootstrap_osd_key,
  }

  ceph::key { 'client.admin':
    secret => $admin_key,
  }
}

On the OSD nodes:

I am creating:

 
# tempfs as  CEPH journal
mkdir /tmp/ramdisk
chmod 777 /tmp/ramdisk

# Create a mount point and reserve small amount of RAM(4G) as filesystem.
sudo mount -t tmpfs -o size=4G tmpfs /tmp/ramdisk/

Suppose If I create the journal files

# Create the journal files for each OSD disk
cd /tmp/ramdisk
touch journal-sdc.journal
touch journal-sdd.journal
touch journal-sde.journal
sudo chmod 664 journal-sd*

and then apply the
`sudo puppet agent -t --waitforcert 1 --server <servername>`

Then I get an error

*Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: ceph-disk: Error: <built-in method capitalize of str object at 0x7fbca1acf180> /tmp/ramdisk/journal-sdc.journal is neither a block device nor regular file*
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]

This is because in `site.pp`, `return` statement is missing in line 1930

         if stat.S_ISREG(mode):
1926             if getattr(args, name + '_dev'):
1927                 raise Error('%s is not a block device' % name.capitalize,
1928                             getattr(args, name))
1929             self.type = self.FILE
1930 
1931         raise Error('%s %s is neither a block device nor regular file' %
1932                     (name.capitalize, getattr(args, name)))
1933

On a local `main.py` if I inserted return statement in `/usr/lib/python2.7/site-packages/ceph_disk/main.py` on my OSD and reran the command it gives an error:

Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + test -b /dev/sdc
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: prepare_file: OSD will not be hot-swappable if journal is not the same device as the osd data
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/sbin/ceph-disk", line 9, in <module>
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5012, in run
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     main(sys.argv[1:])
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4965, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     main_catch(args.func, args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4990, in main_catch
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     func(args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1791, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     Prepare.factory(args).prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1779, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_locked()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1811, in prepare_locked
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.data.prepare(self.journal)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2464, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_device(*to_prepare_list)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2640, in prepare_device
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     to_prepare.prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1984, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_file()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2005, in prepare_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.space_symlink = space_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: UnboundLocalError: local variable 'space_file' referenced before assignment
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Dependency Exec[ceph-osd-prepare-/dev/sdc] has failures: true
Warning: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Skipping because of failed dependencies

On removing the `journal` files in `/tmp/ramdisk` the command goes through, but the journal space does not get used

[jayaajay@osd2 tmp]$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda1                           16G  3.1G   12G  21% /
devtmpfs                            32G     0   32G   0% /dev
tmpfs                               32G     0   32G   0% /dev/shm
tmpfs                               32G   33M   32G   1% /run
tmpfs                               32G     0   32G   0% /sys/fs/cgroup
ops.apt.emulab.net:/proj/idms-PG0  100G  112K  100G   1% /proj/idms-PG0
ops.apt.emulab.net:/share          243G   22G  202G  10% /share
tmpfs                              6.3G     0  6.3G   0% /run/user/20001
tmpfs                              4.0G     0  4.0G   0% /tmp/ramdisk

[jayaajay@osd2 tmp]$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sdc   /dev/sdd   /dev/sde
/dev/sda1  /dev/sda3  /dev/sdb   /dev/sdc1  /dev/sdd1  /dev/sde1

[jayaajay@osd2 ramdisk]$ ls
journal-sdc.journal  journal-sdd.journal  journal-sde.journal

[jayaajay@osd2 tmp]$ cat /etc/ceph/ceph.conf 

[global]
auth_service_required = cephx
mon_initial_members = mon1
fsid = 066F558C-6789-4A93-AAF1-5AF1BA01A3AD
auth_supported = cephx
auth_cluster_required = cephx
mon_host = 10.10.1.1
auth_client_required = cephx

[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/
total 0
drwxr-xr-x 2 root root 100 Oct 21 14:34 .
drwxr-xr-x 9 root root 180 Oct 21 14:34 ..
lrwxrwxrwx 1 root root  10 Oct 21 14:34 60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Oct 21 14:34 6403c621-d064-4724-8e6f-97101862e75f -> ../../sde1
lrwxrwxrwx 1 root root  10 Oct 21 14:34 d4e539de-ca5a-4ae9-b687-14bdb7516531 -> ../../sdd1
[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76
lrwxrwxrwx 1 root root 10 Oct 21 14:34 /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

The space_symlink issue seam to have been fixed in https://github.com/ceph/ceph/commit/d290454cf5f660c1681eefd70a38e79da683525f commit into master. On changing it in jewel release local file `/usr/lib/python2.7/site-packages/ceph_disk/main.py` it does resolve the `UnboundLocalError: local variable 'space_file' referenced before assignment` correctly, But still the journal space is not being used by the OSD.

Actions

Copy link