Bug #17662: ceph-disk: using a regular file as a journal fails - Ceph - Ceph

Custom queries

Backports: mimic
Backports: missing release
Backports: nautilus
Bluestore
Bug queue
Bug queue - no subprojects
Bug triage
Ceph backlog
Crash queue
Crash triage
Feature Requests
Feedback
My issues
Need Review
Pending backports
Priority queue
Product Backlog Scrub
Project Triage
Test Failures

Actions

Copy link

Bug #17662

closed

ceph-disk: using a regular file as a journal fails

Added by Jayashree Ajay Candadai over 7 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Jayashree Ajay Candadai

Category:

Target version:

% Done:

Source:

other

Tags:

Backport:

jewel

Regression:

Severity:

2 - major

Reviewed:

Affected Versions:

ceph-qa-suite:

ceph-disk

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

The return statement is missing when creating the journal space as regular file

1925         if stat.S_ISREG(mode):
1926             if getattr(args, name + '_dev'):
1927                 raise Error('%s is not a block device' % name.capitalize,
1928                             getattr(args, name))
1929             self.type = self.FILE

return is missing in the ceph-disk/main.py

So when we create the journal space we get the error:

*Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: ceph-disk: Error: <built-in method capitalize of str object at 0x7fbca1acf180> /tmp/ramdisk/journal-sdc.journal is neither a block device nor regular file*
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]

Related issues 1 (0 open — 1 closed)

Copied to Ceph - Backport #17707: jewel: ceph-disk: using a regular file as a journal fails

Resolved

Loïc Dachary

Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Loïc Dachary over 7 years ago

Description updated (diff)

Actions

Copy link

Updated by Loïc Dachary over 7 years ago

Could you please explain where exactly the return statement should go ?

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

This needs to be added in line 1930.

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

I have created the pull request with
https://github.com/ceph/ceph/pull/11600

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

For the performance improvement I am trying to use 4G of tmpfs as journal space.

On my puppet master in my `site.pp`, I have the following snippet of code for OSD nodes

 
node /osd*/ {
  class { 'ceph::repo': }

  class { 'ceph':
    fsid                => $fsid,
    mon_initial_members => 'mon1',
    mon_host            => '10.10.1.1',
  }

  #prepare and activate present in ceph::osd class. '/dev/sdb' is the file system - mount point.
  ceph::osd { 
    '/dev/sdc':
    journal => '/tmp/ramdisk/journal-sdc.journal';
    '/dev/sdd':
    journal => '/tmp/ramdisk/journal-sdd.journal';
    '/dev/sde':
    journal => '/tmp/ramdisk/journal-sde.journal';
  }

  ceph::key {'client.bootstrap-osd':
    keyring_path => '/var/lib/ceph/bootstrap-osd/ceph.keyring',
    secret       => $bootstrap_osd_key,
  }

  ceph::key { 'client.admin':
    secret => $admin_key,
  }
}

On the OSD nodes:

I am creating:

 
# tempfs as  CEPH journal
mkdir /tmp/ramdisk
chmod 777 /tmp/ramdisk

# Create a mount point and reserve small amount of RAM(4G) as filesystem.
sudo mount -t tmpfs -o size=4G tmpfs /tmp/ramdisk/

Suppose If I create the journal files

# Create the journal files for each OSD disk
cd /tmp/ramdisk
touch journal-sdc.journal
touch journal-sdd.journal
touch journal-sde.journal
sudo chmod 664 journal-sd*

and then apply the
`sudo puppet agent -t --waitforcert 1 --server <servername>`

Then I get an error

*Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: ceph-disk: Error: <built-in method capitalize of str object at 0x7fbca1acf180> /tmp/ramdisk/journal-sdc.journal is neither a block device nor regular file*
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]

This is because in `site.pp`, `return` statement is missing in line 1930

         if stat.S_ISREG(mode):
1926             if getattr(args, name + '_dev'):
1927                 raise Error('%s is not a block device' % name.capitalize,
1928                             getattr(args, name))
1929             self.type = self.FILE
1930 
1931         raise Error('%s %s is neither a block device nor regular file' %
1932                     (name.capitalize, getattr(args, name)))
1933

On a local `main.py` if I inserted return statement in `/usr/lib/python2.7/site-packages/ceph_disk/main.py` on my OSD and reran the command it gives an error:

Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + test -b /dev/sdc
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: + ceph-disk prepare /dev/sdc /tmp/ramdisk/journal-sdc.journal
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: prepare_file: OSD will not be hot-swappable if journal is not the same device as the osd data
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: Traceback (most recent call last):
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/sbin/ceph-disk", line 9, in <module>
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5012, in run
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     main(sys.argv[1:])
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4965, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     main_catch(args.func, args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4990, in main_catch
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     func(args)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1791, in main
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     Prepare.factory(args).prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1779, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_locked()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1811, in prepare_locked
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.data.prepare(self.journal)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2464, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_device(*to_prepare_list)
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2640, in prepare_device
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     to_prepare.prepare()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1984, in prepare
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.prepare_file()
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2005, in prepare_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns:     self.space_symlink = space_file
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: UnboundLocalError: local variable 'space_file' referenced before assignment
Error: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Error: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-prepare-/dev/sdc]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements
set -ex
if ! test -b /dev/sdc ; then
    mkdir -p /dev/sdc
    if getent passwd ceph >/dev/null 2>&1; then
        chown -h ceph:ceph /dev/sdc
    fi
fi
ceph-disk prepare   /dev/sdc /tmp/ramdisk/journal-sdc.journal
udevadm settle
 returned 1 instead of one of [0]
Notice: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Dependency Exec[ceph-osd-prepare-/dev/sdc] has failures: true
Warning: /Stage[main]/Main/Node[osd]/Ceph::Osd[/dev/sdc]/Exec[ceph-osd-activate-/dev/sdc]: Skipping because of failed dependencies

On removing the `journal` files in `/tmp/ramdisk` the command goes through, but the journal space does not get used

[jayaajay@osd2 tmp]$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/sda1                           16G  3.1G   12G  21% /
devtmpfs                            32G     0   32G   0% /dev
tmpfs                               32G     0   32G   0% /dev/shm
tmpfs                               32G   33M   32G   1% /run
tmpfs                               32G     0   32G   0% /sys/fs/cgroup
ops.apt.emulab.net:/proj/idms-PG0  100G  112K  100G   1% /proj/idms-PG0
ops.apt.emulab.net:/share          243G   22G  202G  10% /share
tmpfs                              6.3G     0  6.3G   0% /run/user/20001
tmpfs                              4.0G     0  4.0G   0% /tmp/ramdisk

[jayaajay@osd2 tmp]$ ls /dev/sd*
/dev/sda   /dev/sda2  /dev/sda4  /dev/sdc   /dev/sdd   /dev/sde
/dev/sda1  /dev/sda3  /dev/sdb   /dev/sdc1  /dev/sdd1  /dev/sde1

[jayaajay@osd2 ramdisk]$ ls
journal-sdc.journal  journal-sdd.journal  journal-sde.journal

[jayaajay@osd2 tmp]$ cat /etc/ceph/ceph.conf 

[global]
auth_service_required = cephx
mon_initial_members = mon1
fsid = 066F558C-6789-4A93-AAF1-5AF1BA01A3AD
auth_supported = cephx
auth_cluster_required = cephx
mon_host = 10.10.1.1
auth_client_required = cephx

[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/
total 0
drwxr-xr-x 2 root root 100 Oct 21 14:34 .
drwxr-xr-x 9 root root 180 Oct 21 14:34 ..
lrwxrwxrwx 1 root root  10 Oct 21 14:34 60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1
lrwxrwxrwx 1 root root  10 Oct 21 14:34 6403c621-d064-4724-8e6f-97101862e75f -> ../../sde1
lrwxrwxrwx 1 root root  10 Oct 21 14:34 d4e539de-ca5a-4ae9-b687-14bdb7516531 -> ../../sdd1
[jayaajay@osd2 tmp]$ ls -la /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76
lrwxrwxrwx 1 root root 10 Oct 21 14:34 /dev/disk/by-partuuid/60de72a1-0945-497d-bc3c-0d7631c07c76 -> ../../sdc1

Actions

Copy link

Updated by Jayashree Ajay Candadai over 7 years ago

The space_symlink issue seam to have been fixed in https://github.com/ceph/ceph/commit/d290454cf5f660c1681eefd70a38e79da683525f commit into master. On changing it in jewel release local file `/usr/lib/python2.7/site-packages/ceph_disk/main.py` it does resolve the `UnboundLocalError: local variable 'space_file' referenced before assignment` correctly, But still the journal space is not being used by the OSD.

Actions

Copy link