Project

General

Profile

Bug #38221

LC_NUMERIC can make lvm.py fail

Added by Patrick Ernzer 5 months ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
02/07/2019
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-ansible
Pull request ID:

Description

(I initialy filed this in the wrong place, https://github.com/ceph/ceph-ansible/issues/3579 )

Bug Report

What happened:
As a European, I am used to writing e.g. "two and a half" as 2,5 (not 2.5). As such I tend to set `LC_NUMERIC=en_DK.UTF-8` on my Linux workstation1.
When I ssh to a RHEL7 box, these get then passed as per the default `AcceptEnv LANG LC_CTYPE LC_NUMERIC`… etc in `/etc/ssh/sshd_config` of RHEL7 (also affects Fedora 28).

So, with this setting, the OSDs will get the var as per the default sshd_config on the OSDs and (as expected) have the comma separator in `vgs` output.

[root@odroid-hc2-00 ~]# env|grep LC
LC_MEASUREMENT=en_DK.UTF-8
LC_MONETARY=de_DE.UTF-8
LC_COLLATE=en_US.UTF-8
LC_NUMERIC=en_DK.UTF-8
LC_TIME=de_DE.UTF-8
[root@odroid-hc2-00 ~]# vgs --noheadings --readonly --units=g --separator=";" 
  ceph-64ff50de-28b8-4bb6-bf76-601df9dc1e8e;1;1;0;wz--n-;931,48g;0g
[root@odroid-hc2-00 ~]# unset LC_NUMERIC
[root@odroid-hc2-00 ~]# vgs --noheadings --readonly --units=g --separator=";" 
  ceph-64ff50de-28b8-4bb6-bf76-601df9dc1e8e;1;1;0;wz--n-;931.48g;0g

It seems that `/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py` on the OSDs chokes on the comma, expecting a dot. (my OSDs are Fedora 28 ARM based, do tell if you need me to reproduce with RHEL7 x86_64 based OSDs)

If I unset the LC_NUMERIC on the host running ceph-ansible, `ansible-playbook site.yml` (copy of site.yml.samle) works just fine.
If `LC_NUMERIC=en_DK.UTF-8`, then I get

2019-02-06 15:49:45,682 p=27999 u=ansible |  TASK [ceph-osd : use ceph-volume lvm batch to create bluestore osds] **************************************************
2019-02-06 15:49:45,683 p=27999 u=ansible |  Wednesday 06 February 2019  15:49:45 +0100 (0:00:00.508)       0:06:06.035 **** 
2019-02-06 15:49:49,216 p=27999 u=ansible |  fatal: [odroid-hc2-03]: FAILED! => {"changed": true, "cmd": ["ceph-volume", "--cluster", "ceph", "lvm", "batch", "--bluestore", "--yes", "--dmcrypt", "/dev/sda"], "delta": "0:00:02.467296", "end": "2019-02-06 15:49:49.161515", "msg": "non-zero return code", "rc": 1, "start": "2019-02-06 15:49:46.694219", "stderr": "Traceback (most recent call last):\n  File \"/usr/sbin/ceph-volume\", line 11, in <module>\n    load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in __init__\n    self.main(self.argv)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc\n    return f(*a, **kw)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 148, in main\n    terminal.dispatch(self.mapper, subcommand_args)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch\n    instance.main()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main\n    terminal.dispatch(self.mapper, self.argv)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch\n    instance.main()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root\n    return func(*a, **kw)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 284, in main\n    self.execute(args)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 175, in execute\n    strategy.execute()\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/strategies/bluestore.py\", line 126, in execute\n    lvs = lvm.create_lvs(create['vg'], parts=create['parts'], name_prefix='osd-data')\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 643, in create_lvs\n    sizing = volume_group.sizing(parts=parts, size=size)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 1067, in sizing\n    size = int(self.free / parts)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 1008, in free\n    return self._parse_size(self.vg_free)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 994, in _parse_size\n    return util.str_to_int(integer)\n  File \"/usr/lib/python2.7/site-packages/ceph_volume/util/__init__.py\", line 40, in str_to_int\n    raise RuntimeError(error_msg)\nRuntimeError: Unable to convert to integer: '1862,98'", "stderr_lines": ["Traceback (most recent call last):", "  File \"/usr/sbin/ceph-volume\", line 11, in <module>", "    load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 38, in __init__", "    self.main(self.argv)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 59, in newfunc", "    return f(*a, **kw)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/main.py\", line 148, in main", "    terminal.dispatch(self.mapper, subcommand_args)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch", "    instance.main()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/main.py\", line 40, in main", "    terminal.dispatch(self.mapper, self.argv)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/terminal.py\", line 182, in dispatch", "    instance.main()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/decorators.py\", line 16, in is_root", "    return func(*a, **kw)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 284, in main", "    self.execute(args)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/batch.py\", line 175, in execute", "    strategy.execute()", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/devices/lvm/strategies/bluestore.py\", line 126, in execute", "    lvs = lvm.create_lvs(create['vg'], parts=create['parts'], name_prefix='osd-data')", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 643, in create_lvs", "    sizing = volume_group.sizing(parts=parts, size=size)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 1067, in sizing", "    size = int(self.free / parts)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 1008, in free", "    return self._parse_size(self.vg_free)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/api/lvm.py\", line 994, in _parse_size", "    return util.str_to_int(integer)", "  File \"/usr/lib/python2.7/site-packages/ceph_volume/util/__init__.py\", line 40, in str_to_int", "    raise RuntimeError(error_msg)", "RuntimeError: Unable to convert to integer: '1862,98'"], "stdout": "Running command: vgcreate --force --yes ceph-8e748fdb-e322-4be0-b3c1-6bc0f1120351 /dev/sda\n stdout: Physical volume \"/dev/sda\" successfully created.\n stdout: Volume group \"ceph-8e748fdb-e322-4be0-b3c1-6bc0f1120351\" successfully created", "stdout_lines": ["Running command: vgcreate --force --yes ceph-8e748fdb-e322-4be0-b3c1-6bc0f1120351 /dev/sda", " stdout: Physical volume \"/dev/sda\" successfully created.", " stdout: Volume group \"ceph-8e748fdb-e322-4be0-b3c1-6bc0f1120351\" successfully created"]}

When this happens, the OSD node have a VG defined but no LV in that VG after the playbook errored out. I'll attach a full log of the ansible-playbook run.

What you expected to happen:
If ceph-ansible has requirements for LANG and/or LC_NUMERIC, it should enforce their values.

How to reproduce it (minimal and precise):

`LC_NUMERIC=en_DK.UTF-8 ansible-playbook site.yml` with the following;

[ansible@ceph-ansible ceph-ansible]$ grep "^[^#;]" group_vars/osds.yml
---
dummy:
copy_admin_key: true
devices:
  - /dev/sda
dmcrypt: True
osd_scenario: lvm
[ansible@ceph-ansible ceph-ansible]$ grep "^[^#;]" group_vars/all.yml
---
dummy:
fetch_directory: ~/ceph-ansible-keys
cluster: ceph
configure_firewall: False
ntp_service_enabled: true
ntp_daemon_type: chronyd
ceph_origin: distro
ceph_repository: rhcs
ceph_rhcs_version: 3
ceph_repository_type: cdn
fsid: "{{ cluster_uuid.stdout }}" 
generate_fsid: True
rbd_cache_writethrough_until_flush: "false" 
rbd_client_directories: false # as per  CEPH125-RHCS3.0-en-1-20180517 pages 45 and 60
monitor_interface: eth0
journal_size: 5120 # OSD journal size in MB
public_network: 192.168.50.0/24 # HouseNet
cluster_network: "{{ public_network | regex_replace(' ', '') }}" 
ceph_conf_overrides:
  global:
    mon_allow_pool_delete: true
  client:
    rbd_default_features: 1
ceph_docker_image: "rhceph-3-rhel7" 
ceph_docker_image_tag: "latest" 
ceph_docker_registry: "registry.access.redhat.com/rhceph/" 
[ansible@ceph-ansible ceph-ansible]$ cat /etc/ansible/hosts
[ceph-arm-nodes]
odroid-hc2-[00:04]

[ceph-x86-nodes]
ceph-ansible

[ceph-housenet:children]
ceph-arm-nodes
ceph-x86-nodes

[ceph-housenet:vars]
ansible_user=ansible

[mons]
odroid-hc2-[00:02]

[mgrs]
odroid-hc2-[00:02]

[osds]
odroid-hc2-[00:04]

[clients]
odroid-hc2-00

[ansible@ceph-ansible ceph-ansible]$ 
Environment:
  • OS (e.g. from /etc/os-release): Red Hat Enterprise Linux Server release 7.6 (Maipo)
  • Kernel (e.g. `uname -a`): Linux ceph-ansible.internal.pcfe.net 3.10.0-862.el7.x86_64 #1 SMP Wed Mar 21 18:14:51 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Docker version if applicable (e.g. `docker version`):
  • Ansible version (e.g. `ansible-playbook --version`): ansible-playbook 2.6.12
  • ceph-ansible version (e.g. `git head or tag or stable branch`): ceph-ansible-3.2.4-1.el7cp.noarch
  • Ceph version (e.g. `ceph -v`): ceph version 12.2.10 (177915764b752804194937482a39e95e0ca3de94) luminous (stable)

note: all except ceph -v collected on the RHEL7 x86_64 machine running ceph-ansible. ceph -v was collected on an affected OSD running Fedora 28 ARM

[1] JFYI; the complete list of language related varoables I normally set is;

LANG=en_US.UTF-8
GDM_LANG=en_GB.UTF-8
LANGUAGE=en_GB:en_US
LC_MONETARY=de_DE.UTF-8
LC_NUMERIC=en_DK.UTF-8
LC_COLLATE=en_US.UTF-8
LC_MEASUREMENT=en_DK.UTF-8
LC_TIME=de_DE.UTF-8

logfile.txt View - log of ceph-ansible run (348 KB) Patrick Ernzer, 02/07/2019 11:49 AM

History

#1 Updated by Patrick Ernzer 5 months ago

do tell if you need more logs, this is stil a test cluster so I can wipe easily if more logs needed. But hopefully the reproducer is sufficient.

Also available in: Atom PDF