Bug #58438
openUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
0%
Description
After upgrading a virtual cluster (running on Ubuntu 22.04 and Podman 3.4.4) from 15.2.17 to 16.2.10 the cluster cannot probe some daemons or devices anymore. I'm not sure if it is related to this issue but since the upgrade I lost one mon which strangely enough isn't even listed as out of quorum, it's simply gone (ceph orch apply mon label:mon didn't resolve that).
Here's the first stacktrace of the result from "ceph health detail" (the others are identical for different nodes):
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host iz-ceph-v1-mon-01 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd -e NODE_NAME=iz-ceph-v1-mon-01 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/run/ceph:z -v /var/log/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/log/ceph:z -v /var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1ifqtpzn:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd inventory --format=json-pretty --filter-for-batch
/usr/bin/podman: stderr exception caught by decorator
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr description=self.help(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr configuration.load(conf.path)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr self.read_file(conffile)
/usr/bin/podman: stderr File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr self._read(f, source)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr line = fp.readline()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr line = self.fp.readline()
/usr/bin/podman: stderr File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
/usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
/usr/bin/podman: stderr self.main(self.argv)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr description=self.help(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr configuration.load(conf.path)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr self.read_file(conffile)
/usr/bin/podman: stderr File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr self._read(f, source)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr line = fp.readline()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr line = self.fp.readline()
/usr/bin/podman: stderr File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
Traceback (most recent call last):
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8971, in <module>
main()
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8959, in main
r = ctx.func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1902, in _infer_config
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1833, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1930, in _infer_image
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1820, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 5172, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1622, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
Updated by Gunther Heinrich over 1 year ago
Update:
I think I found the culprit. Afte some source code checking and tests I managed the reproduce the error. The issue are comment lines containing special characters in the ceph.conf file. So if the conf contains the following line
#test ä
the parsing crashes with the same error as above since "ä" is not part of ascii ("UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)").
The question is: Why are comment lines not completely ignored by the phyton script /usr/lib/python3.6/site-packages/ceph_volume/configuration.py? Like "if line starts with '#': continue to next line" so that a parsing/decode error doesn't occur? And might this problem pop up in other python scripts as well?
Updated by Redouane Kachach Elhichou over 1 year ago
- Project changed from Orchestrator to ceph-volume
- Category deleted (
cephadm)
This is a ceph-volume issue. The code fails in the following lines of configuration.py:
with contextlib.closing(ceph_file): parser.read_conf(trimmed_conf) conf.ceph = parser
Which call later to ConfigParser. Basically ConfigParser is not prepared to parse unicode.