Project

General

Profile

Actions
Copy link

Bug #58438

open

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)

Added by Gunther Heinrich over 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

After upgrading a virtual cluster (running on Ubuntu 22.04 and Podman 3.4.4) from 15.2.17 to 16.2.10 the cluster cannot probe some daemons or devices anymore. I'm not sure if it is related to this issue but since the upgrade I lost one mon which strangely enough isn't even listed as out of quorum, it's simply gone (ceph orch apply mon label:mon didn't resolve that).

Here's the first stacktrace of the result from "ceph health detail" (the others are identical for different nodes):

[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
    host iz-ceph-v1-mon-01 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd -e NODE_NAME=iz-ceph-v1-mon-01 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/run/ceph:z -v /var/log/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/log/ceph:z -v /var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1ifqtpzn:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd inventory --format=json-pretty --filter-for-batch
/usr/bin/podman: stderr exception caught by decorator
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr     return f(*a, **kw)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr     description=self.help(),
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr     ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr     configuration.load(conf.path)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr     parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr     self.read_file(conffile)
/usr/bin/podman: stderr   File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr     self._read(f, source)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr     line = fp.readline()
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr     line = self.fp.readline()
/usr/bin/podman: stderr   File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr     return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr   File "/usr/sbin/ceph-volume", line 11, in <module>
/usr/bin/podman: stderr     load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
/usr/bin/podman: stderr     self.main(self.argv)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr     return f(*a, **kw)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr     description=self.help(),
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr     ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr     configuration.load(conf.path)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr     parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr     self.read_file(conffile)
/usr/bin/podman: stderr   File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr     self._read(f, source)
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr     line = fp.readline()
/usr/bin/podman: stderr   File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr     line = self.fp.readline()
/usr/bin/podman: stderr   File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr     return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
Traceback (most recent call last):
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8971, in <module>
    main()
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8959, in main
    r = ctx.func(ctx)
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1902, in _infer_config
    return func(ctx)
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1833, in _infer_fsid
    return func(ctx)
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1930, in _infer_image
    return func(ctx)
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1820, in _validate_fsid
    return func(ctx)
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 5172, in command_ceph_volume
    out, err, code = call_throws(ctx, c.run_cmd())
  File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1622, in call_throws
    raise RuntimeError('Failed command: %s' % ' '.join(command))
Actions
Copy link
 #1

Updated by Gunther Heinrich over 1 year ago

Update:
I think I found the culprit. Afte some source code checking and tests I managed the reproduce the error. The issue are comment lines containing special characters in the ceph.conf file. So if the conf contains the following line

#test ä

the parsing crashes with the same error as above since "ä" is not part of ascii ("UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)").

The question is: Why are comment lines not completely ignored by the phyton script /usr/lib/python3.6/site-packages/ceph_volume/configuration.py? Like "if line starts with '#': continue to next line" so that a parsing/decode error doesn't occur? And might this problem pop up in other python scripts as well?

Actions
Copy link
 #2

Updated by Redouane Kachach Elhichou about 1 year ago

  • Project changed from Orchestrator to ceph-volume
  • Category deleted (cephadm)

This is a ceph-volume issue. The code fails in the following lines of configuration.py:

        with contextlib.closing(ceph_file):
            parser.read_conf(trimmed_conf)
            conf.ceph = parser

Which call later to ConfigParser. Basically ConfigParser is not prepared to parse unicode.

Actions
Copy link
 #3

Updated by Ilya Dryomov about 1 year ago

  • Target version deleted (v16.2.11)
Actions
Copy link

Also available in: Atom PDF