Actions
Bug #58438
openUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
After upgrading a virtual cluster (running on Ubuntu 22.04 and Podman 3.4.4) from 15.2.17 to 16.2.10 the cluster cannot probe some daemons or devices anymore. I'm not sure if it is related to this issue but since the upgrade I lost one mon which strangely enough isn't even listed as out of quorum, it's simply gone (ceph orch apply mon label:mon didn't resolve that).
Here's the first stacktrace of the result from "ceph health detail" (the others are identical for different nodes):
[WRN] CEPHADM_REFRESH_FAILED: failed to probe daemons or devices
host iz-ceph-v1-mon-01 `cephadm ceph-volume` failed: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd -e NODE_NAME=iz-ceph-v1-mon-01 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/run/ceph:z -v /var/log/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c:/var/log/ceph:z -v /var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp1ifqtpzn:/etc/ceph/ceph.conf:z quay.io/ceph/ceph@sha256:2b68483bcd050472a18e73389c0e1f3f70d34bb7abf733f692e88c935ea0a6bd inventory --format=json-pretty --filter-for-batch
/usr/bin/podman: stderr exception caught by decorator
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr description=self.help(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr configuration.load(conf.path)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr self.read_file(conffile)
/usr/bin/podman: stderr File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr self._read(f, source)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr line = fp.readline()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr line = self.fp.readline()
/usr/bin/podman: stderr File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
/usr/bin/podman: stderr Traceback (most recent call last):
/usr/bin/podman: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
/usr/bin/podman: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
/usr/bin/podman: stderr self.main(self.argv)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
/usr/bin/podman: stderr return f(*a, **kw)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 114, in main
/usr/bin/podman: stderr description=self.help(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 47, in help
/usr/bin/podman: stderr ceph_path=self.stat_ceph_conf(),
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 83, in stat_ceph_conf
/usr/bin/podman: stderr configuration.load(conf.path)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 59, in load
/usr/bin/podman: stderr parser.read_conf(trimmed_conf)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 227, in read_conf
/usr/bin/podman: stderr self.read_file(conffile)
/usr/bin/podman: stderr File "/usr/lib64/python3.6/configparser.py", line 718, in read_file
/usr/bin/podman: stderr self._read(f, source)
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 145, in _read
/usr/bin/podman: stderr line = fp.readline()
/usr/bin/podman: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/configuration.py", line 33, in readline
/usr/bin/podman: stderr line = self.fp.readline()
/usr/bin/podman: stderr File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
/usr/bin/podman: stderr return codecs.ascii_decode(input, self.errors)[0]
/usr/bin/podman: stderr UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 136: ordinal not in range(128)
Traceback (most recent call last):
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8971, in <module>
main()
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 8959, in main
r = ctx.func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1902, in _infer_config
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1833, in _infer_fsid
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1930, in _infer_image
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1820, in _validate_fsid
return func(ctx)
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 5172, in command_ceph_volume
out, err, code = call_throws(ctx, c.run_cmd())
File "/var/lib/ceph/68317c90-b44f-11ea-a0c4-d1443a31407c/cephadm.7ce656a8721deb5054c37b0cfb90381522d521dde51fb0c5a2142314d663f63d", line 1622, in call_throws
raise RuntimeError('Failed command: %s' % ' '.join(command))
Actions