Ceph » Orchestrator

Current theory: common denominator is the ubuntu 18.04 version of podman, which is currently:

2021-01-25T17:04:04.118 INFO:teuthology.orchestra.run.smithi097.stdout:Get:4 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  conmon 2.0.24~1 [32.1 kB]
2021-01-25T17:04:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:Get:5 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  containers-golang 0.6.0~1 [4,020 B]
2021-01-25T17:04:04.119 INFO:teuthology.orchestra.run.smithi097.stdout:Get:6 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  containers-image 5.8.1~1 [24.3 kB]
2021-01-25T17:04:04.121 INFO:teuthology.orchestra.run.smithi097.stdout:Get:7 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  containers-common 1.2.1~2 [13.6 kB]
2021-01-25T17:04:04.122 INFO:teuthology.orchestra.run.smithi097.stdout:Get:8 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  crun 0.16~2 [220 kB]
2021-01-25T17:04:04.218 INFO:teuthology.orchestra.run.smithi097.stdout:Get:9 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  containernetworking-plugins 0.8.7~1 [6,360 kB]
2021-01-25T17:04:04.736 INFO:teuthology.orchestra.run.smithi097.stdout:Get:10 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  podman-plugins 1.1.1~1 [1,035 kB]
2021-01-25T17:04:04.930 INFO:teuthology.orchestra.run.smithi097.stdout:Get:11 https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04  podman 2.2.1~4 [18.5 MB]

All of the failures seem to happen on ubuntu_18.04_podman.yaml. 18.04 with docker and 20.04 with podman seem unaffected.

This sounds like it might be the problem: https://github.com/containers/podman/issues/5046#issuecomment-600330600 ... except that someone claims it is resolved in conmon 2.0.14 and we're using version 2.0.24 and are still seeing (our) problem.

The reproducer on that other bug doesn't work for 20.04.. different bug. However, I can reproduce it on ubuntu 18.04 (and not on fedora).

working fedora box:

[root@gnit ~]# podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ; podman -v ; grep PRETTY /etc/os-release 
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.727671 s, 687 kB/s
-rw-r--r-- 1 root root 500000 Jan 25 17:51 /tmp/test
podman version 2.2.1
PRETTY_NAME="Fedora 32 (Server Edition)" 

bad ubuntu box (not changing size of /tmp/test)

root@teuthology:~# podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ; podman -v ; grep PRETTY /etc/os-release 
-rw-r--r-- 1 root root 497008 Jan 25 23:52 /tmp/test
podman version 2.2.1
PRETTY_NAME="Ubuntu 18.04.4 LTS" 
root@teuthology:~# podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ; podman -v ; grep PRETTY /etc/os-release 
-rw-r--r-- 1 root root 2115 Jan 25 23:52 /tmp/test
podman version 2.2.1
PRETTY_NAME="Ubuntu 18.04.4 LTS" 
root@teuthology:~# podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ; podman -v ; grep PRETTY /etc/os-release 
-rw-r--r-- 1 root root 134809 Jan 25 23:52 /tmp/test
podman version 2.2.1
PRETTY_NAME="Ubuntu 18.04.4 LTS" 
root@teuthology:~# podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ; podman -v ; grep PRETTY /etc/os-release 
-rw-r--r-- 1 root root 1928 Jan 25 23:52 /tmp/test
podman version 2.2.1
PRETTY_NAME="Ubuntu 18.04.4 LTS" 

Filed https://github.com/containers/podman/issues/9096

i have the same issue when testing Ubuntu_20.04 + conmon 2.0.24~1

- /a/kchai-2021-01-26_13:24:13-rados:cephadm-wip-kefu-testing-2021-01-26-1857-distro-basic-smithi/5830522
- /a/kchai-2021-01-26_13:24:13-rados:cephadm-wip-kefu-testing-2021-01-26-1857-distro-basic-smithi/5830523
- /a/kchai-2021-01-26_13:24:13-rados:cephadm-wip-kefu-testing-2021-01-26-1857-distro-basic-smithi/5830527

/ceph/teuthology-archive/yuriw-2021-01-28_19:54:33-rados-wip-yuri4-testing-2021-01-28-0959-octopus-distro-basic-smithi/5835938/teuthology.log