Bug #46529
closedcephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy
0%
Description
/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217488 on centos_7.6
2020-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:57 smithi145 bash[9707]: Error: error removing storage for container "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c": remove /var/lib/containers/storage/overlay/3e200c9a65162f9c55a682cb3ea6b559b07a6569a910eb1057ea4f995067f9eb/merged: device or resource busy 2020-07-12T15:39:58.066 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:58 smithi145 bash[9707]: Error: error creating container storage: the container name "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c" is already in use by "4588abec9d4e9d178e083064e81a5766f0e2d800170592bc51a2d31f247e09b5". You have to remove that container to be able to reuse that name.: that name is already in use
/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217586 on centos_7.6
2020-07-12T16:13:53.582 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 12 16:13:53 smithi101 bash[9557]: Error: error removing storage for container "ceph-0cd81794-c45a-11ea-a06e-001a4aab830c-mon.b": remove /var/lib/containers/storage/overlay/5315326bcdd6353c37a96033711571342823d18d5207df15c7f9bc485e6c1be8/merged: device or resource busy 2020-07-12T16:13:53.918 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 12 16:13:53 smithi101 bash[9557]: Error: error creating container storage: the container name "ceph-0cd81794-c45a-11ea-a06e-001a4aab830c-mon.b" is already in use by "642817748ec62641582f40d49f61b5b92a10277b3f8c4cc8638e68562a0ac1f0". You have to remove that container to be able to reuse that name.: that name is already in use
This might be kernel related.
Updated by Sebastian Wagner almost 4 years ago
- Related to Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory added
Updated by Sebastian Wagner almost 4 years ago
/a/kchai-2020-07-18_13:35:09-rados-wip-kefu-testing-2020-07-18-1927-distro-basic-smithi/5237560
also centos 7.6 (based on master)
Updated by Sebastian Wagner almost 4 years ago
- Priority changed from High to Urgent
Updated by Sebastian Wagner almost 4 years ago
Some thoughts:
- https://github.com/ceph/ceph/pull/35719 remove centos_7 from suites/rados/cephadm
- https://github.com/ceph/ceph/pull/32377 test suites/rados/thrash-old-clients only centos_7!
Right now, this conflict makes `suites/rados/thrash-old-clients´ the only suite that tests cephadm on Centos 7. And it turns out that we likely have a problem with podman on CentOS 7.6. Thus I see two options:
1. We revert https://github.com/ceph/ceph/pull/35719 and continue to test cephadm on CentOS 7. Then, we'd need someone with in-depth podman experience to debug the issue we see with CentOS 7.6.
3. Alternatively, we revert https://github.com/ceph/ceph/pull/32377 and test `thrash-old-clients` using the traditional package based deployment.
Might be related to https://github.com/containers/podman/issues/2553#issuecomment-504229382
Updated by Brad Hubbard almost 4 years ago
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224005
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5223947
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224121
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224035
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224061
Updated by Sebastian Wagner almost 4 years ago
Brad Hubbard wrote:
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224005
thrash-old-clients on centos_7.6
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5223947
thrash-old-clients on centos_7.6
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224121
thrash-old-clients on centos_7.6
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224035
thrash-old-clients on centos_7.6
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224061
thrash-old-clients on centos_7.6
Updated by Sebastian Wagner almost 4 years ago
Seems podman on CentOS 7 is broken?
Updated by Brad Hubbard over 3 years ago
All 7.6.
/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289047
/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289135
/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289017
Updated by Sebastian Wagner over 3 years ago
Updated by Sebastian Wagner over 3 years ago
2020-08-21T02:46:48.316 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: time="2020-08-21T02:46:48Z" level=error msg="unable to remove container 93488819ddcce8e2873eb8ab7665ce8761a15f308a2b5c05a9d579bb4983ed38 after failing to start and attach to it" 2020-08-21T02:46:48.367 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory" 2020-08-21T02:46:48.368 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: : OCI runtime error
/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log
Updated by Sebastian Wagner over 3 years ago
- Related to Bug #46704: container_linux.go:349: "exec: \"stat\": executable file not found added
Updated by Neha Ojha over 3 years ago
/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377136
/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377278
/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377385
Updated by Deepika Upadhyay over 3 years ago
5378365, 5378277, 5378451, 5378510
yuriw-2020-08-26_18:16:40-rados-wip-yuri-testing-2020-08-26-1631-octopus-distro-basic-smithi/
yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379093
Updated by Yuri Weinstein over 3 years ago
Updated by Sebastian Wagner over 3 years ago
- Status changed from New to Fix Under Review
- Assignee set to Yuri Weinstein
- Pull request ID set to 36915
Updated by Yuri Weinstein over 3 years ago
- Assignee changed from Yuri Weinstein to Abhishek Lekshmanan
Updated by Yuri Weinstein over 3 years ago
https://github.com/ceph/ceph/pull/36915 merged
https://github.com/ceph/ceph/pull/36931 - octopus PR
Updated by Neha Ojha over 3 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Yuri Weinstein over 3 years ago
Updated by Sebastian Wagner over 3 years ago
- Status changed from Pending Backport to Resolved
Updated by Sebastian Wagner almost 3 years ago
- Related to Bug #50998: OSD replacement not working added