Project

General

Profile

Actions
Copy link

Bug #46529

closed

cephadm: error removing storage for container "...-mon": remove /var/lib/containers/storage/overlay/.../merged: device or resource busy

Added by Sebastian Wagner almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Abhishek Lekshmanan
Category:
cephadm (binary)
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
36915
Crash signature (v1):
Crash signature (v2):

Description

/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217488 on centos_7.6

2020-07-12T15:39:57.736 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:57 smithi145 bash[9707]: Error: error removing storage for container "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c": remove /var/lib/containers/storage/overlay/3e200c9a65162f9c55a682cb3ea6b559b07a6569a910eb1057ea4f995067f9eb/merged: device or resource busy
2020-07-12T15:39:58.066 INFO:journalctl@ceph.mon.c.smithi145.stdout:Jul 12 15:39:58 smithi145 bash[9707]: Error: error creating container storage: the container name "ceph-567f836e-c455-11ea-a06e-001a4aab830c-mon.c" is already in use by "4588abec9d4e9d178e083064e81a5766f0e2d800170592bc51a2d31f247e09b5". You have to remove that container to be able to reuse that name.: that name is already in use

/a/teuthology-2020-07-12_07:01:02-rados-master-distro-basic-smithi/5217586 on centos_7.6

2020-07-12T16:13:53.582 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 12 16:13:53 smithi101 bash[9557]: Error: error removing storage for container "ceph-0cd81794-c45a-11ea-a06e-001a4aab830c-mon.b": remove /var/lib/containers/storage/overlay/5315326bcdd6353c37a96033711571342823d18d5207df15c7f9bc485e6c1be8/merged: device or resource busy
2020-07-12T16:13:53.918 INFO:journalctl@ceph.mon.b.smithi101.stdout:Jul 12 16:13:53 smithi101 bash[9557]: Error: error creating container storage: the container name "ceph-0cd81794-c45a-11ea-a06e-001a4aab830c-mon.b" is already in use by "642817748ec62641582f40d49f61b5b92a10277b3f8c4cc8638e68562a0ac1f0". You have to remove that container to be able to reuse that name.: that name is already in use

This might be kernel related.

Related issues 3 (0 open — 3 closed)

Related to Orchestrator - Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directoryCan't reproduceSebastian Wagner

Actions
Related to Orchestrator - Bug #46704: container_linux.go:349: "exec: \"stat\": executable file not found Can't reproduce

Actions
Related to Orchestrator - Bug #50998: OSD replacement not workingResolvedJuan Miguel Olmo Martínez

Actions
Actions
Copy link
 #1

Updated by Sebastian Wagner almost 4 years ago

  • Related to Bug #44990: cephadm: exec: "/usr/bin/ceph-mon": stat /usr/bin/ceph-mon: no such file or directory added
Actions
Copy link
 #3

Updated by Sebastian Wagner almost 4 years ago

/a/kchai-2020-07-18_13:35:09-rados-wip-kefu-testing-2020-07-18-1927-distro-basic-smithi/5237560

also centos 7.6 (based on master)

Actions
Copy link
 #5

Updated by Sebastian Wagner almost 4 years ago

  • Priority changed from High to Urgent
Actions
Copy link
 #6

Updated by Sebastian Wagner almost 4 years ago

Some thoughts:

Right now, this conflict makes `suites/rados/thrash-old-clients´ the only suite that tests cephadm on Centos 7. And it turns out that we likely have a problem with podman on CentOS 7.6. Thus I see two options:

1. We revert https://github.com/ceph/ceph/pull/35719 and continue to test cephadm on CentOS 7. Then, we'd need someone with in-depth podman experience to debug the issue we see with CentOS 7.6.
3. Alternatively, we revert https://github.com/ceph/ceph/pull/32377 and test `thrash-old-clients` using the traditional package based deployment.

Might be related to https://github.com/containers/podman/issues/2553#issuecomment-504229382

Actions
Copy link
 #7

Updated by Brad Hubbard over 3 years ago

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224005
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5223947
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224121
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224035
/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224061

Actions
Copy link
 #8

Updated by Sebastian Wagner over 3 years ago

Brad Hubbard wrote:

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224005

thrash-old-clients on centos_7.6

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5223947

thrash-old-clients on centos_7.6

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224121

thrash-old-clients on centos_7.6

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224035

thrash-old-clients on centos_7.6

/a/yuriw-2020-07-13_23:00:15-rados-wip-yuri8-testing-2020-07-13-1946-octopus-distro-basic-smithi/5224061

thrash-old-clients on centos_7.6

Actions
Copy link
 #9

Updated by Sebastian Wagner over 3 years ago

Seems podman on CentOS 7 is broken?

Actions
Copy link
 #10

Updated by Brad Hubbard over 3 years ago

  • Backport set to octopus
Actions
Copy link
 #11

Updated by Brad Hubbard over 3 years ago

All 7.6.

/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289047
/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289135
/a/yuriw-2020-08-05_14:55:18-rados-wip-yuri-testing-2020-08-04-2244-octopus-distro-basic-smithi/5289017

Actions
Copy link
 #13

Updated by Sebastian Wagner over 3 years ago 

2020-08-21T02:46:48.316 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: time="2020-08-21T02:46:48Z" level=error msg="unable to remove container 93488819ddcce8e2873eb8ab7665ce8761a15f308a2b5c05a9d579bb4983ed38 after failing to start and attach to it" 
2020-08-21T02:46:48.367 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: Error: container_linux.go:345: starting container process caused "exec: \"/usr/bin/ceph-mon\": stat /usr/bin/ceph-mon: no such file or directory" 
2020-08-21T02:46:48.368 INFO:journalctl@ceph.mon.c.smithi040.stdout:Aug 21 02:46:48 smithi040 bash[8791]: : OCI runtime error

/a/yuriw-2020-08-20_00:20:21-rados-wip-yuri7-testing-2020-08-19-2051-octopus-distro-basic-smithi/5360912/teuthology.log

Actions
Copy link
 #14

Updated by Sebastian Wagner over 3 years ago

  • Related to Bug #46704: container_linux.go:349: "exec: \"stat\": executable file not found added
Actions
Copy link
 #15

Updated by Neha Ojha over 3 years ago

/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377136
/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377278
/a/teuthology-2020-08-26_07:01:02-rados-master-distro-basic-smithi/5377385

Actions
Copy link
 #16

Updated by Deepika Upadhyay over 3 years ago

5378365, 5378277, 5378451, 5378510
yuriw-2020-08-26_18:16:40-rados-wip-yuri-testing-2020-08-26-1631-octopus-distro-basic-smithi/

yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379093

Actions
Copy link
 #18

Updated by Sebastian Wagner over 3 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Yuri Weinstein
  • Pull request ID set to 36915
Actions
Copy link
 #19

Updated by Yuri Weinstein over 3 years ago

  • Assignee changed from Yuri Weinstein to Abhishek Lekshmanan
Actions
Copy link
 #21

Updated by Neha Ojha over 3 years ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #22

Updated by Yuri Weinstein over 3 years ago

Yuri Weinstein wrote:

https://github.com/ceph/ceph/pull/36931 - octopus PR

merged

Actions
Copy link
 #23

Updated by Sebastian Wagner over 3 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #24

Updated by Sebastian Wagner almost 3 years ago

  • Related to Bug #50998: OSD replacement not working added
Actions
Copy link

Also available in: Atom PDF