Project

General

Profile

Actions

Bug #61589

open

16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create

Added by Zakhar Kirpichenko 10 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Pacific deployed as 16.2.1 and upgraded several times to 16.2.13 with cephadm, ceph-crash throws the following error on all hosts:

ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create

ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z

The mount works correctly:

/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/
total 8
drwx------ 2 nobody nogroup 4096 May  6  2021 .
drwx------ 3 nobody nogroup 4096 May  6  2021 ..

/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a

/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash

/]# ls -al /var/lib/ceph/crash/posted/
total 8
drwx------ 2 nobody nobody 4096 Jun  1 18:26 .
drwx------ 3 nobody nobody 4096 May  6  2021 ..
-rw-r--r-- 1 root   root      0 Jun  1 18:26 a

I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. Someone in ceph-users suggested that it may be "a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13".

Actions #1

Updated by Zakhar Kirpichenko 10 months ago

I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container.

Actions #2

Updated by Mark Glines 6 months ago

I see the same issue on a cluster installed using cephadm 16.2.10, and upgraded to 17.2.6 using "ceph orch upgrade". So, the problem still exists after upgrading the cluster to Pacific.

Setting ownership by hand solved the problem for me too.

Actions #3

Updated by Mehmet Top 3 months ago

i also see this with "quincy".

Initaly installed Ceph Quincy 17.2.3 with cephadm then

Ceph Upgrade 17.2.3 to 17.2.5
Ceph Upgrade 17.2.5 to 17.2.6

drwx------ 3 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash
drwx------ 2 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash/posted/
Actions #4

Updated by Bastian Mäuser 2 months ago

I also see this on reef:

Feb 02 11:28:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create

Actions #5

Updated by Christian Rohmann about 2 months ago

I just ran into a similar issue with the Debian packaged Ceph: https://tracker.ceph.com/issues/64548
(/var/lib/ceph/crash/posted is part of the package, but not properly chowned)

Actions

Also available in: Atom PDF