Bug #61589
open
16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Added by Zakhar Kirpichenko 12 months ago.
Updated 3 months ago.
Description
Pacific deployed as 16.2.1 and upgraded several times to 16.2.13 with cephadm, ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/
total 8
drwx------ 2 nobody nogroup 4096 May 6 2021 .
drwx------ 3 nobody nogroup 4096 May 6 2021 ..
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash
/]# ls -al /var/lib/ceph/crash/posted/
total 8
drwx------ 2 nobody nobody 4096 Jun 1 18:26 .
drwx------ 3 nobody nobody 4096 May 6 2021 ..
-rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. Someone in ceph-users suggested that it may be "a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13".
I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container.
I see the same issue on a cluster installed using cephadm 16.2.10, and upgraded to 17.2.6 using "ceph orch upgrade". So, the problem still exists after upgrading the cluster to Pacific.
Setting ownership by hand solved the problem for me too.
i also see this with "quincy".
Initaly installed Ceph Quincy 17.2.3 with cephadm then
Ceph Upgrade 17.2.3 to 17.2.5
Ceph Upgrade 17.2.5 to 17.2.6
drwx------ 3 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash
drwx------ 2 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash/posted/
I also see this on reef:
Feb 02 11:28:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Also available in: Atom
PDF