Bug #61589
open16.2.13: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
0%
Description
Pacific deployed as 16.2.1 and upgraded several times to 16.2.13 with cephadm, ceph-crash throws the following error on all hosts:
ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
ceph-crash runs in docker, the container has the directory mounted: -v /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash:/var/lib/ceph/crash:z
The mount works correctly:
/var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# ls -al crash/posted/ total 8 drwx------ 2 nobody nogroup 4096 May 6 2021 . drwx------ 3 nobody nogroup 4096 May 6 2021 .. /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# touch crash/posted/a /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86]# docker exec -it c0cd2b8022d8 bash /]# ls -al /var/lib/ceph/crash/posted/ total 8 drwx------ 2 nobody nobody 4096 Jun 1 18:26 . drwx------ 3 nobody nobody 4096 May 6 2021 .. -rw-r--r-- 1 root root 0 Jun 1 18:26 a
I.e. the directory actually exists and is correctly mounted in the crash container, yet ceph-crash says it doesn't exist. Someone in ceph-users suggested that it may be "a permissions issue arising from https://github.com/ceph/ceph/pull/48804, which was included in 16.2.13".
Updated by Zakhar Kirpichenko 11 months ago
I fixed the issue by manually setting directory ownership for /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash and /var/lib/ceph/3f50555a-ae2a-11eb-a2fc-ffde44714d86/crash/posted to 167:167, which in my system is the user ID crash process uses inside the crash container.
Updated by Mark Glines 7 months ago
I see the same issue on a cluster installed using cephadm 16.2.10, and upgraded to 17.2.6 using "ceph orch upgrade". So, the problem still exists after upgrading the cluster to Pacific.
Setting ownership by hand solved the problem for me too.
Updated by Mehmet Top 4 months ago
i also see this with "quincy".
Initaly installed Ceph Quincy 17.2.3 with cephadm then
Ceph Upgrade 17.2.3 to 17.2.5
Ceph Upgrade 17.2.5 to 17.2.6
drwx------ 3 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash
drwx------ 2 nobody nogroup 4.0K Aug 23 16:03 /var/lib/ceph/ced29922-xxxxxx-69f92bxxxxxx/crash/posted/
Updated by Bastian Mäuser 3 months ago
I also see this on reef:
Feb 02 11:28:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:29:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:30:31 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:01 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:31:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:32:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:33:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:34:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:35:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:36:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:37:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:38:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:39:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:02 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Feb 02 11:40:32 ceph1 bash1332: ERROR:ceph-crash:directory /var/lib/ceph/crash/posted does not exist; please create
Updated by Christian Rohmann 2 months ago
I just ran into a similar issue with the Debian packaged Ceph: https://tracker.ceph.com/issues/64548
(/var/lib/ceph/crash/posted is part of the package, but not properly chowned)