Project

General

Profile

Actions

Bug #43862

closed

mkfs fsck found fatal error: (2) No such file or directory during cephadm osd create

Added by Sage Weil about 4 years ago. Updated about 4 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-01-28T13:52:33.931 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 7e92ec97-90a8-49bd-8dc5-dc3f8b8dfa53
2020-01-28T13:52:33.931 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-authtool --gen-print-key
2020-01-28T13:52:33.933 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-11
2020-01-28T13:52:33.933 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -h ceph:ceph /dev/vg_nvme/lv_3
2020-01-28T13:52:33.933 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2
2020-01-28T13:52:33.934 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ln -s /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-11/block
2020-01-28T13:52:33.934 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-11/activate.monmap
2020-01-28T13:52:33.935 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: got monmap epoch 3
2020-01-28T13:52:33.936 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-11/keyring --create-keyring --name osd.11 --add-key AQCcPDBeneBINRAA2YN5xnLkCftUN8Ztxp6QNA==
2020-01-28T13:52:33.936 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stdout: creating /var/lib/ceph/osd/ceph-11/keyring
2020-01-28T13:52:33.937 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr added entity osd.11 auth(key=AQCcPDBeneBINRAA2YN5xnLkCftUN8Ztxp6QNA==)
2020-01-28T13:52:33.937 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-11/keyring
2020-01-28T13:52:33.938 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-11/
2020-01-28T13:52:33.938 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 11 --monmap /var/lib/ceph/osd/ceph-11/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-1
1/ --osd-uuid 7e92ec97-90a8-49bd-8dc5-dc3f8b8dfa53 --setuser ceph --setgroup ceph
2020-01-28T13:52:33.939 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: 2020-01-28T13:52:32.304+0000 7fbcf17a0ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11/) _open_fsid (2) No such file or directory
2020-01-28T13:52:33.940 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: 2020-01-28T13:52:32.304+0000 7fbcf17a0ec0 -1 bluestore(/var/lib/ceph/osd/ceph-11/) mkfs fsck found fatal error: (2) No such file or directory
2020-01-28T13:52:33.940 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: 2020-01-28T13:52:32.304+0000 7fbcf17a0ec0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (2) No such file or directory
2020-01-28T13:52:33.941 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: 2020-01-28T13:52:32.304+0000 7fbcf17a0ec0 -1  ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-11/: (2) No such file or directory
2020-01-28T13:52:33.942 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr --> Was unable to complete a new OSD, will rollback changes
2020-01-28T13:52:33.943 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.11 --yes-i-really-mean-it
2020-01-28T13:52:33.943 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr  stderr: purged osd.11
2020-01-28T13:52:33.944 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: INFO:cephadm:/bin/podman:stderr -->  RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 11 --monmap /var/lib/ceph/osd/ceph-11/activate.monmap --keyfile 
- --osd-data /var/lib/ceph/osd/ceph-11/ --osd-uuid 7e92ec97-90a8-49bd-8dc5-dc3f8b8dfa53 --setuser ceph --setgroup ceph
2020-01-28T13:52:33.944 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: Traceback (most recent call last):
2020-01-28T13:52:33.945 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: File "<stdin>", line 2811, in <module>
2020-01-28T13:52:33.945 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: File "<stdin>", line 669, in _infer_fsid
2020-01-28T13:52:33.946 INFO:ceph.mgr.y.smithi109.stdout:Jan 28 13:52:33 smithi109 bash[6823]: File "<stdin>", line 2118, in command_ceph_volume

/a/sage-2020-01-28_03:52:05-rados-wip-sage2-testing-2020-01-27-1839-distro-basic-smithi/4713126
Actions #1

Updated by Sage Weil about 4 years ago

  • Status changed from New to Can't reproduce
Actions

Also available in: Atom PDF