|
2023-12-18 23:13:36,948 [Dummy-1] [DEBUG] [root] setting log level based on debug_mgr: INFO (2/5)
|
|
2023-12-18 23:13:38,679 [Thread-15] [INFO] [cherrypy.error] [18/Dec/2023:23:13:38] ENGINE Bus STARTING
|
|
2023-12-18 23:13:38,782 [Thread-15] [INFO] [cherrypy.error] [18/Dec/2023:23:13:38] ENGINE Serving on http://10.242.8.219:8765
|
|
2023-12-18 23:13:38,898 [Thread-15] [INFO] [cherrypy.error] [18/Dec/2023:23:13:38] ENGINE Serving on https://10.242.8.219:7150
|
|
2023-12-18 23:13:38,898 [Thread-15] [INFO] [cherrypy.error] [18/Dec/2023:23:13:38] ENGINE Bus STARTED
|
|
2023-12-18 23:13:39,675 [Thread-3] [INFO] [cephadm.serve] Updating storage-13-09002:/etc/ceph/ceph.conf
|
|
2023-12-18 23:13:40,300 [Thread-3] [INFO] [cephadm.serve] Updating storage-13-09002:/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/config/ceph.conf
|
|
2023-12-18 23:13:40,522 [Thread-3] [INFO] [cephadm.serve] Updating storage-13-09002:/etc/ceph/ceph.client.admin.keyring
|
|
2023-12-18 23:13:40,738 [Thread-3] [INFO] [cephadm.serve] Updating storage-13-09002:/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/config/ceph.client.admin.keyring
|
|
2023-12-18 23:13:41,003 [Dummy-22] [INFO] [cephadm.serve] Metadata not up to date on all hosts. Skipping non agent specs
|
|
2023-12-18 23:14:00,170 [Dummy-22] [INFO] [cephadm.serve] Reconfiguring prometheus.storage-13-09002 (dependencies changed)...
|
|
2023-12-18 23:14:00,407 [Thread-23] [INFO] [cephadm.serve] Reconfiguring daemon prometheus.storage-13-09002 on storage-13-09002
|
|
2023-12-18 23:14:41,996 [Dummy-127] [INFO] [root] Marking host: storage-16-09074 for OSDSpec preview refresh.
|
|
2023-12-18 23:14:42,000 [Dummy-127] [INFO] [root] Saving service osd.storage-16-09074 spec with placement storage-16-09074
|
|
2023-12-18 23:15:28,601 [Dummy-22] [ERROR] [cephadm.serve] Failed to apply osd.storage-16-09074 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
|
|
service_id: storage-16-09074
|
|
service_name: osd.storage-16-09074
|
|
placement:
|
|
hosts:
|
|
- storage-16-09074
|
|
spec:
|
|
block_db_size: 60G
|
|
data_devices:
|
|
rotational: 1
|
|
db_devices:
|
|
rotational: 0
|
|
db_slots: 15
|
|
filter_logic: AND
|
|
objectstore: bluestore
|
|
''')): cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpgb9wx_3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptclxu5c8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-30498a7a-669c-4628-b3da-f27ac304d7ff /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3 /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-d0357944-507a-40f0-8886-88e162cf9d6f ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 9d016e9c-ea82-467f-97dc-3d1ab49abc60 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 bdev(0x5596864cbc00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.00763825 s, 1.4 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr stderr: Removing ceph--69b7b714--3f6e--414f--a3c9--517b38f143b3-osd--db--d0357944--507a--40f0--8886--88e162cf9d6f (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0474917 s, 221 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr stderr: Removing ceph--30498a7a--669c--4628--b3da--f27ac304d7ff-osd--block--9d016e9c--ea82--467f--97dc--3d1ab49abc60 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 9d016e9c-ea82-467f-97dc-3d1ab49abc60 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpgb9wx_3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptclxu5c8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
Traceback (most recent call last):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 570, in _apply_all_services
|
|
if self._apply_service(spec):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 685, in _apply_service
|
|
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec
|
|
ret = self.mgr.wait_async(all_hosts())
|
|
File "/usr/share/ceph/mgr/cephadm/module.py", line 700, in wait_async
|
|
return self.event_loop.get_result(coro, timeout)
|
|
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 63, in get_result
|
|
return future.result(timeout)
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
|
|
return self.__get_result()
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
|
|
raise self._exception
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts
|
|
return await gather(*futures)
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one
|
|
replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 97, in create_single_host
|
|
code, '\n'.join(err)))
|
|
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpgb9wx_3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptclxu5c8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-30498a7a-669c-4628-b3da-f27ac304d7ff /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3 /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-d0357944-507a-40f0-8886-88e162cf9d6f ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 9d016e9c-ea82-467f-97dc-3d1ab49abc60 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:24.895+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.047+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 bdev(0x5596864cbc00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:15:25.699+0000 7fbedcc10640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr 10485760 bytes (10 MB, 10 MiB) copied, 0.00763825 s, 1.4 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3
|
|
/usr/bin/docker: stderr stderr: Removing ceph--69b7b714--3f6e--414f--a3c9--517b38f143b3-osd--db--d0357944--507a--40f0--8886--88e162cf9d6f (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-d0357944-507a-40f0-8886-88e162cf9d6f" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff/osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0474917 s, 221 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-30498a7a-669c-4628-b3da-f27ac304d7ff
|
|
/usr/bin/docker: stderr stderr: Removing ceph--30498a7a--669c--4628--b3da--f27ac304d7ff-osd--block--9d016e9c--ea82--467f--97dc--3d1ab49abc60 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-9d016e9c-ea82-467f-97dc-3d1ab49abc60" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-30498a7a-669c-4628-b3da-f27ac304d7ff" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-69b7b714-3f6e-414f-a3c9-517b38f143b3/osd-db-d0357944-507a-40f0-8886-88e162cf9d6f --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 9d016e9c-ea82-467f-97dc-3d1ab49abc60 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmpgb9wx_3o:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmptclxu5c8:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
2023-12-18 23:16:17,973 [Dummy-127] [INFO] [root] Remove service osd.storage-16-09074
|
|
2023-12-18 23:18:06,607 [Dummy-127] [INFO] [root] Marking host: storage-16-09074 for OSDSpec preview refresh.
|
|
2023-12-18 23:18:06,608 [Dummy-127] [INFO] [root] Saving service osd.storage-16-09074 spec with placement storage-16-09074
|
|
2023-12-18 23:19:02,625 [Dummy-22] [ERROR] [cephadm.serve] Failed to apply osd.storage-16-09074 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
|
|
service_id: storage-16-09074
|
|
service_name: osd.storage-16-09074
|
|
placement:
|
|
hosts:
|
|
- storage-16-09074
|
|
spec:
|
|
block_db_size: 60G
|
|
data_devices:
|
|
rotational: 1
|
|
db_devices:
|
|
rotational: 0
|
|
db_slots: 15
|
|
filter_logic: AND
|
|
objectstore: bluestore
|
|
''')): cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp_n1m0kq9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx6n1p46s:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8 /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid b159474d-e34f-4c35-97b4-3566d4b69354 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.027+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 bdev(0x5582473ffc00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00751197 s, 1.4 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr stderr: Removing ceph--9aa62f73--f5dc--410f--99f0--5796f14a874d-osd--db--cee4571a--4a0a--4aca--800e--c10be0769221 (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0498483 s, 210 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr stderr: Removing ceph--b6477598--d185--4b7c--a2f3--dac335e4e3d8-osd--block--b159474d--e34f--4c35--97b4--3566d4b69354 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid b159474d-e34f-4c35-97b4-3566d4b69354 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp_n1m0kq9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx6n1p46s:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
Traceback (most recent call last):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 570, in _apply_all_services
|
|
if self._apply_service(spec):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 685, in _apply_service
|
|
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec
|
|
ret = self.mgr.wait_async(all_hosts())
|
|
File "/usr/share/ceph/mgr/cephadm/module.py", line 700, in wait_async
|
|
return self.event_loop.get_result(coro, timeout)
|
|
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 63, in get_result
|
|
return future.result(timeout)
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
|
|
return self.__get_result()
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
|
|
raise self._exception
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts
|
|
return await gather(*futures)
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one
|
|
replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 97, in create_single_host
|
|
code, '\n'.join(err)))
|
|
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp_n1m0kq9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx6n1p46s:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8 /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid b159474d-e34f-4c35-97b4-3566d4b69354 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.027+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.031+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.183+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 bdev(0x5582473ffc00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:18:59.819+0000 7f46dd378640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00751197 s, 1.4 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d
|
|
/usr/bin/docker: stderr stderr: Removing ceph--9aa62f73--f5dc--410f--99f0--5796f14a874d-osd--db--cee4571a--4a0a--4aca--800e--c10be0769221 (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-cee4571a-4a0a-4aca-800e-c10be0769221" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8/osd-block-b159474d-e34f-4c35-97b4-3566d4b69354 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0498483 s, 210 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8
|
|
/usr/bin/docker: stderr stderr: Removing ceph--b6477598--d185--4b7c--a2f3--dac335e4e3d8-osd--block--b159474d--e34f--4c35--97b4--3566d4b69354 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-b159474d-e34f-4c35-97b4-3566d4b69354" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-b6477598-d185-4b7c-a2f3-dac335e4e3d8" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-9aa62f73-f5dc-410f-99f0-5796f14a874d/osd-db-cee4571a-4a0a-4aca-800e-c10be0769221 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid b159474d-e34f-4c35-97b4-3566d4b69354 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp_n1m0kq9:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpx6n1p46s:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
2023-12-18 23:20:52,453 [Dummy-22] [ERROR] [cephadm.serve] Failed to apply osd.storage-16-09074 spec DriveGroupSpec.from_json(yaml.safe_load('''service_type: osd
|
|
service_id: storage-16-09074
|
|
service_name: osd.storage-16-09074
|
|
placement:
|
|
hosts:
|
|
- storage-16-09074
|
|
spec:
|
|
block_db_size: 60G
|
|
data_devices:
|
|
rotational: 1
|
|
db_devices:
|
|
rotational: 0
|
|
db_slots: 15
|
|
filter_logic: AND
|
|
objectstore: bluestore
|
|
''')): cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp390qncsr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnlr4_bat:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-d3a234da-36d3-4871-9447-f6340f23bc04 /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-2f4fe039-4466-4a17-b193-df09e550b773 ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401 /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 2f4fe039-4466-4a17-b193-df09e550b773 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 bdev(0x5572cf355c00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0592264 s, 177 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr stderr: Removing ceph--d3a234da--36d3--4871--9447--f6340f23bc04-osd--block--2f4fe039--4466--4a17--b193--df09e550b773 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-d3a234da-36d3-4871-9447-f6340f23bc04" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00781535 s, 1.3 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr stderr: Removing ceph--75f57be5--25b9--4fbf--83c8--d5a34cf27401-osd--db--6d9479d9--e8de--4356--98fc--2ad3dc831f46 (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 2f4fe039-4466-4a17-b193-df09e550b773 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp390qncsr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnlr4_bat:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
Traceback (most recent call last):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 570, in _apply_all_services
|
|
if self._apply_service(spec):
|
|
File "/usr/share/ceph/mgr/cephadm/serve.py", line 685, in _apply_service
|
|
self.mgr.osd_service.create_from_spec(cast(DriveGroupSpec, spec))
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 79, in create_from_spec
|
|
ret = self.mgr.wait_async(all_hosts())
|
|
File "/usr/share/ceph/mgr/cephadm/module.py", line 700, in wait_async
|
|
return self.event_loop.get_result(coro, timeout)
|
|
File "/usr/share/ceph/mgr/cephadm/ssh.py", line 63, in get_result
|
|
return future.result(timeout)
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 432, in result
|
|
return self.__get_result()
|
|
File "/lib64/python3.6/concurrent/futures/_base.py", line 384, in __get_result
|
|
raise self._exception
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 76, in all_hosts
|
|
return await gather(*futures)
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 65, in create_from_spec_one
|
|
replace_osd_ids=osd_id_claims_for_host, env_vars=env_vars
|
|
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 97, in create_single_host
|
|
code, '\n'.join(err)))
|
|
RuntimeError: cephadm exited with an error code: 1, stderr:Non-zero exit code 1 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp390qncsr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnlr4_bat:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|
|
/usr/bin/docker: stderr --> passed data devices: 60 physical, 0 LVM
|
|
/usr/bin/docker: stderr --> relative data size: 1.0
|
|
/usr/bin/docker: stderr --> passed block_db devices: 4 physical, 0 LVM
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-d3a234da-36d3-4871-9447-f6340f23bc04 /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/mapper/mpatha" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 5245439 -n osd-block-2f4fe039-4466-4a17-b193-df09e550b773 ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773" created.
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgcreate --force --yes ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401 /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Physical volume "/dev/nvme3n1" successfully created.
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" successfully created
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/lvcreate --yes -l 15360 -n osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46" created.
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-authtool --gen-print-key
|
|
/usr/bin/docker: stderr Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-60
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ln -s /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773 /var/lib/ceph/osd/ceph-1507/block
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-1507/activate.monmap
|
|
/usr/bin/docker: stderr stderr: got monmap epoch 5
|
|
/usr/bin/docker: stderr --> Creating keyring file for osd.1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/keyring
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1507/
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -h ceph:ceph /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46
|
|
/usr/bin/docker: stderr Running command: /usr/bin/chown -R ceph:ceph /dev/dm-61
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 2f4fe039-4466-4a17-b193-df09e550b773 --setuser ceph --setgroup ceph
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label unable to decode label at offset 102: void bluestore_bdev_label_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input [buffer:3]
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.778+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507/) _read_fsid unparsable uuid
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:48.934+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 bluestore(/var/lib/ceph/osd/ceph-1507//block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1507//block: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 bdev(0x5572cf355c00 /var/lib/ceph/osd/ceph-1507//block) open open got: (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 OSD::mkfs: ObjectStore::mkfs failed with error (13) Permission denied
|
|
/usr/bin/docker: stderr stderr: 2023-12-18T23:20:49.562+0000 7fe8624a0640 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1507/: (13) Permission denied[0m
|
|
/usr/bin/docker: stderr --> Was unable to complete a new OSD, will rollback changes
|
|
/usr/bin/docker: stderr Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1507 --yes-i-really-mean-it
|
|
/usr/bin/docker: stderr stderr: purged osd.1507
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773
|
|
/usr/bin/docker: stderr --> Unmounting /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr Running command: /usr/bin/umount -v /var/lib/ceph/osd/ceph-1507
|
|
/usr/bin/docker: stderr stderr: umount: /var/lib/ceph/osd/ceph-1507 unmounted
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-d3a234da-36d3-4871-9447-f6340f23bc04/osd-block-2f4fe039-4466-4a17-b193-df09e550b773 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0592264 s, 177 MB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-d3a234da-36d3-4871-9447-f6340f23bc04
|
|
/usr/bin/docker: stderr stderr: Removing ceph--d3a234da--36d3--4871--9447--f6340f23bc04-osd--block--2f4fe039--4466--4a17--b193--df09e550b773 (253:60)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-d3a234da-36d3-4871-9447-f6340f23bc04" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-block-2f4fe039-4466-4a17-b193-df09e550b773" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/mapper/mpatha" from volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-d3a234da-36d3-4871-9447-f6340f23bc04" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/mapper/mpatha
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/mapper/mpatha" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping: /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46
|
|
/usr/bin/docker: stderr Running command: /usr/bin/dd if=/dev/zero of=/dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 bs=1M count=10 conv=fsync
|
|
/usr/bin/docker: stderr stderr: 10+0 records in
|
|
/usr/bin/docker: stderr 10+0 records out
|
|
/usr/bin/docker: stderr stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00781535 s, 1.3 GB/s
|
|
/usr/bin/docker: stderr --> Only 1 LV left in VG, will proceed to destroy volume group ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/vgremove -v -f ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401
|
|
/usr/bin/docker: stderr stderr: Removing ceph--75f57be5--25b9--4fbf--83c8--d5a34cf27401-osd--db--6d9479d9--e8de--4356--98fc--2ad3dc831f46 (253:61)
|
|
/usr/bin/docker: stderr stderr: Archiving volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" metadata (seqno 5).
|
|
/usr/bin/docker: stderr stderr: Releasing logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46"
|
|
/usr/bin/docker: stderr stderr: Creating volume group backup "/etc/lvm/backup/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" (seqno 6).
|
|
/usr/bin/docker: stderr stdout: Logical volume "osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46" successfully removed
|
|
/usr/bin/docker: stderr stderr: Removing physical volume "/dev/nvme3n1" from volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401"
|
|
/usr/bin/docker: stderr stdout: Volume group "ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401" successfully removed
|
|
/usr/bin/docker: stderr Running command: nsenter --mount=/rootfs/proc/1/ns/mnt --ipc=/rootfs/proc/1/ns/ipc --net=/rootfs/proc/1/ns/net --uts=/rootfs/proc/1/ns/uts /sbin/pvremove -v -f -f /dev/nvme3n1
|
|
/usr/bin/docker: stderr stdout: Labels on physical volume "/dev/nvme3n1" successfully wiped.
|
|
/usr/bin/docker: stderr --> Zapping successful for OSD: 1507
|
|
/usr/bin/docker: stderr Traceback (most recent call last):
|
|
/usr/bin/docker: stderr File "/usr/sbin/ceph-volume", line 11, in <module>
|
|
/usr/bin/docker: stderr load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 41, in __init__
|
|
/usr/bin/docker: stderr self.main(self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
|
|
/usr/bin/docker: stderr return f(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 153, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, subcommand_args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
|
|
/usr/bin/docker: stderr terminal.dispatch(self.mapper, self.argv)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
|
|
/usr/bin/docker: stderr instance.main()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 414, in main
|
|
/usr/bin/docker: stderr self._execute(plan)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 432, in _execute
|
|
/usr/bin/docker: stderr c.create(argparse.Namespace(**args))
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
|
|
/usr/bin/docker: stderr prepare_step.safe_prepare(args)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 196, in safe_prepare
|
|
/usr/bin/docker: stderr self.prepare()
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
|
|
/usr/bin/docker: stderr return func(*a, **kw)
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 285, in prepare
|
|
/usr/bin/docker: stderr osd_fsid,
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 63, in prepare_bluestore
|
|
/usr/bin/docker: stderr db=db
|
|
/usr/bin/docker: stderr File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 459, in osd_mkfs_bluestore
|
|
/usr/bin/docker: stderr raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
|
|
/usr/bin/docker: stderr RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 1507 --monmap /var/lib/ceph/osd/ceph-1507/activate.monmap --keyfile - --bluestore-block-db-path /dev/ceph-75f57be5-25b9-4fbf-83c8-d5a34cf27401/osd-db-6d9479d9-e8de-4356-98fc-2ad3dc831f46 --osdspec-affinity storage-16-09074 --osd-data /var/lib/ceph/osd/ceph-1507/ --osd-uuid 2f4fe039-4466-4a17-b193-df09e550b773 --setuser ceph --setgroup ceph
|
|
Traceback (most recent call last):
|
|
File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
|
|
return _run_code(code, main_globals, None,
|
|
File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
|
|
exec(code, run_globals)
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10700, in <module>
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 10688, in main
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2445, in _infer_config
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2361, in _infer_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2473, in _infer_image
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2348, in _validate_fsid
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 6970, in command_ceph_volume
|
|
File "/var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/cephadm.b72ca0dbf44b4b921d8b8ce04f1e459e69e1ceae24894790b997d795f5dea450/__main__.py", line 2136, in call_throws
|
|
RuntimeError: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint /usr/sbin/ceph-volume --privileged --group-add=disk --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 -e NODE_NAME=storage-16-09074 -e CEPH_USE_RANDOM_NONCE=1 -e CEPH_VOLUME_OSDSPEC_AFFINITY=storage-16-09074 -e CEPH_VOLUME_SKIP_RESTORECON=yes -e CEPH_VOLUME_DEBUG=1 -v /var/run/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/run/ceph:z -v /var/log/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8:/var/log/ceph:z -v /var/lib/ceph/b46162de-9d48-11ee-8c88-2fbb19f34eb8/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /:/rootfs -v /tmp/ceph-tmp390qncsr:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpnlr4_bat:/var/lib/ceph/bootstrap-osd/ceph.keyring:z quay.ceph.io/ceph-ci/ceph@sha256:34f6ff5ae309ec2e196d0edb582e8c4f015e09e959ff2061736df26927c6c776 lvm batch --no-auto /dev/mapper/mpatha /dev/mapper/mpathaa /dev/mapper/mpathab /dev/mapper/mpathac /dev/mapper/mpathad /dev/mapper/mpathae /dev/mapper/mpathaf /dev/mapper/mpathag /dev/mapper/mpathah /dev/mapper/mpathai /dev/mapper/mpathaj /dev/mapper/mpathak /dev/mapper/mpathal /dev/mapper/mpatham /dev/mapper/mpathan /dev/mapper/mpathao /dev/mapper/mpathap /dev/mapper/mpathaq /dev/mapper/mpathar /dev/mapper/mpathas /dev/mapper/mpathat /dev/mapper/mpathau /dev/mapper/mpathav /dev/mapper/mpathaw /dev/mapper/mpathax /dev/mapper/mpathay /dev/mapper/mpathaz /dev/mapper/mpathb /dev/mapper/mpathba /dev/mapper/mpathbb /dev/mapper/mpathbc /dev/mapper/mpathbd /dev/mapper/mpathbe /dev/mapper/mpathbf /dev/mapper/mpathbg /dev/mapper/mpathbh /dev/mapper/mpathc /dev/mapper/mpathd /dev/mapper/mpathe /dev/mapper/mpathf /dev/mapper/mpathg /dev/mapper/mpathh /dev/mapper/mpathi /dev/mapper/mpathj /dev/mapper/mpathk /dev/mapper/mpathl /dev/mapper/mpathm /dev/mapper/mpathn /dev/mapper/mpatho /dev/mapper/mpathp /dev/mapper/mpathq /dev/mapper/mpathr /dev/mapper/mpaths /dev/mapper/mpatht /dev/mapper/mpathu /dev/mapper/mpathv /dev/mapper/mpathw /dev/mapper/mpathx /dev/mapper/mpathy /dev/mapper/mpathz --db-devices /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 --block-db-size 60G --yes --no-systemd
|