Ceph » rbd

Hrm, on quincy the existing tests succeed:

https://pulpito.ceph.com/dis-2022-09-02_08:30:48-rbd-quincy-distro-default-smithi/

Contents of /etc/yum.repos.d/ceph-iscsi.repo file in ":quincy" container:

[ceph-iscsi]
name=ceph-iscsi noarch packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el8/noarch
enabled=1
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md

[ceph-iscsi-source]
name=ceph-iscsi source packages
baseurl=http://download.ceph.com/ceph-iscsi/3/rpm/el8/SRPMS
enabled=0
gpgcheck=1
gpgkey=https://download.ceph.com/keys/release.asc
type=rpm-md

Contents of /etc/yum.repos.d/ceph-iscsi.repo file in ":main" container:

[ceph-iscsi]
name=ceph-iscsi packages for $basearch
baseurl=https://3.chacra.ceph.com/r/ceph-iscsi/main/53e4c85fb7edecd866987731bfa17012e001a1ba/centos/8/flavors/default/$basearch
enabled=1
gpgcheck=0
type=rpm-md

[ceph-iscsi-noarch]
name=ceph-iscsi noarch packages
baseurl=https://3.chacra.ceph.com/r/ceph-iscsi/main/53e4c85fb7edecd866987731bfa17012e001a1ba/centos/8/flavors/default/noarch
enabled=1
gpgcheck=0
type=rpm-md

[ceph-iscsi-source]
name=ceph-iscsi source packages
baseurl=https://3.chacra.ceph.com/r/ceph-iscsi/main/53e4c85fb7edecd866987731bfa17012e001a1ba/centos/8/flavors/default/SRPMS
enabled=1
gpgcheck=0
type=rpm-md

Either http://download.ceph.com/ceph-iscsi/3/rpm/el8/noarch needs to be pushed with new packages or ceph-container needs to be made to pull from the same place for all branches.

Hi Guillaume,

Could you please take a look at this? It seemed like a ceph-container issue to me.

Hitting this on reef now:

http://qa-proxy.ceph.com/teuthology/yuriw-2023-06-22_20:26:47-rbd-wip-yuri3-testing-2023-06-22-0812-reef-distro-default-smithi/7312833/teuthology.log

So ceph-container is pulling from different places on purpose:

    if [[ "${CEPH_VERSION}" =~ master || "${CEPH_VERSION}" == main ]]; then \
      curl -s -L https://shaman.ceph.com/api/repos/ceph-iscsi/main/latest/centos/__ENV_[DISTRO_VERSION]__/repo -o /etc/yum.repos.d/ceph-iscsi.repo ; \
    elif [[ "${CEPH_VERSION}" =~ nautilus|octopus|pacific|quincy|reef ]]; then \
      curl -s -L https://download.ceph.com/ceph-iscsi/3/rpm/el__ENV_[DISTRO_VERSION]__/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ; \
    else \
      curl -s -L https://download.ceph.com/ceph-iscsi/2/rpm/el__ENV_[DISTRO_VERSION]__/ceph-iscsi.repo -o /etc/yum.repos.d/ceph-iscsi.repo ; \
    fi ; \

I think it's right in wanting to pick up a signed ceph-iscsi package when building a release branch, despite it happily picking up an unsigned tcmu-runner package just a line above:

    curl -s -L https://shaman.ceph.com/api/repos/tcmu-runner/main/latest/centos/__ENV_[DISTRO_VERSION]__/repo?arch=$(arch) -o /etc/yum.repos.d/tcmu-runner.repo ; \

Adam, Yuri, can we get https://3.chacra.ceph.com/r/ceph-iscsi/main/53e4c85fb7edecd866987731bfa17012e001a1ba/centos/8/flavors/default/noarch/ceph-iscsi-3.6-1.el8.noarch.rpm signed and synced to https://download.ceph.com/ceph-iscsi/3/rpm/el8/noarch/?

It looks like David used https://jenkins.ceph.com/job/ceph-iscsi-stable/ job to produce the packages to be signed in the past: https://github.com/ceph/ceph-build/blob/main/ceph-iscsi-stable/README.rst

The most obvious difference from https://jenkins.ceph.com/job/ceph-iscsi/ job is that it also pushes SRPMs to chacra so I would try to trigger it instead of grabbing https://3.chacra.ceph.com/r/ceph-iscsi/main/53e4c85fb7edecd866987731bfa17012e001a1ba/centos/8/flavors/default/noarch/ceph-iscsi-3.6-1.el8.noarch.rpm package directly.

Ilya Dryomov wrote:

despite it happily picking up an unsigned tcmu-runner package just a line above:

It turns out that tcmu-runner package wasn't up-to-date either. Hit a bug in the job body there: https://github.com/ceph/ceph-build/pull/2141

Hey Ilya, I upload the signed package please check
https://download.ceph.com/ceph-iscsi/3/rpm/el8/noarch/

adam kraitman wrote:

Hey Ilya, I upload the signed package please check
https://download.ceph.com/ceph-iscsi/3/rpm/el8/noarch/

The corresponding SRPM under https://download.ceph.com/ceph-iscsi/3/rpm/el8/SRPMS/ is missing.

Otherwise, for an end-to-end test, an updated container image is needed. Is there a way to manually trigger the container image build, e.g. for quincy?

Hi Adam,

There seems to be an issue with the repo (or with the mirrors)? I'm trying to test https://github.com/ceph/ceph/pull/52283 and CentOS 8 container builds are failing:

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/70999//consoleFull

================================================================================================================================================
 Package                         Arch    Version                              Repository                                                    Size
================================================================================================================================================
Installing:
 attr                            x86_64  2.4.48-3.el8                         baseos                                                        68 k
 ceph-common                     x86_64  2:17.2.6-745.gdce54d0c.el8           Ceph                                                          23 M
 ceph-exporter                   x86_64  2:17.2.6-745.gdce54d0c.el8           Ceph                                                         357 k
 ceph-fuse                       x86_64  2:17.2.6-745.gdce54d0c.el8           Ceph                                                         874 k
 ceph-grafana-dashboards         noarch  2:17.2.6-745.gdce54d0c.el8           Ceph-noarch                                                   23 k
 ceph-immutable-object-cache     x86_64  2:17.2.6-745.gdce54d0c.el8           Ceph                                                         162 k
 ceph-iscsi                      noarch  3.6-1.el8                            ceph-iscsi                                                   212 k
...
[MIRROR] ceph-iscsi-3.6-1.el8.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 4c40e3b612b507db7169fb1ce46d4ab618178e8a8906729d522c8b67c56683b4(sha256)  Expected: 9ee9d8f0d2b84dcf2c8b2a580f172bcf9e22454057fea97cb42b452da72c78d1(sha256) 
[MIRROR] ceph-iscsi-3.6-1.el8.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 4c40e3b612b507db7169fb1ce46d4ab618178e8a8906729d522c8b67c56683b4(sha256)  Expected: 9ee9d8f0d2b84dcf2c8b2a580f172bcf9e22454057fea97cb42b452da72c78d1(sha256) 
[MIRROR] ceph-iscsi-3.6-1.el8.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 4c40e3b612b507db7169fb1ce46d4ab618178e8a8906729d522c8b67c56683b4(sha256)  Expected: 9ee9d8f0d2b84dcf2c8b2a580f172bcf9e22454057fea97cb42b452da72c78d1(sha256) 
[MIRROR] ceph-iscsi-3.6-1.el8.noarch.rpm: Downloading successful, but checksum doesn't match. Calculated: 4c40e3b612b507db7169fb1ce46d4ab618178e8a8906729d522c8b67c56683b4(sha256)  Expected: 9ee9d8f0d2b84dcf2c8b2a580f172bcf9e22454057fea97cb42b452da72c78d1(sha256) 
[FAILED] ceph-iscsi-3.6-1.el8.noarch.rpm: No more mirrors to try - All mirrors were already tried without success

The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
  ceph-iscsi-3.6-1.el8.noarch: Cannot download, all mirrors were already tried without success
...
make[2]: *** [Makefile:25: build] Error 1
make[2]: Leaving directory '/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/ceph-container/staging/quincy-centos-8-x86_64/daemon-base'
make[1]: *** [Makefile:67: do.image.x86_64,wip-57343-quincy,centos,8] Error 2

The PR just makes a tweak to the test, so it can't be the problem. I retried a few hours later but the failure persisted:

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/70999//consoleFull

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71225/console

adam kraitman wrote:

https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71225/console

Hi Adam,

If you linked this as an example of a successful build, note that it's not an up-to-date package:

================================================================================================================================================
 Package                         Arch    Version                              Repository                                                    Size
================================================================================================================================================
Installing:
...
 ceph-iscsi                      noarch  3.6-1.el8                            ceph-iscsi                                                   212 k

ceph-iscsi was bumped to 3.6-2 last week.

This was marked as a "blocker" for reef

Ilya, I am able to install the new package on my test node, and also this job shows it's installed successfully https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71505/console

Thanks Adam. I'm triggering a build for the iSCSI test and will report back.