Project

General

Profile

Actions
Copy link

Bug #58689

open

Weird Shaman build behavior

Added by Laura Flores about 1 year ago. Updated 11 months ago.

Status:
New
Priority:
Normal
Assignee:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

Problem #1:
Pushing a branch to CI results in some builds duplicating multiple times, and other builds missing completely.
This build is an example: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/. Notice how there are multiple builds of centos 8/9 and opensuse, but no focal. I had to trigger focal manually, so it will likely show up soon.

Problem #2:
Builds are taking an extremely long time for Shaman to pick up. My build has been waiting over an hour, and estimated time to finish is a little over 3 hours remaining (https://jenkins.ceph.com/job/ceph-dev-new/68766)

Related issues 1 (0 open — 1 closed)

Has duplicate Infrastructure - Bug #58951: some branches (reef) seem to build double number of distrosDuplicate

Actions
Actions
Copy link
 #1

Updated by Laura Flores about 1 year ago

Another weird build behavior:

For this build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/
Teuthology says it cannot find centos 8 packages even though there is a successful build for that SHA:

2023-02-10 02:11:55,393.393 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '27a5e68d9b64346d6596f06394f6478d861f4d51' not found
2023-02-10 02:11:55,413.413 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '27a5e68d9b64346d6596f06394f6478d861f4d51' not found
/home/nojha/teuth/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend
Job scheduled with name nojha-2023-02-10_02:11:18-rados-wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt-distro-default-smithi and ID 7166383
2023-02-10 02:11:56,487.487 INFO:teuthology.suite.run:Scheduling rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
Traceback (most recent call last):
  File "./virtualenv/bin/teuthology-suite", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')())
  File "/home/nojha/teuth/scripts/suite.py", line 219, in main
    return teuthology.suite.main(args)
  File "/home/nojha/teuth/teuthology/suite/__init__.py", line 145, in main
    run.prepare_and_schedule()
  File "/home/nojha/teuth/teuthology/suite/run.py", line 416, in prepare_and_schedule
    num_jobs = self.schedule_suite()
  File "/home/nojha/teuth/teuthology/suite/run.py", line 679, in schedule_suite
    self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
  File "/home/nojha/teuth/teuthology/suite/run.py", line 515, in schedule_jobs
    name,
  File "/home/nojha/teuth/teuthology/suite/util.py", line 76, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling nojha-2023-02-10_02:11:18-rados-wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt-distro-default-smithi failed: At least one job needs packages that don't exist for hash 27a5e68d9b64346d6596f06394f6478d861f4d51.

There was one failed centos 8 build and one successful centos 8 build for this SHA. Teuthology looks like it's trying to pull packages for the failed build, even though there is different successful build it should be able to use.

Actions
Copy link
 #3

Updated by Zack Cerza about 1 year ago

I found something I think is pretty strange. Starting from this shaman page: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/

I visited the oldest build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/default/332476/
From there, I visited the Jenkins build: https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/67974/
Then, I visited the build referenced by the message "Started by upstream project ceph-dev-new-build build number 67974": https://jenkins.ceph.com/job/ceph-dev-new-build/67974/
Here, I see a matrix showing that several builds were scheduled by this one, including focal: https://jenkins.ceph.com/job/ceph-dev-new-build/67974/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/
That build even succeeded!
From there I visit the console log and find a reference to: https://shaman.ceph.com/api/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/
As this endpoint returns a JSON payload, I'll paste it here:

[{"status":"ready","sha1":"27a5e68d9b64346d6596f06394f6478d861f4d51","extra":{"build_url":"https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/67988/","root_build_cause":"MANUALTRIGGER","version":"17.2.5-715-g27a5e68d","node_name":"172.21.2.3+braggi03","job_name":"ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic","package_manager_version":"17.2.5-715-g27a5e68d-1focal"},"url":"https://1.chacra.ceph.com/r/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","distro_codename":"focal","modified":"2023-02-10 04:25:09.081873","distro_version":"20.04","project":"ceph","flavor":"default","ref":"wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt","chacra_url":"https://1.chacra.ceph.com/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","archs":["x86_64"],"distro":"ubuntu"},{"status":"ready","sha1":"27a5e68d9b64346d6596f06394f6478d861f4d51","extra":{"build_url":"https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/67974/","root_build_cause":"SCMTRIGGER","version":"17.2.5-715-g27a5e68d","node_name":"172.21.2.2+braggi02","job_name":"ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic","package_manager_version":"17.2.5-715-g27a5e68d-1focal"},"url":"https://4.chacra.ceph.com/r/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","distro_codename":"focal","modified":"2023-02-10 01:41:47.864656","distro_version":"20.04","project":"ceph","flavor":"default","ref":"wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt","chacra_url":"https://4.chacra.ceph.com/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","archs":["x86_64"],"distro":"ubuntu"}]

This is a build that finished at 2023-02-10 01:41:47.864656, yet the shaman UI link does not list it. Why??

Actions
Copy link
 #4

Updated by Zack Cerza about 1 year ago

Laura Flores wrote:

For this build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/

This build was also looking for rhel packages, which I don't see.

Actions
Copy link
 #5

Updated by Laura Flores about 1 year ago

Duplicate distros and flavors were created for my build when I pushed only once to CI:
https://shaman.ceph.com/builds/ceph/wip-lflores-testing-2023-02-17-2149/

Actions
Copy link
 #6

Updated by Laura Flores about 1 year ago

  • Has duplicate Bug #58951: some branches (reef) seem to build double number of distros added
Actions
Copy link
 #7

Updated by Laura Flores about 1 year ago

Problems with this build: https://shaman.ceph.com/builds/ceph/reef/0447d535c14cd711af288300f5ccc6cd3874567a/

Happens when scheduling teuthology-suite:

2023-03-10 18:15:22,978.978 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:22,984.984 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2023-03-10 18:15:22,984.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,126.126 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,136.136 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2023-03-10 18:15:23,136.136 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,279.279 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2023-03-10 18:15:23,279.279 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2023-03-10 18:15:23,279.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,593.593 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2023-03-10 18:15:23,593.593 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson
2023-03-10 18:15:23,594.594 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2023-03-10 18:15:23,594.594 INFO:teuthology.suite.util:build not complete
2023-03-10 18:15:23,594.594 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,607.607 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,629.629 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,655.655 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,682.682 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,693.693 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,714.714 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,720.720 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend
testing12 1
Job scheduled with name lflores-2023-03-10_18:15:16-smoke-reef-distro-default-smithi and ID 7202000
2023-03-10 18:15:24,809.809 INFO:teuthology.suite.run:Scheduling smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_blogbench}}
Traceback (most recent call last):
  File "./teuthology/virtualenv/bin/teuthology-suite", line 33, in <module>
    sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')())
  File "/home/lflores/teuthology/scripts/suite.py", line 219, in main
    return teuthology.suite.main(args)
  File "/home/lflores/teuthology/teuthology/suite/__init__.py", line 145, in main
    run.prepare_and_schedule()
  File "/home/lflores/teuthology/teuthology/suite/run.py", line 422, in prepare_and_schedule
    num_jobs = self.schedule_suite()
  File "/home/lflores/teuthology/teuthology/suite/run.py", line 687, in schedule_suite
    self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
  File "/home/lflores/teuthology/teuthology/suite/run.py", line 522, in schedule_jobs
    dry_run=self.args.dry_run,
  File "/home/lflores/teuthology/teuthology/suite/util.py", line 76, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling lflores-2023-03-10_18:15:16-smoke-reef-distro-default-smithi failed: At least one job needs packages that don't exist for hash 0447d535c14cd711af288300f5ccc6cd3874567a.

The issue is apparent in https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a.
There are two builds, 0 and 1. If you click on "build_url" for both, you can see that the build failed for 0, but passed for 1.
Teuthology is trying to access packages from the failed build (although there is a successful one there, it is not first in line).

Actions
Copy link
 #9

Updated by Yuri Weinstein 12 months ago

Also related issue
Here is the build https://shaman.ceph.com/builds/ceph/wip-yuri2-testing-2023-05-15-0810-pacific/25796df0fedbe757877f6bad6ff202a3d2ca4abf/

However, `teuthology-suite` failed scheduling runs with the error: "At least one job needs packages that don't exist for hash 25796df0fedbe757877f6bad6ff202a3d2ca4abf" (full output too large to attach here)

.......
2023-05-15 19:34:00,140.140 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2023-05-15 19:34:00,140.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=25796df0fedbe757877f6bad6ff202a3d2ca4abf
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/20.04 x86_64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 x86_64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: centos/8 arm64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 x86_64 default
2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 arm64 default
2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: ubuntu/18.04 arm64 default
2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: windows/1809 x86_64 default
2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: ubuntu/20.04 x86_64 default
2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2023-05-15 19:34:00,443.443 INFO:teuthology.suite.util:build not complete
2023-05-15 19:34:00,443.443 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,454.454 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,470.470 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,486.486 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,501.501 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,524.524 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,534.534 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,542.542 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,552.552 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,560.560 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,568.568 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,577.577 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,592.592 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,603.603 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,613.613 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,621.621 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,636.636 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,643.643 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,656.656 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,670.670 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,679.679 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,687.687 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,707.707 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,714.714 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,723.723 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,749.749 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,757.757 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,765.765 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,785.785 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,802.802 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,810.810 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,818.818 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,825.825 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,838.838 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,850.850 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,860.860 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,892.892 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,899.899 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,906.906 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,924.924 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,931.931 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,949.949 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,972.972 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:00,979.979 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,010.010 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,018.018 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,028.028 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,045.045 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,053.053 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,062.062 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,071.071 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,081.081 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,089.089 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,124.124 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,131.131 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,146.146 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,149.149 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,158.158 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,165.165 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,172.172 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,183.183 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,190.190 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,204.204 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,225.225 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,231.231 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,239.239 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,249.249 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,255.255 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,262.262 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,275.275 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,283.283 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,290.290 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,307.307 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,316.316 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,326.326 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,343.343 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,350.350 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,358.358 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,375.375 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,382.382 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,389.389 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,396.396 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,406.406 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,414.414 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,422.422 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,430.430 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,456.456 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,464.464 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,471.471 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,479.479 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,485.485 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,515.515 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,534.534 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2023-05-15 19:34:01,535.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=25796df0fedbe757877f6bad6ff202a3d2ca4abf
2023-05-15 19:34:01,687.687 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,703.703 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,709.709 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,717.717 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,723.723 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,750.750 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,758.758 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,766.766 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,774.774 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,783.783 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,791.791 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,800.800 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,823.823 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,835.835 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,859.859 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,872.872 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,886.886 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,893.893 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,903.903 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,910.910 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,921.921 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
2023-05-15 19:34:01,937.937 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found
Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274491
2023-05-15 19:34:02,674.674 INFO:teuthology.suite.run:Scheduling fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc}
Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274492
2023-05-15 19:34:03,537.537 INFO:teuthology.suite.run:Scheduling fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}}
Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274493
2023-05-15 19:34:04,330.330 INFO:teuthology.suite.run:Scheduling fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead}
Traceback (most recent call last):
  File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 8, in <module>
    sys.exit(main())
  File "/home/yuriw/teuthology/scripts/suite.py", line 223, in main
    return teuthology.suite.main(args)
  File "/home/yuriw/teuthology/teuthology/suite/__init__.py", line 145, in main
    run.prepare_and_schedule()
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 425, in prepare_and_schedule
    num_jobs = self.schedule_suite()
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 690, in schedule_suite
    self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
  File "/home/yuriw/teuthology/teuthology/suite/run.py", line 521, in schedule_jobs
    util.schedule_fail(
  File "/home/yuriw/teuthology/teuthology/suite/util.py", line 76, in schedule_fail
    raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi failed: At least one job needs packages that don't exist for hash 25796df0fedbe757877f6bad6ff202a3d2ca4abf.
Actions
Copy link
 #10

Updated by Zack Cerza 11 months ago

This hash:
https://shaman.ceph.com/builds/ceph/wip-yuri7-testing-2023-06-05-1505/65cebeadd1b8670eeab54addbd95365d0898a99f/

Has duplicate builds, i.e.:
https://jenkins.ceph.com/job/ceph-dev-new-trigger/45802/
https://jenkins.ceph.com/job/ceph-dev-new-trigger/45803/

Looking for differences, I see this at the beginning of 02's console log:

Started by an SCM change

At the beginning of 03's console log:

Started by GitHub push by yuriw
Started by GitHub push by yuriw

jenkins.ceph.com:~jenkins/0.log:

jenkins@jenkins:~$ grep 4580[12] 0.log
Jun 5, 2023 10:06:20,488 PMjob/ceph-dev-new-trigger/ #45801 Started by GitHub push by yuriw, Started by GitHub push by yuriw, Parameters:[]
Jun 5, 2023 10:08:20,313 PM'ceph-dev-new-trigger#45801' (class hudson.model.FreeStyleBuild) used credentials 'shaman-api-key' (class org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl).
Jun 5, 2023 10:08:21,735 PMceph-dev-new-trigger #45801 Started by GitHub push by yuriw, Started by GitHub push by yuriw, Parameters:[] on node Jenkins started at 2023-06-05T22:06:19Z completed in 119934ms completed: SUCCESS
Jun 5, 2023 10:08:21,823 PMjob/ceph-dev-new-trigger/ #45802 Started by an SCM change, Parameters:[]
Jun 5, 2023 10:09:37,512 PM'ceph-dev-new-trigger#45802' (class hudson.model.FreeStyleBuild) used credentials 'shaman-api-key' (class org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl).
Jun 5, 2023 10:09:38,778 PMceph-dev-new-trigger #45802 Started by an SCM change, Parameters:[] on node Jenkins started at 2023-06-05T22:08:21Z completed in 75905ms completed: SUCCESS

Back at the Jenkins UI, I noticed this job is claiming to have been running for 24h:
https://jenkins.ceph.com/job/ceph-dev-new-build/70346/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic
It was started by 45802, and is hung on:

+ ls -la
total 392
drwxr-xr-x.  6 jenkins-build jenkins-build    162 Jun  5 22:28 .
drwxr-xr-x.  3 jenkins-build jenkins-build     22 Mar  7 20:33 ..
-rw-r--r--.  1 jenkins-build users            110 Jun  5 22:28 build_info
-rw-r--r--.  1 jenkins-build users            847 Jun  5 22:28 build_status.json
-rw-r--r--.  1 jenkins-build users         381328 Jun  5 22:24 cephadm
drwxr-xr-x. 11 jenkins-build users            251 Jun  5 22:28 ceph-container
drwxr-xr-x.  2 jenkins-build users              6 Jun  5 22:28 ceph-container@tmp
drwxr-xr-x.  3 jenkins-build users           4096 Jun  5 21:12 dist
drwxr-xr-x.  3 jenkins-build users             35 Jun  5 21:12 release
-rw-r--r--.  1 jenkins-build users            471 Jun  5 22:24 repo-extra.json
+ sudo rm -rf dist
+ rm -rf venv
+ sudo rm -rf release

It's allegedly running on braggi04, but I didn't see any relevant processes. Also:

[jenkins-build@braggi04 ~]$ find build/workspace/
build/workspace/
build/workspace/workspaces.txt
jenkins+  355993  0.5  0.1 38402512 446776 ?     Ssl  06:39   5:06 /usr/bin/java -Dfile.encoding=UTF8 -jar /home/jenkins-build/agent.jar -jnlpUrl https://jenkins.ceph.com/computer/172.21.2.4+braggi04/slave-agent.jnlp -jnlpCredentials @/etc/systemd/system/jenkins.secret
root      488043  0.5  0.6 38114332 1522472 ?    Sl   Mar06 771:12 /usr/lib/jvm/java-11-openjdk-11.0.18.0.9-0.3.ea.el8.x86_64/bin/java -jar agent.jar -jnlpUrl https://jenkins.ceph.com/manage/computer/172%2E21%2E2%2E4%2Bbraggi04/jenkins-agent.jnlp -secret 6d84a7652ba06fb8711c1474eaff7f0be0ed65874a470d9ac256c96e6a9eef88 -workDir /home/jenkins-build/build

Actions
Copy link

Also available in: Atom PDF