Bug #58689
openWeird Shaman build behavior
Added by Laura Flores over 1 year ago. Updated 12 months ago.
0%
Description
Problem #1:
Pushing a branch to CI results in some builds duplicating multiple times, and other builds missing completely.
This build is an example: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/. Notice how there are multiple builds of centos 8/9 and opensuse, but no focal. I had to trigger focal manually, so it will likely show up soon.
Problem #2:
Builds are taking an extremely long time for Shaman to pick up. My build has been waiting over an hour, and estimated time to finish is a little over 3 hours remaining (https://jenkins.ceph.com/job/ceph-dev-new/68766)
Updated by Laura Flores over 1 year ago
Another weird build behavior:
For this build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/
Teuthology says it cannot find centos 8 packages even though there is a successful build for that SHA:
2023-02-10 02:11:55,393.393 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '27a5e68d9b64346d6596f06394f6478d861f4d51' not found
2023-02-10 02:11:55,413.413 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '27a5e68d9b64346d6596f06394f6478d861f4d51' not found
/home/nojha/teuth/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
Job scheduled with name nojha-2023-02-10_02:11:18-rados-wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt-distro-default-smithi and ID 7166383
2023-02-10 02:11:56,487.487 INFO:teuthology.suite.run:Scheduling rados/singleton/{all/dump-stuck mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}
Traceback (most recent call last):
File "./virtualenv/bin/teuthology-suite", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')())
File "/home/nojha/teuth/scripts/suite.py", line 219, in main
return teuthology.suite.main(args)
File "/home/nojha/teuth/teuthology/suite/__init__.py", line 145, in main
run.prepare_and_schedule()
File "/home/nojha/teuth/teuthology/suite/run.py", line 416, in prepare_and_schedule
num_jobs = self.schedule_suite()
File "/home/nojha/teuth/teuthology/suite/run.py", line 679, in schedule_suite
self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
File "/home/nojha/teuth/teuthology/suite/run.py", line 515, in schedule_jobs
name,
File "/home/nojha/teuth/teuthology/suite/util.py", line 76, in schedule_fail
raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling nojha-2023-02-10_02:11:18-rados-wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt-distro-default-smithi failed: At least one job needs packages that don't exist for hash 27a5e68d9b64346d6596f06394f6478d861f4d51.
There was one failed centos 8 build and one successful centos 8 build for this SHA. Teuthology looks like it's trying to pull packages for the failed build, even though there is different successful build it should be able to use.
Updated by Yuri Weinstein over 1 year ago
See an example on `main` - https://shaman.ceph.com/builds/ceph/main/567963111c306402d43911e64819861a1a30de81/
Updated by Zack Cerza over 1 year ago
I found something I think is pretty strange. Starting from this shaman page: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/
I visited the oldest build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/default/332476/
From there, I visited the Jenkins build: https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/67974/
Then, I visited the build referenced by the message "Started by upstream project ceph-dev-new-build build number 67974": https://jenkins.ceph.com/job/ceph-dev-new-build/67974/
Here, I see a matrix showing that several builds were scheduled by this one, including focal: https://jenkins.ceph.com/job/ceph-dev-new-build/67974/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/
That build even succeeded!
From there I visit the console log and find a reference to: https://shaman.ceph.com/api/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/
As this endpoint returns a JSON payload, I'll paste it here:
[{"status":"ready","sha1":"27a5e68d9b64346d6596f06394f6478d861f4d51","extra":{"build_url":"https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/67988/","root_build_cause":"MANUALTRIGGER","version":"17.2.5-715-g27a5e68d","node_name":"172.21.2.3+braggi03","job_name":"ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic","package_manager_version":"17.2.5-715-g27a5e68d-1focal"},"url":"https://1.chacra.ceph.com/r/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","distro_codename":"focal","modified":"2023-02-10 04:25:09.081873","distro_version":"20.04","project":"ceph","flavor":"default","ref":"wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt","chacra_url":"https://1.chacra.ceph.com/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","archs":["x86_64"],"distro":"ubuntu"},{"status":"ready","sha1":"27a5e68d9b64346d6596f06394f6478d861f4d51","extra":{"build_url":"https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic/67974/","root_build_cause":"SCMTRIGGER","version":"17.2.5-715-g27a5e68d","node_name":"172.21.2.2+braggi02","job_name":"ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=focal,DIST=focal,MACHINE_SIZE=gigantic","package_manager_version":"17.2.5-715-g27a5e68d-1focal"},"url":"https://4.chacra.ceph.com/r/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","distro_codename":"focal","modified":"2023-02-10 01:41:47.864656","distro_version":"20.04","project":"ceph","flavor":"default","ref":"wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt","chacra_url":"https://4.chacra.ceph.com/repos/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/ubuntu/focal/flavors/default/","archs":["x86_64"],"distro":"ubuntu"}]
This is a build that finished at
2023-02-10 01:41:47.864656
, yet the shaman UI link does not list it. Why??Updated by Zack Cerza over 1 year ago
Laura Flores wrote:
For this build: https://shaman.ceph.com/builds/ceph/wip-sseshasa-quincy-2023-02-09-mclk-cost-fixes-opt/27a5e68d9b64346d6596f06394f6478d861f4d51/
This build was also looking for rhel packages, which I don't see.
Updated by Laura Flores about 1 year ago
Duplicate distros and flavors were created for my build when I pushed only once to CI:
https://shaman.ceph.com/builds/ceph/wip-lflores-testing-2023-02-17-2149/
Updated by Laura Flores about 1 year ago
- Has duplicate Bug #58951: some branches (reef) seem to build double number of distros added
Updated by Laura Flores about 1 year ago
Problems with this build: https://shaman.ceph.com/builds/ceph/reef/0447d535c14cd711af288300f5ccc6cd3874567a/
Happens when scheduling teuthology-suite:
2023-03-10 18:15:22,978.978 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:22,984.984 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2023-03-10 18:15:22,984.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,126.126 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,136.136 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb
2023-03-10 18:15:23,136.136 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,279.279 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2023-03-10 18:15:23,279.279 INFO:teuthology.suite.util:container build centos/8, checking for build_complete
2023-03-10 18:15:23,279.279 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a
2023-03-10 18:15:23,593.593 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default
2023-03-10 18:15:23,593.593 DEBUG:teuthology.packaging:build: centos/8 x86_64 crimson
2023-03-10 18:15:23,594.594 DEBUG:teuthology.packaging:build: centos/8 x86_64 default
2023-03-10 18:15:23,594.594 INFO:teuthology.suite.util:build not complete
2023-03-10 18:15:23,594.594 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,607.607 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,629.629 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,655.655 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,682.682 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,693.693 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,714.714 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
2023-03-10 18:15:23,720.720 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '0447d535c14cd711af288300f5ccc6cd3874567a' not found
/home/lflores/teuthology/virtualenv/lib/python3.6/site-packages/paramiko/transport.py:33: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
from cryptography.hazmat.backends import default_backend
testing12 1
Job scheduled with name lflores-2023-03-10_18:15:16-smoke-reef-distro-default-smithi and ID 7202000
2023-03-10 18:15:24,809.809 INFO:teuthology.suite.run:Scheduling smoke/basic/{clusters/{fixed-3-cephfs openstack} objectstore/bluestore-bitmap supported-random-distro$/{centos_8} tasks/{0-install test/cfuse_workunit_suites_blogbench}}
Traceback (most recent call last):
File "./teuthology/virtualenv/bin/teuthology-suite", line 33, in <module>
sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')())
File "/home/lflores/teuthology/scripts/suite.py", line 219, in main
return teuthology.suite.main(args)
File "/home/lflores/teuthology/teuthology/suite/__init__.py", line 145, in main
run.prepare_and_schedule()
File "/home/lflores/teuthology/teuthology/suite/run.py", line 422, in prepare_and_schedule
num_jobs = self.schedule_suite()
File "/home/lflores/teuthology/teuthology/suite/run.py", line 687, in schedule_suite
self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name)
File "/home/lflores/teuthology/teuthology/suite/run.py", line 522, in schedule_jobs
dry_run=self.args.dry_run,
File "/home/lflores/teuthology/teuthology/suite/util.py", line 76, in schedule_fail
raise ScheduleFailError(message, name)
teuthology.exceptions.ScheduleFailError: Scheduling lflores-2023-03-10_18:15:16-smoke-reef-distro-default-smithi failed: At least one job needs packages that don't exist for hash 0447d535c14cd711af288300f5ccc6cd3874567a.
The issue is apparent in https://shaman.ceph.com/api/search/?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=0447d535c14cd711af288300f5ccc6cd3874567a.
There are two builds, 0 and 1. If you click on "build_url" for both, you can see that the build failed for 0, but passed for 1.
Teuthology is trying to access packages from the failed build (although there is a successful one there, it is not first in line).
Updated by Yuri Weinstein about 1 year ago
Please 3x the number of distro built for this =>
https://shaman.ceph.com/builds/ceph/reef/6a1183ca1670d9ed678e2162459161e1ad2cdd45/
Updated by Yuri Weinstein about 1 year ago
Also related issue
Here is the build https://shaman.ceph.com/builds/ceph/wip-yuri2-testing-2023-05-15-0810-pacific/25796df0fedbe757877f6bad6ff202a3d2ca4abf/
However, `teuthology-suite` failed scheduling runs with the error: "At least one job needs packages that don't exist for hash 25796df0fedbe757877f6bad6ff202a3d2ca4abf" (full output too large to attach here)
....... 2023-05-15 19:34:00,140.140 INFO:teuthology.suite.util:container build centos/8, checking for build_complete 2023-05-15 19:34:00,140.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=25796df0fedbe757877f6bad6ff202a3d2ca4abf 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:looking for centos/8 x86_64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: centos/8 arm64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/20.04 x86_64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 x86_64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: centos/8 arm64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 x86_64 default 2023-05-15 19:34:00,442.442 DEBUG:teuthology.packaging:build: ubuntu/18.04 arm64 default 2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: ubuntu/18.04 arm64 default 2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: windows/1809 x86_64 default 2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: ubuntu/20.04 x86_64 default 2023-05-15 19:34:00,443.443 DEBUG:teuthology.packaging:build: centos/8 x86_64 default 2023-05-15 19:34:00,443.443 INFO:teuthology.suite.util:build not complete 2023-05-15 19:34:00,443.443 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,454.454 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,470.470 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,486.486 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,501.501 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,524.524 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,534.534 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,542.542 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,552.552 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,560.560 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,568.568 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,577.577 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,592.592 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,603.603 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,613.613 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,621.621 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,636.636 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,643.643 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,656.656 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,670.670 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,679.679 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,687.687 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,707.707 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,714.714 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,723.723 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,749.749 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,757.757 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,765.765 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,785.785 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,802.802 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,810.810 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,818.818 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,825.825 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,838.838 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,850.850 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,860.860 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,892.892 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,899.899 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,906.906 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,924.924 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,931.931 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,949.949 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,972.972 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:00,979.979 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,010.010 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,018.018 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,028.028 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,045.045 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,053.053 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,062.062 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,071.071 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,081.081 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,089.089 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,124.124 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,131.131 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,146.146 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,149.149 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,158.158 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,165.165 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,172.172 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,183.183 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,190.190 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,204.204 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,225.225 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,231.231 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,239.239 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,249.249 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,255.255 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,262.262 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,275.275 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,283.283 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,290.290 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,307.307 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,316.316 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,326.326 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,343.343 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,350.350 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,358.358 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,375.375 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,382.382 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,389.389 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,396.396 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,406.406 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,414.414 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,422.422 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,430.430 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,456.456 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,464.464 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,471.471 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,479.479 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,485.485 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,515.515 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,534.534 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro ubuntu: arch=x86_64, release=ubuntu/16.04, pkg_type=deb 2023-05-15 19:34:01,535.535 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=25796df0fedbe757877f6bad6ff202a3d2ca4abf 2023-05-15 19:34:01,687.687 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,703.703 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,709.709 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,717.717 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,723.723 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,750.750 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,758.758 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,766.766 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,774.774 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,783.783 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,791.791 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,800.800 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,823.823 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,835.835 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,859.859 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,872.872 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,886.886 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,893.893 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,903.903 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,910.910 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,921.921 ERROR:teuthology.suite.run:Packages for os_type 'centos', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found 2023-05-15 19:34:01,937.937 ERROR:teuthology.suite.run:Packages for os_type 'rhel', flavor default and ceph hash '25796df0fedbe757877f6bad6ff202a3d2ca4abf' not found Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274491 2023-05-15 19:34:02,674.674 INFO:teuthology.suite.run:Scheduling fs/traceless/{begin clusters/fixed-2-ucephfs conf/{client mds mon osd} distro/{ubuntu_latest} mount/fuse objectstore-ec/bluestore-comp overrides/{frag ignorelist_health ignorelist_wrongly_marked_down} tasks/cfuse_workunit_suites_dbench traceless/50pc} Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274492 2023-05-15 19:34:03,537.537 INFO:teuthology.suite.run:Scheduling fs/workload/{begin clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/ubuntu_latest k-testing} ms-die-on-skipped}} ms_mode/{crc} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/3 scrub/no standby-replay tasks/{0-check-counter workunit/suites/fsx} wsync/{no}} Job scheduled with name yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi and ID 7274493 2023-05-15 19:34:04,330.330 INFO:teuthology.suite.run:Scheduling fs/functional/{begin clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount overrides/{distro/testing/{flavor/centos_latest k-testing} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/client-readahead} Traceback (most recent call last): File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 8, in <module> sys.exit(main()) File "/home/yuriw/teuthology/scripts/suite.py", line 223, in main return teuthology.suite.main(args) File "/home/yuriw/teuthology/teuthology/suite/__init__.py", line 145, in main run.prepare_and_schedule() File "/home/yuriw/teuthology/teuthology/suite/run.py", line 425, in prepare_and_schedule num_jobs = self.schedule_suite() File "/home/yuriw/teuthology/teuthology/suite/run.py", line 690, in schedule_suite self.schedule_jobs(jobs_missing_packages, jobs_to_schedule, name) File "/home/yuriw/teuthology/teuthology/suite/run.py", line 521, in schedule_jobs util.schedule_fail( File "/home/yuriw/teuthology/teuthology/suite/util.py", line 76, in schedule_fail raise ScheduleFailError(message, name) teuthology.exceptions.ScheduleFailError: Scheduling yuriw-2023-05-15_19:33:12-fs-wip-yuri2-testing-2023-05-15-0810-pacific-distro-default-smithi failed: At least one job needs packages that don't exist for hash 25796df0fedbe757877f6bad6ff202a3d2ca4abf.
Updated by Zack Cerza 12 months ago
Has duplicate builds, i.e.:
https://jenkins.ceph.com/job/ceph-dev-new-trigger/45802/
https://jenkins.ceph.com/job/ceph-dev-new-trigger/45803/
Looking for differences, I see this at the beginning of 02's console log:
Started by an SCM change
At the beginning of 03's console log:
Started by GitHub push by yuriw Started by GitHub push by yuriw
jenkins.ceph.com:~jenkins/0.log:
jenkins@jenkins:~$ grep 4580[12] 0.log Jun 5, 2023 10:06:20,488 PMjob/ceph-dev-new-trigger/ #45801 Started by GitHub push by yuriw, Started by GitHub push by yuriw, Parameters:[] Jun 5, 2023 10:08:20,313 PM'ceph-dev-new-trigger#45801' (class hudson.model.FreeStyleBuild) used credentials 'shaman-api-key' (class org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl). Jun 5, 2023 10:08:21,735 PMceph-dev-new-trigger #45801 Started by GitHub push by yuriw, Started by GitHub push by yuriw, Parameters:[] on node Jenkins started at 2023-06-05T22:06:19Z completed in 119934ms completed: SUCCESS Jun 5, 2023 10:08:21,823 PMjob/ceph-dev-new-trigger/ #45802 Started by an SCM change, Parameters:[] Jun 5, 2023 10:09:37,512 PM'ceph-dev-new-trigger#45802' (class hudson.model.FreeStyleBuild) used credentials 'shaman-api-key' (class org.jenkinsci.plugins.plaincredentials.impl.StringCredentialsImpl). Jun 5, 2023 10:09:38,778 PMceph-dev-new-trigger #45802 Started by an SCM change, Parameters:[] on node Jenkins started at 2023-06-05T22:08:21Z completed in 75905ms completed: SUCCESS
Back at the Jenkins UI, I noticed this job is claiming to have been running for 24h:
https://jenkins.ceph.com/job/ceph-dev-new-build/70346/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic
It was started by 45802, and is hung on:
+ ls -la total 392 drwxr-xr-x. 6 jenkins-build jenkins-build 162 Jun 5 22:28 . drwxr-xr-x. 3 jenkins-build jenkins-build 22 Mar 7 20:33 .. -rw-r--r--. 1 jenkins-build users 110 Jun 5 22:28 build_info -rw-r--r--. 1 jenkins-build users 847 Jun 5 22:28 build_status.json -rw-r--r--. 1 jenkins-build users 381328 Jun 5 22:24 cephadm drwxr-xr-x. 11 jenkins-build users 251 Jun 5 22:28 ceph-container drwxr-xr-x. 2 jenkins-build users 6 Jun 5 22:28 ceph-container@tmp drwxr-xr-x. 3 jenkins-build users 4096 Jun 5 21:12 dist drwxr-xr-x. 3 jenkins-build users 35 Jun 5 21:12 release -rw-r--r--. 1 jenkins-build users 471 Jun 5 22:24 repo-extra.json + sudo rm -rf dist + rm -rf venv + sudo rm -rf release
It's allegedly running on braggi04, but I didn't see any relevant processes. Also:
[jenkins-build@braggi04 ~]$ find build/workspace/ build/workspace/ build/workspace/workspaces.txt jenkins+ 355993 0.5 0.1 38402512 446776 ? Ssl 06:39 5:06 /usr/bin/java -Dfile.encoding=UTF8 -jar /home/jenkins-build/agent.jar -jnlpUrl https://jenkins.ceph.com/computer/172.21.2.4+braggi04/slave-agent.jnlp -jnlpCredentials @/etc/systemd/system/jenkins.secret root 488043 0.5 0.6 38114332 1522472 ? Sl Mar06 771:12 /usr/lib/jvm/java-11-openjdk-11.0.18.0.9-0.3.ea.el8.x86_64/bin/java -jar agent.jar -jnlpUrl https://jenkins.ceph.com/manage/computer/172%2E21%2E2%2E4%2Bbraggi04/jenkins-agent.jnlp -secret 6d84a7652ba06fb8711c1474eaff7f0be0ed65874a470d9ac256c96e6a9eef88 -workDir /home/jenkins-build/build