Bug #57655
openqa: fs:mixed-clients kernel_untar_build failure
Added by Patrick Donnelly over 1 year ago. Updated 6 months ago.
0%
Description
2022-09-12T12:12:00.425 INFO:tasks.workunit.client.1.smithi176.stderr:fs/compat.o: warning: objtool: missing symbol for section .text 2022-09-12T12:12:00.487 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/binfmt_misc.o 2022-09-12T12:12:00.842 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/binfmt_script.o 2022-09-12T12:12:00.980 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/binfmt_elf.o 2022-09-12T12:12:01.273 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/compat_binfmt_elf.o 2022-09-12T12:12:01.278 INFO:tasks.workunit.client.1.smithi176.stdout: AR kernel/built-in.a 2022-09-12T12:12:01.714 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/mbcache.o 2022-09-12T12:12:01.739 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/posix_acl.o 2022-09-12T12:12:01.742 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/coredump.o 2022-09-12T12:12:01.777 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/drop_caches.o 2022-09-12T12:12:01.795 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/fhandle.o 2022-09-12T12:12:02.186 INFO:tasks.workunit.client.1.smithi176.stdout: CC fs/dcookies.o 2022-09-12T12:12:02.982 INFO:tasks.workunit.client.1.smithi176.stderr:fs/dcookies.o: warning: objtool: missing symbol for section .text 2022-09-12T12:12:02.999 INFO:tasks.workunit.client.1.smithi176.stdout: AR fs/built-in.a 2022-09-12T12:12:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-09-12T12:12:03.196 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.1...
Seen: /ceph/teuthology-archive/dparmar-2022-09-12_11:38:14-fs:mixed-clients-main-distro-default-smithi/7029223/teuthology.log
and more recently: /ceph/teuthology-archive/pdonnell-2022-09-22_12:22:37-fs-wip-pdonnell-testing-20220920.234701-distro-default-smithi/7041086/teuthology.log
Updated by Patrick Donnelly over 1 year ago
- Related to Bug #57280: qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman added
Updated by Venky Shankar over 1 year ago
- Related to deleted (Bug #57280: qa: tasks/kernel_cfuse_workunits_untarbuild_blogbench fails - Failed to fetch package version from shaman)
Updated by Venky Shankar over 1 year ago
- Category set to Correctness/Safety
- Status changed from New to Triaged
- Assignee set to Milind Changire
- Backport set to pacific,quincy
Updated by Dhairya Parmar over 1 year ago
Maybe this can help here https://lore.kernel.org/all/alpine.LSU.2.21.2004031057320.25955@pobox.suse.cz/.
Updated by Dhairya Parmar over 1 year ago
Hitting similar issue in my today's run against ubuntu 22.04:
http://qa-proxy.ceph.com/teuthology/dparmar-2022-10-10_12:03:15-fs:mixed-clients-main-distro-default-smithi/7060610/teuthology.log
http://qa-proxy.ceph.com/teuthology/dparmar-2022-10-10_12:03:15-fs:mixed-clients-main-distro-default-smithi/7060611/teuthology.log
http://qa-proxy.ceph.com/teuthology/dparmar-2022-10-10_12:03:15-fs:mixed-clients-main-distro-default-smithi/7060616/teuthology.log
Updated by Venky Shankar over 1 year ago
Updated by Venky Shankar 12 months ago
- Target version changed from v18.0.0 to v19.0.0
- Backport changed from pacific,quincy to reef,quincy,pacific
Milind, do we know if this (kernel build failure) is a compiler related things or an issue (bug) in cephfs in the IO path?
Updated by Kotresh Hiremath Ravishankar 12 months ago
Milind, PTAL at the discussion in https://tracker.ceph.com/issues/59342 if it helps. I closed that as duplicate of this.
Updated by Kotresh Hiremath Ravishankar 12 months ago
reef
https://pulpito.ceph.com/yuriw-2023-05-10_18:53:39-fs-wip-yuri3-testing-2023-05-10-0851-reef-distro-default-smithi/7270187
https://pulpito.ceph.com/yuriw-2023-05-10_18:53:39-fs-wip-yuri3-testing-2023-05-10-0851-reef-distro-default-smithi/7270297
https://pulpito.ceph.com/yuriw-2023-05-15_15:47:06-fs-wip-yuri3-testing-2023-05-10-0851-reef-distro-default-smithi/7274460
Updated by Milind Changire 12 months ago
kernel_untar_build.sh test passes with latest code (HEAD 17f4abe9c9c) in the main branch
Xiubo pointed that a revert PR helps with that
so we just need to get this backported
Updated by Venky Shankar 8 months ago
/a/https://pulpito.ceph.com/vshankar-2023-09-12_06:47:30-fs-wip-vshankar-testing-20230908.065909-testing-default-smithi/7394622/
This time with a slightly different failure (but along the same lines):
2023-09-12T08:12:53.754 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr.o: In function `sr_block_check_events': 2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr.c:(.text+0x1062): undefined reference to `cdrom_check_events' 2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr.o: In function `sr_block_revalidate_disk': 2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr.c:(.text+0x11f2): undefined reference to `cdrom_get_last_written' 2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:drivers/scsi/sr_ioctl.o: In function `sr_drive_status': 2023-09-12T08:12:53.755 INFO:tasks.workunit.client.0.smithi064.stderr:sr_ioctl.c:(.text+0x6b0): undefined reference to `cdrom_get_media_event' 2023-09-12T08:12:54.083 INFO:tasks.workunit.client.0.smithi064.stderr:make: *** [Makefile:1077: vmlinux] Error 1
Updated by Patrick Donnelly 6 months ago
- Priority changed from Normal to Immediate
2023-10-26T06:08:42.898 INFO:tasks.workunit.client.1.smithi155.stderr:ld: arch/x86/boot/compressed/pgtable_64.o:(.bss+0x0): multiple definition of `__force_order'; arch/x86/boot/compressed/kaslr_64.o:(.bss+0x0): first defined here 2023-10-26T06:08:42.902 INFO:tasks.workunit.client.1.smithi155.stderr:ld: arch/x86/boot/compressed/head_64.o: warning: relocation in read-only section `.head.text' 2023-10-26T06:08:42.931 INFO:tasks.workunit.client.1.smithi155.stderr:ld: warning: creating DT_TEXTREL in a PIE 2023-10-26T06:08:42.936 INFO:tasks.workunit.client.1.smithi155.stderr:make[2]: *** [arch/x86/boot/compressed/Makefile:118: arch/x86/boot/compressed/vmlinux] Error 1 2023-10-26T06:08:42.937 INFO:tasks.workunit.client.1.smithi155.stderr:make[1]: *** [arch/x86/boot/Makefile:112: arch/x86/boot/compressed/vmlinux] Error 2
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Updated by Milind Changire 6 months ago
Patrick Donnelly wrote:
[...]
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel build
e.g. there's no multiple definition of this variable in the testing kernel sources
Updated by Patrick Donnelly 6 months ago
Milind Changire wrote:
Patrick Donnelly wrote:
[...]
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel builde.g. there's no multiple definition of this variable in the testing kernel sources
You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.
Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.
Updated by Milind Changire 6 months ago
Patrick Donnelly wrote:
Milind Changire wrote:
Patrick Donnelly wrote:
[...]
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel builde.g. there's no multiple definition of this variable in the testing kernel sources
You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.
Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.
centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logs
so this is 100% reproducible for centos9 and ubuntu 22.04 as well
we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 job
Updated by Venky Shankar 6 months ago
Milind Changire wrote:
Patrick Donnelly wrote:
Milind Changire wrote:
Patrick Donnelly wrote:
[...]
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel builde.g. there's no multiple definition of this variable in the testing kernel sources
You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.
Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.
centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logsso this is 100% reproducible for centos9 and ubuntu 22.04 as well
we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 job
Are you saying it was an issue with the kernel tarball the test was fetching and using a latest tarball gives a clean build?
EDIT: Since the current tarball fails build in other distros, then we could update our tests to fetch the latest tarball to be used.
Updated by Milind Changire 6 months ago
- Pull request ID set to 54414
Venky Shankar wrote:
Milind Changire wrote:
Patrick Donnelly wrote:
Milind Changire wrote:
Patrick Donnelly wrote:
[...]
/teuthology/pdonnell-2023-10-26_05:21:22-fs-wip-batrick-testing-20231024.144545-distro-default-smithi/7438447/teuthology.log
This one is pretty scary. I'm raising priority.
Can we replace the kernel source tarball with a newer one ?
... because I did find multiple definition of the variable unsigned long __force_order ... one in arch/x86/boot/compressed/pgtable_64.c and the other in arch/x86/boot/compressed/kaslr_64.c as the per the error dumped during the kernel builde.g. there's no multiple definition of this variable in the testing kernel sources
You can try; maybe this is related to centos09 but I was worried there was some kind of corruption.
Just update the workunit to clone some recent linux tag and run 1 job to test. Also double-check this is 100% reproducible only for centos9.
centos9 job failed with the exact 'multiple definition' error
ubuntu 22.04 job failed as well, but the error is not obvious from the teuthology logsso this is 100% reproducible for centos9 and ubuntu 22.04 as well
we have a successful rhel_8 job
unfortunately, I couldn't find a filter to specifically launch a centos9 jobAre you saying it was an issue with the kernel tarball the test was fetching and using a latest tarball gives a clean build?
EDIT: Since the current tarball fails build in other distros, then we could update our tests to fetch the latest tarball to be used.
yes, using a newer tarball fixed the build issue
btw, the old tarball wasn't corrupted, it just a had a 'multiple deinition' of a symbol
I've added the PR number as well
Updated by Backport Bot 6 months ago
- Copied to Backport #63588: pacific: qa: fs:mixed-clients kernel_untar_build failure added
Updated by Backport Bot 6 months ago
- Copied to Backport #63589: quincy: qa: fs:mixed-clients kernel_untar_build failure added
Updated by Backport Bot 6 months ago
- Copied to Backport #63590: reef: qa: fs:mixed-clients kernel_untar_build failure added