Project

General

Profile

Actions

Bug #65116

open

squid: kclient: "ld: final link failed: Resource temporarily unavailable"

Added by Patrick Donnelly about 1 month ago. Updated about 1 month ago.

Status:
Triaged
Priority:
High
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
squid,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client, MDS, kceph
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2024-03-24T06:05:25.606 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception:
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/fwd_scrub.py", line 38, in _run
    self.do_scrub()
  File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/fwd_scrub.py", line 55, in do_scrub
    self._scrub()
  File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/fwd_scrub.py", line 79, in _scrub
    self._check_damage()
  File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/fwd_scrub.py", line 90, in _check_damage
    raise RuntimeError(f"rank damage found: {types}")
RuntimeError: rank damage found: {'backtrace'}
...
2024-03-24T06:12:00.854 INFO:tasks.workunit.client.0.smithi044.stderr:ld: final link failed: Resource temporarily unavailable
2024-03-24T06:12:00.900 INFO:tasks.workunit.client.0.smithi044.stderr:make[2]: *** [scripts/Makefile.vmlinux:36: vmlinux] Error 1
2024-03-24T06:12:00.901 INFO:tasks.workunit.client.0.smithi044.stderr:make[1]: *** [/home/ubuntu/cephtest/mnt.0/client.0/tmp/t/linux-6.5.11/Makefile:1250: vmlinux] Error 2
2024-03-24T06:12:00.901 INFO:tasks.workunit.client.0.smithi044.stderr:make: *** [Makefile:234: __sub-make] Error 2
2024-03-24T06:12:00.903 DEBUG:teuthology.orchestra.run:got remote process result: 2
2024-03-24T06:12:00.904 INFO:tasks.workunit:Stopping ['kernel_untar_build.sh'] on client.0...

From: /teuthology/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619024/teuthology.log

Note: there was a fwd_scrub traceback but it doesn't appear related to why the kernel_untar_build task failed.

The kernel was logging this approximately near the failure:

Mar 24 06:12:09 smithi044 kernel: evict_inodes inode 00000000db25ac9a, i_count = 1, was skipped!
Mar 24 06:12:09 smithi044 kernel: evict_inodes inode 0000000093846fa3, i_count = 1, was skipped!
Mar 24 06:12:09 smithi044 kernel: evict_inodes inode 00000000262ef84e, i_count = 1, was skipped!
Mar 24 06:12:09 smithi044 kernel: evict_inodes inode 0000000088763049, i_count = 1, was skipped!
Mar 24 06:12:09 smithi044 kernel: evict_inodes inode 00000000c8817dad, i_count = 1, was skipped!

From: /teuthology/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619024/remote/smithi044/syslog/journalctl-b0.gz

Actions #1

Updated by Venky Shankar about 1 month ago

  • Status changed from New to Triaged
  • Assignee set to Xiubo Li
  • Component(FS) Client, MDS, kceph added
Actions #2

Updated by Xiubo Li about 1 month ago

Patrick Donnelly wrote:

[...]

From: /teuthology/pdonnell-2024-03-24_04:56:01-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7619024/teuthology.log

Note: there was a fwd_scrub traceback but it doesn't appear related to why the kernel_untar_build task failed.

The kernel was logging this approximately near the failure:

[...]

This kclient logs are okay, they are just debug logs I added before and the most important is that these inodes are not ceph ones, they should be the local filesystems' inode numbers.

I will check the ceph logs to find what has happened.

Actions

Also available in: Atom PDF