Bug #63488: smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined" - CephFS - Ceph

Actions

Copy link

Bug #63488

open

smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"

Added by Laura Flores 6 months ago. Updated 3 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Venky Shankar

Category:

Testing

Target version:

Ceph - v19.0.0

% Done:

Source:

Tags:

backport_processed

Backport:

quincy,reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Labels (FS):

qa, qa-failure

Pull request ID:

54515

Crash signature (v1):

Crash signature (v2):

Description

/a/yuriw-2023-11-07_20:47:24-smoke-reef-release-distro-default-smithi/7451517

2023-11-07T21:12:59.283 DEBUG:teuthology.orchestra.run.smithi183:> def get_id_to_dir():
2023-11-07T21:12:59.285 DEBUG:teuthology.orchestra.run.smithi183:>     result = {}
2023-11-07T21:12:59.285 DEBUG:teuthology.orchestra.run.smithi183:>     for dir in glob.glob("/sys/kernel/debug/ceph/*"):
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>         if os.path.basename(dir) == DEBUGFS_META_DIR:
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>             continue
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>         mds_sessions_lines = open(os.path.join(dir, "mds_sessions")).readlines()
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         global_id = mds_sessions_lines[0].split()[1].strip('"'"'"'"'"')
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         client_id = mds_sessions_lines[1].split()[1].strip('"'"'"'"'"')
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         result[client_id] = global_id
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:>     return result
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:> print(json.dumps(get_id_to_dir()))
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:> '
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:Traceback (most recent call last):
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:  File "<string>", line 16, in <module>
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:  File "<string>", line 9, in get_id_to_dir
2023-11-07T21:12:59.332 INFO:teuthology.orchestra.run.smithi183.stderr:NameError: name 'DEBUGFS_META_DIR' is not defined
2023-11-07T21:12:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-11-07T21:12:59.336 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/kernel_mount.py", line 245, in _get_global_id
    p = self.run_shell_payload("getfattr --only-values -n ceph.client_id .", stdout=StringIO())
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/mount.py", line 789, in run_shell_payload
    return self.run_shell(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/mount.py", line 780, in run_shell
    return self.client_remote.run(args=args, cwd=cwd, timeout=timeout,
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 1: "(cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .')"

Related issues 2 (1 open — 1 closed)

Actions

Copy link

Updated by Laura Flores 6 months ago

/a/yuriw-2023-11-07_20:47:24-smoke-reef-release-distro-default-smithi/7451519

Actions

Copy link

Updated by Laura Flores 6 months ago

Subject changed from kclient_workunit_direct_ioworkload fails from NameError: name 'DEBUGFS_META_DIR' is not defined to cephfs smoke workload fails from NameError: name 'DEBUGFS_META_DIR' is not defined

Actions

Copy link

Updated by Laura Flores 6 months ago

Subject changed from cephfs smoke workload fails from NameError: name 'DEBUGFS_META_DIR' is not defined to cephfs smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"

Actions

Copy link

Updated by Laura Flores 6 months ago

Subject changed from cephfs smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined" to smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"

Actions

Copy link

Updated by Venky Shankar 6 months ago

Status changed from New to In Progress
Assignee set to Venky Shankar
Target version set to v19.0.0
Labels (FS) qa, qa-failure added

Actions

Copy link

Updated by Venky Shankar 6 months ago

This failure is odd - nothing has changed w.r.t. DEBUGFS_META_DIR when it got merged in commit

commit ff88d7de52b347486a626375816fc119ee8fd72f (vshankar/wip-kclient-ignore-meta, origin/wip-kclient-ignore-meta)
Author: Venky Shankar <vshankar@redhat.com>
Date:   Thu Oct 7 00:40:13 2021 -0400

    qa: skip internal metadata directory when scanning ceph debugfs directory

    kclient patchset

            https://patchwork.kernel.org/project/ceph-devel/list/?series=556049

    introduces `meta` directory to add debugging entries. This needs to be filtered
    when scanning ceph debugfs directory.

    Fixes: https://tracker.ceph.com/issues/52824
    Signed-off-by: Venky Shankar <vshankar@redhat.com>

Actions

Copy link

Updated by Venky Shankar 6 months ago

So, this failure was also seen in earlier runs. Testing up a fix.

Actions

Copy link

Updated by Venky Shankar 6 months ago

Realized that ubuntu 20.04 doesn't have the kclient changes that adds functionality to fetch client-id via an xattr query thereby causing the getfattr call here

https://github.com/ceph/ceph/blob/main/qa/tasks/cephfs/kernel_mount.py#L245

to fallback to fetching the client-id from debugfs, however, that code path has a bug. Once I moved past fixing that, I ran into more problems. The client instance addr requires fetching from a file in debugfs, e.g.:

/sys/kernel/debug/ceph/214ddfa0-c3ba-4e15-80dc-d44fecf09144.client4466/status

However, that isn't available in ubuntu 20.04 either. So, we have hit a road block.

The best approach seems like to override the kernel for test that use kernel-driver with ubuntu 20.04 (which the fs suite does btw), so the smoke suite can be changes accordingly.

EDIT: Xiubo confirmed in #cephfs (on slack) that ubuntu 20.04 does not have the following commits:

- 5a9e2f5d5590 ceph: add ceph.{cluster_fsid/client_id} vxattrs
- 247b1f19dbeb ceph: add status debugfs file

which causes the kernel_mount.py to not find a specific file in debugfs to extract the client addr.

Actions

Copy link

Updated by Laura Flores 6 months ago

@Venky want to attach your PR to this tracker?

Actions

Copy link

#10

Updated by Venky Shankar 6 months ago

Laura Flores wrote:

@Venky want to attach your PR to this tracker?

My changes aren't fully ready yet - it solves the issue related to DEBUGFS_META_DIR, but once that's fixed there are a host of issue following that. These issues were uncovered with Ubuntu 20.04 kernel since that kernel has some missing changes.

I'll push a PR soon.

Actions

Copy link

#11