Project

General

Profile

Actions

Bug #63488

open

smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"

Added by Laura Flores 5 months ago. Updated 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/yuriw-2023-11-07_20:47:24-smoke-reef-release-distro-default-smithi/7451517

2023-11-07T21:12:59.283 DEBUG:teuthology.orchestra.run.smithi183:> def get_id_to_dir():
2023-11-07T21:12:59.285 DEBUG:teuthology.orchestra.run.smithi183:>     result = {}
2023-11-07T21:12:59.285 DEBUG:teuthology.orchestra.run.smithi183:>     for dir in glob.glob("/sys/kernel/debug/ceph/*"):
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>         if os.path.basename(dir) == DEBUGFS_META_DIR:
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>             continue
2023-11-07T21:12:59.286 DEBUG:teuthology.orchestra.run.smithi183:>         mds_sessions_lines = open(os.path.join(dir, "mds_sessions")).readlines()
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         global_id = mds_sessions_lines[0].split()[1].strip('"'"'"'"'"')
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         client_id = mds_sessions_lines[1].split()[1].strip('"'"'"'"'"')
2023-11-07T21:12:59.287 DEBUG:teuthology.orchestra.run.smithi183:>         result[client_id] = global_id
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:>     return result
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:> print(json.dumps(get_id_to_dir()))
2023-11-07T21:12:59.288 DEBUG:teuthology.orchestra.run.smithi183:> '
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:Traceback (most recent call last):
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:  File "<string>", line 16, in <module>
2023-11-07T21:12:59.331 INFO:teuthology.orchestra.run.smithi183.stderr:  File "<string>", line 9, in get_id_to_dir
2023-11-07T21:12:59.332 INFO:teuthology.orchestra.run.smithi183.stderr:NameError: name 'DEBUGFS_META_DIR' is not defined
2023-11-07T21:12:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1
2023-11-07T21:12:59.336 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/kernel_mount.py", line 245, in _get_global_id
    p = self.run_shell_payload("getfattr --only-values -n ceph.client_id .", stdout=StringIO())
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/mount.py", line 789, in run_shell_payload
    return self.run_shell(**kwargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_55e3239498650453ff76a9b06a37f1a6f488c8fd/qa/tasks/cephfs/mount.py", line 780, in run_shell
    return self.client_remote.run(args=args, cwd=cwd, timeout=timeout,
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 1: "(cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .')" 


Related issues 2 (1 open1 closed)

Copied to CephFS - Backport #64249: quincy: smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"In ProgressVenky ShankarActions
Copied to CephFS - Backport #64250: reef: smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"ResolvedVenky ShankarActions
Actions #1

Updated by Laura Flores 5 months ago

/a/yuriw-2023-11-07_20:47:24-smoke-reef-release-distro-default-smithi/7451519

Actions #2

Updated by Laura Flores 5 months ago

  • Subject changed from kclient_workunit_direct_ioworkload fails from NameError: name 'DEBUGFS_META_DIR' is not defined to cephfs smoke workload fails from NameError: name 'DEBUGFS_META_DIR' is not defined
Actions #3

Updated by Laura Flores 5 months ago

  • Subject changed from cephfs smoke workload fails from NameError: name 'DEBUGFS_META_DIR' is not defined to cephfs smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"
Actions #4

Updated by Laura Flores 5 months ago

  • Subject changed from cephfs smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined" to smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined"
Actions #5

Updated by Venky Shankar 5 months ago

  • Status changed from New to In Progress
  • Assignee set to Venky Shankar
  • Target version set to v19.0.0
  • Labels (FS) qa, qa-failure added
Actions #6

Updated by Venky Shankar 5 months ago

This failure is odd - nothing has changed w.r.t. DEBUGFS_META_DIR when it got merged in commit

commit ff88d7de52b347486a626375816fc119ee8fd72f (vshankar/wip-kclient-ignore-meta, origin/wip-kclient-ignore-meta)
Author: Venky Shankar <vshankar@redhat.com>
Date:   Thu Oct 7 00:40:13 2021 -0400

    qa: skip internal metadata directory when scanning ceph debugfs directory

    kclient patchset

            https://patchwork.kernel.org/project/ceph-devel/list/?series=556049

    introduces `meta` directory to add debugging entries. This needs to be filtered
    when scanning ceph debugfs directory.

    Fixes: https://tracker.ceph.com/issues/52824
    Signed-off-by: Venky Shankar <vshankar@redhat.com>
Actions #7

Updated by Venky Shankar 5 months ago

So, this failure was also seen in earlier runs. Testing up a fix.

Actions #8

Updated by Venky Shankar 5 months ago

Realized that ubuntu 20.04 doesn't have the kclient changes that adds functionality to fetch client-id via an xattr query thereby causing the getfattr call here

https://github.com/ceph/ceph/blob/main/qa/tasks/cephfs/kernel_mount.py#L245

to fallback to fetching the client-id from debugfs, however, that code path has a bug. Once I moved past fixing that, I ran into more problems. The client instance addr requires fetching from a file in debugfs, e.g.:

/sys/kernel/debug/ceph/214ddfa0-c3ba-4e15-80dc-d44fecf09144.client4466/status

However, that isn't available in ubuntu 20.04 either. So, we have hit a road block.

The best approach seems like to override the kernel for test that use kernel-driver with ubuntu 20.04 (which the fs suite does btw), so the smoke suite can be changes accordingly.

EDIT: Xiubo confirmed in #cephfs (on slack) that ubuntu 20.04 does not have the following commits:

- 5a9e2f5d5590 ceph: add ceph.{cluster_fsid/client_id} vxattrs
- 247b1f19dbeb ceph: add status debugfs file

which causes the kernel_mount.py to not find a specific file in debugfs to extract the client addr.

Actions #9

Updated by Laura Flores 5 months ago

@Venky want to attach your PR to this tracker?

Actions #10

Updated by Venky Shankar 5 months ago

Laura Flores wrote:

@Venky want to attach your PR to this tracker?

My changes aren't fully ready yet - it solves the issue related to DEBUGFS_META_DIR, but once that's fixed there are a host of issue following that. These issues were uncovered with Ubuntu 20.04 kernel since that kernel has some missing changes.

I'll push a PR soon.

Actions #11

Updated by Venky Shankar 5 months ago

  • Category set to Testing
  • Status changed from In Progress to Fix Under Review
  • Backport changed from reef to quincy,reef
  • Pull request ID set to 54515
Actions #12

Updated by Venky Shankar 2 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions #13

Updated by Backport Bot 2 months ago

  • Copied to Backport #64249: quincy: smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined" added
Actions #14

Updated by Backport Bot 2 months ago

  • Copied to Backport #64250: reef: smoke test fails from "NameError: name 'DEBUGFS_META_DIR' is not defined" added
Actions #15

Updated by Backport Bot 2 months ago

  • Tags set to backport_processed
Actions

Also available in: Atom PDF