Project

General

Profile

Bug #49309

nautilus: qa: "Assertion `cb_done' failed."

Added by Patrick Donnelly 3 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-16T04:57:21.926 INFO:teuthology.orchestra.run.gibba006.stdout:waiting for crash
2021-02-16T04:57:21.930 DEBUG:teuthology.orchestra.run.gibba006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb
2021-02-16T04:57:23.880 INFO:teuthology.orchestra.run.gibba006.stderr:ceph_test_ino_release_cb: /build/ceph-14.2.16-310-gca4a449b/src/test/fs/test_ino_release_cb.cc:75: int main(int, char**): Assertion `cb_done' failed.
2021-02-16T04:57:23.881 DEBUG:teuthology.orchestra.run:got remote process result: None
2021-02-16T04:57:23.881 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/task/exec.py", line 54, in task
    c],
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 179, in _raise_for_status
    raise CommandCrashedError(command=self.command)
teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb'
2021-02-16T04:57:24.245 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=443652519d1c46858c89099e3ff59ef0
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 91, in run_tasks
    manager = run_one_task(taskname, ctx=ctx, config=config)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 70, in run_one_task
    return task(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/task/exec.py", line 54, in task
    c],
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 179, in _raise_for_status
    raise CommandCrashedError(command=self.command)
teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb'

From: /ceph/teuthology-archive/yuriw-2021-02-15_21:26:29-fs-wip-yuri5-testing-2021-02-15-1156-nautilus-distro-basic-gibba/5885477/teuthology.log

Looking at the test, I'm wondering if this may be caused by cb_done being a volatile variable rather than atomic.


Related issues

Copied to CephFS - Backport #49473: nautilus: nautilus: qa: "Assertion `cb_done' failed." Resolved
Copied to CephFS - Backport #49474: pacific: nautilus: qa: "Assertion `cb_done' failed." Resolved
Copied to CephFS - Backport #49475: octopus: nautilus: qa: "Assertion `cb_done' failed." In Progress
Copied to CephFS - Bug #49500: qa: "Assertion `cb_done' failed." Pending Backport

History

#1 Updated by Jeff Layton 3 months ago

Could be. It wouldn't hurt to rework that with atomics and see whether it's reproducible. I'll do that and put up a PR.

The other possibility is that it just never got a callback for some reason, but let's rule out the volatile/atomic thing first.

#2 Updated by Patrick Donnelly 2 months ago

  • Status changed from New to Pending Backport

Hoping that simple fix is all what's needed...

#3 Updated by Backport Bot 2 months ago

  • Copied to Backport #49473: nautilus: nautilus: qa: "Assertion `cb_done' failed." added

#4 Updated by Backport Bot 2 months ago

  • Copied to Backport #49474: pacific: nautilus: qa: "Assertion `cb_done' failed." added

#5 Updated by Backport Bot 2 months ago

  • Copied to Backport #49475: octopus: nautilus: qa: "Assertion `cb_done' failed." added

#6 Updated by Patrick Donnelly 2 months ago

  • Copied to Bug #49500: qa: "Assertion `cb_done' failed." added

#7 Updated by Patrick Donnelly 2 months ago

This is still alive: #49500

Cloned this ticket so that the good fix, which didn't help, will still get backported normally.

#8 Updated by Patrick Donnelly about 2 months ago

  • Pull request ID set to 39498

Also available in: Atom PDF