Actions
Bug #49309
closednautilus: qa: "Assertion `cb_done' failed."
% Done:
0%
Source:
Q/A
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2021-02-16T04:57:21.926 INFO:teuthology.orchestra.run.gibba006.stdout:waiting for crash
2021-02-16T04:57:21.930 DEBUG:teuthology.orchestra.run.gibba006:> sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb
2021-02-16T04:57:23.880 INFO:teuthology.orchestra.run.gibba006.stderr:ceph_test_ino_release_cb: /build/ceph-14.2.16-310-gca4a449b/src/test/fs/test_ino_release_cb.cc:75: int main(int, char**): Assertion `cb_done' failed.
2021-02-16T04:57:23.881 DEBUG:teuthology.orchestra.run:got remote process result: None
2021-02-16T04:57:23.881 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/task/exec.py", line 54, in task
c],
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 179, in _raise_for_status
raise CommandCrashedError(command=self.command)
teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb'
2021-02-16T04:57:24.245 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=443652519d1c46858c89099e3ff59ef0
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/task/exec.py", line 54, in task
c],
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/remote.py", line 215, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_64d5f18051c2fa3ad001b1c48f945134f60a8f76/teuthology/orchestra/run.py", line 179, in _raise_for_status
raise CommandCrashedError(command=self.command)
teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo TESTDIR=/home/ubuntu/cephtest bash -c ceph_test_ino_release_cb'
From: /ceph/teuthology-archive/yuriw-2021-02-15_21:26:29-fs-wip-yuri5-testing-2021-02-15-1156-nautilus-distro-basic-gibba/5885477/teuthology.log
Looking at the test, I'm wondering if this may be caused by cb_done being a volatile variable rather than atomic.
Updated by Jeff Layton about 3 years ago
Could be. It wouldn't hurt to rework that with atomics and see whether it's reproducible. I'll do that and put up a PR.
The other possibility is that it just never got a callback for some reason, but let's rule out the volatile/atomic thing first.
Updated by Patrick Donnelly about 3 years ago
- Status changed from New to Pending Backport
Hoping that simple fix is all what's needed...
Updated by Backport Bot about 3 years ago
- Copied to Backport #49473: nautilus: nautilus: qa: "Assertion `cb_done' failed." added
Updated by Backport Bot about 3 years ago
- Copied to Backport #49474: pacific: nautilus: qa: "Assertion `cb_done' failed." added
Updated by Backport Bot about 3 years ago
- Copied to Backport #49475: octopus: nautilus: qa: "Assertion `cb_done' failed." added
Updated by Patrick Donnelly about 3 years ago
- Copied to Bug #49500: qa: "Assertion `cb_done' failed." added
Updated by Patrick Donnelly about 3 years ago
This is still alive: #49500
Cloned this ticket so that the good fix, which didn't help, will still get backported normally.
Updated by Loïc Dachary almost 3 years ago
- Status changed from Pending Backport to Resolved
While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".
Actions