Actions
Bug #58756
closedqa: error during scrub thrashing
Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
quincy
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
fs
Component(FS):
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
error during scrub thrashing in [1]
2023-02-16T20:18:55.286 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub
2023-02-16T20:18:55.319 INFO:tasks.fwd_scrub:joining ForwardScrubbers
2023-02-16T20:18:55.320 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_fbbadb5ff5cfccce0d20e136f8956e65ec955359/teuthology/run_tasks.py", line 188, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
next(self.gen)
File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/fwd_scrub.py", line 151, in task
stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers)
File "/home/teuthworker/src/github.com_ceph_ceph-c_fd249f18f047d5b86a29796a05bfeba7a7666d84/qa/tasks/fwd_scrub.py", line 86, in stop_all_fwd_scrubbers
raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}")
RuntimeError: error during scrub thrashing: Command failed on smithi073 with status 13: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive'
Updated by Xiubo Li about 1 year ago
This should be a known issue with https://tracker.ceph.com/issues/58564:
2023-02-16T19:43:36.006343+00:00 smithi073 kernel:
2023-02-16T19:43:36.006470+00:00 smithi073 kernel: ======================================================
2023-02-16T19:43:36.006499+00:00 smithi073 kernel: WARNING: possible circular locking dependency detected
2023-02-16T19:43:36.006523+00:00 smithi073 kernel: 6.2.0-rc7-ceph-gd9ba97321a89 #1 Tainted: G S
2023-02-16T19:43:36.006546+00:00 smithi073 kernel: ------------------------------------------------------
2023-02-16T19:43:36.006569+00:00 smithi073 kernel: runc/73796 is trying to acquire lock:
2023-02-16T19:43:36.006592+00:00 smithi073 kernel: ffffffff82463f30 (cpu_hotplug_lock){++++}-{0:0}, at: static_key_slow_inc+0x12/0x30
2023-02-16T19:43:36.006625+00:00 smithi073 kernel: #012but task is already holding lock:
2023-02-16T19:43:36.006651+00:00 smithi073 kernel: ffffffff8256abe8 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0x91/0x560
2023-02-16T19:43:36.006674+00:00 smithi073 kernel: #012which lock already depends on the new lock.
2023-02-16T19:43:36.006698+00:00 smithi073 kernel: #012the existing dependency chain (in reverse order) is:
2023-02-16T19:43:36.006721+00:00 smithi073 kernel: #012-> #2 (freezer_mutex){+.+.}-{3:3}:
2023-02-16T19:43:36.006744+00:00 smithi073 kernel: __mutex_lock+0x9c/0xf30
2023-02-16T19:43:36.006766+00:00 smithi073 kernel: freezer_attach+0x30/0xf0
2023-02-16T19:43:36.006790+00:00 smithi073 kernel: cgroup_migrate_execute+0x3f3/0x4c0
2023-02-16T19:43:36.006824+00:00 smithi073 kernel: cgroup_attach_task+0x23a/0x3f0
2023-02-16T19:43:36.006852+00:00 smithi073 kernel: __cgroup1_procs_write.constprop.12+0xfb/0x140
2023-02-16T19:43:36.006876+00:00 smithi073 kernel: cgroup_file_write+0x95/0x230
2023-02-16T19:43:36.006900+00:00 smithi073 kernel: kernfs_fop_write_iter+0x13b/0x1d0
2023-02-16T19:43:36.006922+00:00 smithi073 kernel: vfs_write+0x348/0x4d0
2023-02-16T19:43:36.006945+00:00 smithi073 kernel: ksys_write+0x60/0xe0
2023-02-16T19:43:36.006967+00:00 smithi073 kernel: do_syscall_64+0x38/0x80
2023-02-16T19:43:36.006991+00:00 smithi073 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
2023-02-16T19:43:36.007016+00:00 smithi073 kernel: #012-> #1 (cgroup_threadgroup_rwsem){++++}-{0:0}:
2023-02-16T19:43:36.007040+00:00 smithi073 kernel: percpu_down_write+0x49/0x2d0
2023-02-16T19:43:36.007079+00:00 smithi073 kernel: cgroup_procs_write_start+0x88/0x270
2023-02-16T19:43:36.007105+00:00 smithi073 kernel: __cgroup1_procs_write.constprop.12+0x57/0x140
2023-02-16T19:43:36.007128+00:00 smithi073 kernel: cgroup_file_write+0x95/0x230
2023-02-16T19:43:36.007149+00:00 smithi073 kernel: kernfs_fop_write_iter+0x13b/0x1d0
2023-02-16T19:43:36.007171+00:00 smithi073 kernel: vfs_write+0x348/0x4d0
2023-02-16T19:43:36.007193+00:00 smithi073 kernel: ksys_write+0x60/0xe0
2023-02-16T19:43:36.007215+00:00 smithi073 kernel: do_syscall_64+0x38/0x80
2023-02-16T19:43:36.007237+00:00 smithi073 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
2023-02-16T19:43:36.007258+00:00 smithi073 kernel: #012-> #0 (cpu_hotplug_lock){++++}-{0:0}:
2023-02-16T19:43:36.007280+00:00 smithi073 kernel: __lock_acquire+0x108b/0x1e40
2023-02-16T19:43:36.007302+00:00 smithi073 kernel: lock_acquire+0xd8/0x300
2023-02-16T19:43:36.007326+00:00 smithi073 kernel: cpus_read_lock+0x40/0xd0
2023-02-16T19:43:36.007348+00:00 smithi073 kernel: static_key_slow_inc+0x12/0x30
2023-02-16T19:43:36.007370+00:00 smithi073 kernel: freezer_apply_state+0x98/0xb0
2023-02-16T19:43:36.007393+00:00 smithi073 kernel: freezer_write+0x327/0x560
2023-02-16T19:43:36.007415+00:00 smithi073 kernel: cgroup_file_write+0x95/0x230
2023-02-16T19:43:36.007437+00:00 smithi073 kernel: kernfs_fop_write_iter+0x13b/0x1d0
2023-02-16T19:43:36.007458+00:00 smithi073 kernel: vfs_write+0x348/0x4d0
2023-02-16T19:43:36.007480+00:00 smithi073 kernel: ksys_write+0x60/0xe0
2023-02-16T19:43:36.007501+00:00 smithi073 kernel: do_syscall_64+0x38/0x80
2023-02-16T19:43:36.007522+00:00 smithi073 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
2023-02-16T19:43:36.007544+00:00 smithi073 kernel: #012other info that might help us debug this:
2023-02-16T19:43:36.007565+00:00 smithi073 kernel: Chain exists of:#012 cpu_hotplug_lock --> cgroup_threadgroup_rwsem --> freezer_mutex
2023-02-16T19:43:36.007590+00:00 smithi073 kernel: Possible unsafe locking scenario:
2023-02-16T19:43:36.007613+00:00 smithi073 kernel: CPU0 CPU1
2023-02-16T19:43:36.007636+00:00 smithi073 kernel: ---- ----
2023-02-16T19:43:36.007657+00:00 smithi073 kernel: lock(freezer_mutex);
2023-02-16T19:43:36.007679+00:00 smithi073 kernel: lock(cgroup_threadgroup_rwsem);
2023-02-16T19:43:36.007700+00:00 smithi073 kernel: lock(freezer_mutex);
2023-02-16T19:43:36.007723+00:00 smithi073 kernel: lock(cpu_hotplug_lock);
2023-02-16T19:43:36.007746+00:00 smithi073 kernel: #012 *** DEADLOCK ***
2023-02-16T19:43:36.007768+00:00 smithi073 kernel: 5 locks held by runc/73796:
2023-02-16T19:43:36.007792+00:00 smithi073 kernel: #0: ffff88810e8104e8 (&f->f_pos_lock){+.+.}-{3:3}, at: __fdget_pos+0x48/0x60
2023-02-16T19:43:36.007842+00:00 smithi073 kernel: #1: ffff88810ed6e448 (sb_writers#6){.+.+}-{0:0}, at: ksys_write+0x60/0xe0
2023-02-16T19:43:36.007866+00:00 smithi073 kernel: #2: ffff88816f8fdc88 (&of->mutex){+.+.}-{3:3}, at: kernfs_fop_write_iter+0x108/0x1d0
2023-02-16T19:43:36.007889+00:00 smithi073 kernel: #3: ffff88810aa27c80 (kn->active#169){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x111/0x1d0
2023-02-16T19:43:36.007912+00:00 smithi073 kernel: #4: ffffffff8256abe8 (freezer_mutex){+.+.}-{3:3}, at: freezer_write+0x91/0x560
2023-02-16T19:43:36.007935+00:00 smithi073 kernel: #012stack backtrace:
2023-02-16T19:43:36.007958+00:00 smithi073 kernel: CPU: 1 PID: 73796 Comm: runc Tainted: G S 6.2.0-rc7-ceph-gd9ba97321a89 #1
2023-02-16T19:43:36.007981+00:00 smithi073 kernel: Hardware name: Supermicro SYS-5018R-WR/X10SRW-F, BIOS 2.0 12/17/2015
2023-02-16T19:43:36.008004+00:00 smithi073 kernel: Call Trace:
2023-02-16T19:43:36.008029+00:00 smithi073 kernel: <TASK>
2023-02-16T19:43:36.008053+00:00 smithi073 kernel: dump_stack_lvl+0x59/0x71
2023-02-16T19:43:36.008076+00:00 smithi073 kernel: check_noncircular+0xfe/0x110
2023-02-16T19:43:36.008099+00:00 smithi073 kernel: ? __lock_acquire+0xf12/0x1e40
2023-02-16T19:43:36.008138+00:00 smithi073 kernel: __lock_acquire+0x108b/0x1e40
2023-02-16T19:43:36.008162+00:00 smithi073 kernel: lock_acquire+0xd8/0x300
2023-02-16T19:43:36.008186+00:00 smithi073 kernel: ? static_key_slow_inc+0x12/0x30
2023-02-16T19:43:36.008210+00:00 smithi073 kernel: ? freezer_write+0x1d6/0x560
2023-02-16T19:43:36.008231+00:00 smithi073 kernel: cpus_read_lock+0x40/0xd0
2023-02-16T19:43:36.008253+00:00 smithi073 kernel: ? static_key_slow_inc+0x12/0x30
2023-02-16T19:43:36.008274+00:00 smithi073 kernel: static_key_slow_inc+0x12/0x30
2023-02-16T19:43:36.008296+00:00 smithi073 kernel: freezer_apply_state+0x98/0xb0
2023-02-16T19:43:36.008317+00:00 smithi073 kernel: freezer_write+0x327/0x560
2023-02-16T19:43:36.008339+00:00 smithi073 kernel: cgroup_file_write+0x95/0x230
2023-02-16T19:43:36.008360+00:00 smithi073 kernel: kernfs_fop_write_iter+0x13b/0x1d0
2023-02-16T19:43:36.008382+00:00 smithi073 kernel: vfs_write+0x348/0x4d0
2023-02-16T19:43:36.008406+00:00 smithi073 kernel: ksys_write+0x60/0xe0
2023-02-16T19:43:36.008428+00:00 smithi073 kernel: do_syscall_64+0x38/0x80
2023-02-16T19:43:36.008450+00:00 smithi073 kernel: entry_SYSCALL_64_after_hwframe+0x63/0xcd
2023-02-16T19:43:36.008472+00:00 smithi073 kernel: RIP: 0033:0x55d73f7179db
2023-02-16T19:43:36.008494+00:00 smithi073 kernel: Code: fa ff eb bf e8 e6 b4 fa ff e9 61 ff ff ff cc e8 db 83 fa ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
2023-02-16T19:43:36.008518+00:00 smithi073 kernel: RSP: 002b:000000c0001b82e0 EFLAGS: 00000206 ORIG_RAX: 0000000000000001
2023-02-16T19:43:36.008540+00:00 smithi073 kernel: RAX: ffffffffffffffda RBX: 000000c00002e800 RCX: 000055d73f7179db
2023-02-16T19:43:36.008563+00:00 smithi073 kernel: RDX: 0000000000000006 RSI: 000000c0001b8518 RDI: 000000000000000d
2023-02-16T19:43:36.008585+00:00 smithi073 kernel: RBP: 000000c0001b8330 R08: 000000c0001b8301 R09: 0000000000000004
2023-02-16T19:43:36.008609+00:00 smithi073 kernel: R10: 00007fa09c042fd8 R11: 0000000000000206 R12: 00000000000000f2
2023-02-16T19:43:36.008631+00:00 smithi073 kernel: R13: 0000000000000000 R14: 000055d73fb9b35e R15: 0000000000000000
2023-02-16T19:43:36.008654+00:00 smithi073 kernel: </TASK>
Actions