Project

General

Profile

Actions
Copy link

Bug #62485

open

quincy (?): pybind/mgr/volumes: subvolume rm timeout

Added by Patrick Donnelly 9 months ago. Updated 9 months ago.

Status:
New
Priority:
High
Assignee:
Venky Shankar
Category:
-
Target version:
Ceph - v19.0.0
% Done:

0%

Source:
Q/A
Tags:
Backport:
reef,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
mgr/volumes
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2023-08-09T05:20:40.464 INFO:teuthology.orchestra.run.smithi105.stderr:2023-08-09T05:20:40.459+0000 7f5f51e6b700  1 -- 172.21.15.105:0/2270184776 --> [v2:172.21.15.105:6832/32722,v1:172.21.15.105:6833/32722] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "subvolume_0000000000681210", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f4c069bc0 con 0x7f5f38067c80
2023-08-09T05:21:05.977 DEBUG:teuthology.orchestra.run.smithi105:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:21:05.984 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:21:36.141 DEBUG:teuthology.orchestra.run.smithi105:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:21:36.148 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:22:06.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:22:06.230 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:22:36.336 DEBUG:teuthology.orchestra.run.smithi105:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:22:36.345 DEBUG:teuthology.orchestra.run.smithi154:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2023-08-09T05:22:40.137 DEBUG:teuthology.orchestra.run:got remote process result: 124

From: /teuthology/yuriw-2023-08-09_01:12:27-fs-wip-yuri5-testing-2023-08-08-0807-quincy-distro-default-smithi/7363541/teuthology.log

2023-08-09T05:20:40.491+0000 7f0e05951700  0 [volumes INFO volumes.fs.operations.versions.subvolume_base] subvolume path 'b'/volumes/_nogroup/subvolume_0000000000681210'' moved to trashcan
...
9.076813+0000 mtime=2023-08-09T05:20:39.125042+0000 ctime=2023-08-09T05:20:40.495262+0000 change_attr=4 caps=pAsLsXs(0=pAsLsXs) parents=0x1.head["volumes"] 0x564c5df5be00) n = 1             
2023-08-09T05:20:40.495+0000 7f0e05951700  0 [volumes INFO volumes.fs.async_job] queuing job for volume 'cephfs'                                                                              
2023-08-09T05:20:40.495+0000 7f0e05951700  0 [volumes DEBUG mgr_util] locking <locked _thread.lock object at 0x7f0e2841f600> with 5 timeout

From: /teuthology/yuriw-2023-08-09_01:12:27-fs-wip-yuri5-testing-2023-08-08-0807-quincy-distro-default-smithi/7363541/remote/smithi105/log/ceph-mgr.y.log.gz

Actions
Copy link
 #1

Updated by Patrick Donnelly 9 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Venky Shankar 9 months ago

  • Assignee set to Venky Shankar

2023-08-09T05:20:40.495+0000 7f0e05951700 0 [volumes DEBUG mgr_util] locking <locked _thread.lock object at 0x7f0e2841f600> with 5 timeout

That's probably locked by another async thread that is blocked on the libcephfs call (never to return due to the bug we spoke about in standup - client not handling fsmap updates with MDSs gone/reassigned). I'll have a look..

Actions
Copy link

Also available in: Atom PDF