Actions
Bug #50112
closedMDS stuck at stopping when reducing max_mds
% Done:
0%
Source:
Community (user)
Tags:
Backport:
pacific,octopus,nautilus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We are trying to upgrade to v16 today. Cephadm is trying to reduce max_mds to 1 automatically. However, MDS.1 is stuck at stopping state for about 10 minutes and seems making no progress. We can't wait longer because some MDS requests are blocked.
We are running 2 active MDS for weeks, with both kernel and userspace clients.
On the web dashboard, when stuck at stopping state, MDS.1 has 0 dentries, 2 inodes, 1 dirs, 0 caps.
We've tried 3 times. All get the same result.
To get out of this state, we reset max_mds to 2 and restart the stopping MDS.
Here are the logs from MDS.1
2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps 2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets 2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 4.40e-01] 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache shutdown_pass 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a '' 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:28.832+0000 7f17f888f700 5 mds.1.cache lru size now 0/0 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes 2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache looking for subtrees to export 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0 2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.cache still have replicated objects 2021-04-02T10:36:28.832+0000 7f17f888f700 7 mds.1.23151 shutdown_pass=false 2021-04-02T10:36:29.056+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4d0800 client.5274411 v1:222.201.187.248:0/44200992 state closed 2021-04-02T10:36:29.056+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274411 2021-04-02T10:36:29.056+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:29.236+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:29.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000067010s ago 2021-04-02T10:36:29.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:29.528+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:29.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:29.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:29.528+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:29.528+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:29.528+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:29.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:29.664+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4d1200 client.6374310 v1:222.201.187.246:0/1497586911 state closed 2021-04-02T10:36:29.664+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.6374310 2021-04-02T10:36:29.664+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:30.236+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:30.336+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4cf400 client.6514667 v1:222.201.187.170:0/3293323087 state closed 2021-04-02T10:36:30.336+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 2) from client.6514667 2021-04-02T10:36:30.336+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:30.368+0000 7f17f888f700 20 mds.1.23151 get_task_status 2021-04-02T10:36:30.368+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task 2021-04-02T10:36:30.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000068049s ago 2021-04-02T10:36:30.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:30.528+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:30.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:30.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:30.528+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:30.528+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:30.528+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:30.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:30.840+0000 7f17f808e700 5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7654 2021-04-02T10:36:30.840+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s 2021-04-02T10:36:30.844+0000 7f17fd098700 5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7654 rtt 0.00400027 2021-04-02T10:36:31.164+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88956800 client.5274448 v1:222.201.187.233:0/638074764 state closed 2021-04-02T10:36:31.164+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274448 2021-04-02T10:36:31.164+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:31.236+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:31.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000069055s ago 2021-04-02T10:36:31.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:31.528+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:31.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:31.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:31.528+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:31.528+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:31.528+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:31.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:32.236+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:32.364+0000 7f17f888f700 20 mds.1.23151 get_task_status 2021-04-02T10:36:32.364+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task 2021-04-02T10:36:32.524+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 0.996069741s ago 2021-04-02T10:36:32.524+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:32.524+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:32.524+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:32.524+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:32.528+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:32.528+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:32.528+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:32.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:33.236+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:33.352+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600ad00 client.5457150 v1:222.201.187.241:0/2631762979 state closed 2021-04-02T10:36:33.352+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5457150 2021-04-02T10:36:33.352+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:33.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000070970s ago 2021-04-02T10:36:33.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:33.528+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:33.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:33.528+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:33.528+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:33.528+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:33.528+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:33.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 0.995397667s 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.server find_idle_sessions. last cleared laggy state 5.81484e+06s ago 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.locker scatter_tick 2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps 2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets 2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 3.11e-01] 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache shutdown_pass 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a '' 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:33.828+0000 7f17f888f700 5 mds.1.cache lru size now 0/0 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes 2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache looking for subtrees to export 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0 2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.cache still have replicated objects 2021-04-02T10:36:33.828+0000 7f17f888f700 7 mds.1.23151 shutdown_pass=false 2021-04-02T10:36:34.237+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:34.365+0000 7f17f888f700 20 mds.1.23151 get_task_status 2021-04-02T10:36:34.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task 2021-04-02T10:36:34.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 0.996071593s ago 2021-04-02T10:36:34.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:34.525+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:34.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:34.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:34.525+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:34.525+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:34.525+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:34.525+0000 7f17f608a700 10 mds.1.cache releasing free memory 2021-04-02T10:36:34.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:34.797+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4cea00 client.6574441 222.201.187.249:0/4238770464 state closed 2021-04-02T10:36:34.797+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 2) v1 from client.6574441 2021-04-02T10:36:34.797+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:34.837+0000 7f17f808e700 5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7655 2021-04-02T10:36:34.837+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s 2021-04-02T10:36:34.841+0000 7f17fd098700 5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7655 rtt 0.00400029 2021-04-02T10:36:35.237+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:35.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000072765s ago 2021-04-02T10:36:35.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:35.525+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:35.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:35.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:35.525+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:35.525+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:35.525+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:35.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:36.237+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:36.365+0000 7f17f888f700 20 mds.1.23151 get_task_status 2021-04-02T10:36:36.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task 2021-04-02T10:36:36.369+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88955900 client.5274453 v1:222.201.187.231:0/3972282249 state closed 2021-04-02T10:36:36.369+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5274453 2021-04-02T10:36:36.369+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:36.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000073624s ago 2021-04-02T10:36:36.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:36.525+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:36.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:36.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:36.525+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:36.525+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:36.525+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:36.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:36.949+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600a800 client.5218886 v1:116.56.143.171:0/2506259812 state closed 2021-04-02T10:36:36.949+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5218886 2021-04-02T10:36:36.949+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:37.237+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:37.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000074454s ago 2021-04-02T10:36:37.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:37.525+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:37.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:37.525+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:37.529+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:37.529+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:37.529+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:37.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:37.737+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600a300 client.6544148 v1:222.201.187.244:0/4103172439 state closed 2021-04-02T10:36:37.737+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.6544148 2021-04-02T10:36:37.737+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:38.237+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:38.365+0000 7f17f888f700 20 mds.1.23151 get_task_status 2021-04-02T10:36:38.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task 2021-04-02T10:36:38.529+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000075263s ago 2021-04-02T10:36:38.529+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:38.529+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:38.529+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:38.529+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:38.529+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:38.529+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:38.529+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:38.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:38.737+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88955400 client.5274416 v1:222.201.187.242:0/3687007628 state closed 2021-04-02T10:36:38.737+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274416 2021-04-02T10:36:38.737+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.server find_idle_sessions. last cleared laggy state 5.81484e+06s ago 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.locker scatter_tick 2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps 2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets 2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 2.20e-01] 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache shutdown_pass 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a '' 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:38.829+0000 7f17f888f700 5 mds.1.cache lru size now 0/0 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes 2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache looking for subtrees to export 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0 2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.cache still have replicated objects 2021-04-02T10:36:38.829+0000 7f17f888f700 7 mds.1.23151 shutdown_pass=false 2021-04-02T10:36:38.837+0000 7f17f808e700 5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7656 2021-04-02T10:36:38.837+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s 2021-04-02T10:36:38.841+0000 7f17fd098700 5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7656 rtt 0.0040003 2021-04-02T10:36:39.229+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c47ed00 client.5274447 v1:222.201.187.247:0/816986335 state closed 2021-04-02T10:36:39.229+0000 7f17fa893700 3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274447 2021-04-02T10:36:39.229+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed) 2021-04-02T10:36:39.253+0000 7f17fa893700 7 mds.1.cache cache_expire from mds.0 2021-04-02T10:36:39.529+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000076044s ago 2021-04-02T10:36:39.529+0000 7f17f608a700 10 mds.1.cache trim_client_leases 2021-04-02T10:36:39.529+0000 7f17f608a700 7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0 2021-04-02T10:36:39.529+0000 7f17f608a700 7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0 2021-04-02T10:36:39.529+0000 7f17f608a700 7 mds.1.cache trim_lru trimmed 0 items 2021-04-02T10:36:39.529+0000 7f17f608a700 2 mds.1.cache Memory usage: total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode 2021-04-02T10:36:39.529+0000 7f17f608a700 7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa 2021-04-02T10:36:39.529+0000 7f17f608a700 7 mds.1.server recalled 0 client caps. 2021-04-02T10:36:39.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s 2021-04-02T10:36:44.529+0000 7f17fa893700 1 mds.cephfs.gpu006.ddpekw Updating MDS map to version 23242 from mon.1
Actions