Ceph » CephFS

We are trying to upgrade to v16 today. Cephadm is trying to reduce max_mds to 1 automatically. However, MDS.1 is stuck at stopping state for about 10 minutes and seems making no progress. We can't wait longer because some MDS requests are blocked.

We are running 2 active MDS for weeks, with both kernel and userspace clients.

On the web dashboard, when stuck at stopping state, MDS.1 has 0 dentries, 2 inodes, 1 dirs, 0 caps.

We've tried 3 times. All get the same result.

To get out of this state, we reset max_mds to 2 and restart the stopping MDS.

Here are the logs from MDS.1

2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps
2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets
2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 4.40e-01]
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache shutdown_pass
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a ''
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:28.832+0000 7f17f888f700  5 mds.1.cache lru size now 0/0
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes
2021-04-02T10:36:28.832+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache looking for subtrees to export
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.cache   examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0
2021-04-02T10:36:28.832+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.cache still have replicated objects
2021-04-02T10:36:28.832+0000 7f17f888f700  7 mds.1.23151 shutdown_pass=false
2021-04-02T10:36:29.056+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4d0800 client.5274411 v1:222.201.187.248:0/44200992 state closed
2021-04-02T10:36:29.056+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274411
2021-04-02T10:36:29.056+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:29.236+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:29.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000067010s ago
2021-04-02T10:36:29.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:29.528+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:29.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:29.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:29.528+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:29.528+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:29.528+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:29.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:29.664+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4d1200 client.6374310 v1:222.201.187.246:0/1497586911 state closed
2021-04-02T10:36:29.664+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.6374310
2021-04-02T10:36:29.664+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:30.236+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:30.336+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4cf400 client.6514667 v1:222.201.187.170:0/3293323087 state closed
2021-04-02T10:36:30.336+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 2) from client.6514667
2021-04-02T10:36:30.336+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:30.368+0000 7f17f888f700 20 mds.1.23151 get_task_status
2021-04-02T10:36:30.368+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task
2021-04-02T10:36:30.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000068049s ago
2021-04-02T10:36:30.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:30.528+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:30.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:30.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:30.528+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:30.528+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:30.528+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:30.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:30.840+0000 7f17f808e700  5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7654
2021-04-02T10:36:30.840+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s
2021-04-02T10:36:30.844+0000 7f17fd098700  5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7654 rtt 0.00400027
2021-04-02T10:36:31.164+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88956800 client.5274448 v1:222.201.187.233:0/638074764 state closed
2021-04-02T10:36:31.164+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274448
2021-04-02T10:36:31.164+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:31.236+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:31.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000069055s ago
2021-04-02T10:36:31.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:31.528+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:31.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:31.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:31.528+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:31.528+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:31.528+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:31.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:32.236+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:32.364+0000 7f17f888f700 20 mds.1.23151 get_task_status
2021-04-02T10:36:32.364+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task
2021-04-02T10:36:32.524+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 0.996069741s ago
2021-04-02T10:36:32.524+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:32.524+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:32.524+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:32.524+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:32.528+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:32.528+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:32.528+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:32.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:33.236+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:33.352+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600ad00 client.5457150 v1:222.201.187.241:0/2631762979 state closed
2021-04-02T10:36:33.352+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5457150
2021-04-02T10:36:33.352+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:33.528+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000070970s ago
2021-04-02T10:36:33.528+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:33.528+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:33.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:33.528+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:33.528+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:33.528+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:33.528+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:33.528+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 0.995397667s
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.server find_idle_sessions. last cleared laggy state 5.81484e+06s ago
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.locker scatter_tick
2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps
2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets
2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 3.11e-01]
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache shutdown_pass
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a ''
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:33.828+0000 7f17f888f700  5 mds.1.cache lru size now 0/0
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes
2021-04-02T10:36:33.828+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache looking for subtrees to export
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.cache   examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0
2021-04-02T10:36:33.828+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.cache still have replicated objects
2021-04-02T10:36:33.828+0000 7f17f888f700  7 mds.1.23151 shutdown_pass=false
2021-04-02T10:36:34.237+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:34.365+0000 7f17f888f700 20 mds.1.23151 get_task_status
2021-04-02T10:36:34.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task
2021-04-02T10:36:34.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 0.996071593s ago
2021-04-02T10:36:34.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:34.525+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:34.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:34.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:34.525+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289596, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:34.525+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:34.525+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:34.525+0000 7f17f608a700 10 mds.1.cache releasing free memory
2021-04-02T10:36:34.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:34.797+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c4cea00 client.6574441 222.201.187.249:0/4238770464 state closed
2021-04-02T10:36:34.797+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 2) v1 from client.6574441
2021-04-02T10:36:34.797+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:34.837+0000 7f17f808e700  5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7655
2021-04-02T10:36:34.837+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s
2021-04-02T10:36:34.841+0000 7f17fd098700  5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7655 rtt 0.00400029
2021-04-02T10:36:35.237+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:35.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000072765s ago
2021-04-02T10:36:35.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:35.525+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:35.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:35.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:35.525+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:35.525+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:35.525+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:35.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:36.237+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:36.365+0000 7f17f888f700 20 mds.1.23151 get_task_status
2021-04-02T10:36:36.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task
2021-04-02T10:36:36.369+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88955900 client.5274453 v1:222.201.187.231:0/3972282249 state closed
2021-04-02T10:36:36.369+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5274453
2021-04-02T10:36:36.369+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:36.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000073624s ago
2021-04-02T10:36:36.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:36.525+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:36.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:36.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:36.525+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:36.525+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:36.525+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:36.525+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:36.949+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600a800 client.5218886 v1:116.56.143.171:0/2506259812 state closed
2021-04-02T10:36:36.949+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.5218886
2021-04-02T10:36:36.949+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:37.237+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:37.525+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000074454s ago
2021-04-02T10:36:37.525+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:37.525+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:37.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:37.525+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:37.529+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:37.529+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:37.529+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:37.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:37.737+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ada600a300 client.6544148 v1:222.201.187.244:0/4103172439 state closed
2021-04-02T10:36:37.737+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 12) from client.6544148
2021-04-02T10:36:37.737+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:38.237+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:38.365+0000 7f17f888f700 20 mds.1.23151 get_task_status
2021-04-02T10:36:38.365+0000 7f17f888f700 20 mds.1.23151 schedule_update_timer_task
2021-04-02T10:36:38.529+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000075263s ago
2021-04-02T10:36:38.529+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:38.529+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:38.529+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:38.529+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:38.529+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:38.529+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:38.529+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:38.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:38.737+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad88955400 client.5274416 v1:222.201.187.242:0/3687007628 state closed
2021-04-02T10:36:38.737+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274416
2021-04-02T10:36:38.737+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.server find_idle_sessions. last cleared laggy state 5.81484e+06s ago
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.locker scatter_tick
2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.locker caps_tick 0 revoking caps
2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.23151 updating export targets, currently 1 ranks are targets
2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.23151 export target mds.0 is [C 2.20e-01]
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim 1 / 128 segments, 1 / -1 events, 0 (0) expiring, 0 (0) expired
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache shutdown_pass
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache shutdown_export_strays 0x60a ''
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=18446744073709551615
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache trim_lru trimming 18446744073709551615 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:38.829+0000 7f17f888f700  5 mds.1.cache lru size now 0/0
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache Migrating any ephemerally pinned inodes
2021-04-02T10:36:38.829+0000 7f17f888f700 20 mds.1.bal handle_export_pins export_pin_queue size=0
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache looking for subtrees to export
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.cache   examining [dir 0x101 ~mds1/ [2,head] auth v=70451715 cv=70451715/70451715 dir_auth=1 state=1073741824 f(v0 10=0+10) n(v322133 rc2021-04-02T10:26:20.324976+0000 b50 45=35+10) hs=0+0,ss=0+0 | child=0 subtree=1 subtreetemp=0 replicated=0 dirty=0 waiter=0 authpin=0 0x55ad7c233600] bounds
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log trim_all: 1/0/0
2021-04-02T10:36:38.829+0000 7f17f888f700 10 mds.1.log _trim_expired_segments waiting for 181543359/644740121223 to expire
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.cache still have replicated objects
2021-04-02T10:36:38.829+0000 7f17f888f700  7 mds.1.23151 shutdown_pass=false
2021-04-02T10:36:38.837+0000 7f17f808e700  5 mds.beacon.cephfs.gpu006.ddpekw Sending beacon up:stopping seq 7656
2021-04-02T10:36:38.837+0000 7f17f808e700 20 mds.beacon.cephfs.gpu006.ddpekw sender thread waiting interval 4s
2021-04-02T10:36:38.841+0000 7f17fd098700  5 mds.beacon.cephfs.gpu006.ddpekw received beacon reply up:stopping seq 7656 rtt 0.0040003
2021-04-02T10:36:39.229+0000 7f17fa893700 20 mds.1.23151 get_session have 0x55ad7c47ed00 client.5274447 v1:222.201.187.247:0/816986335 state closed
2021-04-02T10:36:39.229+0000 7f17fa893700  3 mds.1.server handle_client_session client_session(request_renewcaps seq 11) from client.5274447
2021-04-02T10:36:39.229+0000 7f17fa893700 10 mds.1.server ignoring renewcaps on non open|stale session (closed)
2021-04-02T10:36:39.253+0000 7f17fa893700  7 mds.1.cache cache_expire from mds.0
2021-04-02T10:36:39.529+0000 7f17f608a700 20 mds.1.cache upkeep thread trimming cache; last trim 1.000076044s ago
2021-04-02T10:36:39.529+0000 7f17f608a700 10 mds.1.cache trim_client_leases
2021-04-02T10:36:39.529+0000 7f17f608a700  7 mds.1.cache trim bytes_used=5kB limit=16GB reservation=0.05% count=0
2021-04-02T10:36:39.529+0000 7f17f608a700  7 mds.1.cache trim_lru trimming 0 items from LRU size=0 mid=0 pintail=0 pinned=0
2021-04-02T10:36:39.529+0000 7f17f608a700  7 mds.1.cache trim_lru trimmed 0 items
2021-04-02T10:36:39.529+0000 7f17f608a700  2 mds.1.cache Memory usage:  total 7697216, rss 289588, heap 332004, baseline 332004, 0 / 2 inodes have caps, 0 caps, 0 caps per inode
2021-04-02T10:36:39.529+0000 7f17f608a700  7 mds.1.server recall_client_state: min=100 max=8388608 total=0 flags=0xa
2021-04-02T10:36:39.529+0000 7f17f608a700  7 mds.1.server recalled 0 client caps.
2021-04-02T10:36:39.529+0000 7f17f608a700 20 mds.1.cache upkeep thread waiting interval 1s
2021-04-02T10:36:44.529+0000 7f17fa893700  1 mds.cephfs.gpu006.ddpekw Updating MDS map to version 23242 from mon.1