Project

General

Profile

Bug #11499

ceph-fuse: don't try and remount during shutdown

Added by Greg Farnum over 4 years ago. Updated about 3 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Category:
Correctness/Safety
Target version:
-
Start date:
04/29/2015
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Client, ceph-fuse
Labels (FS):
Pull request ID:

Description

http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-04-26_16:04:01-fs-hammer-testing-basic-typica/5768/

2015-04-27T15:57:19.079 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@typica151.front.sepia.ceph.com...
2015-04-27T15:57:19.079 INFO:teuthology.orchestra.run.typica151:Running: 'sudo fusermount -u /home/ubuntu/cephtest/mnt.0'
2015-04-27T15:57:19.169 INFO:teuthology.orchestra.run.typica151.stderr:2015-04-27 15:57:19.166129 7f0adf49d700  1 -- 172.20.133.151:0/1019868 --> 172.20.133.164:6789/0 -- mon_command({"prefix": "mds dump", "format": "json"} v 0) v1 -- ?+0 0x7f0ad002a9c0 con 0x7f0ad0026aa0
2015-04-27T15:57:19.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:mount: /home/ubuntu/cephtest/mnt.0 not mounted or bad option
2015-04-27T15:57:19.170 INFO:teuthology.orchestra.run.typica151.stderr:2015-04-27 15:57:19.166911 7f0acffff700  1 -- 172.20.133.151:0/1019868 <== mon.1 172.20.133.164:6789/0 12 ==== mon_command_ack([{"prefix": "mds dump", "format": "json"}]=0 dumped mdsmap epoch 56 v56) v1 ==== 96+0+905 (3493472965 0 201683057) 0x7f0ac40009b0 con 0x7f0ad0026aa0
2015-04-27T15:57:19.170 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:2015-04-27 15:57:19.167342 7fd053fff700 -1 client.4136 tried to remount (to trim kernel dentries) and got error 32
2015-04-27T15:57:19.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:client/Client.cc: In function 'virtual void C_Client_Remount::finish(int)' thread 7fd053fff700 time 2015-04-27 15:57:19.167399
2015-04-27T15:57:19.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:client/Client.cc: 3457: FAILED assert(0 == "failed to remount for kernel dentry trimming")
2015-04-27T15:57:19.173 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: ceph version 0.94.1-8-g74c2dc1 (74c2dc1f3924fa05e2c40f4cceb2ab060493bdfb)
2015-04-27T15:57:19.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x70e81b]
2015-04-27T15:57:19.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 2: (C_Client_Remount::finish(int)+0xb5) [0x5ac2f5]
2015-04-27T15:57:19.174 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 3: (Context::complete(int)+0x9) [0x5aad59]
2015-04-27T15:57:19.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 4: (Finisher::finisher_thread_entry()+0x158) [0x633428]
2015-04-27T15:57:19.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 5: (()+0x8182) [0x7fd07a7d0182]
2015-04-27T15:57:19.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 6: (clone()+0x6d) [0x7fd07913a47d]
2015-04-27T15:57:19.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-04-27T15:57:19.176 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:2015-04-27 15:57:19.168551 7fd053fff700 -1 client/Client.cc: In function 'virtual void C_Client_Remount::finish(int)' thread 7fd053fff700 time 2015-04-27 15:57:19.167399
2015-04-27T15:57:19.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:client/Client.cc: 3457: FAILED assert(0 == "failed to remount for kernel dentry trimming")
2015-04-27T15:57:19.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:
2015-04-27T15:57:19.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: ceph version 0.94.1-8-g74c2dc1 (74c2dc1f3924fa05e2c40f4cceb2ab060493bdfb)
2015-04-27T15:57:19.177 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x8b) [0x70e81b]
2015-04-27T15:57:19.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 2: (C_Client_Remount::finish(int)+0xb5) [0x5ac2f5]
2015-04-27T15:57:19.178 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 3: (Context::complete(int)+0x9) [0x5aad59]
2015-04-27T15:57:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 4: (Finisher::finisher_thread_entry()+0x158) [0x633428]
2015-04-27T15:57:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 5: (()+0x8182) [0x7fd07a7d0182]
2015-04-27T15:57:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: 6: (clone()+0x6d) [0x7fd07913a47d]
2015-04-27T15:57:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2015-04-27T15:57:19.179 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.typica151.stderr:

I think maybe this we can fix this crash by checking whether we're unmounting during the remount callback? If we are then obviously we don't want to try and remount.

History

#1 Updated by Zheng Yan over 4 years ago

the test is already there

      if (client->require_remount && !client->unmounting) {
        assert(0 == "failed to remount for kernel dentry trimming");
      }

#2 Updated by Greg Farnum over 4 years ago

I thought that solution seemed familiar. I guess there's still a race between when the unmount reaches the Client versus when the OS knows about it...blech. :/

#3 Updated by Greg Farnum over 4 years ago

  • Assignee deleted (Greg Farnum)

#4 Updated by Greg Farnum about 3 years ago

  • Category changed from 45 to Correctness/Safety
  • Status changed from New to Can't reproduce
  • Component(FS) Client, ceph-fuse added

We haven't seen this again.

Also available in: Atom PDF