Project

General

Profile

Actions
Copy link

Bug #38059

open

ceph.py: fstrim during shutdown may prevent osd umount

Added by Sage Weil about 5 years ago. Updated almost 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

2019-01-28T00:00:05.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi139.front.sepia.ceph.com
2019-01-28T00:00:05.273 INFO:teuthology.orchestra.run.smithi139:Running:
2019-01-28T00:00:05.273 INFO:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0
2019-01-28T00:00:05.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi139.front.sepia.ceph.com
2019-01-28T00:00:05.997 INFO:teuthology.orchestra.run.smithi139:Running:
2019-01-28T00:00:05.997 INFO:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1
2019-01-28T00:00:06.199 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi139.front.sepia.ceph.com
2019-01-28T00:00:06.200 INFO:teuthology.orchestra.run.smithi139:Running:
2019-01-28T00:00:06.200 INFO:teuthology.orchestra.run.smithi139:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2
2019-01-28T00:00:06.322 INFO:teuthology.orchestra.run.smithi139.stderr:umount: /var/lib/ceph/osd/ceph-2: target is busy.
2019-01-28T00:00:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 32
2019-01-28T00:00:06.323 INFO:teuthology.orchestra.run.smithi139:Running:
2019-01-28T00:00:06.324 INFO:teuthology.orchestra.run.smithi139:> sudo PATH=/usr/sbin:$PATH lsof ; ps auxf
2019-01-28T00:00:06.417 INFO:teuthology.orchestra.run.smithi139.stdout:COMMAND     PID  TID            USER   FD      TYPE             DEVICE SIZE/OFF       NODE NAME
2019-01-28T00:00:06.417 INFO:teuthology.orchestra.run.smithi139.stdout:systemd       1                 root  cwd       DIR                8,1     4096          2 /
2019-01-28T00:00:06.417 INFO:teuthology.orchestra.run.smithi139.stdout:systemd       1                 root  rtd       DIR                8,1     4096          2 /
...
2019-01-28T00:00:06.936 INFO:teuthology.orchestra.run.smithi139.stdout:fstrim    60786                 root    3r      DIR              253,3      164         32 /var/lib/ceph/osd/ceph-2
...
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2019-01-27-1015/qa/tasks/ceph.py", line 1955, in task
    '--no-mon-health-to-clog',
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/contextutil.py", line 46, in nested
    if exit(*exc):
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2019-01-27-1015/qa/tasks/ceph.py", line 1193, in cluster
    raise e
CommandFailedError: Command failed on smithi139 with status 32: 'sync && sudo umount -f /var/lib/ceph/osd/ceph-2'

/a/sage-2019-01-27_21:24:50-rados-wip-sage2-testing-2019-01-27-1015-distro-basic-smithi/3516547
Actions
Copy link
 #1

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link
 #2

Updated by Ilya Dryomov almost 4 years ago

Encountered this with nothing referencing /var/lib/ceph/osd/ceph-2 in lsof output. fstrim could have raced and finished though, so adding a note here:

2020-07-13T00:00:30.427 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com
2020-07-13T00:00:30.427 INFO:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0
2020-07-13T00:00:30.928 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com
2020-07-13T00:00:30.928 INFO:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1
2020-07-13T00:00:31.277 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi037.front.sepia.ceph.com
2020-07-13T00:00:31.277 INFO:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2
2020-07-13T00:00:31.425 INFO:teuthology.orchestra.run.smithi037.stderr:umount: /var/lib/ceph/osd/ceph-2: target is busy.
2020-07-13T00:00:31.427 DEBUG:teuthology.orchestra.run:got remote process result: 32

/a/dis-2020-07-12_22:31:12-krbd:fsx-wip-krbd-58-options-octopus-testing-basic-smithi/5218125

Actions
Copy link

Also available in: Atom PDF