Project

General

Profile

Bug #19556

upgrade/hammer-x failing with "OSD has the store locked" when Thrasher runs ceph-objectstore-tool on down PG

Added by Nathan Cutler 8 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
04/09/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
jewel,kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

Test:

  description: upgrade:hammer-x/stress-split-erasure-code-x86_64/{0-cluster/{openstack.yaml
    start.yaml} 0-tz-eastern.yaml 0-x86_64.yaml 1-hammer-install/hammer.yaml 2-partial-upgrade/firsthalf.yaml
    3-thrash/default.yaml 4-mon/mona.yaml 5-workload/ec-rados-default.yaml 6-next-mon/monb.yaml
    8-finish-upgrade/last-osds-and-monc.yaml 9-workload/ec-rados-plugin=isa-k=2-m=1.yaml}

Test fails with:

2017-04-09T13:38:11.936 INFO:teuthology.orchestra.run.vpm163:Running: 'type ceph-objectstore-tool'
2017-04-09T13:38:11.942 INFO:teuthology.orchestra.run.vpm163.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool
2017-04-09T13:38:11.942 INFO:teuthology.orchestra.run.vpm163:Running: 'sudo adjust-ulimits ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'
2017-04-09T13:38:12.033 INFO:teuthology.orchestra.run.vpm163.stderr:OSD has the store locked
...
2017-04-09T13:38:17.038 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 680, in wrapper
    return func(self)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 734, in do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-jewel-backports/qa/tasks/ceph_manager.py", line 231, in kill_osd
    format(ret=proc.exitstatus))
Exception: ceph-objectstore-tool: exp list-pgs failure with status 1

Related issues

Copied to ceph-qa-suite - Backport #19557: jewel: upgrade/hammer-x failing with "OSD has the store locked" when Thrasher runs ceph-objectstore-tool on down PG Resolved
Copied to ceph-qa-suite - Backport #19558: kraken: upgrade/hammer-x failing with "OSD has the store locked" when Thrasher runs ceph-objectstore-tool on down PG Rejected

History

#1 Updated by Nathan Cutler 8 months ago

  • Backport set to hammer,jewel,kraken

#2 Updated by Nathan Cutler 8 months ago

  • Backport changed from hammer,jewel,kraken to jewel,kraken

#3 Updated by Nathan Cutler 8 months ago

  • Status changed from New to Need Review

#4 Updated by Nathan Cutler 8 months ago

  • Status changed from Need Review to Pending Backport

#5 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #19557: jewel: upgrade/hammer-x failing with "OSD has the store locked" when Thrasher runs ceph-objectstore-tool on down PG added

#6 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #19558: kraken: upgrade/hammer-x failing with "OSD has the store locked" when Thrasher runs ceph-objectstore-tool on down PG added

#7 Updated by Nathan Cutler 8 months ago

  • Status changed from Pending Backport to Need Review

#8 Updated by Nathan Cutler 7 months ago

  • Status changed from Need Review to Pending Backport

#9 Updated by Nathan Cutler 3 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF