Project

General

Profile

Bug #18799

AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'

Added by Sage Weil 9 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
02/02/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
hammer, jewel, kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Release:
Needs Doc:
No

Description

2017-02-02T17:01:40.466 INFO:tasks.ceph.osd.1:Stopped
2017-02-02T17:01:40.479 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-backoff/qa/tasks/ceph_manager.py", line 714, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-backoff/qa/tasks/ceph_manager.py", line 831, in do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-backoff/qa/tasks/ceph_manager.py", line 201, in kill_osd
    if self.ceph_objectstore_tool:
AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'

2017-02-02T17:01:40.496 CRITICAL:root:  File "gevent.libev.corecext.pyx", line 367, in gevent.libev.corecext.loop.handle_error (src/gevent/libev/gevent.corecext.c:5051)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 558, in handle_error
    self.print_exception(context, type, value, tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 605, in print_exception
    traceback.print_exception(type, value, tb, file=errstream)
  File "/usr/lib/python2.7/traceback.py", line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File "/usr/lib/python2.7/traceback.py", line 13, in _print
    file.write(str+terminator)

2017-02-02T17:01:40.502 CRITICAL:root:IOError

/a/sage-2017-02-02_14:58:21-rados:thrash-wip-backoff---basic-smithi/778202

Related issues

Duplicated by ceph-qa-suite - Bug #19417: "AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'" in rados Duplicate 03/29/2017
Copied to ceph-qa-suite - Backport #19063: jewel: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' Resolved
Copied to ceph-qa-suite - Backport #19064: kraken: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' Resolved
Copied to ceph-qa-suite - Backport #19576: hammer: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' Rejected

History

#1 Updated by Nathan Cutler 9 months ago

  • Status changed from New to In Progress
  • Assignee set to Nathan Cutler

#2 Updated by Nathan Cutler 9 months ago

description: rados:thrash/{0-size-min-size-overrides/2-size-1-min-size.yaml 1-pg-log-overrides/normal_pg_log.yaml backoff/peering_and_recovery.yaml clusters/{fixed-2.yaml openstack.yaml} fs/btrfs.yaml hobj-sort.yaml msgr-failures/fastclose.yaml msgr/async.yaml objectstore/bluestore.yaml rados.yaml rocksdb.yaml thrashers/default.yaml workloads/cache-agent-big.yaml}

#3 Updated by Nathan Cutler 9 months ago

The problem, I think, is that Thrasher.__init__() spawns the do_thrash thread before initializing the ceph_objectstore_tool attribute. Essentially, do_thrash is racing with the rest of Thrasher.__init__() and in this case do_thrash triggered kill_osd() before Trasher.__init__() got much farther. It was very close, but on one of the nodes kill_osd() got to if self.ceph_objectstore_tool before that attribute got initialized.

The fix, I guess, is to move the ceph_objectstore_tool initialization block up so it runs before do_thrash is spawned.

#4 Updated by Nathan Cutler 9 months ago

  • Status changed from In Progress to Need Review
  • Backport set to jewel, kraken

#5 Updated by Nathan Cutler 9 months ago

  • Status changed from Need Review to Pending Backport

#6 Updated by Nathan Cutler 8 months ago

  • Project changed from teuthology to ceph-qa-suite

#7 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #19063: jewel: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' added

#8 Updated by Nathan Cutler 8 months ago

  • Copied to Backport #19064: kraken: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' added

#9 Updated by Kefu Chai 7 months ago

  • Duplicated by Bug #19417: "AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool'" in rados added

#10 Updated by Nathan Cutler 7 months ago

  • Backport changed from jewel, kraken to hammer, jewel, kraken

Need to backport this to hammer as well, because the bug is showing up in upgrade/hammer-x runs, e.g. http://pulpito.ceph.com/smithfarm-2017-04-10_18:49:29-upgrade:hammer-x-wip-jewel-backports-distro-basic-vps/1009149/

#11 Updated by Nathan Cutler 7 months ago

  • Copied to Backport #19576: hammer: AttributeError: Thrasher instance has no attribute 'ceph_objectstore_tool' added

#12 Updated by Nathan Cutler about 2 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF