Bug #4430: teuthology: powercycling hits NoneType object has no attribute 'power_off' - teuthology - Ceph

Actions

Copy link

Bug #4430

closed

teuthology: powercycling hits NoneType object has no attribute 'power_off'

Added by Sam Lang about 11 years ago. Updated about 11 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Sam Lang

Category:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

To avoid confusion, this is the current powercycling issue being investigated:

19473: (293s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:admin_socket_objecter_requests.yaml
'NoneType' object has no attribute 'power_off'
19476: (649s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:cfuse_workunit_suites_blogbench.yaml
'NoneType' object has no attribute 'power_off'
19480: (1916s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:cfuse_workunit_suites_fsx.yaml
'NoneType' object has no attribute 'power_off'
19483: (501s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:cfuse_workunit_suites_pjd.yaml
'NoneType' object has no attribute 'power_off'
19485: (221s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:cfuse_workunit_suites_truncate_delay.yaml
'NoneType' object has no attribute 'power_off'
19488: (2101s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:radosbench.yaml
'NoneType' object has no attribute 'power_off'
19490: (1328s) collection:osd_powercycle clusters:3osd-1per-target.yaml powercycle:default.yaml tasks:snaps-few-objects.yaml
'NoneType' object has no attribute 'power_off'

Actions

Copy link

Updated by Sage Weil about 11 years ago

Priority changed from Normal to Urgent

Actions

Copy link

Updated by Sam Lang about 11 years ago

Status changed from In Progress to Fix Under Review

I think the issue here is ipmi flakiness. If the ipmi user was specified in the .teuthology.yaml, the internal task would always try to initialize the ipmi consoles on all targets. If that failed, the console wouldn't get initialized but the test would still run. I pushed a bunch of fixes to wip-powercycling-fixes that should at least report an ipmi connect failure properly, including:

moved the ipmi console connect code to the osdthrasher so that we only try to ipmi console connect to targets when powercycling is specified.
report a better error if we can't connect to ipmi and we need it for powercycling
re-create /var/run/ceph on reboot

Actions

Copy link