Bug #20700: mira022, mira042, mira077 and mira088 is broken - sepia - Ceph

Actions

Copy link

Bug #20700

closed

mira022, mira042, mira077 and mira088 is broken

Added by Kefu Chai almost 7 years ago. Updated over 5 years ago.

Status:

Rejected

Priority:

High

Assignee:

David Galloway

Category:

Test Node

Target version:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

ceph-objectstore-tool: exp list-pgs failure with status 1

is a typical error message of this sort of error.

[  198.353999]  sdd: unknown partition table
[  199.505865]  sde: unknown partition table
[  200.643640]  sdf: unknown partition table
[  201.793157]  sdg: unknown partition table
[  202.944696]  sdb: unknown partition table
[  204.085415]  sdc: unknown partition table
[  205.222828]  sdh: unknown partition table

in dmesg output.

and http://qa-proxy.ceph.com/teuthology/kchai-2017-07-20_03:05:27-rados-wip-kefu-testing-distro-basic-mira/1422160/teuthology.log shows an I/O error.

2017-07-20T09:29:14.829 INFO:teuthology.orchestra.run.mira077.stderr:Mount failed with '(5) Input/output error'

the same applies to mira022, mira042 and mira088.

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

Subject changed from mira077 is broken to mira077 and mira088 is broken
Description updated (diff)

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

Assignee set to David Galloway

David, mind taking a look?

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

Subject changed from mira077 and mira088 is broken to mira042, mira077 and mira088 is broken
Description updated (diff)
Priority changed from Normal to High

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

Subject changed from mira042, mira077 and mira088 is broken to mira022, mira042, mira077 and mira088 is broken
Description updated (diff)

Actions

Copy link

Updated by Kefu Chai almost 7 years ago

Assignee deleted (~~David Galloway~~)

thanks to dmick, he helped to look at the involved test nodes, the partition tables of the reported devices are zeroed, but that should not be fatal.

in the case of mira077, the osd.3 was up and running before ceph-objectstore-tools list pg:

 ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --log-file=/var/log/ceph/objectstore_tool.\\$pid.log --op list-pgs'

and failed with

2017-07-20T09:29:14.829 INFO:teuthology.orchestra.run.mira077.stderr:Mount failed with '(5) Input/output error'

and another interesting thing is that teuthology reports I/O error also

2017-07-20T09:29:14.832 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 608, in _notify_links
    link(self)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 538, in run
    self._report_error(sys.exc_info())
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/greenlet.py", line 526, in _report_error
    self.parent.handle_error(self, *exc_info)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/__init__.py", line 106, in custom_handle_error
    self._origin_handle_error(context, type, value, tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 558, in handle_error
    self.print_exception(context, type, value, tb)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/gevent/hub.py", line 605, in print_exception
    traceback.print_exception(type, value, tb, file=errstream)
  File "/usr/lib/python2.7/traceback.py", line 124, in print_exception
    _print(file, 'Traceback (most recent call last):')
  File "/usr/lib/python2.7/traceback.py", line 13, in _print
    file.write(str+terminator)
IOError: [Errno 5] Input/output error

Actions

Copy link