Project

General

Profile

Actions

Bug #44500

closed

ceph-objectstore-tool error on apply-layout-settings+bluestore

Added by Sage Weil about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-03-07T03:29:24.584 INFO:teuthology.orchestra.run.smithi025:> CEPH_ARGS='--filestore-merge-threshold 1 --filestore-split-multiple 1' sudo -E ceph-objectstore-tool --no-mon-config --log-file=/var/log/ceph/objectstore_tool.$pid.log --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op apply-layout-settings --pool device_health_metrics
...
2020-03-07T03:29:25.455 INFO:teuthology.orchestra.run.smithi025.stderr:Nothing to do for non-filestore backend
...
2020-03-07T03:29:25.999 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 123, in wrapper
    return func(self)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 1205, in _do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 461, in kill_osd
    " failed with {status}".format(status=proc.exitstatus))
Exception: ceph-objectstore-tool apply-layout-settings failed with 10

/a/sage-2020-03-06_21:03:21-rados-wip-sage2-testing-2020-03-06-1128-distro-basic-smithi/4832203

the relevant code is

  if (!fs) {
    cerr << "Nothing to do for non-filestore backend" << std::endl;
    return 0; // making this return success makes testing easier
  }

not sure where the 10 came from! during shutdown there is this,

  int r = fs->umount();
  if (r < 0) {
    cerr << "umount failed: " << cpp_strerror(r) << std::endl;
    // If no previous error, then use umount() error
    if (ret == 0)
      ret = r;
  }

but the tool log shows a successful shutdown (copied to archive dir)

Actions #1

Updated by Sage Weil about 4 years ago

  • Subject changed from ceph-objectstore-tool exit code 10 on apply-layout-settings+bluestore to ceph-objectstore-tool error on apply-layout-settings+bluestore
  • Priority changed from Normal to Urgent

another instance in teh same test batch:

2020-03-07T03:58:40.938 INFO:teuthology.orchestra.run.smithi018.stderr:Nothing to do for non-filestore backend
...
2020-03-07T03:58:41.459 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 123, in wrapper
    return func(self)
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 1205, in _do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/git.ceph.com_ceph-c_wip-sage2-testing-2020-03-06-1128/qa/tasks/ceph_manager.py", line 461, in kill_osd
    " failed with {status}".format(status=proc.exitstatus))
Exception: ceph-objectstore-tool apply-layout-settings failed with 12

/a/sage-2020-03-06_21:03:21-rados-wip-sage2-testing-2020-03-06-1128-distro-basic-smithi/4832261

Actions #2

Updated by Sage Weil about 4 years ago

a third one! /a/sage-2020-03-06_21:03:21-rados-wip-sage2-testing-2020-03-06-1128-distro-basic-smithi/4832341

Actions #3

Updated by Sage Weil about 4 years ago

/a/sage-2020-03-07_14:00:00-rados-master-distro-basic-smithi/4834662
/a/sage-2020-03-07_14:00:00-rados-master-distro-basic-smithi/4834732

Actions #4

Updated by Sage Weil about 4 years ago

/a/sage-2020-03-07_15:51:10-rados-wip-sage4-testing-2020-03-07-0705-distro-basic-smithi/4835208
description: rados/verify/{centos_latest.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml}
d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v2only.yaml
objectstore/bluestore-low-osd-mem-target.yaml rados.yaml tasks/rados_cls_all.yaml
validater/valgrind.yaml}

/a/sage-2020-03-07_15:51:10-rados-wip-sage4-testing-2020-03-07-0705-distro-basic-smithi/4835272
description: rados/verify/{centos_latest.yaml ceph.yaml clusters/{fixed-2.yaml openstack.yaml}
d-thrash/default/{default.yaml thrashosds-health.yaml} msgr-failures/few.yaml msgr/async-v1only.yaml
objectstore/filestore-xfs.yaml rados.yaml tasks/rados_api_tests.yaml validater/valgrind.yaml}

Actions #5

Updated by Kefu Chai about 4 years ago

  • Status changed from New to Fix Under Review
  • Assignee set to Kefu Chai
  • Pull request ID set to 33805
Actions #6

Updated by Sage Weil about 4 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF