Project

General

Profile

Actions
Copy link

Bug #45944

open

osd/osd-markdown.sh: TEST_osd_stop failed

Added by Neha Ojha almost 4 years ago. Updated almost 4 years ago.

Status:
Triaged
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2020-06-07T01:32:18.288 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop:  sleep 15
2020-06-07T01:32:33.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop:  ceph osd tree
2020-06-07T01:32:33.290 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop:  grep down
2020-06-07T01:32:33.291 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop:  grep osd.0
2020-06-07T01:32:33.644 INFO:tasks.workunit.client.0.smithi105.stdout: 0    hdd  0.09769          osd.0         down   1.00000  1.00000
2020-06-07T01:32:33.645 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop:  ps -p 640980
2020-06-07T01:32:33.650 INFO:tasks.workunit.client.0.smithi105.stdout:    PID TTY          TIME CMD
2020-06-07T01:32:33.650 INFO:tasks.workunit.client.0.smithi105.stdout: 640980 ?        00:00:00 ceph-osd
2020-06-07T01:32:33.650 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop:  return 1
2020-06-07T01:32:33.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run:  return 1
2020-06-07T01:32:33.651 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main:  code=1

/a/yuriw-2020-06-06_21:25:23-rados-wip-yuri-master_6.6.20-distro-basic-smithi/5122912

Actions
Copy link
 #1

Updated by Neha Ojha almost 4 years ago

  • Status changed from New to Triaged

This appears to be a rare condition when 15 seconds sleep was not enough.

Actions
Copy link
 #2

Updated by Neha Ojha almost 4 years ago

/a/teuthology-2020-06-19_07:01:02-rados-master-distro-basic-smithi/5164221

Actions
Copy link
 #3

Updated by Neha Ojha almost 4 years ago 

2020-06-07T01:32:17.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop:  ceph osd tree
2020-06-07T01:32:17.419 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop:  grep osd.0
2020-06-07T01:32:17.420 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop:  grep up
2020-06-07T01:32:17.761 INFO:tasks.workunit.client.0.smithi105.stdout: 0    hdd  0.09769          osd.0           up   1.00000  1.00000
2020-06-07T01:32:17.761 INFO:tasks.workunit.client.0.smithi105.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop:  ceph osd stop osd.0
2020-06-07T01:32:18.280 INFO:tasks.workunit.client.0.smithi105.stderr:osd.0 is already stopped.

The weird part is that the stop command actually did not work, somehow "osd.0 is already stopped" which will only happen if is_stop() returns true!

bool is_stop(int osd) const {
    return exists(osd) && is_down(osd) &&
           (osd_state[osd] & CEPH_OSD_STOP);
  }

This test was introduced in https://github.com/ceph/ceph/pull/27595.

Actions
Copy link

Also available in: Atom PDF