Project

General

Profile

Actions

Bug #49524

closed

ceph_test_rados_delete_pools_parallel didn't start

Added by Sage Weil about 3 years ago. Updated about 1 year ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-26T23:49:08.979 INFO:tasks.workunit.client.0.smithi035.stderr:+ echo 'test delete_pools_parallel on pid 32088'
...
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ pid=32088
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ wait 32088
...
2021-02-27T05:49:08.686 INFO:tasks.workunit.client.0.smithi035.stderr:++ cleanup
2021-02-27T05:49:08.687 INFO:tasks.workunit.client.0.smithi035.stderr:++ pkill -P 31603
2021-02-27T05:49:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-02-27T05:49:08.712 INFO:tasks.workunit.client.0.smithi035.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 32088 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 

/a/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917049

Reminds me of #38416

Actions #1

Updated by Neha Ojha about 3 years ago

Actions #2

Updated by Neha Ojha over 2 years ago

2021-12-14T00:05:14.196 INFO:tasks.workunit.client.0.smithi045.stdout:test delete_pools_parallel on pid 47101
...
2021-12-14T06:05:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:++ pkill -P 46564
2021-12-14T06:05:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-12-14T06:05:13.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 47101 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 

/a/yuriw-2021-12-13_23:18:05-rados-wip-yuri-testing-2021-12-13-1210-distro-default-smithi/6561154

Actions #3

Updated by Laura Flores over 1 year ago

These failures are diagnosed by noting the failed pid (in this case 59576), and backtracking to see which test it was attached to (in this case, delete_pools_parallel).

/a/yuriw-2022-09-04_23:20:10-rados-wip-yuri10-testing-2022-09-04-0811-quincy-distro-default-smithi/7011689

2022-09-05T06:01:29.044 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test delete_pools_parallel on pid 59576

...

2022-09-05T12:01:28.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 59576 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\"" 

Actions #4

Updated by Laura Flores over 1 year ago

  • Translation missing: en.field_tag_list set to test-failure
Actions #5

Updated by Neha Ojha over 1 year ago

  • Assignee set to Nitzan Mordechai

Nitzan, can you please take a look at this issue? seems intermittent, but still exists

Actions #6

Updated by Nitzan Mordechai over 1 year ago

The printing will be flushed only after the process complete, in that case of ceph_test_rados_delete_pools_parallel, we will first create 50 object, since there is no mention in the log for delete_pools_parallel, i think that the first process (create objects) didn't complete (probably failed in some way that we don't know yet).
osds and mons logs do show that the pool created, i need to track if and how many object created for that pool ceph_test_rados_delete_pools_parallel.smithi104-59583

Actions #7

Updated by Nitzan Mordechai over 1 year ago

My theory is that fork failed, which caused all the test not to run, this is the only place we won't get any printing.

Running teuthology tests against branch that will add some more message to debug that issue.

Actions #8

Updated by Nitzan Mordechai over 1 year ago

  • Status changed from New to In Progress
Actions #9

Updated by Nitzan Mordechai about 1 year ago

  • Status changed from In Progress to Resolved
  • Pull request ID set to 49109

Should be fixed by PR 49109

Actions

Also available in: Atom PDF