Project

General

Profile

Bug #49524

ceph_test_rados_delete_pools_parallel didn't start

Added by Sage Weil over 2 years ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Nitzan Mordechai
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
49109
Crash signature (v1):
Crash signature (v2):
Tags:
test-failure

Description

2021-02-26T23:49:08.979 INFO:tasks.workunit.client.0.smithi035.stderr:+ echo 'test delete_pools_parallel on pid 32088'
...
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ pid=32088
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ wait 32088
...
2021-02-27T05:49:08.686 INFO:tasks.workunit.client.0.smithi035.stderr:++ cleanup
2021-02-27T05:49:08.687 INFO:tasks.workunit.client.0.smithi035.stderr:++ pkill -P 31603
2021-02-27T05:49:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-02-27T05:49:08.712 INFO:tasks.workunit.client.0.smithi035.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 32088 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""

/a/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917049

Reminds me of #38416

History

#2 Updated by Neha Ojha almost 2 years ago 

2021-12-14T00:05:14.196 INFO:tasks.workunit.client.0.smithi045.stdout:test delete_pools_parallel on pid 47101
...
2021-12-14T06:05:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:++ pkill -P 46564
2021-12-14T06:05:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-12-14T06:05:13.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 47101 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""

/a/yuriw-2021-12-13_23:18:05-rados-wip-yuri-testing-2021-12-13-1210-distro-default-smithi/6561154

#3 Updated by Laura Flores about 1 year ago

These failures are diagnosed by noting the failed pid (in this case 59576), and backtracking to see which test it was attached to (in this case, delete_pools_parallel).

/a/yuriw-2022-09-04_23:20:10-rados-wip-yuri10-testing-2022-09-04-0811-quincy-distro-default-smithi/7011689

2022-09-05T06:01:29.044 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test delete_pools_parallel on pid 59576

...

2022-09-05T12:01:28.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 59576 Terminated              bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""

#4 Updated by Laura Flores about 1 year ago

  • Tags set to test-failure

#5 Updated by Neha Ojha about 1 year ago

  • Assignee set to Nitzan Mordechai

Nitzan, can you please take a look at this issue? seems intermittent, but still exists

#6 Updated by Nitzan Mordechai about 1 year ago

The printing will be flushed only after the process complete, in that case of ceph_test_rados_delete_pools_parallel, we will first create 50 object, since there is no mention in the log for delete_pools_parallel, i think that the first process (create objects) didn't complete (probably failed in some way that we don't know yet).
osds and mons logs do show that the pool created, i need to track if and how many object created for that pool ceph_test_rados_delete_pools_parallel.smithi104-59583

#7 Updated by Nitzan Mordechai about 1 year ago

My theory is that fork failed, which caused all the test not to run, this is the only place we won't get any printing.

Running teuthology tests against branch that will add some more message to debug that issue.

#8 Updated by Nitzan Mordechai about 1 year ago

  • Status changed from New to In Progress

#9 Updated by Nitzan Mordechai 8 months ago

  • Status changed from In Progress to Resolved
  • Pull request ID set to 49109

Should be fixed by PR 49109

Also available in: Atom PDF