Bug #49524
closed
ceph_test_rados_delete_pools_parallel didn't start
Added by Sage Weil about 3 years ago.
Updated about 1 year ago.
Description
2021-02-26T23:49:08.979 INFO:tasks.workunit.client.0.smithi035.stderr:+ echo 'test delete_pools_parallel on pid 32088'
...
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ pid=32088
2021-02-27T00:01:08.332 INFO:tasks.workunit.client.0.smithi035.stderr:+ wait 32088
...
2021-02-27T05:49:08.686 INFO:tasks.workunit.client.0.smithi035.stderr:++ cleanup
2021-02-27T05:49:08.687 INFO:tasks.workunit.client.0.smithi035.stderr:++ pkill -P 31603
2021-02-27T05:49:08.710 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-02-27T05:49:08.712 INFO:tasks.workunit.client.0.smithi035.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 32088 Terminated bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""
/a/sage-2021-02-26_22:19:00-rados-wip-sage-testing-2021-02-26-1412-distro-basic-smithi/5917049
Reminds me of #38416
2021-12-14T00:05:14.196 INFO:tasks.workunit.client.0.smithi045.stdout:test delete_pools_parallel on pid 47101
...
2021-12-14T06:05:13.860 INFO:tasks.workunit.client.0.smithi045.stderr:++ pkill -P 46564
2021-12-14T06:05:13.873 DEBUG:teuthology.orchestra.run:got remote process result: 124
2021-12-14T06:05:13.873 INFO:tasks.workunit.client.0.smithi045.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 47101 Terminated bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""
/a/yuriw-2021-12-13_23:18:05-rados-wip-yuri-testing-2021-12-13-1210-distro-default-smithi/6561154
These failures are diagnosed by noting the failed pid (in this case 59576), and backtracking to see which test it was attached to (in this case, delete_pools_parallel).
/a/yuriw-2022-09-04_23:20:10-rados-wip-yuri10-testing-2022-09-04-0811-quincy-distro-default-smithi/7011689
2022-09-05T06:01:29.044 INFO:tasks.workunit.client.0.smithi104.stderr:+ echo 'test delete_pools_parallel on pid 59576
...
2022-09-05T12:01:28.773 INFO:tasks.workunit.client.0.smithi104.stderr:/home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh: line 10: 59576 Terminated bash -o pipefail -exc "ceph_test_rados_$f $color 2>&1 | tee ceph_test_rados_$ff.log | sed \"s/^/$r: /\""
- Translation missing: en.field_tag_list set to test-failure
- Assignee set to Nitzan Mordechai
Nitzan, can you please take a look at this issue? seems intermittent, but still exists
The printing will be flushed only after the process complete, in that case of ceph_test_rados_delete_pools_parallel, we will first create 50 object, since there is no mention in the log for delete_pools_parallel, i think that the first process (create objects) didn't complete (probably failed in some way that we don't know yet).
osds and mons logs do show that the pool created, i need to track if and how many object created for that pool ceph_test_rados_delete_pools_parallel.smithi104-59583
My theory is that fork failed, which caused all the test not to run, this is the only place we won't get any printing.
Running teuthology tests against branch that will add some more message to debug that issue.
- Status changed from New to In Progress
- Status changed from In Progress to Resolved
- Pull request ID set to 49109
Should be fixed by PR 49109
Also available in: Atom
PDF