Bug #24057: cbt fails to copy results to the archive dir - RADOS - Ceph

Actions

Copy link

Bug #24057

closed

cbt fails to copy results to the archive dir

Added by Kefu Chai almost 6 years ago. Updated about 2 years ago.

Status:

Rejected

Priority:

Normal

Assignee:

Neha Ojha

Category:

Tests

Target version:

% Done:

Source:

Development

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

/a/kchai-2018-05-08_12:15:21-rados-wip-kefu-testing2-2018-05-08-1834-distro-basic-mira/2501280

2018-05-08T20:15:55.636 INFO:teuthology.orchestra.run.mira041.stderr:20:15:55 - DEBUG    - cbt      - CheckedPopen continue_if_error=False args=rpdcp -f 10 -R ssh -w ubuntu@ubuntu@m
ira041.front.sepia.ceph.com -r /tmp/cbt/00000000/LibrbdFio/osd_ra-00004096/op_size-00004096/concurrent_procs-002/iodepth-032/randrw/* /home/ubuntu/cephtest/archive/cbt/results/00000
000/id6196024225295968574
2018-05-08T20:15:56.015 INFO:teuthology.orchestra.run.mira041.stderr:20:15:56 - INFO     - cbt      - Convert results to json format.
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:20:15:56 - ERROR    - cbt      - During tests
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:Traceback (most recent call last):
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:  File "/home/ubuntu/cephtest/cbt/cbt.py", line 71, in main
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:    b.run()
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:  File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 131, in run
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:    self.analyze(self.out_dir)
2018-05-08T20:15:56.031 INFO:teuthology.orchestra.run.mira041.stderr:  File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 212, in analyze
2018-05-08T20:15:56.032 INFO:teuthology.orchestra.run.mira041.stderr:    self.parse(out_dir)
2018-05-08T20:15:56.032 INFO:teuthology.orchestra.run.mira041.stderr:  File "/home/ubuntu/cephtest/cbt/benchmark/librbdfio.py", line 198, in parse
2018-05-08T20:15:56.032 INFO:teuthology.orchestra.run.mira041.stderr:    with open(out_file) as fd:
2018-05-08T20:15:56.032 INFO:teuthology.orchestra.run.mira041.stderr:IOError: [Errno 2] No such file or directory: '/home/ubuntu/cephtest/archive/cbt/results/00000000/id6196024225295968574/output.0.ubuntu@mira041.front.sepia.ceph.com'

Actions

Copy link

Updated by Kefu Chai almost 6 years ago

Description updated (diff)

Neha, mind taking a look? i've run into this failure couple times.

Actions

Copy link

Updated by Neha Ojha almost 6 years ago

This seems to be an issue with cbt not being able to copy output files to its archive dir, and hence we don't find them while converting them to json format. The result files missing under the cbt dir in the teuthology archives, suggests the same.

Noticed a similar issue with radosbench as well here: http://pulpito.ceph.com/teuthology-2018-04-26_03:57:02-perf-basic-master-distro-basic-smithi/2440039/

Actions

Copy link

Updated by Neha Ojha almost 6 years ago

Subject changed from rados/perf/ceph.yaml fails in librbdfio.py to cbt fails to copy results to the archive dir
Status changed from New to 12

Actions

Copy link

Updated by Neha Ojha over 4 years ago

Status changed from 12 to Can't reproduce

Actions

Copy link

Updated by Sridhar Seshasayee almost 4 years ago

Observed the issue during this run:
/a/sseshasa-2020-06-24_17:46:09-rados-wip-sseshasa-testing-2020-06-24-1858-distro-basic-smithi/5176293

2020-06-24T21:23:31.065 INFO:teuthology.orchestra.run.smithi072.stderr:21:23:31 - INFO - cbt - Convert results to json format.
2020-06-24T21:23:31.066 INFO:teuthology.orchestra.run.smithi072.stderr:21:23:31 - DEBUG - cbt - Nodes : ubuntu@smithi072.front.sepia.ceph.com
2020-06-24T21:23:31.094 INFO:teuthology.orchestra.run.smithi072.stderr:21:23:31 - ERROR - cbt - During tests
2020-06-24T21:23:31.095 INFO:teuthology.orchestra.run.smithi072.stderr:Traceback (most recent call last):
2020-06-24T21:23:31.095 INFO:teuthology.orchestra.run.smithi072.stderr: File "/home/ubuntu/cephtest/cbt/cbt.py", line 76, in main
2020-06-24T21:23:31.096 INFO:teuthology.orchestra.run.smithi072.stderr: b.run()
2020-06-24T21:23:31.096 INFO:teuthology.orchestra.run.smithi072.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/radosbench.py", line 105, in run
2020-06-24T21:23:31.096 INFO:teuthology.orchestra.run.smithi072.stderr: runtime=self.write_time)
2020-06-24T21:23:31.097 INFO:teuthology.orchestra.run.smithi072.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/radosbench.py", line 208, in _run
2020-06-24T21:23:31.097 INFO:teuthology.orchestra.run.smithi072.stderr: self.analyze(out_dir)
2020-06-24T21:23:31.097 INFO:teuthology.orchestra.run.smithi072.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/radosbench.py", line 248, in analyze
2020-06-24T21:23:31.097 INFO:teuthology.orchestra.run.smithi072.stderr: self.parse(out_dir)
2020-06-24T21:23:31.098 INFO:teuthology.orchestra.run.smithi072.stderr: File "/home/ubuntu/cephtest/cbt/benchmark/radosbench.py", line 233, in parse
2020-06-24T21:23:31.098 INFO:teuthology.orchestra.run.smithi072.stderr: with open(out_file) as fd:
2020-06-24T21:23:31.099 INFO:teuthology.orchestra.run.smithi072.stderr:FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/cephtest/archive/cbt/results/00000000/id-8b8185cb/write/output.0.smithi072.front.sepia.ceph.com'