Bug #62188
closedAttributeError: 'RemoteProcess' object has no attribute 'read'
0%
Description
/a/vshankar-2023-07-26_04:54:56-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7352553
2023-07-26T07:12:26.945 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2023-07-26T07:12:26.964 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.S5KnYoRutf 2023-07-26T07:12:26.965 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-07-26T07:12:26.965 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/tmp/tmp.S5KnYoRutf 2023-07-26T07:12:27.009 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/run.py", line 309, in copy_and_close shutil.copyfileobj(src, fdst) File "/usr/lib/python3.8/shutil.py", line 199, in copyfileobj fsrc_read = fsrc.read AttributeError: 'RemoteProcess' object has no attribute 'read'
Updated by Venky Shankar 9 months ago
- Project changed from teuthology to CephFS
- Assignee set to Rishabh Dave
FWIW - this seems to be happening with multifs-auth tests in fs suite
/a/vshankar-2023-07-26_04:54:56-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7352571
Rishabh, please RCA. I remember we talking about this but not sure if any RCA was done.
Updated by Rishabh Dave 9 months ago
I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in test_multifs_auth.py
.
I suspect the cause of failure is this line at the end of method CephFSTestCase.create_client()
- return run_ceph_cmd(f'auth get {self.client_name}')
. I can't be sure because there's nothing in logs to confirm this. I suspect so because same test (test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent
) failed recently while I was working with "fs authorize" PR.
Running this test locally (with vstart_runner.py) fails because the method create_client()
doesn't return the keyring.
Updated by Venky Shankar 9 months ago
Rishabh Dave wrote:
I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in
test_multifs_auth.py
.I suspect the cause of failure is this line at the end of method
CephFSTestCase.create_client()
-return run_ceph_cmd(f'auth get {self.client_name}')
. I can't be sure because there's nothing in logs to confirm this.
You could dump out the return value just before that line - if some output is expected, then the empty return value is most likely the issue.
I suspect so because same test (
test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent
) failed recently while I was working with "fs authorize" PR.Running this test locally (with vstart_runner.py) fails because the method
create_client()
doesn't return the keyring.
That sounds like a bug to me.
Updated by Rishabh Dave 9 months ago
Venky Shankar wrote:
Rishabh Dave wrote:
I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in
test_multifs_auth.py
.I suspect the cause of failure is this line at the end of method
CephFSTestCase.create_client()
-return run_ceph_cmd(f'auth get {self.client_name}')
. I can't be sure because there's nothing in logs to confirm this.You could dump out the return value just before that line - if some output is expected, then the empty return value is most likely the issue.
Yes. I suspect the tracker ticket I've linked is the same issue.
I suspect so because same test (
test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent
) failed recently while I was working with "fs authorize" PR.Running this test locally (with vstart_runner.py) fails because the method
create_client()
doesn't return the keyring.That sounds like a bug to me.
It is. I've raised a tracker ticket for it (https://tracker.ceph.com/issues/62246) and also a commit has been written for it.
Updated by Venky Shankar 9 months ago
- Related to Bug #62246: qa/cephfs: test_mount_mon_and_osd_caps_present_mds_caps_absent fails added
Updated by Venky Shankar 26 days ago
- Priority changed from Normal to Urgent
- Target version set to v20.0.0
- Backport set to quincy,reef,squid
- Severity changed from 3 - minor to 2 - major
- Labels (FS) qa, qa-failure added
Rishabh, this is seen in reef run: /a/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641588
2024-04-05T21:05:00.846 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 309, in copy_and_close shutil.copyfileobj(src, fdst) File "/usr/lib/python3.8/shutil.py", line 202, in copyfileobj fsrc_read = fsrc.read AttributeError: 'RemoteProcess' object has no attribute 'read'
Please RCA this asap.
Updated by Venky Shankar 26 days ago
Another instance: /a/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641634
and more dead jobs probably due to the same reason: https://pulpito.ceph.com/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/
Updated by Rishabh Dave 23 days ago ยท Edited
- Status changed from New to Duplicate
All the recent failures are from QA runs for Reef, this is because the fix for this issue (https://tracker.ceph.com/issues/62246) wasn't backported. Backports have been created now - quincy: https://github.com/ceph/ceph/pull/56836 and reef: https://github.com/ceph/ceph/pull/56837.
Updated by Venky Shankar 22 days ago
Rishabh Dave wrote in #note-9:
All the recent failures are from QA runs for Reef, this is because the fix for this issue (https://tracker.ceph.com/issues/62246) wasn't backported. Backports have been created now - quincy: https://github.com/ceph/ceph/pull/56836 and reef: https://github.com/ceph/ceph/pull/56837.
kk, thx for confirming,