Project

General

Profile

Actions
Copy link

Bug #62188

closed

AttributeError: 'RemoteProcess' object has no attribute 'read'

Added by Venky Shankar 9 months ago. Updated 21 days ago.

Status:
Duplicate
Priority:
Urgent
Assignee:
Rishabh Dave
Category:
-
Target version:
Ceph - v20.0.0
% Done:

0%

Source:
Tags:
Backport:
quincy,reef,squid
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

/a/vshankar-2023-07-26_04:54:56-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7352553

2023-07-26T07:12:26.945 DEBUG:teuthology.orchestra.run.smithi121:> mktemp
2023-07-26T07:12:26.964 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.S5KnYoRutf
2023-07-26T07:12:26.965 DEBUG:teuthology.orchestra.run.smithi121:> set -ex
2023-07-26T07:12:26.965 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/tmp/tmp.S5KnYoRutf
2023-07-26T07:12:27.009 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_teuthology_407880c6d3fb77318fff01c863715090f9c2de69/teuthology/orchestra/run.py", line 309, in copy_and_close
    shutil.copyfileobj(src, fdst)
  File "/usr/lib/python3.8/shutil.py", line 199, in copyfileobj
    fsrc_read = fsrc.read
AttributeError: 'RemoteProcess' object has no attribute 'read'

Related issues 1 (1 open — 0 closed)

Related to CephFS - Bug #62246: qa/cephfs: test_mount_mon_and_osd_caps_present_mds_caps_absent failsPending BackportRishabh Dave

Actions
Actions
Copy link
 #1

Updated by Venky Shankar 9 months ago

  • Project changed from teuthology to CephFS
  • Assignee set to Rishabh Dave

FWIW - this seems to be happening with multifs-auth tests in fs suite

/a/vshankar-2023-07-26_04:54:56-fs-wip-vshankar-testing-20230725.053049-testing-default-smithi/7352571

Rishabh, please RCA. I remember we talking about this but not sure if any RCA was done.

Actions
Copy link
 #2

Updated by Rishabh Dave 9 months ago

I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in test_multifs_auth.py.

I suspect the cause of failure is this line at the end of method CephFSTestCase.create_client() - return run_ceph_cmd(f'auth get {self.client_name}'). I can't be sure because there's nothing in logs to confirm this. I suspect so because same test (test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent) failed recently while I was working with "fs authorize" PR.

Running this test locally (with vstart_runner.py) fails because the method create_client() doesn't return the keyring.

Actions
Copy link
 #3

Updated by Venky Shankar 9 months ago

Rishabh Dave wrote:

I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in test_multifs_auth.py.

I suspect the cause of failure is this line at the end of method CephFSTestCase.create_client() - return run_ceph_cmd(f'auth get {self.client_name}'). I can't be sure because there's nothing in logs to confirm this.

You could dump out the return value just before that line - if some output is expected, then the empty return value is most likely the issue.

I suspect so because same test (test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent) failed recently while I was working with "fs authorize" PR.

Running this test locally (with vstart_runner.py) fails because the method create_client() doesn't return the keyring.

That sounds like a bug to me.

Actions
Copy link
 #4

Updated by Rishabh Dave 9 months ago

Venky Shankar wrote:

Rishabh Dave wrote:

I spent a good amount of time with this ticket. The reason for this failure is unclear from logs. There's no traceback pointing to any code in test_multifs_auth.py.

I suspect the cause of failure is this line at the end of method CephFSTestCase.create_client() - return run_ceph_cmd(f'auth get {self.client_name}'). I can't be sure because there's nothing in logs to confirm this.

You could dump out the return value just before that line - if some output is expected, then the empty return value is most likely the issue.

Yes. I suspect the tracker ticket I've linked is the same issue.

I suspect so because same test (test_multifs_auth.TestClientsWithoutAuth.test_mount_mon_and_osd_caps_present_mds_caps_absent) failed recently while I was working with "fs authorize" PR.

Running this test locally (with vstart_runner.py) fails because the method create_client() doesn't return the keyring.

That sounds like a bug to me.

It is. I've raised a tracker ticket for it (https://tracker.ceph.com/issues/62246) and also a commit has been written for it.

Actions
Copy link
 #5

Updated by Venky Shankar 9 months ago

  • Related to Bug #62246: qa/cephfs: test_mount_mon_and_osd_caps_present_mds_caps_absent fails added
Actions
Copy link
 #7

Updated by Venky Shankar 24 days ago

  • Priority changed from Normal to Urgent
  • Target version set to v20.0.0
  • Backport set to quincy,reef,squid
  • Severity changed from 3 - minor to 2 - major
  • Labels (FS) qa, qa-failure added

Rishabh, this is seen in reef run: /a/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641588

2024-04-05T21:05:00.846 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_teuthology_6c637841c215537a4502385240412f1966e0faab/teuthology/orchestra/run.py", line 309, in copy_and_close
    shutil.copyfileobj(src, fdst)
  File "/usr/lib/python3.8/shutil.py", line 202, in copyfileobj
    fsrc_read = fsrc.read
AttributeError: 'RemoteProcess' object has no attribute 'read'

Please RCA this asap.

Actions
Copy link
 #8

Updated by Venky Shankar 24 days ago

Another instance: /a/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/7641634

and more dead jobs probably due to the same reason: https://pulpito.ceph.com/yuriw-2024-04-05_18:13:21-fs-wip-yuri4-testing-2024-04-04-1128-reef-distro-default-smithi/

Actions
Copy link
 #9

Updated by Rishabh Dave 21 days ago ยท Edited

  • Status changed from New to Duplicate

All the recent failures are from QA runs for Reef, this is because the fix for this issue (https://tracker.ceph.com/issues/62246) wasn't backported. Backports have been created now - quincy: https://github.com/ceph/ceph/pull/56836 and reef: https://github.com/ceph/ceph/pull/56837.

Actions
Copy link
 #10

Updated by Venky Shankar 21 days ago

Rishabh Dave wrote in #note-9:

All the recent failures are from QA runs for Reef, this is because the fix for this issue (https://tracker.ceph.com/issues/62246) wasn't backported. Backports have been created now - quincy: https://github.com/ceph/ceph/pull/56836 and reef: https://github.com/ceph/ceph/pull/56837.

kk, thx for confirming,

Actions
Copy link

Also available in: Atom PDF