Bug #49466: qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'" - CephFS - Ceph

Actions

Copy link

Bug #49466

closed

qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"

Added by Patrick Donnelly about 3 years ago. Updated about 3 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Rishabh Dave

Category:

Target version:

Ceph - v17.0.0

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

qa-suite

Labels (FS):

qa-failure

Pull request ID:

40431

Crash signature (v1):

Crash signature (v2):

Description

2021-02-22T23:48:38.116 INFO:tasks.cephfs_test_runner:test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) ... ERROR
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:======================================================================
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:ERROR: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_963d854aba2d8969ed239500ad8f88439f7451db/qa/tasks/cephfs/test_multifs_auth.py", line 279, in test_mount_all_caps_absent
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:    keyring)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_963d854aba2d8969ed239500ad8f88439f7451db/qa/tasks/cephfs/cephfs_test_case.py", line 453, in create_keyring_file
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:    sudo_write_file(remote, keyring_path, keyring)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 546, in sudo_write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    remote.sudo_write_file(path, data, mode=perms, owner=owner)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 600, in sudo_write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self.write_file(path, data, sudo=True, **kwargs)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 594, in write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self.run(args=args, stdin=data, quiet=True)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 455, in run
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    r.wait()
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 161, in wait
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-02-22T23:48:38.119 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'

From: /ceph/teuthology-archive/teuthology-2021-02-21_03:15:03-fs-master-distro-basic-gibba/5899697/teuthology.log

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Rishabh Dave about 3 years ago

just for record: same issue hit me yesterday night on vstart cluster too.

Actions

Copy link

Updated by Rishabh Dave about 3 years ago

Status changed from New to In Progress

Actions

Copy link

Updated by Rishabh Dave about 3 years ago

Description updated (diff)

Cause of the bug: Write was attempted on /tmp file with the root user. Files in /tmp can't be written by any user, including the root user, besides the file owner even when permission mode for the file is 777. This error doesn't occur everytime because RHEL uses older kernel version and Ubuntu uses newer kernel versions that enforces this policy.

I triggered a RHEL and Ubuntu job yesterday to confirm this -
RHEL job: http://pulpito.front.sepia.ceph.com/rishabh-2021-03-17_13:40:14-fs-wip-rishabh-vr-set-shell-to-True-distro-basic-smithi/#
Ubuntu job: http://pulpito.front.sepia.ceph.com/rishabh-2021-03-25_14:23:03-fs-wip-rishabh-vr-set-shell-to-True-distro-basic-smithi/

Job in bug description failed because it is a Ubuntu job.

The best fix is to add a method to teuthology.orchestra.remote.Remote. It would call teuthology.orchestra.remote.Remote.write_file() with sudo set to False and using that method instead of teuthology.misc.sudo_write_file.

Actions

Copy link

Updated by Rishabh Dave about 3 years ago

Pull request ID set to 40431

Actions

Copy link

Updated by Kyrylo Shatskyy about 3 years ago

Thanks Rishabh for your analysis.

The best fix is to add a method to teuthology.orchestra.remote.Remote. It would call teuthology.orchestra.remote.Remote.write_file() with sudo set to False and using that method instead of teuthology.misc.sudo_write_file.

The write_file is supposed to be used sudo defaults False, by design. Where the sudo_write_file is designed to always use sudo=True. So it should be just enough to use write_file, in other words, it just one-line fix. No need to add new API or extending others.

Actions

Copy link