Project

General

Profile

Bug #49466

qa: "Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'"

Added by Patrick Donnelly 9 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa-failure
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2021-02-22T23:48:38.116 INFO:tasks.cephfs_test_runner:test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth) ... ERROR
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:======================================================================
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:ERROR: test_mount_all_caps_absent (tasks.cephfs.test_multifs_auth.TestClientsWithoutAuth)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:----------------------------------------------------------------------
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:Traceback (most recent call last):
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_963d854aba2d8969ed239500ad8f88439f7451db/qa/tasks/cephfs/test_multifs_auth.py", line 279, in test_mount_all_caps_absent
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:    keyring)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_ceph_963d854aba2d8969ed239500ad8f88439f7451db/qa/tasks/cephfs/cephfs_test_case.py", line 453, in create_keyring_file
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:    sudo_write_file(remote, keyring_path, keyring)
2021-02-22T23:48:38.117 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/misc.py", line 546, in sudo_write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    remote.sudo_write_file(path, data, mode=perms, owner=owner)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 600, in sudo_write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self.write_file(path, data, sudo=True, **kwargs)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 594, in write_file
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self.run(args=args, stdin=data, quiet=True)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/remote.py", line 215, in run
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 455, in run
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    r.wait()
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 161, in wait
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    self._raise_for_status()
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:  File "/home/teuthworker/src/git.ceph.com_git_teuthology_9a6b849c7975e1b07e5495aaa5b961c0d22cad1c/teuthology/orchestra/run.py", line 183, in _raise_for_status
2021-02-22T23:48:38.118 INFO:tasks.cephfs_test_runner:    node=self.hostname, label=self.label
2021-02-22T23:48:38.119 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on gibba030 with status 1: 'set -ex\nsudo dd of=/tmp/tmp.ZEeZBasJer'

From: /ceph/teuthology-archive/teuthology-2021-02-21_03:15:03-fs-master-distro-basic-gibba/5899697/teuthology.log


Related issues

Related to teuthology - Bug #51169: qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp Won't Fix

History

#1 Updated by Rishabh Dave 9 months ago

just for record: same issue hit me yesterday night on vstart cluster too.

#2 Updated by Rishabh Dave 8 months ago

  • Status changed from New to In Progress

#3 Updated by Rishabh Dave 8 months ago

  • Description updated (diff)

Cause of the bug: Write was attempted on /tmp file with the root user. Files in /tmp can't be written by any user, including the root user, besides the file owner even when permission mode for the file is 777. This error doesn't occur everytime because RHEL uses older kernel version and Ubuntu uses newer kernel versions that enforces this policy.

I triggered a RHEL and Ubuntu job yesterday to confirm this -
RHEL job: http://pulpito.front.sepia.ceph.com/rishabh-2021-03-17_13:40:14-fs-wip-rishabh-vr-set-shell-to-True-distro-basic-smithi/#
Ubuntu job: http://pulpito.front.sepia.ceph.com/rishabh-2021-03-25_14:23:03-fs-wip-rishabh-vr-set-shell-to-True-distro-basic-smithi/

Job in bug description failed because it is a Ubuntu job.

The best fix is to add a method to teuthology.orchestra.remote.Remote. It would call teuthology.orchestra.remote.Remote.write_file() with sudo set to False and using that method instead of teuthology.misc.sudo_write_file.

#4 Updated by Rishabh Dave 8 months ago

  • Pull request ID set to 40431

#5 Updated by Kyrylo Shatskyy 8 months ago

Thanks Rishabh for your analysis.

The best fix is to add a method to teuthology.orchestra.remote.Remote. It would call teuthology.orchestra.remote.Remote.write_file() with sudo set to False and using that method instead of teuthology.misc.sudo_write_file.

The write_file is supposed to be used sudo defaults False, by design. Where the sudo_write_file is designed to always use sudo=True. So it should be just enough to use write_file, in other words, it just one-line fix. No need to add new API or extending others.

#6 Updated by Patrick Donnelly 8 months ago

  • Status changed from In Progress to Resolved

#7 Updated by Patrick Donnelly 6 months ago

  • Related to Bug #51169: qa: ubuntu 20.04 sys protections prevent multiuser file access in /tmp added

Also available in: Atom PDF