Project

General

Profile

Bug #44399

qa: /usr/libexec/platform-python fails / core dumps

Added by Patrick Donnelly 9 months ago. Updated 3 months ago.

Status:
Triaged
Priority:
Normal
Assignee:
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

2020-03-03T16:28:22.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2020-03-03T16:28:22.320 INFO:teuthology.task.internal:Transferring archived files...
2020-03-03T16:28:22.320 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063
2020-03-03T16:28:22.320 INFO:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
2020-03-03T16:28:24.154 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2020-03-03T16:28:24.193 INFO:teuthology.orchestra.run.smithi063:> which /usr/libexec/platform-python
2020-03-03T16:28:24.232 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/libexec/platform-python
2020-03-03T16:28:24.289 DEBUG:teuthology.orchestra.remote:smithi063:/usr/libexec/platform-python is 12KB
2020-03-03T16:28:24.335 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 363, in archive
    fetch_binaries_for_coredumps(path, rem)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
    dump_program))
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 320, in _sftp_get_file
    sftp.get(remote_path, local_path)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 801, in get
    with open(localpath, "wb") as fl:
IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'
2020-03-03T16:28:24.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2020-03-03T16:28:24.360 INFO:teuthology.task.internal:Not uploading archives.
2020-03-03T16:28:24.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2020-03-03T16:28:24.374 INFO:teuthology.task.internal:Tidying up after the test...
2020-03-03T16:28:24.374 INFO:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.381 INFO:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.384 INFO:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout:   262149      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Mar  3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout:   262150      4 drwxr-xr-x   5  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout:   262170      4 drwxrwxr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout:   262174     92 -rw-------   1  ubuntu   ubuntu      91917 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout:   262152      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout:   262154      4 -rw-rw-rw-   1  ubuntu   ubuntu        678 Mar  3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout:   262172     68 -rw-rw-rw-   1  ubuntu   ubuntu      65894 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout:   262151      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Mar  3 15:54 /home/ubuntu/cephtest/archive/coredump
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout:   262159  47596 -rw-------   1  root     root     57581568 Mar  3 15:54 /home/ubuntu/cephtest/archive/coredump/1583250867.14575.core
2020-03-03T16:28:24.398 INFO:teuthology.orchestra.run.smithi063.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout:   262153      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Mar  3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout:   262154      4 drwxr-xr-x   4  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout:   262170      4 drwxrwxr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout:   262174    100 -rw-------   1  ubuntu   ubuntu     100166 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout:   262156      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout:   262158      4 -rw-rw-rw-   1  ubuntu   ubuntu        686 Mar  3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout:   262172     72 -rw-rw-rw-   1  ubuntu   ubuntu      69870 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.404 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout:   393248      4 drwxr-xr-x   3  ubuntu   ubuntu       4096 Mar  3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout:   393249      4 drwxr-xr-x   4  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout:   524332      4 drwxrwxr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout:   524609     92 -rw-------   1  ubuntu   ubuntu      90692 Mar  3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout:   393251      4 drwxr-xr-x   2  ubuntu   ubuntu       4096 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout:   393237      4 -rw-rw-rw-   1  ubuntu   ubuntu       2921 Mar  3 16:02 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout:   393236     68 -rw-rw-rw-   1  ubuntu   ubuntu      66499 Mar  3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-03-03T16:28:24.409 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 49, in base
    wait=False,
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 457, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
    node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'
2020-03-03T16:28:24.411 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2020-03-03T16:28:24.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2020-03-03T16:28:24.443 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'

From: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/teuthology.log

Core: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063/coredump/1583250867.14575.core

./remote/smithi063/coredump/1583250867.14575.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/platform-python /usr/bin/dnf makecache --timer'

History

#1 Updated by Patrick Donnelly 9 months ago

  • Subject changed from qa: to qa: /usr/libexec/platform-python fails / core dumps

#2 Updated by Brad Hubbard 9 months ago

  • Status changed from New to Triaged
  • Assignee set to Brad Hubbard
  • Priority changed from High to Normal

https://bugs.centos.org/view.php?id=17116

Patrick, How many times have you seen this?

#4 Updated by Brad Hubbard 9 months ago

Patrick Donnelly wrote:

Brad Hubbard wrote:

https://bugs.centos.org/view.php?id=17116

Patrick, How many times have you seen this?

A few times. Here's the most recent one:

http://pulpito.ceph.com/pdonnell-2020-03-04_14:32:48-multimds-wip-pdonnell-testing-20200303.175301-distro-basic-smithi/4825249/

OK, hopefully this is a bad file being pulled down and that gets resolved but, if it persists, we might need to try inserting a 'dnf downgrade dnf' into the process somewhere.

#6 Updated by Deepika Upadhyay 3 months ago

/a/yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379176/

#7 Updated by Brad Hubbard 3 months ago

Deepika, /a/yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379176/ is actually https://tracker.ceph.com/issues/46405 so I'm adding it there. /a/ideepika-2020-07-07_12:58:57-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5206719 appears to be https://tracker.ceph.com/issues/46054 although that should be resolved. Can you confirm the branch used included https://github.com/ceph/ceph/pull/35811 ?

#8 Updated by Deepika Upadhyay 3 months ago

umm, yes Brad, I rechecked seems like pr:35811 was included in my batch analysis, thanks for verifying my analysis, might have missed something.

Also available in: Atom PDF