Actions
Bug #44399
openqa: /usr/libexec/platform-python fails / core dumps
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
2020-03-03T16:28:22.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2020-03-03T16:28:22.320 INFO:teuthology.task.internal:Transferring archived files...
2020-03-03T16:28:22.320 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063
2020-03-03T16:28:22.320 INFO:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
2020-03-03T16:28:24.154 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2020-03-03T16:28:24.193 INFO:teuthology.orchestra.run.smithi063:> which /usr/libexec/platform-python
2020-03-03T16:28:24.232 INFO:teuthology.orchestra.run.smithi063.stdout:/usr/libexec/platform-python
2020-03-03T16:28:24.289 DEBUG:teuthology.orchestra.remote:smithi063:/usr/libexec/platform-python is 12KB
2020-03-03T16:28:24.335 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
self.gen.next()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 363, in archive
fetch_binaries_for_coredumps(path, rem)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 318, in fetch_binaries_for_coredumps
dump_program))
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 320, in _sftp_get_file
sftp.get(remote_path, local_path)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/sftp_client.py", line 801, in get
with open(localpath, "wb") as fl:
IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'
2020-03-03T16:28:24.346 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload
2020-03-03T16:28:24.360 INFO:teuthology.task.internal:Not uploading archives.
2020-03-03T16:28:24.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.base
2020-03-03T16:28:24.374 INFO:teuthology.task.internal:Tidying up after the test...
2020-03-03T16:28:24.374 INFO:teuthology.orchestra.run.smithi040:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.381 INFO:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.384 INFO:teuthology.orchestra.run.smithi099:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262149 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262150 4 drwxr-xr-x 5 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262170 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262174 92 -rw------- 1 ubuntu ubuntu 91917 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.396 INFO:teuthology.orchestra.run.smithi063.stdout: 262152 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262154 4 -rw-rw-rw- 1 ubuntu ubuntu 678 Mar 3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262172 68 -rw-rw-rw- 1 ubuntu ubuntu 65894 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262151 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 15:54 /home/ubuntu/cephtest/archive/coredump
2020-03-03T16:28:24.397 INFO:teuthology.orchestra.run.smithi063.stdout: 262159 47596 -rw------- 1 root root 57581568 Mar 3 15:54 /home/ubuntu/cephtest/archive/coredump/1583250867.14575.core
2020-03-03T16:28:24.398 INFO:teuthology.orchestra.run.smithi063.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262153 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262154 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.401 INFO:teuthology.orchestra.run.smithi099.stdout: 262170 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262174 100 -rw------- 1 ubuntu ubuntu 100166 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262156 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262158 4 -rw-rw-rw- 1 ubuntu ubuntu 686 Mar 3 16:19 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.402 INFO:teuthology.orchestra.run.smithi099.stdout: 262172 72 -rw-rw-rw- 1 ubuntu ubuntu 69870 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.404 INFO:teuthology.orchestra.run.smithi099.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 393248 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 3 16:26 /home/ubuntu/cephtest
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 393249 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 524332 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit
2020-03-03T16:28:24.407 INFO:teuthology.orchestra.run.smithi040.stdout: 524609 92 -rw------- 1 ubuntu ubuntu 90692 Mar 3 16:28 /home/ubuntu/cephtest/archive/audit/audit.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393251 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393237 4 -rw-rw-rw- 1 ubuntu ubuntu 2921 Mar 3 16:02 /home/ubuntu/cephtest/archive/syslog/kern.log.gz
2020-03-03T16:28:24.408 INFO:teuthology.orchestra.run.smithi040.stdout: 393236 68 -rw-rw-rw- 1 ubuntu ubuntu 66499 Mar 3 16:28 /home/ubuntu/cephtest/archive/syslog/misc.log.gz
2020-03-03T16:28:24.409 INFO:teuthology.orchestra.run.smithi040.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2020-03-03T16:28:24.409 DEBUG:teuthology.orchestra.run:got remote process result: 1
2020-03-03T16:28:24.409 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/run_tasks.py", line 159, in run_tasks
suppress = manager.__exit__(*exc_info)
File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
self.gen.throw(type, value, traceback)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/internal/__init__.py", line 49, in base
wait=False,
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 457, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 158, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 180, in _raise_for_status
node=self.hostname, label=self.label
CommandFailedError: Command failed on smithi040 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'
2020-03-03T16:28:24.411 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2020-03-03T16:28:24.427 DEBUG:teuthology.run_tasks:Unwinding manager internal.lock_machines
2020-03-03T16:28:24.443 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: IOError: [Errno 13] Permission denied: '/usr/libexec/platform-python'
From: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/teuthology.log
Core: /ceph/teuthology-archive/pdonnell-2020-03-03_14:22:02-kcephfs-master-distro-basic-smithi/4820744/remote/smithi063/coredump/1583250867.14575.core
./remote/smithi063/coredump/1583250867.14575.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/libexec/platform-python /usr/bin/dnf makecache --timer'
Updated by Patrick Donnelly about 4 years ago
- Subject changed from qa: to qa: /usr/libexec/platform-python fails / core dumps
Updated by Brad Hubbard about 4 years ago
- Status changed from New to Triaged
- Assignee set to Brad Hubbard
- Priority changed from High to Normal
https://bugs.centos.org/view.php?id=17116
Patrick, How many times have you seen this?
Updated by Patrick Donnelly about 4 years ago
Brad Hubbard wrote:
https://bugs.centos.org/view.php?id=17116
Patrick, How many times have you seen this?
A few times. Here's the most recent one:
Updated by Brad Hubbard about 4 years ago
Patrick Donnelly wrote:
Brad Hubbard wrote:
https://bugs.centos.org/view.php?id=17116
Patrick, How many times have you seen this?
A few times. Here's the most recent one:
OK, hopefully this is a bad file being pulled down and that gets resolved but, if it persists, we might need to try inserting a 'dnf downgrade dnf' into the process somewhere.
Updated by Deepika Upadhyay almost 4 years ago
Updated by Deepika Upadhyay over 3 years ago
/a/yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379176/
Updated by Brad Hubbard over 3 years ago
Deepika, /a/yuriw-2020-08-27_00:49:53-rados-wip-yuri8-testing-2020-08-26-2329-octopus-distro-basic-smithi/5379176/ is actually https://tracker.ceph.com/issues/46405 so I'm adding it there. /a/ideepika-2020-07-07_12:58:57-rados-wip-deepika-testing-2020-06-25-2058-distro-basic-smithi/5206719 appears to be https://tracker.ceph.com/issues/46054 although that should be resolved. Can you confirm the branch used included https://github.com/ceph/ceph/pull/35811 ?
Updated by Deepika Upadhyay over 3 years ago
umm, yes Brad, I rechecked seems like pr:35811 was included in my batch analysis, thanks for verifying my analysis, might have missed something.
Actions