Project

General

Profile

Actions
Copy link

Bug #53206

closed

Found coredumps on ubuntu@smithi115.front.sepia.ceph.com | IndexError: list index out of range

Added by Laura Flores over 2 years ago. Updated 11 months ago.

Status:
Resolved
Priority:
High
Assignee:
Kamoltat (Junior) Sirivadhna
Category:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

This error has occurred in some master and batch runs.

2021-11-06T21:37:59.718 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump
2021-11-06T21:37:59.731 DEBUG:teuthology.orchestra.run.smithi115:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2021-11-06T21:37:59.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump
2021-11-06T21:37:59.757 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = core
2021-11-06T21:37:59.780 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core
2021-11-06T21:37:59.793 DEBUG:teuthology.orchestra.run.smithi115:> test -e /home/ubuntu/cephtest/archive/coredump
2021-11-06T21:37:59.826 WARNING:teuthology.task.internal:Found coredumps on ubuntu@smithi115.front.sepia.ceph.com, flagging run as failed
2021-11-06T21:37:59.827 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump
2021-11-06T21:37:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1
2021-11-06T21:37:59.848 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive
2021-11-06T21:37:59.862 INFO:teuthology.task.internal:Transferring archived files...
2021-11-06T21:37:59.864 DEBUG:teuthology.misc:Transferring archived files from smithi115:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2021-11-06_17:01:58-rados-wip-yuri-testing-master-11.5.21-distro-basic-smithi/6488712/remote/smithi115
2021-11-06T21:37:59.871 DEBUG:teuthology.orchestra.run.smithi115:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- .
2021-11-06T21:38:12.226 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2021-11-06T21:38:12.258 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/run_tasks.py", line 176, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/task/internal/__init__.py", line 398, in archive
    fetch_binaries_for_coredumps(path, rem)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_27954452159076fda5642b6c0eb0c4998b99d2e4/teuthology/task/internal/__init__.py", line 320, in fetch_binaries_for_coredumps
    dump_program = dump_out.split("from '")[1].split(' ')[0]
IndexError: list index out of range

master example: /yuriw-2021-11-06_17:01:58-rados-wip-yuri-testing-master-11.5.21-distro-basic-smithi/6488712/
batch example: /yuriw-2021-11-03_15:48:02-rados-wip-yuri7-testing-2021-11-01-1748-distro-basic-smithi/6481624/

Related issues 1 (1 open — 0 closed)

Related to teuthology - Bug #61586: fetch_binaries_for_coredumps() attempts to run "which" in the entire command with argumentsNewKamoltat (Junior) Sirivadhna

Actions
Actions
Copy link
 #1

Updated by Laura Flores over 2 years ago

  • Source set to Q/A

Affected versions: master, pacific

master example: /yuriw-2021-11-06_17:01:58-rados-wip-yuri-testing-master-11.5.21-distro-basic-smithi/6488712/
pacific example: /yuriw-2021-11-02_19:49:55-rados-wip-yuri8-testing-2021-11-02-1009-pacific-distro-basic-smithi/6479111/

Actions
Copy link
 #2

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

  • Assignee set to Kamoltat (Junior) Sirivadhna
Actions
Copy link
 #3

Updated by Ilya Dryomov about 2 years ago

  • Priority changed from Normal to High
2022-02-24T00:55:35.550 INFO:teuthology.task.internal:Transferring binaries for coredumps...
2022-02-24T00:55:35.586 INFO:teuthology.task.internal: core looks like: /home/teuthworker/archive/dis-2022-02-23_21:10:50-rbd-wip-rbd-quincy-batch-4-distro-default-smithi/6703486/remote/smithi133/coredump/1645664031.59278.core.gz: gzip compressed data, was "1645664031.59278.core", last modified: Thu Feb 24 00:55:10 2022, max compression

2022-02-24T00:55:35.587 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b607b46862a6b148fb1608e09b31d367f094555/teuthology/run_tasks.py", line 176, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b607b46862a6b148fb1608e09b31d367f094555/teuthology/task/internal/__init__.py", line 399, in archive
    fetch_binaries_for_coredumps(path, rem)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b607b46862a6b148fb1608e09b31d367f094555/teuthology/task/internal/__init__.py", line 321, in fetch_binaries_for_coredumps
    dump_program = dump_out.split("from '")[1].split(' ')[0]
IndexError: list index out of range

http://qa-proxy.ceph.com/teuthology/dis-2022-02-23_21:10:50-rbd-wip-rbd-quincy-batch-4-distro-default-smithi/6703486/teuthology.log

Actions
Copy link
 #4

Updated by Laura Flores about 2 years ago

  • Backport set to quincy, pacific
Actions
Copy link
 #5

Updated by Ilya Dryomov about 2 years ago

  • Backport deleted (quincy, pacific)

This is a bug in teuthology, no backports here ;)

Actions
Copy link
 #6

Updated by Laura Flores about 2 years ago

Whoops, thanks Ilya

Actions
Copy link
 #7

Updated by Kamoltat (Junior) Sirivadhna about 2 years ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #8

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

/a/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943763/

core looks like: /home/teuthworker/archive/yuriw-2022-07-22_03:30:40-rados-wip-yuri3-testing-2022-07-21-1604-distro-default-smithi/6943763/remote/smithi083/coredump/1658515951.94790.core: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from 'ceph_test_rados_api_io', real uid: 1000, effective uid: 1000, real gid: 1265, effective gid: 1265, execfn: '/usr/bin/ceph_test_rados_api_io', platform: 'x86_64'

2022-07-22T19:11:45.927 DEBUG:teuthology.orchestra.run.smithi083:> which 'ceph_test_rados_api_io'"'"','
2022-07-22T19:11:45.953 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-07-22T19:11:45.954 INFO:teuthology.orchestra.run.smithi083.stderr:/usr/bin/which: no ceph_test_rados_api_io', in (/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
2022-07-22T19:11:45.956 ERROR:teuthology.run_tasks:Manager failed: internal.archive
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/run_tasks.py", line 188, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/task/internal/__init__.py", line 399, in archive
    fetch_binaries_for_coredumps(path, rem)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/task/internal/__init__.py", line 324, in fetch_binaries_for_coredumps
    remote_path = remote.sh(['which', dump_program]).rstrip()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/remote.py", line 96, in sh
    proc = self.run(**kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/remote.py", line 510, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 455, in run
    r.wait()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_8d598431210977f8caccec83230b4bfec7bd5d3f/teuthology/orchestra/run.py", line 183, in _raise_for_status
    node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi083 with status 1: 'which \'ceph_test_rados_api_io\'"\'"\',\''

Actions
Copy link
 #9

Updated by Kamoltat (Junior) Sirivadhna 11 months ago

  • Status changed from Fix Under Review to Resolved
Actions
Copy link
 #10

Updated by Ilya Dryomov 11 months ago

  • Related to Bug #61586: fetch_binaries_for_coredumps() attempts to run "which" in the entire command with arguments added
Actions
Copy link

Also available in: Atom PDF