Bug #6776: nightly failure: timed out waiting for admin_socket after osd restarted - Ceph - Ceph

Actions

Copy link

Bug #6776

closed

nightly failure: timed out waiting for admin_socket after osd restarted

Added by Tamilarasi muthamizhan over 10 years ago. Updated over 10 years ago.

Status:

Duplicate

Priority:

Urgent

Assignee:

Category:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

logs: ubuntu@teuthology:/a/teuthology-2013-11-13_14:42:07-upgrade-parallel-next-testing-basic-vps/97235

2013-11-13T18:52:14.267 INFO:teuthology.task.thrashosds:joining thrashosds
2013-11-13T18:52:14.267 ERROR:teuthology.run_tasks:Manager failed: <contextlib.GeneratorContextManager object at 0x289e890>
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-master/teuthology/run_tasks.py", line 82, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/home/teuthworker/teuthology-master/teuthology/task/thrashosds.py", line 170, in task
    thrash_proc.do_join()
  File "/home/teuthworker/teuthology-master/teuthology/task/ceph_manager.py", line 105, in do_join
    self.thread.get()
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 308, in get
    raise self._exception
Exception: timed out waiting for admin_socket to appear after osd.4 restart

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Tamilarasi muthamizhan over 10 years ago

ubuntu@teuthology:/a/teuthology-2013-11-13_14:42:07-upgrade-parallel-next-testing-basic-vps/97235$ cat config.yaml 
archive_path: /var/lib/teuthworker/archive/teuthology-2013-11-13_14:42:07-upgrade-parallel-next-testing-basic-vps/97235
description: upgrade-parallel/stress-split/{0-cluster/start.yaml 1-dumpling-install/dumpling.yaml
  2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/more.yaml 5-workload/readwrite.yaml
  6-next-mon/monb.yaml 7-workload/rados_api_tests.yaml distro/fedora_18.yaml}
email: null
job_id: '97235'
kernel:
  kdb: true
  sha1: 68174f0c97e7c0561aa844059569e3cbf0a43de1
last_in_suite: false
machine_type: vps
name: teuthology-2013-11-13_14:42:07-upgrade-parallel-next-testing-basic-vps
nuke-on-error: true
os_type: fedora
os_version: '18'
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug ms: 1
        debug osd: 5
    log-whitelist:
    - slow request
    - wrongly marked me down
    - objects unfound and apparently lost
    sha1: aef3378bd721ff4b73ad3a7a8b07e5f6e2e578f8
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: aef3378bd721ff4b73ad3a7a8b07e5f6e2e578f8
  s3tests:
    branch: master
  workunit:
    sha1: aef3378bd721ff4b73ad3a7a8b07e5f6e2e578f8
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.b
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - osd.3
  - osd.4
  - osd.5
  - client.0
  - mon.c
targets:
  ubuntu@vpm082.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVCEQbPYoHi4qj0r8HI59TFbSCBQSREhXi6DgQ9EwG/94tLnmfMwRUBzWbTN/HkyWEU+M/2bXZhjshUa/LttBPbrfn0CLYzijs5ZrkWDhtMzZ5c0zyOM8DzJhy2IJ1sBMOI47T2BlihJBRvX/tqix2wcimTb7feD5p4uO+I0oeDZHZlYoJ5ZG8B/DAuac7hf4zzDMG1kl88kbIDOXpt4nW0o7nuMjtLir5xSx1auHB7l0HDSTw9X6BF6ianCWzQ20UK0yVlctMmzOcnRujv9OeCOkcVag3mPLItuoMgANyQrVgUlYbewE/P8uSvkcFBRc56NSXUNkBHL+YuOIZyJOP
  ubuntu@vpm083.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDWVdt6WcEsB3nxxSYwFb1cm2O1TOpO6jIt/M5gvNhxRy/TpwjbtlvqQEZVoRg1sFGufBuqO2YSM76Bcvg3N0LtC3NgYi4CrRsQsj8s4tZn3s4bLQjmtbyWEnIXMAhYB/HFATsBo7SSKUKyEXmVMd/JTppFstue3XKeZZWS2N+VNxniqxN8P4oeQbkcdld9itdg7n5Kkqsc8gSEtB0dQ3dMxYpG0WC7s4GgOe/NpVwUZvCuk6vKHiYq1x0ivF/bbk/+ynvpSFeGE52dQZhsSYI0kEKBOvh5PAyK/fz3yEfkE1N0myEzn3CiRuSVX6wY4MIaBHig9H5eLotJaNVVF2O1
tasks:
- internal.lock_machines:
  - 2
  - vps
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    branch: dumpling
- ceph: null
- install.upgrade:
    osd.0: null
- ceph.restart:
    daemons:
    - osd.0
    - osd.1
    - osd.2
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- ceph.restart:
    daemons:
    - mon.a
    wait-for-healthy: false
    wait-for-osds-up: true
- rados:
    clients:
    - client.0
    objects: 500
    op_weights:
      delete: 10
      read: 45
      write: 45
    ops: 4000
- ceph.restart:
    daemons:
    - mon.b
    wait-for-healthy: false
    wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
  - a
  - b
- workunit:
    branch: dumpling
    clients:
      client.0:
      - rados/test.sh
teuthology_branch: master
verbose: true

Actions

Copy link