Project

General

Profile

Bug #8847

"Error initializing cluster client" in upgrade:firefly-firefly-testing-basic-plana suite

Added by Yuri Weinstein over 9 years ago. Updated over 9 years ago.

Status:
Can't reproduce
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-15_19:12:01-upgrade:firefly-firefly-testing-basic-plana/363981/

2014-07-15T22:54:57.394 DEBUG:teuthology.orchestra.run:Running [10.214.131.32]: 'sudo apt-key list | grep Ceph'
2014-07-15T22:54:57.447 DEBUG:teuthology.orchestra.run:Running [10.214.131.32]: 'lsb_release -sc'
2014-07-15T22:54:57.464 INFO:teuthology.orchestra.run.err:[10.214.131.14]: Error initializing cluster client: Error
2014-07-15T22:54:57.475 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_firefly/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_firefly/teuthology/task/rgw.py", line 511, in create_nonregion_pools
    create_replicated_pool(remote, data_pool, 64)
  File "/home/teuthworker/src/teuthology_firefly/teuthology/task_util/rados.py", line 40, in create_replicated_pool
    'ceph', 'osd', 'pool', 'create', name, str(pgnum), str(pgnum),
  File "/home/teuthworker/src/teuthology_firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/src/teuthology_firefly/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/src/teuthology_firefly/teuthology/orchestra/run.py", line 326, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.14 with status 1: 'ceph osd pool create .rgw.buckets 64 64'
archive_path: /var/lib/teuthworker/archive/teuthology-2014-07-15_19:12:01-upgrade:firefly-firefly-testing-basic-plana/363981
branch: firefly
description: upgrade:firefly/{0-cluster/start.yaml 1-install/v0.80.3.yaml 2-workload/s3tests.yaml
  3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml
  testrgw.yaml}}
email: ceph-qa@ceph.com
job_id: '363981'
kernel: &id001
  kdb: true
  sha1: 20bdcbcb52ac95397901581086368dd79c06091f
last_in_suite: false
machine_type: plana
name: teuthology-2014-07-15_19:12:01-upgrade:firefly-firefly-testing-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    fs: xfs
    log-whitelist:
    - slow request
    - scrub
    - wrongly marked me down
    - objects unfound and apparently lost
    - log bound mismatch
    sha1: 67b5193f73a2c9ec9e503ad3431473998217375d
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 67b5193f73a2c9ec9e503ad3431473998217375d
  s3tests:
    branch: firefly
  workunit:
    sha1: 67b5193f73a2c9ec9e503ad3431473998217375d
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mon.c
  - osd.3
  - osd.4
  - osd.5
  - client.0
suite: upgrade:firefly
suite_branch: firefly
targets:
  ubuntu@plana08.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCmrPbNrKBRrF2sFfhWGBXBrxP1c9wi06uF6USYU0ubQXxm+xTpudp52IT+VnIL12Lnkg/552C6VJEQhZbWyH1Y/0Udx6lkzW+jedLgQVjeK8gPVxRl3/xtP0C5b9Spao/iX8EiBZ1ijq+CIb6hAej90nEUrfh8dBrpSwX3d2b5ECpkjypoboF7OOWYOEOtUrxVxXzTC6VysQeVoJh15u3lMa1otYbOvlFvdeFE5fQqfg7yPsQqX48CptUT3H6UZXMfpXY5axu69Wqhpj4wQdAFGWW6oZtGeRYyUyowK+aFJRWQKHXIZGi4KOvH8Bgf87u+BOg/t3bFJGuUUZYokOPJ
  ubuntu@plana26.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7ggZ7aqae7QJjZ2+KyYSYys2vhqpMkjksuRNBQ+pjdzcHSSbxSRtDxGyZcucG0e3K0O2KJUkco/38peR9dU4iyL0oEdAEs6+391mBDBd5PmLhGeINaIpE6Q9TUGh7jMVdNqsKX03K8cmyd0ryR+QGJZC8aKlRCo0ZsbY9Pb5/tML1zDy/9V/n9bHxvqxSdh8LGJAOzt6+zxMKbLyTULT89lPni/uxB+dmErEDCzG3LG3BghU+t1LjIKSL5N6bhJqBztdGjctX6as4mpmatbwkYowCrJH3PV7DVy3MuXDxl93yLyrm7TiVjb3KjhT9RcfHVq23M4otok9OVV0AXvNV
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
    tag: v0.80.3
- parallel:
  - workload
  - upgrade-sequence
- sequential:
  - mon_thrash:
      revive_delay: 20
      thrash_delay: 1
  - ceph-fuse: null
  - workunit:
      clients:
        client.0:
        - suites/dbench.sh
- sequential:
  - thrashosds:
      chance_pgnum_grow: 1
      chance_pgpnum_fix: 1
      timeout: 1200
  - ceph-fuse: null
  - workunit:
      clients:
        all:
        - suites/iogen.sh
- sequential:
  - rgw:
    - client.0
  - s3tests:
      client.0:
        rgw_server: client.0
teuthology_branch: firefly
tube: plana
upgrade-sequence:
  sequential:
  - install.upgrade:
      all:
        branch: firefly
  - ceph.restart:
    - mon.a
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.b
  - sleep:
      duration: 60
  - ceph.restart:
    - mon.c
  - sleep:
      duration: 60
  - ceph.restart:
    - mds.a
  - sleep:
      duration: 60
  - ceph.restart:
    - osd.0
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.1
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.2
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.3
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.4
  - sleep:
      duration: 30
  - ceph.restart:
    - osd.5
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.4976
workload:
  sequential:
  - rgw:
    - client.0
  - s3tests:
      client.0:
        force-branch: firefly-original
        rgw_server: client.0
description: upgrade:firefly/{0-cluster/start.yaml 1-install/v0.80.3.yaml 2-workload/s3tests.yaml
  3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml
  testrgw.yaml}}
duration: 163.7384009361267
failure_reason: 'Command failed on 10.214.131.14 with status 1: ''ceph osd pool create
  .rgw.buckets 64 64'''
flavor: basic
mon.a-kernel-sha1: 20bdcbcb52ac95397901581086368dd79c06091f
mon.b-kernel-sha1: 20bdcbcb52ac95397901581086368dd79c06091f
owner: scheduled_teuthology@teuthology
success: false

History

#1 Updated by Yuri Weinstein over 9 years ago

  • Project changed from Ceph to rgw

#2 Updated by Sage Weil over 9 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF