Project

General

Profile

Actions

Bug #7426

closed

Tests in rgw-dumpling-master-basic-plana fail, error "Couldn't init storage provider (RADOS)"

Added by Yuri Weinstein about 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/wusui-2014-02-13_12:23:01-rgw-dumpling-master-basic-plana/81039/

2014-02-13T16:29:19.246 INFO:teuthology.task.s3readwrite:Creating rgw users...
2014-02-13T16:29:19.246 DEBUG:teuthology.task.s3readwrite:creating user foo.client.0 on client.0
2014-02-13T16:29:19.247 DEBUG:teuthology.orchestra.run:Running [10.214.131.11]: "/home/ubuntu/cephtest/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key QULVBUZXGXMAEQEDIOUK --secret aCN5f+xesS+psCqOItanAywMEGpt8kxJXXj122CgIItCe1fCbhCYlw== --email foo.client.0+test@test.test" 
2014-02-13T16:29:20.515 INFO:teuthology.orchestra.run.err:[10.214.131.11]: 2014-02-13 16:29:20.514350 7ffb14cbd780  0 couldn't find old data placement pools config, setting up new ones for the zone
2014-02-13T16:29:20.573 INFO:teuthology.task.rgw.client.0.out:[10.214.131.11]: 2014-02-13 16:29:20.573195 7ff7aeeb6780 -1 error storing region info: (17) File exists
2014-02-13T16:29:20.574 INFO:teuthology.task.rgw.client.0.out:[10.214.131.11]: 2014-02-13 16:29:20.574251 7ff7aeeb6780 -1 failed reading region info from .rgw.root:region_info.default: (2) No such file or directory
2014-02-13T16:29:20.575 INFO:teuthology.task.rgw.client.0.out:[10.214.131.11]: 2014-02-13 16:29:20.574967 7ff7aeeb6780 -1 Couldn't init storage provider (RADOS)
2014-02-13T16:29:20.597 INFO:teuthology.orchestra.run.err:[10.214.131.11]: couldn't init storage provider
2014-02-13T16:29:20.599 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-dumpling/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/s3readwrite.py", line 116, in create_users
    '--email', s3tests_conf[section]['email'],
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/cluster.py", line 52, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/remote.py", line 47, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 267, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 263, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.11 with status 5: "/home/ubuntu/cephtest/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key QULVBUZXGXMAEQEDIOUK --secret aCN5f+xesS+psCqOItanAywMEGpt8kxJXXj122CgIItCe1fCbhCYlw== --email foo.client.0+test@test.test" 
2014-02-13T16:29:20.600 INFO:teuthology.task.s3readwrite:Removing s3-tests...
2014-02-13T16:29:20.600 DEBUG:teuthology.orchestra.run:Running [10.214.131.11]: 'rm -rf /home/ubuntu/cephtest/s3-tests'
2014-02-13T16:29:20.609 ERROR:teuthology.run_tasks:Saw exception from tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-dumpling/teuthology/run_tasks.py", line 27, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/s3readwrite.py", line 316, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/s3readwrite.py", line 116, in create_users
    '--email', s3tests_conf[section]['email'],
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/cluster.py", line 52, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/remote.py", line 47, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 267, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 263, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.11 with status 5: "/home/ubuntu/cephtest/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key QULVBUZXGXMAEQEDIOUK --secret aCN5f+xesS+psCqOItanAywMEGpt8kxJXXj122CgIItCe1fCbhCYlw== --email foo.client.0+test@test.test" 
2014-02-13T16:29:20.609 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x2b16250>
2014-02-13T16:29:20.609 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-dumpling/teuthology/contextutil.py", line 27, in nested
    yield vars
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/rgw.py", line 602, in task
    yield
  File "/home/teuthworker/teuthology-dumpling/teuthology/run_tasks.py", line 27, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/s3readwrite.py", line 316, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-dumpling/teuthology/task/s3readwrite.py", line 116, in create_users
    '--email', s3tests_conf[section]['email'],
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/cluster.py", line 52, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/remote.py", line 47, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 267, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-dumpling/teuthology/orchestra/run.py", line 263, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.11 with status 5: "/home/ubuntu/cephtest/adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user create --uid foo.client.0 --display-name 'Mr. foo.client.0' --access-key QULVBUZXGXMAEQEDIOUK --secret aCN5f+xesS+psCqOItanAywMEGpt8kxJXXj122CgIItCe1fCbhCYlw== --email foo.client.0+test@test.test" 
2014-02-13T16:29:20.610 INFO:teuthology.task.rgw:Stopping apache...
archive_path: /var/lib/teuthworker/archive/wusui-2014-02-13_12:23:01-rgw-dumpling-master-basic-plana/81039
description: rgw/multifs/{clusters/fixed-2.yaml fs/btrfs.yaml tasks/rgw_readwrite.yaml}
email: null
job_id: '81039'
kernel: &id001
  kdb: true
  sha1: 125d725c923527a85876c031028c7f55c28b74b3
last_in_suite: false
machine_type: plana
name: wusui-2014-02-13_12:23:01-rgw-dumpling-master-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: dumpling
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug ms: 1
        debug osd: 5
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 1c42a6e34bb8e318e001384270cda003add946a7
  ceph-deploy:
    branch:
      dev: dumpling
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 1c42a6e34bb8e318e001384270cda003add946a7
  s3tests:
    branch: dumpling
  workunit:
    sha1: 1c42a6e34bb8e318e001384270cda003add946a7
owner: scheduled_wusui@aardvark
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
targets:
  ubuntu@plana29.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6yHeLkt88spLGm1otJ4MM2SDrBJesgz1uoFxqD5WX+t9P7ge/qXAPvn/ySslAtaseWnsEdCbSXGN2GHplEgL0FB3qlb948qz1F5YM++QCMsXV9qql8dDPgW3P3S6AeHNyn+jS37TUDDJuxLO9i9C1A59/Edx6pSSQLQ+OlodC64LD00RQY2Qs/PE+q64NtJnwYbdre01b6MPWFzh68cm3c1p47rK7VDeEuermEM8D5gHuOeudW5TzzzE2YzFiyF9P9kfQSLZlMEEF++yqHXuUtWgQEN8YdGDfH/xhEi2Uj/sVIG6kqTJi/cPhHyXA9VuVExARJuXRM2gZSYeaimEh
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDX5qLQ05TWlUoX78wIXMpSqq+6+J3UEXLM4bo0IvoyJr6dl1EG1z+EEvhdKGGobvj+zX9+JEeu7g4lrFUiJpmDs3YuqS6ECyuQJFRHQHImR1u+r5w2hqKGyRqoW0yG82q6D+8VLNyRL1B1+gAfWP3Eva9IS5e9iLQqwmCkPXa8J2m9C+k7zzE80IoOUCOzUMMYmrrgiuMub2sN9UPFv5cK0S/suJiNcktg8YQI8XONRdf3LChudRdMOckhvCwH71H/fXPPxv9MOfs36ixCW8TdP0L+9+QvCrcA0p5MCl2OcZcoexCtM9G6rXzjQ1zcOdrpV6sV+XBwHXoNEqHo2JLd
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph: null
- rgw:
  - client.0
- s3readwrite:
    client.0:
      readwrite:
        bucket: rwtest
        duration: 300
        files:
          num: 10
          size: 2000
          stddev: 500
        readers: 10
        writers: 3
      rgw_server: client.0
teuthology_branch: dumpling
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.8946
description: rgw/multifs/{clusters/fixed-2.yaml fs/btrfs.yaml tasks/rgw_readwrite.yaml}
duration: 180.8530511856079
failure_reason: 'Command failed on 10.214.131.11 with status 5: "/home/ubuntu/cephtest/adjust-ulimits
  ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user
  create --uid foo.client.0 --display-name ''Mr. foo.client.0'' --access-key QULVBUZXGXMAEQEDIOUK
  --secret aCN5f+xesS+psCqOItanAywMEGpt8kxJXXj122CgIItCe1fCbhCYlw== --email foo.client.0+test@test.test"'
flavor: basic
mon.a-kernel-sha1: 125d725c923527a85876c031028c7f55c28b74b3
mon.b-kernel-sha1: 125d725c923527a85876c031028c7f55c28b74b3
owner: scheduled_wusui@aardvark
success: false
Actions #1

Updated by Yehuda Sadeh about 10 years ago

  • Status changed from New to Resolved

This was an issue with the osd, fixed by commit:2795339d3632df975486d51e69762f9f567fb43d.

Actions

Also available in: Atom PDF