Project

General

Profile

Bug #7906

"adjust-ulimits ... --rgw-region zero'''" filed in rgw-firefly-distro-basic-plana suite

Added by Yuri Weinstein about 10 years ago. Updated about 10 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana/149644/

2014-03-28T14:22:52.377 INFO:teuthology.task_util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'region', 'default', '--rgw-region', 'zero']
2014-03-28T14:22:52.377 INFO:teuthology.task_util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'region', 'default', '--rgw-region', 'zero']
2014-03-28T14:22:52.377 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'
2014-03-28T14:22:52.772 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 564, in configure_regions_and_zones
    check_status=True)
  File "/home/teuthworker/teuthology-firefly/teuthology/task_util/rgw.py", line 38, in rgwadmin
    stdin=stdin,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'
2014-03-28T14:22:52.850 INFO:teuthology.task.rgw:Cleaning up apache directories...
2014-03-28T14:22:52.850 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'rm -rf /home/ubuntu/cephtest/apache/tmp.client.1 && rmdir /home/ubuntu/cephtest/apache/htdocs.client.1'
2014-03-28T14:22:52.863 DEBUG:teuthology.orchestra.run:Running [10.214.131.13]: 'rm -rf /home/ubuntu/cephtest/apache/tmp.client.0 && rmdir /home/ubuntu/cephtest/apache/htdocs.client.0'
2014-03-28T14:22:52.883 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'rmdir /home/ubuntu/cephtest/apache'
2014-03-28T14:22:52.920 DEBUG:teuthology.orchestra.run:Running [10.214.131.13]: 'rmdir /home/ubuntu/cephtest/apache'
2014-03-28T14:22:52.929 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 33, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 725, in task
    lambda: start_apache(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 27, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 564, in configure_regions_and_zones
    check_status=True)
  File "/home/teuthworker/teuthology-firefly/teuthology/task_util/rgw.py", line 38, in rgwadmin
    stdin=stdin,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'
2014-03-28T14:22:53.052 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/inktank/teuthology/search?q=aa7220b1a0504a67b38ddc4f6bdb8d73
CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'
2014-03-28T14:22:53.052 DEBUG:teuthology.run_tasks:Unwinding manager ceph
2014-03-28T14:22:53.052 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json'
2014-03-28T14:22:53.159 ERROR:sentry.errors:Unable to reach Sentry log server: HTTP Error 502: Bad Gateway (url: http://sentry.ceph.com/api/2/store/, body: <html>
<head><title>502 Bad Gateway</title></head>
<body bgcolor="white">
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.1.19</center>
</body>
</html>
)
Traceback (most recent call last):
  File "/var/lib/teuthworker/.local/lib/python2.7/site-packages/raven/transport/threaded.py", line 110, in send_sync
    super(ThreadedHTTPTransport, self).send(data, headers)
  File "/var/lib/teuthworker/.local/lib/python2.7/site-packages/raven/transport/http.py", line 40, in send
    response = urlopen(req, data, self.timeout).read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 406, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 444, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 502: Bad Gateway
2014-03-28T14:22:53.161 ERROR:sentry.errors:Failed to submit message: u"CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'" 
2014-03-28T14:22:54.326 INFO:teuthology.orchestra.run.err:[10.214.132.30]: dumped all in format json
2014-03-28T14:22:54.366 INFO:teuthology.task.ceph:Waiting for all osds to be active and clean.
2014-03-28T14:23:04.348 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json'
archive_path: /var/lib/teuthworker/archive/teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana/149644
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml
  rgw_pool_type/replicated.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml}
email: null
job_id: '149644'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: plana
name: teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
        osd op thread timeout: 60
        osd sloppy crc: true
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7
    valgrind:
      mds:
      - --tool=memcheck
      mon:
      - --tool=memcheck
      - --leak-check=full
      - --show-reachable=yes
      osd:
      - --tool=memcheck
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7
  rgw:
    ec-data-pool: false
  s3tests:
    branch: master
  workunit:
    sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
  - client.0
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.1
targets:
  ubuntu@plana27.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBXl4mTKaVsBa9HTYX2cTy1UBVb5bdqBoUMZDvP/9fXGgf+qTVpxAAbSDrxmTLDeGhCVToM2MxAuvZ7V/E1aVDEaifgPylNJLQ0BApaOD02SKUAxPEYgkI0hWRNZ2e5AvVky6JR0wc7fxxR1NE/MbWO2SGLIIdN9oxFi0+uGRZal/vx56R6HTm1IUIC0tuGSPkiTUy1SnGIVMCtLiFeYUHueeki+3Ul2Pm8X5OMXpskiSTM+ikEQIO0VEw+vUkFjK5SLZ9UIHP0jQ+Jr/cOQhB1P1pBdoR7MnpFaOEr4RkJO/GzocBlmwHlllMIabL6jxhq7KxGS8GPffDZMAyBrI3
  ubuntu@plana48.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBHhRwSkvAeRW6VrqAd9uejc+y4wydw26ndSiYjo1vv3WiQcGVKa9pNEuAYb/TRjmrTKHDO9lYl9fQwMEyY3BDNceKYJz43YyUHeNd8RNiAczRI4VTuMxJ6/GWv8n0t+24GkMjs88fnpDQ3xS4IdF5nJDzl9QnAzLFZbWGxt/h43rMQ4WOcAp57D9e7ZF/YtagB714HT2fEJfVnw6Q5EgPXnQ/xuxX9FolixiPNdP3iY998OXWS6PWAgbx1yw2t2CLIll7jXlDrBb/9fYIDa8qdV4JakzfQLGG/Ws3UnoP7Bx5V1edSLHHNmyXZ60H+82fVFe3woK+0jBcmL6BEObJ
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph:
    conf:
      client.0:
        rgw gc pool: .rgw.gc.0
        rgw log data: true
        rgw log meta: true
        rgw region: zero
        rgw region root pool: .rgw.region.0
        rgw user keys pool: .users.0
        rgw user uid pool: .users.uid.0
        rgw zone: r0z1
        rgw zone root pool: .rgw.zone.0
      client.1:
        rgw gc pool: .rgw.gc.1
        rgw log data: false
        rgw log meta: false
        rgw region: one
        rgw region root pool: .rgw.region.1
        rgw user keys pool: .users.1
        rgw user uid pool: .users.uid.1
        rgw zone: r1z1
        rgw zone root pool: .rgw.zone.1
- rgw:
    client.0:
      system user:
        access key: 1te6NH5mcdcq0Tc5i8i2
        name: client0-system-user
        secret key: 1y4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
      valgrind:
      - --tool=memcheck
    client.1:
      system user:
        access key: 0te6NH5mcdcq0Tc5i8i2
        name: client1-system-user
        secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
      valgrind:
      - --tool=memcheck
    regions:
      one:
        api name: api1
        is master: false
        master zone: r1z1
        zones:
        - r1z1
      zero:
        api name: api1
        is master: true
        master zone: r0z1
        zones:
        - r0z1
- radosgw-agent:
    client.0:
      dest: client.1
      metadata-only: true
      src: client.0
- s3tests:
    client.0:
      rgw_server: client.0
teuthology_branch: firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.16902
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml
  rgw_pool_type/replicated.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml}
duration: 567.8836269378662
failure_reason: 'Command failed on 10.214.132.30 with status 5: ''adjust-ulimits ceph-coverage
  /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json
  -n client.1 -n client.1 region default --rgw-region zero'''
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=aa7220b1a0504a67b38ddc4f6bdb8d73
success: false

History

#1 Updated by Yehuda Sadeh about 10 years ago

  • Status changed from New to Duplicate

Duplicates #7903

Also available in: Atom PDF