Bug #7906
"adjust-ulimits ... --rgw-region zero'''" filed in rgw-firefly-distro-basic-plana suite
Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana/149644/
2014-03-28T14:22:52.377 INFO:teuthology.task_util.rgw:rgwadmin: client.1 : ['-n', 'client.1', 'region', 'default', '--rgw-region', 'zero'] 2014-03-28T14:22:52.377 INFO:teuthology.task_util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '-n', 'client.1', 'region', 'default', '--rgw-region', 'zero'] 2014-03-28T14:22:52.377 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero' 2014-03-28T14:22:52.772 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 564, in configure_regions_and_zones check_status=True) File "/home/teuthworker/teuthology-firefly/teuthology/task_util/rgw.py", line 38, in rgwadmin stdin=stdin, File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run r = self._runner(client=self.ssh, **kwargs) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run r.exitstatus = _check_status(r.exitstatus) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status raise CommandFailedError(command=r.command, exitstatus=status, node=host) CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero' 2014-03-28T14:22:52.850 INFO:teuthology.task.rgw:Cleaning up apache directories... 2014-03-28T14:22:52.850 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'rm -rf /home/ubuntu/cephtest/apache/tmp.client.1 && rmdir /home/ubuntu/cephtest/apache/htdocs.client.1' 2014-03-28T14:22:52.863 DEBUG:teuthology.orchestra.run:Running [10.214.131.13]: 'rm -rf /home/ubuntu/cephtest/apache/tmp.client.0 && rmdir /home/ubuntu/cephtest/apache/htdocs.client.0' 2014-03-28T14:22:52.883 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'rmdir /home/ubuntu/cephtest/apache' 2014-03-28T14:22:52.920 DEBUG:teuthology.orchestra.run:Running [10.214.131.13]: 'rmdir /home/ubuntu/cephtest/apache' 2014-03-28T14:22:52.929 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 33, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 725, in task lambda: start_apache(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/rgw.py", line 564, in configure_regions_and_zones check_status=True) File "/home/teuthworker/teuthology-firefly/teuthology/task_util/rgw.py", line 38, in rgwadmin stdin=stdin, File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run r = self._runner(client=self.ssh, **kwargs) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 330, in run r.exitstatus = _check_status(r.exitstatus) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 326, in _check_status raise CommandFailedError(command=r.command, exitstatus=status, node=host) CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero' 2014-03-28T14:22:53.052 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/inktank/teuthology/search?q=aa7220b1a0504a67b38ddc4f6bdb8d73 CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero' 2014-03-28T14:22:53.052 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2014-03-28T14:22:53.052 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json' 2014-03-28T14:22:53.159 ERROR:sentry.errors:Unable to reach Sentry log server: HTTP Error 502: Bad Gateway (url: http://sentry.ceph.com/api/2/store/, body: <html> <head><title>502 Bad Gateway</title></head> <body bgcolor="white"> <center><h1>502 Bad Gateway</h1></center> <hr><center>nginx/1.1.19</center> </body> </html> ) Traceback (most recent call last): File "/var/lib/teuthworker/.local/lib/python2.7/site-packages/raven/transport/threaded.py", line 110, in send_sync super(ThreadedHTTPTransport, self).send(data, headers) File "/var/lib/teuthworker/.local/lib/python2.7/site-packages/raven/transport/http.py", line 40, in send response = urlopen(req, data, self.timeout).read() File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 406, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 519, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 444, in error return self._call_chain(*args) File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain result = func(*args) File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) HTTPError: HTTP Error 502: Bad Gateway 2014-03-28T14:22:53.161 ERROR:sentry.errors:Failed to submit message: u"CommandFailedError: Command failed on 10.214.132.30 with status 5: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero'" 2014-03-28T14:22:54.326 INFO:teuthology.orchestra.run.err:[10.214.132.30]: dumped all in format json 2014-03-28T14:22:54.366 INFO:teuthology.task.ceph:Waiting for all osds to be active and clean. 2014-03-28T14:23:04.348 DEBUG:teuthology.orchestra.run:Running [10.214.132.30]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph pg dump --format json'
archive_path: /var/lib/teuthworker/archive/teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana/149644 description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml} email: null job_id: '149644' kernel: &id001 kdb: true sha1: distro last_in_suite: false machine_type: plana name: teuthology-2014-03-27_23:00:17-rgw-firefly-distro-basic-plana nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: firefly ceph: conf: global: ms inject socket failures: 5000 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 osd op thread timeout: 60 osd sloppy crc: true fs: btrfs log-whitelist: - slow request sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: branch: dev: firefly conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7 rgw: ec-data-pool: false s3tests: branch: master workunit: sha1: c5682e78e9fb9d49c03ca983e0b4100696055eb7 owner: scheduled_teuthology@teuthology roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - client.0 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - client.1 targets: ubuntu@plana27.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBXl4mTKaVsBa9HTYX2cTy1UBVb5bdqBoUMZDvP/9fXGgf+qTVpxAAbSDrxmTLDeGhCVToM2MxAuvZ7V/E1aVDEaifgPylNJLQ0BApaOD02SKUAxPEYgkI0hWRNZ2e5AvVky6JR0wc7fxxR1NE/MbWO2SGLIIdN9oxFi0+uGRZal/vx56R6HTm1IUIC0tuGSPkiTUy1SnGIVMCtLiFeYUHueeki+3Ul2Pm8X5OMXpskiSTM+ikEQIO0VEw+vUkFjK5SLZ9UIHP0jQ+Jr/cOQhB1P1pBdoR7MnpFaOEr4RkJO/GzocBlmwHlllMIabL6jxhq7KxGS8GPffDZMAyBrI3 ubuntu@plana48.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBHhRwSkvAeRW6VrqAd9uejc+y4wydw26ndSiYjo1vv3WiQcGVKa9pNEuAYb/TRjmrTKHDO9lYl9fQwMEyY3BDNceKYJz43YyUHeNd8RNiAczRI4VTuMxJ6/GWv8n0t+24GkMjs88fnpDQ3xS4IdF5nJDzl9QnAzLFZbWGxt/h43rMQ4WOcAp57D9e7ZF/YtagB714HT2fEJfVnw6Q5EgPXnQ/xuxX9FolixiPNdP3iY998OXWS6PWAgbx1yw2t2CLIll7jXlDrBb/9fYIDa8qdV4JakzfQLGG/Ws3UnoP7Bx5V1edSLHHNmyXZ60H+82fVFe3woK+0jBcmL6BEObJ tasks: - internal.lock_machines: - 2 - plana - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: null - ceph: conf: client.0: rgw gc pool: .rgw.gc.0 rgw log data: true rgw log meta: true rgw region: zero rgw region root pool: .rgw.region.0 rgw user keys pool: .users.0 rgw user uid pool: .users.uid.0 rgw zone: r0z1 rgw zone root pool: .rgw.zone.0 client.1: rgw gc pool: .rgw.gc.1 rgw log data: false rgw log meta: false rgw region: one rgw region root pool: .rgw.region.1 rgw user keys pool: .users.1 rgw user uid pool: .users.uid.1 rgw zone: r1z1 rgw zone root pool: .rgw.zone.1 - rgw: client.0: system user: access key: 1te6NH5mcdcq0Tc5i8i2 name: client0-system-user secret key: 1y4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv valgrind: - --tool=memcheck client.1: system user: access key: 0te6NH5mcdcq0Tc5i8i2 name: client1-system-user secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv valgrind: - --tool=memcheck regions: one: api name: api1 is master: false master zone: r1z1 zones: - r1z1 zero: api name: api1 is master: true master zone: r0z1 zones: - r0z1 - radosgw-agent: client.0: dest: client.1 metadata-only: true src: client.0 - s3tests: client.0: rgw_server: client.0 teuthology_branch: firefly verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.16902
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml rgw_pool_type/replicated.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml} duration: 567.8836269378662 failure_reason: 'Command failed on 10.214.132.30 with status 5: ''adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 -n client.1 region default --rgw-region zero''' flavor: basic owner: scheduled_teuthology@teuthology sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=aa7220b1a0504a67b38ddc4f6bdb8d73 success: false