Bug #7526
"ERROR:radosgw_agent.worker:syncing entries for shard 59" in rgw-firefly-distro-basic-plana suite
% Done:
0%
Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Logs are in qa-proxy.ceph.com/teuthology/teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana/98508
DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/log?lock&locker-id=plana32%3A6161&length=60&zone-id=r1z1&type=metadata&id=59 HTTP/1.1" 200 None 2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]: ERROR:radosgw_agent.worker:syncing entries for shard 59 failed 2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]: Traceback (most recent call last): 2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]: File
...
2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Host: plana32.front.sepia.ceph.com:7280 2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Port: 7280 2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Params: {} 2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Token: None 2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: StringToSign: 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: GET 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Mon, 24 Feb 2014 05:06:27 GMT 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: /test-client.0-ehifidx0qn23xwi-234/testobj-dest 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Signature: 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: AWS RTCSXBSATCKJJPZJORHC:rezIj5jT5Cf5Ltmt13YxnUBd5tI= 2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: --------------------- >> end captured logging << --------------------- 2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: ---------------------------------------------------------------------- 2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Ran 285 tests in 2301.213s 2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: FAILED (failures=1) 2014-02-23T21:20:44.404 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 25, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 305, in run_tests args=args, File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/cluster.py", line 61, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run r = self._runner(client=self.ssh, **kwargs) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run r.exitstatus = _check_status(r.exitstatus) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status raise CommandFailedError(command=r.command, exitstatus=status, node=host) CommandFailedError: Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'" 2014-02-23T21:20:44.404 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data' 2014-02-23T21:20:48.675 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data' 2014-02-23T21:20:58.947 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23 21:20:58.947043 7f4c6defa700 0 -- 10.214.131.8:0/1007361 >> 10.214.131.8:6800/4334 pipe(0x123e910 sd=8 :40779 s=2 pgs=35 cs=1 l=1 c=0x123eb70).injecting socket failure 2014-02-23T21:21:00.876 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23 21:21:00.875119 7f4c6d6f2700 0 -- 10.214.131.8:0/1007361 >> 10.214.131.8:6805/4337 pipe(0x1247b50 sd=12 :56391 s=2 pgs=26 cs=1 l=1 c=0x1247db0).injecting socket failure 2014-02-23T21:21:21.507 INFO:teuthology.task.s3tests:Removing s3-tests... 2014-02-23T21:21:21.508 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'rm -rf /home/ubuntu/cephtest/s3-tests' 2014-02-23T21:21:21.581 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 33, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 399, in task lambda: run_tests(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 25, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 305, in run_tests args=args, File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/cluster.py", line 61, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run r = self._runner(client=self.ssh, **kwargs) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run r.exitstatus = _check_status(r.exitstatus) File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status raise CommandFailedError(command=r.command, exitstatus=status, node=host) CommandFailedError: Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'"
archive_path: /var/lib/teuthworker/archive/teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana/98508 description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml} email: null job_id: '98508' kernel: &id001 kdb: true sha1: distro last_in_suite: false machine_type: plana name: teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: firefly ceph: conf: global: ms inject socket failures: 5000 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug ms: 1 debug osd: 20 osd op thread timeout: 60 osd sloppy crc: true fs: btrfs log-whitelist: - slow request sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: branch: dev: firefly conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc s3tests: branch: master workunit: sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc owner: scheduled_teuthology@teuthology roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - client.0 - - mon.b - mds.a - osd.3 - osd.4 - osd.5 - client.1 targets: ubuntu@plana32.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmERfjlurKX631Ys98uSfL1mMJkRRZRRV5Hhen56sub04bFDz7W9zjh3Zs9pNMfdc1dWLf8IcpbdfcbR7cmkyfxQlLl+KmCwvRED+ZCR8P5HlkMFb+HnTdvyLAbu/4pvQRxrjy1GyQdNRUpxA8WWbfHrlz8leZPz3u3+hsHaCt8W0Y8cBpqmdTUtSgaGa9JTo/GWSkavF81o5xuVD+A4TGwNwTqIbb1f/HXAytffUwKr5fHHs1+hm1aT9GzQSumDHVCf9ykbcvO2uR70JZl3lZW2pVeFwQmq0AwmD5SetofuQK4ykVweONstnPwNGBqZJ/1A8jbxcby94RhDztzTqb ubuntu@plana39.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9rGG4WG4JuK1VecigkCM7IzMRmwhlrpnoYTzYW7Uh6xVnAAWaa4x6CcQIJOwoifIM8tpi2LAf+Bo3idMYMHq3ppq0i42ifC9W2ZoVAsN1pvMgPPIYXaL+A1Jfk5Pf1yPUNLljHLFJMrXpEfnbU8uow6ieQ/zwi1o9IpNJ1fsm+woKHwjXuC7W0QEn1q38WJGqFAH1do4JZErJZH1TD/rU/0oaSFhG+lykZd0copMlZgVS8EsS40aXUUqLvlAXi5TvzSDe+jYBigSjq6FRHNqcFB6Rd6VJ9BZTha9UZgSAPZsaFyJNL8Mb6wsU287+eqLqFoz33Vb73lSOwC63lG+j tasks: - internal.lock_machines: - 2 - plana - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - kernel: *id001 - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: null - ceph: conf: client.0: rgw gc pool: .rgw.gc.0 rgw log data: true rgw log meta: true rgw region: zero rgw region root pool: .rgw.region.0 rgw user keys pool: .users.0 rgw user uid pool: .users.uid.0 rgw zone: r0z1 rgw zone root pool: .rgw.zone.0 client.1: rgw gc pool: .rgw.gc.1 rgw log data: false rgw log meta: false rgw region: one rgw region root pool: .rgw.region.1 rgw user keys pool: .users.1 rgw user uid pool: .users.uid.1 rgw zone: r1z1 rgw zone root pool: .rgw.zone.1 - rgw: client.0: system user: access key: 0te6NH5mcdcq0Tc5i8i2 name: client0-system-user secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv valgrind: - --tool=memcheck client.1: system user: access key: 0te6NH5mcdcq0Tc5i8i2 name: client1-system-user secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv valgrind: - --tool=memcheck regions: one: api name: api1 is master: false master zone: r1z1 zones: - r1z1 zero: api name: api1 is master: true master zone: r0z1 zones: - r0z1 - radosgw-agent: client.0: dest: client.1 metadata-only: true src: client.0 - s3tests: client.0: rgw_server: client.0 teuthology_branch: firefly verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.3296
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml} duration: 2883.6131539344788 failure_reason: 'Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a ''!fails_on_rgw''"' flavor: basic owner: scheduled_teuthology@teuthology sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=d15dcfd4591d45dfa18e4038b37eef69 success: false
History
#1 Updated by Yehuda Sadeh about 10 years ago
I think the real issue is here, the specific agent error looks like a red herring:
2014-02-23T21:20:44.334 INFO:teuthology.orchestra.run.err:[10.214.131.8]: FAIL: s3tests.functional.test_s3.test_region_copy_object 2014-02-23T21:20:44.335 INFO:teuthology.orchestra.run.err:[10.214.131.8]: ---------------------------------------------------------------------- 2014-02-23T21:20:44.335 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Traceback (most recent call last): 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: self.test(*self.arg) 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4760, in test_region_copy_object 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: _verify_atomic_key_data(dest_key, file_size, 'A') 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4466, in _verify_atomic_key_data 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: key.get_contents_to_file(fp_verify) 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1603, in get_contents_to_file 2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]: response_headers=response_headers) 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1435, in get_file 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: query_args=None) 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1489, in _get_file_internal 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: fp.write(bytes) 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4459, in write 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: eq(data, self.char*size) 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: AssertionError: 'Status: 404\nContent-Length: 75\nAccept-Ranges: bytes\nContent-type: application/xml\r\n\r\n<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code></Error>' != 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' 2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: -------------------- >> begin captured stdout << ---------------------
#2 Updated by Yehuda Sadeh about 10 years ago
- Assignee set to Josh Durgin
hmm.. the timestamps actually match, so the shard sync failure might be the culprit. Seems like a sync agent issue (according to Josh).
#3 Updated by Josh Durgin about 10 years ago
- Assignee changed from Josh Durgin to Yehuda Sadeh
Fixing the agent problem did not make the test pass. I don't think the remainder is an agent issue, but haven't looked closely at the gateway logs. This test last passed on feb 20th.
#4 Updated by Yehuda Sadeh about 10 years ago
This fails due to issue #7703.
#5 Updated by Yehuda Sadeh about 10 years ago
- Status changed from New to Resolved