Project

General

Profile

Bug #7526

"ERROR:radosgw_agent.worker:syncing entries for shard 59" in rgw-firefly-distro-basic-plana suite

Added by Yuri Weinstein about 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Logs are in qa-proxy.ceph.com/teuthology/teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana/98508

DEBUG:requests.packages.urllib3.connectionpool:"POST /admin/log?lock&locker-id=plana32%3A6161&length=60&zone-id=r1z1&type=metadata&id=59 HTTP/1.1" 200 None
2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]: ERROR:radosgw_agent.worker:syncing entries for shard 59 failed
2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]: Traceback (most recent call last):
2014-02-23T21:20:34.904 INFO:teuthology.task.radosgw-agent.ubuntu@plana32.front.sepia.ceph.com.8000.syncdaemon.err:[10.214.131.8]:   File 

...

2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Host: plana32.front.sepia.ceph.com:7280
2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Port: 7280
2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Params: {}
2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Token: None
2014-02-23T21:20:44.401 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: StringToSign:
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: GET
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Mon, 24 Feb 2014 05:06:27 GMT
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: /test-client.0-ehifidx0qn23xwi-234/testobj-dest
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: boto: DEBUG: Signature:
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: AWS RTCSXBSATCKJJPZJORHC:rezIj5jT5Cf5Ltmt13YxnUBd5tI=
2014-02-23T21:20:44.402 INFO:teuthology.orchestra.run.err:[10.214.131.8]: --------------------- >> end captured logging << ---------------------
2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 
2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: ----------------------------------------------------------------------
2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Ran 285 tests in 2301.213s
2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 
2014-02-23T21:20:44.403 INFO:teuthology.orchestra.run.err:[10.214.131.8]: FAILED (failures=1)
2014-02-23T21:20:44.404 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 305, in run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/cluster.py", line 61, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'" 
2014-02-23T21:20:44.404 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data'
2014-02-23T21:20:48.675 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data'
2014-02-23T21:20:58.947 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23 21:20:58.947043 7f4c6defa700  0 -- 10.214.131.8:0/1007361 >> 10.214.131.8:6800/4334 pipe(0x123e910 sd=8 :40779 s=2 pgs=35 cs=1 l=1 c=0x123eb70).injecting socket failure
2014-02-23T21:21:00.876 INFO:teuthology.orchestra.run.err:[10.214.131.8]: 2014-02-23 21:21:00.875119 7f4c6d6f2700  0 -- 10.214.131.8:0/1007361 >> 10.214.131.8:6805/4337 pipe(0x1247b50 sd=12 :56391 s=2 pgs=26 cs=1 l=1 c=0x1247db0).injecting socket failure
2014-02-23T21:21:21.507 INFO:teuthology.task.s3tests:Removing s3-tests...
2014-02-23T21:21:21.508 DEBUG:teuthology.orchestra.run:Running [10.214.131.8]: 'rm -rf /home/ubuntu/cephtest/s3-tests'
2014-02-23T21:21:21.581 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-firefly/teuthology/run_tasks.py", line 33, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 399, in task
    lambda: run_tests(ctx=ctx, config=config),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-firefly/teuthology/task/s3tests.py", line 305, in run_tests
    args=args,
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/cluster.py", line 61, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/remote.py", line 106, in run
    r = self._runner(client=self.ssh, **kwargs)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 328, in run
    r.exitstatus = _check_status(r.exitstatus)
  File "/home/teuthworker/teuthology-firefly/teuthology/orchestra/run.py", line 324, in _check_status
    raise CommandFailedError(command=r.command, exitstatus=status, node=host)
CommandFailedError: Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'" 
archive_path: /var/lib/teuthworker/archive/teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana/98508
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml
  tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml}
email: null
job_id: '98508'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: plana
name: teuthology-2014-02-22_23:02:21-rgw-firefly-distro-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: firefly
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        debug filestore: 20
        debug ms: 1
        debug osd: 20
        osd op thread timeout: 60
        osd sloppy crc: true
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc
    valgrind:
      mds:
      - --tool=memcheck
      mon:
      - --tool=memcheck
      - --leak-check=full
      - --show-reachable=yes
      osd:
      - --tool=memcheck
  ceph-deploy:
    branch:
      dev: firefly
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc
  s3tests:
    branch: master
  workunit:
    sha1: 6d8cb22e5887b3289bf3e0809f207442874aaccc
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
  - client.0
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.1
targets:
  ubuntu@plana32.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmERfjlurKX631Ys98uSfL1mMJkRRZRRV5Hhen56sub04bFDz7W9zjh3Zs9pNMfdc1dWLf8IcpbdfcbR7cmkyfxQlLl+KmCwvRED+ZCR8P5HlkMFb+HnTdvyLAbu/4pvQRxrjy1GyQdNRUpxA8WWbfHrlz8leZPz3u3+hsHaCt8W0Y8cBpqmdTUtSgaGa9JTo/GWSkavF81o5xuVD+A4TGwNwTqIbb1f/HXAytffUwKr5fHHs1+hm1aT9GzQSumDHVCf9ykbcvO2uR70JZl3lZW2pVeFwQmq0AwmD5SetofuQK4ykVweONstnPwNGBqZJ/1A8jbxcby94RhDztzTqb
  ubuntu@plana39.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9rGG4WG4JuK1VecigkCM7IzMRmwhlrpnoYTzYW7Uh6xVnAAWaa4x6CcQIJOwoifIM8tpi2LAf+Bo3idMYMHq3ppq0i42ifC9W2ZoVAsN1pvMgPPIYXaL+A1Jfk5Pf1yPUNLljHLFJMrXpEfnbU8uow6ieQ/zwi1o9IpNJ1fsm+woKHwjXuC7W0QEn1q38WJGqFAH1do4JZErJZH1TD/rU/0oaSFhG+lykZd0copMlZgVS8EsS40aXUUqLvlAXi5TvzSDe+jYBigSjq6FRHNqcFB6Rd6VJ9BZTha9UZgSAPZsaFyJNL8Mb6wsU287+eqLqFoz33Vb73lSOwC63lG+j
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph:
    conf:
      client.0:
        rgw gc pool: .rgw.gc.0
        rgw log data: true
        rgw log meta: true
        rgw region: zero
        rgw region root pool: .rgw.region.0
        rgw user keys pool: .users.0
        rgw user uid pool: .users.uid.0
        rgw zone: r0z1
        rgw zone root pool: .rgw.zone.0
      client.1:
        rgw gc pool: .rgw.gc.1
        rgw log data: false
        rgw log meta: false
        rgw region: one
        rgw region root pool: .rgw.region.1
        rgw user keys pool: .users.1
        rgw user uid pool: .users.uid.1
        rgw zone: r1z1
        rgw zone root pool: .rgw.zone.1
- rgw:
    client.0:
      system user:
        access key: 0te6NH5mcdcq0Tc5i8i2
        name: client0-system-user
        secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
      valgrind:
      - --tool=memcheck
    client.1:
      system user:
        access key: 0te6NH5mcdcq0Tc5i8i2
        name: client1-system-user
        secret key: Oy4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
      valgrind:
      - --tool=memcheck
    regions:
      one:
        api name: api1
        is master: false
        master zone: r1z1
        zones:
        - r1z1
      zero:
        api name: api1
        is master: true
        master zone: r0z1
        zones:
        - r0z1
- radosgw-agent:
    client.0:
      dest: client.1
      metadata-only: true
      src: client.0
- s3tests:
    client.0:
      rgw_server: client.0
teuthology_branch: firefly
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.3296
description: rgw/verify/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml
  tasks/rgw_s3tests_multiregion.yaml validater/valgrind.yaml}
duration: 2883.6131539344788
failure_reason: 'Command failed on 10.214.131.8 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf
  /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests
  -v -a ''!fails_on_rgw''"'
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=d15dcfd4591d45dfa18e4038b37eef69
success: false

History

#1 Updated by Yehuda Sadeh about 10 years ago

I think the real issue is here, the specific agent error looks like a red herring:

2014-02-23T21:20:44.334 INFO:teuthology.orchestra.run.err:[10.214.131.8]: FAIL: s3tests.functional.test_s3.test_region_copy_object
2014-02-23T21:20:44.335 INFO:teuthology.orchestra.run.err:[10.214.131.8]: ----------------------------------------------------------------------
2014-02-23T21:20:44.335 INFO:teuthology.orchestra.run.err:[10.214.131.8]: Traceback (most recent call last):
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     self.test(*self.arg)
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4760, in test_region_copy_object
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     _verify_atomic_key_data(dest_key, file_size, 'A')
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4466, in _verify_atomic_key_data
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     key.get_contents_to_file(fp_verify)
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1603, in get_contents_to_file
2014-02-23T21:20:44.336 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     response_headers=response_headers)
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1435, in get_file
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     query_args=None)
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1489, in _get_file_internal
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     fp.write(bytes)
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 4459, in write
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]:     eq(data, self.char*size)
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: AssertionError: 'Status: 404\nContent-Length: 75\nAccept-Ranges: bytes\nContent-type: application/xml\r\n\r\n<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchKey</Code></Error>' != 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
2014-02-23T21:20:44.337 INFO:teuthology.orchestra.run.err:[10.214.131.8]: -------------------- >> begin captured stdout << ---------------------

#2 Updated by Yehuda Sadeh about 10 years ago

  • Assignee set to Josh Durgin

hmm.. the timestamps actually match, so the shard sync failure might be the culprit. Seems like a sync agent issue (according to Josh).

#3 Updated by Josh Durgin about 10 years ago

  • Assignee changed from Josh Durgin to Yehuda Sadeh

Fixing the agent problem did not make the test pass. I don't think the remainder is an agent issue, but haven't looked closely at the gateway logs. This test last passed on feb 20th.

#4 Updated by Yehuda Sadeh about 10 years ago

This fails due to issue #7703.

#5 Updated by Yehuda Sadeh about 10 years ago

  • Status changed from New to Resolved

Also available in: Atom PDF