Project

General

Profile

Bug #7345

LibRadosTier.Evict failed in rados suite

Added by Yuri Weinstein over 6 years ago. Updated over 6 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Logs are in - http://qa-proxy.ceph.com/teuthology/teuthology-2014-02-04_23:00:02-rados-next-distro-basic-plana/68114

Note: see in the summary file that these errors wee not recorded.

2014-02-05T01:54:15.521 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: test/librados/tier.cc:634: Failure
2014-02-05T01:54:15.522 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: Value of: base_ioctx.read("foo", bl, 1, 0)
2014-02-05T01:54:15.522 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]:   Actual: -2
2014-02-05T01:54:15.522 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: Expected: 1
2014-02-05T01:54:15.523 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [  FAILED  ] LibRadosTier.Evict (8645 ms)
2014-02-05T01:54:15.523 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [ RUN      ] LibRadosTier.EvictSnap
2014-02-05T01:54:17.839 DEBUG:teuthology.orchestra.run:Running [10.214.132.20]: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph quorum_status'

...

1391594215,1391594217,1391594218,1391594220,1391594221,1391594223,1391594224,0
2014-02-05T01:57:03.896 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: first now 1391594215, trimmed
2014-02-05T01:57:04.206 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [       OK ] LibRadosTier.HitSetTrim (19652 ms)
2014-02-05T01:57:04.207 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [----------] 18 tests from LibRadosTier (254887 ms total)
2014-02-05T01:57:04.207 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: 
2014-02-05T01:57:04.207 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [----------] Global test environment tear-down
2014-02-05T01:57:04.207 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [==========] 18 tests from 1 test case ran. (254887 ms total)
2014-02-05T01:57:04.207 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [  PASSED  ] 17 tests.
2014-02-05T01:57:04.208 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [  FAILED  ] 1 test, listed below:
2014-02-05T01:57:04.208 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: [  FAILED  ] LibRadosTier.Evict
2014-02-05T01:57:04.208 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]: 
2014-02-05T01:57:04.208 INFO:teuthology.task.workunit.client.0.out:[10.214.132.18]:  1 FAILED TEST

archive_path: /var/lib/teuthworker/archive/teuthology-2014-02-04_23:00:02-rados-next-distro-basic-plana/68114
description: rados/monthrash/{ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml msgr-failures/few.yaml
  thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml}
email: null
job_id: '68114'
kernel: &id001
  kdb: true
  sha1: distro
last_in_suite: false
machine_type: plana
name: teuthology-2014-02-04_23:00:02-rados-next-distro-basic-plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      global:
        ms inject socket failures: 5000
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon min osdmap epochs: 25
        paxos service trim min: 5
      osd:
        debug ms: 1
        debug osd: 5
        osd sloppy crc: true
    fs: xfs
    log-whitelist:
    - slow request
    sha1: eb18c0a8d3b7ae2ba4f0ffba4dcff983152437ec
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      sha1: eb18c0a8d3b7ae2ba4f0ffba4dcff983152437ec
  s3tests:
    branch: next
  workunit:
    sha1: eb18c0a8d3b7ae2ba4f0ffba4dcff983152437ec
owner: scheduled_teuthology@teuthology
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
  - client.0
targets:
  ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDX5qLQ05TWlUoX78wIXMpSqq+6+J3UEXLM4bo0IvoyJr6dl1EG1z+EEvhdKGGobvj+zX9+JEeu7g4lrFUiJpmDs3YuqS6ECyuQJFRHQHImR1u+r5w2hqKGyRqoW0yG82q6D+8VLNyRL1B1+gAfWP3Eva9IS5e9iLQqwmCkPXa8J2m9C+k7zzE80IoOUCOzUMMYmrrgiuMub2sN9UPFv5cK0S/suJiNcktg8YQI8XONRdf3LChudRdMOckhvCwH71H/fXPPxv9MOfs36ixCW8TdP0L+9+QvCrcA0p5MCl2OcZcoexCtM9G6rXzjQ1zcOdrpV6sV+XBwHXoNEqHo2JLd
  ubuntu@plana60.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD6LGy3we7gWdPmxX1aJlK5UGi4r9G4tcY/GiVKeGb2zlBwM75KyLA84xEQZG59GxjTzALftFB5bgtL2AmMz1NNHI7u+NX9PqVVeEyPYqJcMZpIx6YqsxWmLCfkGYhhHMtRqX1SRoFEnvtrvq2wKLpAmKYCW2reAkajhcDmCul3ZrkGJaCncHuVlLFLQX5DqROIsmX0jFhcdKWHN0KWoSy/PN7aFTEHoMah/+Vj9NcDgcVUH8lhmld8IizoioIRXxz9RCP1kl6Z+hYx/vhTli3F96BhKG5d6O2125VOQdASG+ILmVg0U3m4ugkTFYf9AIPYJHsZk1Kn58AIfhz1qnCN
tasks:
- internal.lock_machines:
  - 2
  - plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install: null
- ceph: null
- mon_thrash:
    revive_delay: 90
    thrash_delay: 1
    thrash_many: true
    thrash_store: true
- workunit:
    clients:
      client.0:
      - rados/test.sh
teuthology_branch: next
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.8913
description: rados/monthrash/{ceph/ceph.yaml clusters/3-mons.yaml fs/xfs.yaml msgr-failures/few.yaml
  thrashers/force-sync-many.yaml workloads/rados_api_tests.yaml}
duration: 1053.7851231098175
failure_reason: 'Command failed on 10.214.132.18 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp
  && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1
  CEPH_REF=eb18c0a8d3b7ae2ba4f0ffba4dcff983152437ec TESTDIR="/home/ubuntu/cephtest" 
  CEPH_ID="0" adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage
  /home/ubuntu/cephtest/workunit.client.0/rados/test.sh'''
flavor: basic
owner: scheduled_teuthology@teuthology
sentry_event: http://sentry.ceph.com/inktank/teuthology/search?q=efae4f91df374302baa93edbd6fe0f5c
success: false

History

#1 Updated by Ian Colle over 6 years ago

  • Project changed from teuthology to Ceph
  • Priority changed from Normal to Urgent

#2 Updated by Ian Colle over 6 years ago

  • Assignee changed from Josh Durgin to Greg Farnum

#3 Updated by Greg Farnum over 6 years ago

  • Status changed from New to In Progress
  • Priority changed from Urgent to High

This hasn't failed in the nightlies since then (although there are other related failures which Sage is looking at), but I don't see anything in the commit history that I think should have fixed it so I'm trying to reproduce.

#4 Updated by Greg Farnum over 6 years ago

  • Status changed from In Progress to Need More Info
  • Assignee deleted (Greg Farnum)
  • Target version set to v0.77
  • Source changed from other to Q/A

Deferring this since there are a bunch of other outstanding changes to this code.

#5 Updated by Sage Weil over 6 years ago

  • Status changed from Need More Info to Can't reproduce

Also available in: Atom PDF