Ceph » CephFS

2012-04-12T13:09:27.496 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chown/00.t (Wstat: 0 Tests: 171 Failed: 1)
2012-04-12T13:09:27.497 INFO:teuthology.task.workunit.client.0.out: Failed test: 112

on ubuntu@teuthology:/a/nightly_coverage_2012-04-12-b/666$

2012-05-22T12:08:06.355 INFO:teuthology.task.workunit.client.0.out:Test Summary Report
2012-05-22T12:08:06.355 INFO:teuthology.task.workunit.client.0.out:-------------------
2012-05-22T12:08:06.355 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chown/00.t (Wstat: 0 Tests: 171 Failed: 3)
2012-05-22T12:08:06.356 INFO:teuthology.task.workunit.client.0.out: Failed tests: 97, 102, 112

ubuntu@teuthology:/a/nightly_coverage_2012-05-22-b/3038kernel: &id001
sha1: 374cd6946b9ed423992739a9036e59171bd63cdf
nuke-on-error: true
overrides:
ceph:
conf:
osd:
osd op complaint time: 120
coverage: true
fs: btrfs
log-whitelist:
- clocks not synchronized
- slow request
sha1: 884aa8a6616b03cc53e8691bfc73a48fa25db555

- kclient: null
- workunit:
all:
- suites/pjd.sh

ceph-fuse and kclient affected.. mds?

2012-05-29T00:00:52.273 DEBUG:teuthology.run:Config:
  kernel:
    sha1: f5f56aef8402124dc331d3c24651e4b52b3b5b1a
  nuke-on-error: true
  overrides:
    ceph:
      conf:
        osd:
          osd op complaint time: 120
      coverage: true
      fs: btrfs
      log-whitelist:
      - clocks not synchronized
      - slow request
      sha1: 73e29b658dfba231ea491178314ad419c8813f93
  roles:
  - - mon.a
    - mon.c
    - osd.0
    - osd.1
    - osd.2
  - - mon.b
    - mds.a
    - osd.3
    - osd.4
    - osd.5
  - - client.0
  tasks:
  - chef: null
  - ceph: null
  - ceph-fuse: null
  - workunit:
      all:
      - suites/pjd.sh

2012-05-29T00:13:05.967 INFO:teuthology.task.workunit.client.0.out:Test Summary Report
2012-05-29T00:13:05.967 INFO:teuthology.task.workunit.client.0.out:-------------------
2012-05-29T00:13:05.967 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chmod/00.t   (Wstat: 0 Tests: 58 Failed: 2)
2012-05-29T00:13:05.967 INFO:teuthology.task.workunit.client.0.out:  Failed tests:  27, 31
2012-05-29T00:13:05.967 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chown/00.t   (Wstat: 0 Tests: 171 Failed: 6)
2012-05-29T00:13:05.968 INFO:teuthology.task.workunit.client.0.out:  Failed tests:  97, 102, 118, 124, 141, 145
2012-05-29T00:13:05.969 INFO:teuthology.task.workunit.client.0.out:Files=184, Tests=1957, 109 wallclock secs ( 0.83 usr  0.55 sys +  0.76 cusr  2.39 csys =  4.53 CPU)
2012-05-29T00:13:05.970 INFO:teuthology.task.workunit.client.0.out:Result: FAIL

I haven't gone through the annoyance of counting it out, but it looks like the initially-reported test (112) is in the middle of the timing-based ones. If any of those are broken it's likely due to out-of-sync clocks causing times to not always be changed when it expects them to be.

That's not what's causing 27 and 31 to fail, though; those are just regular permission changes...

ubuntu@teuthology:/a/teuthology-2012-07-18_00:00:05-regression-next-testing-basic/13245$ cat config.yaml 
kernel: &id001
  kdb: true
  sha1: 14240f8208136dbbe7e825caedc0104806027aae
nuke-on-error: true
overrides:
  ceph:
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: d78235be1b5c20440f3a55abb3f4b980aa3f3596
  workunit:
    sha1: d78235be1b5c20440f3a55abb3f4b980aa3f3596
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
targets:
  ubuntu@plana78.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5u611hXVaR3i+fQJ3IWIw9y4mhS0WrusfWBkk4lDeuEyaJy0SqxbN6D5GhB7KzLZfpyiVcmdFzxk4w9Kc3XgQpJ42mNx4EDTen5ndI8HL8GclRd71RaocVIerynAD94lCAVzLSQCRBOA3HLH35OITPMR+ztZG8zSXUhhdYpwCLTJVb2BE8bA47GzyhCvu4L1JYwnIRzVUrufwugSS4odhlBJvge++omtL+r3Cm8M8W3Ahow+Yk8Ni40m60j82CEs7/Mc95j/CXYIQzwJWkIeRoQMMgT3HmK9koJRzEBkK8FQS9KMnrEi6mJAa+8eL/AgVMvyxpz+W7z77cH5p2Kab
  ubuntu@plana80.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCv9ah6KrNde/qjaxGaNCBWMlzzprTwkPmQetLrGhY+MsTz9VN6CRBegYq94TcOjE6A+w3swGNkuPzgG+meNGKV3fq+rggCEDJq+lPPgvM2dU86d0eoPr3m4BYgHcEdxcIzkaZ6f+F+vC956Lc1a1DGRyz6ygAuD8HJh/4FKhGe+PaaQ2EtvrwY8zEj+i9PARj9PHttLYLgsaOrdhszJkXBPGHGSJ7jaBI/AViv12WmYN9WpXWE+khICRpqfm2PIhzKZOgrRBMUuH8YpHTh8hUH5HjBtC2kNS6FPowrNhJU4luFQWV5un85CDRHthfGW+e8/ZelH56NfS1oOxnRM7jL
  ubuntu@plana81.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdD/y74z7frMimAwHJVN1Po5vf7CkyaI3Csixv3ruym7XoIP3/1Sj1kGmIKADaRu4rQKAIMywILuHXggMoFjvEzXY0PuJnw5og39oriY6cMMo8aVpqPZCukHNjQCJv7EIl0GQaG1uTXfjEXbrbbMGi2NXiYVzj+2xgpSL/WS3CAYO6BNsXSF4EJFFeUrEbvHZ14oPqy0TYN4SHN6EdvpY2lcPV8SlCPkivLtGs07WPqiPpfe9P8pS5ohWXEdMJERJcPcS6iQkkNhB6hAryOX3QvO2KTobJEI/9+aIx4eTnICmuxtywVkfZstpOUw5oT3NXSt2eLvU2cF4PaXtMQ4KX
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- kclient: null
- workunit:
    clients:
      all:
      - suites/pjd.sh

grr

Latest logs: ubuntu@teuthology:/a/teuthology-2012-07-21_05:00:03-regression-stable-master-basic/15237

ubuntu@teuthology:/a/teuthology-2012-07-21_05:00:03-regression-stable-master-basic/15237$ cat config.yaml
kernel: &id001
kdb: true
sha1: 26ce171915f348abd1f41da1ed139d93750d987f
nuke-on-error: true
overrides:
ceph:
fs: btrfs
log-whitelist:
- slow request
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
workunit:
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
roles:
- - mon.a
- mon.c
- osd.0
- osd.1
- osd.2
- - mon.b
- mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana27.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHsjsqjgaLtco9/n4XOXwoTaBmkHxVgzNy3ElRQUccEn3LxQB4a5k3GFuju7Zdq4I9eHYajcjc8h2cHZWSYaDObbJE4I7X1QByEhBtIeyKIFajOVHQvczuVktpWXCoyeg/4M9b7E14AypFy4HpHA6KWAfwPJbvvEUp5jCFw+GHi/XhkrtU/7ydRwQsAql39YzpbDSacsyzblj685akhyY/Dn5sHVnmFaIkMmc64LJMsu9k93EKqD4LxtUn3A4G0qpC4lKA2RatTjYBVNBUPqNyCou6l88gnPgZOjLuWte0g1Met6WbtUJrE+nvAfpf1CAipDFyejIgFIt2U4LbUX9j
ubuntu@plana35.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtCEUU7ZjJxMBOYq9dlPZppwlahU15OIq1a/xqI0YIpAF7ZLf1eis04s6Kzvl45ooKP2J6DW598Lr9zTtFb6j9RoAkgIbIOkWZmLQjeZesi//7B8uXBw7niNzU1iBI83K8dPL2cwTsgwXXtoaD03pW9HCv4ADhH12/CsfoqNPfql7ze4JlI07seHmUqkBuBFxLVgdZDXb+Cjdg2sQnMS5qZo0OGBp2yfBla01Pk8V1sHGg39miF0EDJZyIOL2ziZOcm1f02WVkUAD8MMOtfQbf33DT/ya4Th20YtRMQxNLkUYG6rxcjutzILFKDr7ugbvOwtBhVS7qplqLWAwHdi0p
ubuntu@plana36.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9Ru6XkJBGiUQK9AtlFt82TzpaWuKams26i0FcItt3hbniR1yxpWVHM3dQI5Gft3liumnOD+cPZiZJzGYyj2KDBCZ8G9V65YqCbzO+moJmv5wDWKg1pEIIW040aLrlsOPbZlEL7htT14MHTTstyTQCOLkrySCpexwYrA2wQBhsHc7pxL+XLa+WM1zTXSQe6QrS8iYxITGRibEMSjcXlOuLFnst42O6o4WQHd31WS9pbniBmso7KVgTFxmcN5rvEo1YAJJYwVxGfmorWrXan1ULY6CksasatbCuohmVNNZfsnE8KdyYsPYCbKIPp9NnmBL3Pp/oPqqyPsj36Wgj5e4/
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
clients:
all:
- suites/pjd.sh
ubuntu@teuthology:/a/teuthology-2012-07-21_05:00:03-regression-stable-master-basic/15237$ cat summary.yaml
ceph-sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
client.0-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
description: collection:basic clusters:fixed-3.yaml fs:btrfs.yaml tasks:cfuse_workunit_suites_pjd.yaml
duration: 292.90552878379822
failure_reason: 'Command failed with status 1: ''mkdir -- /tmp/cephtest/mnt.0/client.0/tmp
&& cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=c99671201de9d9cdf03bbf0f4e28e8afb70c280c
PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib"
CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret"
CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages"
/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
/tmp/cephtest/workunit.client.0/suites/pjd.sh && rm rf - /tmp/cephtest/mnt.0/client.0/tmp'''
flavor: basic
mon.a-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
mon.b-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
owner: scheduled_teuthology@teuthology
success: false

Recent logs: ubuntu@teuthology: /a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16416

From teuthology.log:
2012-07-23T05:16:07.396 INFO:teuthology.task.workunit.client.0.out:ok 144
2012-07-23T05:16:07.396 INFO:teuthology.task.workunit.client.0.out:not ok 145
2012-07-23T05:16:07.396 INFO:teuthology.task.workunit.client.0.out:ok 146
2012-07-23T05:16:07.396 INFO:teuthology.task.workunit.client.0.out:ok 147

ubuntu@teuthology:/a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16416$ cat summary.yaml
ceph-sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
client.0-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
description: collection:basic clusters:fixed-3.yaml fs:btrfs.yaml tasks:cfuse_workunit_suites_pjd.yaml
duration: 442.30905795097351
failure_reason: 'Command failed with status 1: ''mkdir -- /tmp/cephtest/mnt.0/client.0/tmp
&& cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=c99671201de9d9cdf03bbf0f4e28e8afb70c280c
PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib"
CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret"
CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages"
/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
/tmp/cephtest/workunit.client.0/suites/pjd.sh && rm rf - /tmp/cephtest/mnt.0/client.0/tmp'''
flavor: basic
mon.a-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
mon.b-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
owner: scheduled_teuthology@teuthology
success: false
ubuntu@teuthology:/a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16416$ cat config.yaml
kernel: &id001
kdb: true
sha1: 26ce171915f348abd1f41da1ed139d93750d987f
nuke-on-error: true
overrides:
ceph:
fs: btrfs
log-whitelist:
- slow request
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
workunit:
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
roles:
- - mon.a
- mon.c
- osd.0
- osd.1
- osd.2
- - mon.b
- mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana24.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQClf8RuTWfOjpMgfDd3/Km+B8HeCQA3P0vqSF5UXyq5ZZ//pVogPo40/kJR//UvErWLMYM1udR+NVdRpecFZHObFMWeMVSvGTIaaz6JSWRNjBZ5sG8qoaViW4Q2CMN1htz1sfUkrafCALTPjtIAfKIJJf6h2gcU/UHQEWXed4nyLmzA6XLHJUUg89tkCimfWqQ5A0vZz+MdlgWkjHgmv502AYaWMl838q6v6iMiqOcVEjw/p4B+v9NTGlqdByzOCIS80Ps8AKWLE6VoyF+BEgLnnvO2kARufJvokt6Q8HHsIlvCOKz1d0EiXEu5TRw6D779QpveaoRLKMnr2dsSN6gn
ubuntu@plana25.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuXajaQgHe9XnbLOzI8WWFYVz6+TnOiTzbkIJPGOZpzQEjnUtJraQIEt5ABSeovMjiEj+V4XvunfyuSmEd0H9giRSyjmCHTPGlpndfTeCdVtCBpNqf5GkUqHaEY1Hp57XPbya2rGlwtFm0NeIDYx6pfkejKnsTOUqwhgUb6950TRhjHQhMjFgyALSyfAm/4y6vGZfjm57+yyih6XgDkqWiiQ6Y/aJVR2n+iCzvqEzV7JSCU+Brn+k8IQLHho1fadYqc5PjYct5BaVlHcP6c+T8nJE/DvqGwZ4gQaVJcuWJiDfLOPPYo1g/0AFicxauLwVNJ6HFR9FjLLGtGU+2DcVN
ubuntu@plana27.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHsjsqjgaLtco9/n4XOXwoTaBmkHxVgzNy3ElRQUccEn3LxQB4a5k3GFuju7Zdq4I9eHYajcjc8h2cHZWSYaDObbJE4I7X1QByEhBtIeyKIFajOVHQvczuVktpWXCoyeg/4M9b7E14AypFy4HpHA6KWAfwPJbvvEUp5jCFw+GHi/XhkrtU/7ydRwQsAql39YzpbDSacsyzblj685akhyY/Dn5sHVnmFaIkMmc64LJMsu9k93EKqD4LxtUn3A4G0qpC4lKA2RatTjYBVNBUPqNyCou6l88gnPgZOjLuWte0g1Met6WbtUJrE+nvAfpf1CAipDFyejIgFIt2U4LbUX9j
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
clients:
all:
- suites/pjd.sh

also,

ubuntu@teuthology: /a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16425

from teuthology.log
2012-07-23T05:16:53.247 INFO:teuthology.task.workunit.client.0.out:not ok 141
2012-07-23T05:16:53.247 INFO:teuthology.task.workunit.client.0.out:ok 142
2012-07-23T05:16:53.247 INFO:teuthology.task.workunit.client.0.out:ok 143
2012-07-23T05:16:53.247 INFO:teuthology.task.workunit.client.0.out:ok 144
2012-07-23T05:16:53.247 INFO:teuthology.task.workunit.client.0.out:not ok 145

ubuntu@teuthology:/a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16425$ cat summary.yaml
ceph-sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
client.0-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
description: collection:basic clusters:fixed-3.yaml fs:btrfs.yaml tasks:kclient_workunit_suites_pjd.yaml
duration: 483.58079195022583
failure_reason: 'Command failed with status 1: ''mkdir -- /tmp/cephtest/mnt.0/client.0/tmp
&& cd -- /tmp/cephtest/mnt.0/client.0/tmp && CEPH_REF=c99671201de9d9cdf03bbf0f4e28e8afb70c280c
PATH="$PATH:/tmp/cephtest/binary/usr/local/bin" LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/tmp/cephtest/binary/usr/local/lib"
CEPH_CONF="/tmp/cephtest/ceph.conf" CEPH_SECRET_FILE="/tmp/cephtest/data/client.0.secret"
CEPH_ID="0" PYTHONPATH="$PYTHONPATH:/tmp/cephtest/binary/usr/local/lib/python2.7/dist-packages:/tmp/cephtest/binary/usr/local/lib/python2.6/dist-packages"
/tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage
/tmp/cephtest/workunit.client.0/suites/pjd.sh && rm rf - /tmp/cephtest/mnt.0/client.0/tmp'''
flavor: basic
mon.a-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
mon.b-kernel-sha1: 26ce171915f348abd1f41da1ed139d93750d987f
owner: scheduled_teuthology@teuthology
success: false
ubuntu@teuthology:/a/teuthology-2012-07-23_05:00:03-regression-stable-master-basic/16425$ cat config.yaml
kernel: &id001
kdb: true
sha1: 26ce171915f348abd1f41da1ed139d93750d987f
nuke-on-error: true
overrides:
ceph:
fs: btrfs
log-whitelist:
- slow request
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
workunit:
sha1: c99671201de9d9cdf03bbf0f4e28e8afb70c280c
roles:
- - mon.a
- mon.c
- osd.0
- osd.1
- osd.2
- - mon.b
- mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana66.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9QpKzRhLR+H4oCXhZtMxFDAC7E9cmtqcEPmQiW1ycxHNpk+Vy/uNNtlSh8Ljz7/R595beZe4JKgAkxZCCDmmWIEcm01bJgxVq0cbFLz+9dvyiirmr+RxbWqayu2VDC6uuiVAuxz5RdAw6+5Y/4gCrSdQKfQ8dUJZb4e/4Kz/TLr/+R5z+WqCGeutwb9QvK2anvhPJy+wa/JWHwtpTBjZVa5RFOkz9lfNDYayw3j1rylKk0d39J4VB/ch/qIBqUfxD7Hc4exu9sbG9bt5VKdomqdQvTjQXRw083+Nlj6RqJxYZfdJTWG/gYV3MXC4pwqP/ovcjoqZ9cvrsicdSezKp
ubuntu@plana67.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqiZsiT7h3fNR9yzwK2WToaotO4olIxmVdh+aSf3ILwEpHjFYbWXymL0C77hn0MdGRbaOzWOSMMng3MAHKy9xR3/CGNXXqO7iEK1fJOvSfmypkvJDyrMY/RuSvdifcXJyREvFsSK6cdmRpO235ODhfui4FC5BLmgv/VvasH/1Ur4ALfe7UE9L+cU4VeoJdl082oYeo1nn1beERgaypX67MXepG2NKbEY77jG5FXbGVpKWmsgIEWiiX8p6+afTOP+8cGsM3vsAG7nTJeFVKkEHc7A8cPkT4l/iXKjSiwWAtU5NV0QmRC/1ad78+xTOWNzJaTrIxoKuuGpB+DjdvrJgN
ubuntu@plana69.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDE4lTHELBl2d+BXuaxMKtJX4r3GN2qSaxHTfewZxutqEC+rSNbD5Otiqwm16GtmCYklbYJL0yr6mizCZ0KzL3lVwNSczb/CxfvLTLegmBv9YzPRvoL1ymxdHggPIw47JTNRg6+UO9EConQpp7LOktT7fYf3AFx/BS9Ux6UeKyxifkb/1GUlzVT2rk1D39frBhI7AcPHovvbd/6uHvamrpUS3wSDWdz6BNRTQDaK307TJS9LPva6Cj6WAgmTZZesjWB4mOpWywrA9sva7GqSrnTMGKnIHPcsI6U57CALwHS1g/zIz73QuolZA2NkbhA7jjrfsUaWYJorE9dynIga9pf
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- kclient: null
- workunit:
clients:
all:
- suites/pjd.sh

Latest log: ubuntu@teuthology:/a/teuthology-2012-07-23_19:00:03-regression-master-testing-gcov/16530

2012-07-23T19:16:10.185 INFO:teuthology.task.workunit.client.0.out:not ok 43
2012-07-23T19:16:10.186 INFO:teuthology.task.workunit.client.0.out:not ok 44

OLD ubuntu@teuthology:/a/teuthology-2012-07-23_19:00:03-regression-master-testing-gcov/16530$ cat config.yaml
kernel: &id001
kdb: true
sha1: 77dca1ac33894de22b1740bb9cf6b8ef6429c700
nuke-on-error: true
overrides:
ceph:
coverage: true
fs: btrfs
log-whitelist:
- slow request
sha1: d9bfe9547d7ee610e94a4d485988287f8143903c
workunit:
sha1: d9bfe9547d7ee610e94a4d485988287f8143903c
roles:
- - mon.a
- mon.c
- osd.0
- osd.1
- osd.2
- - mon.b
- mds.a
- osd.3
- osd.4
- osd.5
- - client.0
targets:
ubuntu@plana49.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsodFx35LkTvKGlOrQT7Zt/AuvpUOoNz4sM8ovIxIjn/AJliZfup2KAhO/VRUpcndtJMy0eAp/v6wMy9bP0tqTNbeZ3q+zzzLuwnoCOQ0relggohTE7lzLDVLb/MHCQlxpLifwUrwQpVdiUfJ2B5mcGyMr2Lku4TcC3BrgPKuaXLkGHZl0aaShLfr6PbTvDjqI+IDT4E4iKd65KhzLuzDqAIwuZyoifiG+5KYYsoIZmCHidTZYHdf6utnBzP9jTQHroZR5kE/qbVnoL1tT9vvXWLcG7y2PB3UMmIOJmRXlys2/My5iRSC+1Wd9nBxcwE3BgWHknGQhrwkMl0uC+vH3
ubuntu@plana50.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVJ+lkgUdkr27WFzrmwSQU22m+pFIiqzhfcO4Hinu8A8uyP4FIephrEcq4Rrt4hp14Syb1pxXisV6UKwAZKikDoD1Wl0LSro4TzOs6HuMEhfvzdnISvyzE3f2w0cj1zE61rHFYfPNF14b9fkE3wBf2Vb4i6ReaN2/Yd12J/xO52tJH1lPxgsFoAIRMjdQMbfVwPU6kK9SY4ngt9iLjge6gZ0O9Jwe2vrgD6+LNoMY9qvNjgRvQdCTi85OQwitU0ZMZdGC0cQ/oNbKd+yW92rW9Wu6dcyKSisesRcm7lbtS6X2uUup+u3vWze7coT+Py3TdNW6nGpIg4muyvqHfSinz
ubuntu@plana51.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDLTsxC+nR+xtTXMbOtazCh7MOzgBKjX/oCLMP16k0AtH8Ui92tlqsfNxcHczUol0DNzxCITgrhF8FTvgM3EgkbOUVAxGj+xLqfxsdlf58nTVXbm/pOGYnvOI8CvA4DgISHDbkzuFH4FKtR8qNTTFVmtEXaZ+jpSvn7vrYuI/Uu9XZOQh73phYW8zvVB1x8770czM0Gy2wgxdNguKy6L/Q9ShsLcFfm8Uvxf6aXb3qmuxwGhqYsMlNl0X3AjoOwmow74rodlcMvQP/pAQdjMZfe1lBPqsjmU518BE5eo7zV3O9iF6ahOrm8igOu9bfki0G52R22pA3hE9BPKPfzA0hL
tasks:
- internal.lock_machines: 3
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.check_conflict: null
- kernel: *id001
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock: null
- ceph: null
- ceph-fuse: null
- workunit:
clients:
all:
- suites/pjd.sh

Happened again on Friday. Time to add the delay injection to the nightlies?

2013-01-11T07:32:37.489 INFO:teuthology.task.workunit.client.0.out:Test Summary Report
2013-01-11T07:32:37.489 INFO:teuthology.task.workunit.client.0.out:-------------------
2013-01-11T07:32:37.489 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chmod/00.t (Wstat: 0 Tests: 58 Failed: 1)
2013-01-11T07:32:37.489 INFO:teuthology.task.workunit.client.0.out: Failed test: 31
2013-01-11T07:32:37.489 INFO:teuthology.task.workunit.client.0.out:../pjd-fstest-20080816/tests/chown/00.t (Wstat: 0 Tests: 171 Failed: 1)
2013-01-11T07:32:37.490 INFO:teuthology.task.workunit.client.0.out: Failed test: 145
2013-01-11T07:32:37.490 INFO:teuthology.task.workunit.client.0.out:Files=184, Tests=1957, 124 wallclock secs ( 3.56 usr 1.83 sys + 4.07 cusr 3.68 csys = 13.14 CPU)
2013-01-11T07:32:37.490 INFO:teuthology.task.workunit.client.0.out:Result: FAIL