Ceph

none of the failed are related to erasure coded pools ( the config file does not contain the string ec_ hence no erasure coded pool was created ).

the job died because it could not reach paddles, not because the job itself died.

2014-06-12T12:50:21.894 INFO:teuthology.run:Summary data:
{description: 'rados/thrash/{clusters/fixed-2.yaml fs/btrfs.yaml msgr-failures/few.yaml
    thrashers/mapgap.yaml workloads/ec-radosbench.yaml}', duration: 3391.9412620067596,
  flavor: basic, mon.a-kernel-sha1: 22001f619f29ddf66582d834223dcff4c0b74595, mon.b-kernel-sha1: 22001f619f29ddf66582d834223dcff4c0b74595,
  owner: scheduled_loic@fold, success: true}

running teuthology test run

Fixed a few problems and running a firefly upgrade suite

Cancel the teuthology run that did not contain any LRC workload and run another

canceled the previous job because it did not have enough OSD to complete (the LRC rule requires a minimum of 8 for each PG). schedule another job

There is no need to test upgrade on a plugin that does not exist in LRC.

Reserved three machines and run the following job on them:

os_type: ubuntu
os_version: '14.04'
overrides:
  ceph:
    conf:
      global:
        osd heartbeat grace: 40
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      branch: wip-7238-lrc-plugin
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
  - osd.3
- - mon.b
  - mon.c
  - osd.4
  - osd.5
  - osd.6
  - osd.7
- - client.0
  - osd.8
  - osd.9
  - osd.10
  - osd.11
  - osd.12
  - osd.13
  - osd.14
  - osd.15
  - osd.16
  - osd.17
  - osd.18
  - osd.19
  - osd.20
suite_path: /home/loic/software/ceph/ceph-qa-suite
tasks:
- install:
    branch: wip-7238-lrc-plugin
- ceph:
    fs: xfs
- rados:
    clients: [client.0]
    ops: 4000
    objects: 500
    ec_pool: true
    erasure_code_profile:
      name: LRCprofile
      plugin: LRC
      k: 4
      m: 2
      l: 3
      ruleset-failure-domain: osd
    op_weights:
      read: 45
      write: 0
      append: 45
      delete: 10

Fixed a bug that made the plugin incorrectly claiming it could not recover when the last OSD was out, running tests again.

The rados test is running, with thrashing after a rebase on master.

It crashes the OSD in decode() but the plugin is silent on the reason why it refuses to decode. Adding debug information to analyze in the logs and running the tests again.

IsRecoverablePredicate must not assume the object is recoverable if the first K chunks are available. This is no longer true since data chunks can be remapped.

the above teuthology test ran successfully, with trashosd

2014-08-28 07:36:32,336.336 INFO:teuthology.run:Summary data:
{duration: 6015.952927112579, flavor: basic, owner: loic@dachary.org, success: true}

thrashosd passed because it had enough OSD to never be in a situation where mapping fails. When this happens, it triggers http://tracker.ceph.com/issues/9263 which comes from the fact that decode_chunks skips the layer which do not contain chunks that are to be read. This is find most of the time but when a layer is required to repair a chunk that will help an upper layer, it fails.

The problem has been fixed and the thrasher runs again with a number of OSD that has been verified to create bad mappings.

os_type: ubuntu
os_version: '14.04'
overrides:
  ceph:
    conf:
      global:
        osd heartbeat grace: 40
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  install:
    ceph:
      branch: wip-7238-lrc-plugin
roles:
- - mon.a
  - osd.0
  - osd.1
  - osd.2
  - osd.3
- - mon.b
  - mon.c
  - osd.4
  - osd.5
  - osd.6
  - osd.7
- - client.0
  - osd.8
  - osd.9
  - osd.10
  - osd.11
  - osd.12
suite_path: /home/loic/software/ceph/ceph-qa-suite
tasks:
- install:
    branch: wip-7238-lrc-plugin
- ceph:
    fs: xfs
- thrashosds:
    chance_pgnum_grow: 1
    chance_pgpnum_fix: 1
    timeout: 1200
- rados:
    clients: [client.0]
    ops: 4000
    objects: 500
    ec_pool: true
    erasure_code_profile:
      name: LRCprofile
      plugin: LRC
      k: 4
      m: 2
      l: 3
      ruleset-failure-domain: osd
    op_weights:
      read: 45
      write: 0
      append: 45
      delete: 10

the above teuthology test ran successfully, with trashosd

2014-08-29 07:13:31,922.922 INFO:teuthology.run:Summary data:
{duration: 5186.044275045395, flavor: basic, owner: loic@dachary.org, success: true}