Ceph

Also in run:
http://pulpito.ceph.com/teuthology-2016-02-28_02:10:01-upgrade:hammer-x-jewel-distro-basic-vps/31527/
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-02-28_02:10:01-upgrade:hammer-x-jewel-distro-basic-vps/31527/teuthology.log

From IRC:

yuriw: actually, perhaps it's just a race condition on package install jdillaman
yuriw: it failed creating a pool, so maybe the crushmap tool was partially installed jdillaman @ 9:31    
yuriw - we started seeing  those after changes in packages, maybe be yuriw @ 9:31    
jdillaman ... so most likely the same issue

The ceph SHA1 for this job is "d6031c2 Merge pull request #7700 from SUSE/wip-14811"

As of that SHA1 the rados-classes libraries were not packaged correctly. This issue was later fixed by https://github.com/ceph/ceph/commit/5083980f4619bf282dfecde5078f1a73a2ed562a

I don't know for sure whether that commit fixed this bug, but might be worth rerunning these tests on latest jewel.

On pre-packadges split code tests passed http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-01_23:53:26-upgrade:hammer-x-wip-jewel-no-package-split-distro-basic-openstack/

Last run on jewel: http://teuthology.ovh.sepia.ceph.com/teuthology/teuthology-2016-03-02_02:52:36-upgrade:hammer-x-jewel-distro-basic-openstack/8041/teuthology.log

side-by-side runs before and after the split code, same test

http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-02_19:34:51-upgrade:hammer-x-wip-jewel-no-package-split-distro-basic-openstack/

http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-02_19:33:38-upgrade:hammer-x-jewel-distro-basic-openstack/

run on wip-jewel-no-package-split
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-02_19:34:51-upgrade:hammer-x-wip-jewel-no-package-split-distro-basic-openstack/

@Yuri, so the test passes on latest jewel, but fails on wip-jewel-no-package-split? Weird.

@Nathan Weinberg I created a little noise in this ticket, my apologies.

Now on plan of attack.

Dan and I ran several simplified tests rados:clsonly@ and they all passed:
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-07_18:11:19-rados:clsonly-jewel-distro-basic-openstack/
http://pulpito.ovh.sepia.ceph.com:8081/teuthology-2016-03-07_17:57:03-rados:clsonly-jewel-distro-basic-openstack/
http://pulpito.ceph.com/teuthology-2016-03-07_10:11:02-rados:clsonly-jewel-distro-basic-smithi/
http://pulpito.ceph.com/teuthology-2016-03-07_10:12:13-rados:clsonly-jewel-distro-basic-vps/

So I think this indicates to us that the problem is really in upgrade part.
I think it'd be useful to simplify the faulty upgrade job, leave only cls tests, shorten run time and reproduce and then maybe add more logging for troubleshooting.

Latest test with errors failed with error -22
http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-07_02:10:10-upgrade:hammer-x-jewel-distro-basic-vps/44586/teuthology.log

Here is a simplified upgrade test that can be used to reproduce the issue (can run on teuthology box):

ubuntu@teuthology:~$ cat ~/yuriw/_run_tests.yaml 
interactive-on-error: true

kernel:
  kdb: true
  sha1: distro

nuke-on-error: true
os_type: ubuntu
os_version: '14.04'
overrides:
  admin_socket:
    branch: jewel
  ceph:
    conf:
      global:
        mon lease: 15
        mon lease ack timeout: 25
        osd heartbeat grace: 100
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
        mon warn on legacy crush tunables: false
      osd:
        debug filestore: 20
        debug journal: 20
        debug ms: 1
        debug osd: 20
    log-whitelist:
    - slow request
    - scrub mismatch
    - ScrubResult
    - failed to encode map
    - wrongly marked me down
    - soft lockup
    - detected stalls on CPUs
    sha1: 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d
  ceph-deploy:
    branch:
      dev-commit: 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
        osd default pool size: 2
  ceph-fuse:
    client.0:
      mount_timeout: 120
      mount_wait: 60
  install:
    ceph:
      sha1: 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d
  rgw:
    default_idle_timeout: 1200
  s3tests:
    idle_timeout: 1200
  workunit:
    sha1: 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d
priority: 100
roles:
- - mon.a
  - mds.a
  - osd.0
  - osd.1
- - mon.b
  - mon.c
  - osd.2
  - osd.3
- - client.0
  - client.1
sha1: 2c23b6a9d3da77a8ea423b9e578a4f2cfc1f8d3d
suite: upgrade:hammer-x:parallel
suite_branch: wip-testing-14927-jewel
suite_path: /var/lib/teuthworker/src/ceph-qa-suite_wip-testing-14927-jewel
suite_sha1: a400d00a73d2bf5eac67304b475f46a99a6bd41f
tasks:
- exec:
    all:
    - echo America/New_York | sudo tee /etc/timezone
- install:
    branch: hammer
- print: '**** done installing hammer'
- ceph:
    fs: xfs
- print: '**** done ceph'
- parallel:
  - workload
  - upgrade-sequence
- print: '**** done parallel'
- install.upgrade:
    client.0: null
- exec:
    osd.0:
    - ceph osd set sortbitwise
    - for p in `ceph osd pool ls` ; do ceph osd pool set $p use_gmt_hitset true ;
      done
- workunit:
    clients:
      client.1:
      - cls/test_cls_rbd.sh
- print: '**** done cls/test_cls_rbd.sh 4-final-workload'
teuthology_branch: master
tube: smithi
upgrade-sequence:
  sequential:
  - install.upgrade:
      mon.a: null
      mon.b: null
  - print: '**** done install.upgrade mon.a and mon.b'
  - ceph.restart:
    - mon.a
    - mon.b
    - mon.c
    - mds.a
    - osd.0
    - osd.1
    - osd.2
    - osd.3
  - print: '**** done ceph.restart all'
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.smithi.21695
workload:
  sequential:
  - workunit:
      branch: hammer
      clients:
        client.0:
        - cls
  - print: '**** done cls 2-workload'

Per IRC discussion with Josh and Dan:

the next thing I'd do is add logging to the monitor showing all the pool create parameters and before all the returns in OSDMonitor::prepare_new_pool() so we can figure out at least where the EINVAL is coming from joshd1 @ 2:53
existing debug statements don't have enough detail for that
......
we need to change the ceph code