Ceph

sjust@teuthology:/a/samuelj-2016-06-21_10:29:56-rados-wip-sam-testing-distro-basic-smithi/268946

sjust@teuthology:/a/samuelj-2016-06-21_10:29:56-rados-wip-sam-testing-distro-basic-smithi/268958

sjust@teuthology:/a/yuriw-2016-06-20_09:29:55-rados-master_2016_6_20-distro-basic-smithi/267205

sjust@teuthology:/a/yuriw-2016-06-20_09:29:55-rados-master_2016_6_20-distro-basic-smithi/267215

@Samuel Hassine http://pulpito.ceph.com/yuriw-2016-06-20_09:29:55-rados-master_2016_6_20-distro-basic-smithi/267205/ and http://pulpito.ceph.com/samuelj-2016-06-21_10:29:56-rados-wip-sam-testing-distro-basic-smithi/268688/ both use simple messenger...

http://pulpito.ceph.com/haomai-2016-06-22_23:47:00-rados-wip-haomai-testing-distro-basic-smithi/272023/:
this job do ec-lost-unfound task, and hang too. From job file, it first install infernails then do upgrade. Is this cause async msgr also used in infernails since async msgr is buggy in that version.
2016-06-23 05:28:46.760003 osd.3 172.21.15.22:6800/31612 34 : cluster [WRN] failed to encode map e23 with expected crc
2016-06-23 05:28:47.728869 osd.3 172.21.15.22:6800/31612 35 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.730454 osd.3 172.21.15.22:6800/31612 36 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.732226 osd.3 172.21.15.22:6800/31612 37 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:56.631793 osd.3 172.21.15.22:6800/31612 38 : cluster [WRN] failed to encode map e25 with expected crc
2016-06-23 05:28:47.728869 osd.3 172.21.15.22:6800/31612 35 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.730454 osd.3 172.21.15.22:6800/31612 36 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.732226 osd.3 172.21.15.22:6800/31612 37 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:56.631793 osd.3 172.21.15.22:6800/31612 38 : cluster [WRN] failed to encode map e25 with expected crc
2016-06-23 05:28:47.728869 osd.3 172.21.15.22:6800/31612 35 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.730454 osd.3 172.21.15.22:6800/31612 36 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:47.732226 osd.3 172.21.15.22:6800/31612 37 : cluster [WRN] failed to encode map e24 with expected crc
2016-06-23 05:28:56.631793 osd.3 172.21.15.22:6800/31612 38 : cluster [WRN] failed to encode map e25 with expected crc
2016-06-23 05:28:57.801069 osd.3 172.21.15.22:6800/31612 39 : cluster [WRN] failed to encode map e27 with expected crc
2016-06-23 05:28:59.162718 osd.3 172.21.15.22:6800/31612 40 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.164540 osd.3 172.21.15.22:6800/31612 41 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.166803 osd.3 172.21.15.22:6800/31612 42 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.212481 osd.3 172.21.15.22:6800/31612 43 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:57.801069 osd.3 172.21.15.22:6800/31612 39 : cluster [WRN] failed to encode map e27 with expected crc
2016-06-23 05:28:59.162718 osd.3 172.21.15.22:6800/31612 40 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.164540 osd.3 172.21.15.22:6800/31612 41 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.166803 osd.3 172.21.15.22:6800/31612 42 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.212481 osd.3 172.21.15.22:6800/31612 43 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:57.801069 osd.3 172.21.15.22:6800/31612 39 : cluster [WRN] failed to encode map e27 with expected crc
2016-06-23 05:28:59.162718 osd.3 172.21.15.22:6800/31612 40 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.164540 osd.3 172.21.15.22:6800/31612 41 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.166803 osd.3 172.21.15.22:6800/31612 42 : cluster [WRN] failed to encode map e28 with expected crc
2016-06-23 05:28:59.212481 osd.3 172.21.15.22:6800/31612 43 : cluster [WRN] failed to encode map e28 with expected crc

Then from osd.3.log:
2016-06-23 05:29:15.579763 7f9874f8f700 2 osd.3 28 got incremental 29 but failed to encode full with correct crc; requesting
2016-06-23 05:29:15.579779 7f9874f8f700 0 log_channel(cluster) log [WRN] : failed to encode map e29 with expected crc
2016-06-23 05:29:15.579784 7f9874f8f700 20 osd.3 28 my encoded map was:
0000 : 08 07 46 13 00 00 03 01 86 0b 00 00 6e d8 6c 7c : ..F.........n.l|

health HEALTH_ERR
            18 pgs are stuck inactive for more than 300 seconds
            12 pgs degraded
            15 pgs incomplete
            3 pgs stale
            15 pgs stuck inactive
            3 pgs stuck stale
            12 pgs stuck unclean
            12 pgs undersized
            13 requests are blocked > 32 sec
            1/3 in osds are down
     monmap e1: 3 mons at {a=172.21.15.49:6789/0,b=172.21.15.49:6790/0,c=172.21.15.49:6791/0}
            election epoch 12, quorum 0,1,2 a,b,c
     osdmap e34: 4 osds: 2 up, 3 in; 27 remapped pgs
            flags sortbitwise
      pgmap v246: 32 pgs, 2 pools, 26160 bytes data, 10 objects
            400 MB used, 278 GB / 279 GB avail
                  15 incomplete
                  12 active+undersized+degraded
                   3 stale+active+clean
                   2 active+clean

The strange to me is osd.3 and osd.0 is disappear without any shutdown/coredump log.

c841ac0e84b63e50ce5fc31441800ad5a39bc5a0 changed min_size to default to k+1. Need to adjust the test.