Ceph

See #381

I manually wiped out the class state machien, but didn't delete class_impl content, and saw:

root@node14:/srv/ceph/mon.node14/class# cclass -a
Loading class: /usr/lib/rados-classes/libcls_rbd.so.1.0.0: rbd 1.2 x86-64
read 197658 bytes from /usr/lib/rados-classes/libcls_rbd.so.1.0.0
10.08.28_16:47:26.151134 mon <- [class,add,rbd,1.2,x86-64,changed]
10.08.28_16:47:27.012301 mon1 -> 'updated' (0)
Loading class: /usr/lib/rados-classes/libcls_sync.so.1.0.0: sync 1.0 x86-64
read 184515 bytes from /usr/lib/rados-classes/libcls_sync.so.1.0.0
10.08.28_16:47:27.051525 mon <- [class,add,sync,1.0,x86-64,changed]
10.08.28_16:47:28.173041 mon0 -> 'updated' (0)
root@node14:/srv/ceph/mon.node14/class# class list
No command 'class' found, did you mean:
 Command 'iclass' from package 'ivtools-bin' (universe)
class: command not found
root@node14:/srv/ceph/mon.node14/class# ceph class list
10.08.28_16:47:33.834434 mon <- [class,list]
10.08.28_16:47:33.835442 mon1 -> 'installed classes: 
 (v []) [active]
 (v []) [active]
' (0)

From the mon0 log,

10.08.28_16:47:26.436554 7f2c5d8a5710 mon.node13@0(leader).class v1 preprocess_query mon_command(class add rbd 1.2 x86-64 changed v 0) v1 from client? [2001:16f8:10:2::c3c3:2e5c]:0/12393
10.08.28_16:47:26.436581 7f2c5d8a5710 mon.node13@0(leader).class v1 prepare_update mon_command(class add rbd 1.2 x86-64 changed v 0) v1 from client? [2001:16f8:10:2::c3c3:2e5c]:0/12393
10.08.28_16:47:26.436640 7f2c5d8a5710 store(/srv/ceph/mon.node13) reading at off 0 of 197688
10.08.28_16:47:26.436970 7f2c5d8a5710 store(/srv/ceph/mon.node13) get_bl class_impl/rbd.1.2.x86-64 = 197688 bytes
10.08.28_16:47:26.436982 7f2c5d8a5710 mon.node13@0(leader).class v1 class name exists
10.08.28_16:47:26.437070 7f2c5d8a5710 mon.node13@0(leader).class v1 class content has not changed, not doing anything

but notice the v1.. the stray file is in class_impl but it's not in the library. Maybe this is where the weird blankness comes from?

In any case, the "bad" class library states are in a 't' subdir on node13, node14. hexdump of 'latest' below:

root@node13:/srv/ceph/mon.node13/class/t# hexdump -C latest
00000000  38 00 00 00 00 00 00 00  73 00 00 00 01 38 00 00  |8.......s....8..|
00000010  00 00 00 00 00 02 00 00  00 03 00 00 00 72 62 64  |.............rbd|
00000020  01 01 00 00 00 01 03 00  00 00 31 2e 32 06 00 00  |..........1.2...|
00000030  00 78 38 36 2d 36 34 01  00 00 00 00 01 00 00 00  |.x86-64.........|
00000040  00 00 00 00 00 04 00 00  00 73 79 6e 63 01 01 00  |.........sync...|
00000050  00 00 01 03 00 00 00 31  2e 30 06 00 00 00 78 38  |.......1.0....x8|
00000060  36 2d 36 34 01 04 00 00  00 73 79 6e 63 01 03 00  |6-64.....sync...|
00000070  00 00 31 2e 30 06 00 00  00 78 38 36 2d 36 34     |..1.0....x86-64|
0000007f