Ceph

Last week we've upgraded to firefily 80.1 on a production cluster (3 nodes, 18 osds). Upgrade went fine but the speed of the entire cluster dropped drastically. We've benched all OSD's but none seem to be slow. Also no weird messages in dmesg. If i map a rbd device on one of the nodes and do a dd with conv=sync oflag=direct, i get only 10mb /sec. Used to be 170. We use 10gbit networking dedicated osd and cluster network. Problems started after rebooting an osd server. Recovery of 6% degraded data took ages and was exponentionally slower. Untill recovery stalled completly at 1.1%. Until i stopped some clients, then recovery kicked back in again and finished. Still the cluster is very slow. Especially read.

Before We upgraded from 0.72 everything was super fast. I know this isn't a proper sized cluster but it shouldn't be slower after a upgrade right?

Please not we're in te process of imgrating everyhing off this cluster (at 10mb /sec) and after finishing will i be able to restart deamons and such. Please not this is new hardware and all enterprise disks.