Ceph

<vikhyat> kefu: Hello good morning o/
<kefu> morning vikhyat
<vikhyat> kefu: Cons of co-locating mon with osd node
<vikhyat> kefu: we have got one more input
<vikhyat> could you please verify
<kefu> sure.
<vikhyat> if this will also impact the performance
<vikhyat> Monitor do lots of cluster maps updates for example we do have lots of `pg map` updates as soon as pg state changes and when MON daemon wants to sync the map to its DB it will call `sync` call which will flush the data from in memory to the disk but `sync` is not per device it will flush the data for all devices in the node it could be root device or OSD devices and if we we co-locate Monitor daemons with OSD daemons then OSD devices will also be busy
<vikhyat> when `sync` will be called by MON daemon.
<kefu> i will check if leveldb are using sync() for syncing.
<vikhyat> kefu: sure
<kefu> looks like we don't call sync(2) directly.
<vikhyat> ahh okay
<kefu> vikhyat, may i have the quotation of this input?
<vikhyat> http://docs.ceph.com/docs/hammer/rados/configuration/mon-config-ref/#data
<vikhyat> kefu: i think this link has it
<kefu> "Ceph Monitors do lots of fsync(), which can interfere with Ceph OSD Daemon workloads."
<kefu> this is what i found.
<vikhyat> right
<vikhyat> yes we got it from here only
<vikhyat> but I was adding more into it
<kefu> "Ceph Monitors do lots of fsync()": this is not accurate.
<vikhyat> kefu: ahh okay
<vikhyat> then we need to fix it

<kefu> vikhyat, ceph monitor does not use fsync() very often.
<vikhyat> kefu: ahh okay
<kefu> it does so when preparing the keyring. and this only happens when the monitor is initialized.
<vikhyat> right only once in life time of that mon
<kefu> but the underlying leveldb uses fdatasync() and msync() for the data consistency.
<vikhyat> okay
<kefu> for unmapped memory and mapped memory chunks.
<vikhyat> okay
<vikhyat> fsync, fdatasync - synchronize a file's in-core state with storage device
<kefu> unlike sync(2), i don't think either of them are trying to sync all the devices on that host.
<vikhyat> ack
<vikhyat> any good one from your side
<kefu> Ceph Monitors flush their data from memory to disk very often, which can interfere with Ceph OSD Daemon workloads if the data store is co-located with the OSD Daemons.
<kefu> as leveldb is using mmap for writing the data.
<vikhyat> got it
<vikhyat> As leveldb is using mmap for writing the data, Ceph Monitors flush their data from memory to disk very often, which can interfere with Ceph OSD Daemon workloads if the data store is co-located with the OSD Daemons.
<vikhyat> what about this

Fixed here : http://docs.ceph.com/docs/master/rados/configuration/mon-config-ref/#data