Ceph

I just noticed the second osd is now consuming 100% cpu too. Before it was properly running for around 15 minutes. Guess I'll need to downgrade now... :(

Here's the output of dstat http://pastie.org/5687470.text

I'm not sure why it is writing so much now, before the update there was almost no disk activity.

Output of "iostat -xdm 5 /dev/sd*" is here http://pastie.org/pastes/5687479/text

osd.8 is on /dev/sda1 (the one taking 100% cpu right after starting it)
osd.9 is on /dev/sdb1 (the one taking 100% cpu around 15 minutes after starting it)

I just downgraded to 0.48.2argonaut and everything seems to be running normally again now:

Before downgrade:
ii ceph 0.48.3argonaut-1quantal amd64 distributed storage and file system
ii ceph-common 0.48.3argonaut-1quantal amd64 common utilities to mount and interact with a ceph storage cluster
ii ceph-fs-common 0.48.3argonaut-1quantal amd64 common utilities to mount and interact with a ceph file system
ii ceph-fuse 0.48.3argonaut-1quantal amd64 FUSE-based client for the Ceph distributed file system
ii ceph-mds 0.48.3argonaut-1quantal amd64 metadata server for the ceph distributed file system
ii libcephfs1 0.48.3argonaut-1quantal amd64 Ceph distributed file system client library
ii librados2 0.48.3argonaut-1quantal amd64 RADOS distributed object store client library
ii librbd1 0.48.3argonaut-1quantal amd64 RADOS block device client library
ii python-ceph 0.48.3argonaut-1quantal amd64 Python libraries for the Ceph distributed filesystem

After downgrade:
ii ceph 0.48.2-0ubuntu2 amd64 distributed storage and file system
ii ceph-common 0.48.2-0ubuntu2 amd64 common utilities to mount and interact with a ceph filesystem
ii ceph-fs-common 0.48.2-0ubuntu2 amd64 common utilities to mount and interact with a ceph filesystem
ii ceph-mds 0.48.2-0ubuntu2 amd64 distributed filesystem service
ii libcephfs1 0.48.2-0ubuntu2 amd64 Ceph distributed file system client library
ii librados2 0.48.2-0ubuntu2 amd64 RADOS distributed object store client library
ii librbd1 0.48.2-0ubuntu2 amd64 RADOS block device client library
ii python-ceph 0.48.2-0ubuntu2 amd64 Python libraries for the Ceph distributed filesystem

Can you try reupgrading one of the nodes and start it with debug file store = 20? That will tell is what it is writing.

Corin-

Have you tried 0.48.3 again since then? I'd like to get to the bottom of this, if possible... :)

Just a small update - nothing changed so far. The cluster is still healthy, but the osd is still using 100% of one core and writing 11MB/s.

27045 root 20 0 1245m 57m 4188 S 76.6 0.4 40:12.77 ceph-osd

Just another small update - nothing changed so far. The cluster is still healthy, but the osd is still using 100% of one core and writing 11MB/s.

root 27045 72.4 0.3 1290560 57896 ? Ssl 12:54 156:28 /usr/bin/ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

Do you need more information? Do you think the bug will really soon be fixed and a new version released? Otherwise I'd downgrad to 0.48.2 within the next 1-2 days.

Hi Corin-

Can you enable 'debug osd = 20' for a bit and attach that log? I think this is related to 830b8ffa2e53fde37b74de12a053e037ff3f15b0 which fixed a bug in snap trimming, but i'm not quite sure. Thanks!

looks like levedb spinning on background compaction.

his .2 package is quantals, which is leveldb 1.5.. newer than argonaut .3 but older than what we were about to merge in for master. pushing wip-argonaut-leveldb for him to test (that has 1.6).

that fixed it it seems. we could

- update argonaut and bobtail to newer leveldb :/
- link dynamically for quantal builds only
- link dynamically for all deb builds :/

Ok, after around 10 minutes of runtime everything seems normal. Thanks for the fast and great help! :-)

ceph version 0.48.3argonaut-14-g5df37c2 (commit:5df37c228a7992e7878c6acc0f45b1ad1808481c)

root 4281 5.1 0.3 1243308 50288 ? Ssl 19:51 0:28 /usr/bin/ceph-osd -i 8 --pid-file /var/run/ceph/osd.8.pid -c /etc/ceph/ceph.conf

Ok, everything still looks good :). Last question: should I upgrade my whole cluster to this version or will a new argonaut release made the following days? :)

The conclusion:

- quantal had a newer libleveldb than we built statically into our debs
- downgrading made compaction spin
- a test branch that staticall linked a newer version that quantals' (the version that gary prepared for master) fixed it.

Corin: for now just upgrade the others to the branch i pushed. we're going to redo the way we link to leveldb to avoid situations like this, but it is a non-trivial change.

thanks!