Bug #21603: rocksdb is using slow crc - RADOS - Ceph

Actions

Copy link

Bug #21603

closed

rocksdb is using slow crc

Added by Sage Weil over 6 years ago. Updated over 6 years ago.

Status:

Resolved

Priority:

Urgent

Assignee:

Kefu Chai

Category:

Performance/Resource Usage

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(RADOS):

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

        | |     + 33.60% rocksdb::DBImpl::WriteToWAL
        | |     | + 33.40% rocksdb::DBImpl::WriteToWAL
        | |     | | + 33.35% rocksdb::log::Writer::AddRecord
        | |     | |   + 33.30% rocksdb::log::Writer::EmitPhysicalRecord
        | |     | |     + 22.60% rocksdb::crc32c::ExtendImpl<rocksdb::crc32c::Slow_CRC32>
        | |     | |     | + 21.50% Slow_CRC32

this is from a recent master build. this is probably fallout from 652bc5e8328873c67c6e5f405fd6a48c53ae0a80 ?

strangely we are using the ss4 version of memcmp:

        | |     | | | | | |   |   | + 7.55% rocksdb::(anonymous namespace)::BytewiseComparatorImpl::Compare
        | |     | | | | | |   |   | | + 7.00% compare
        | |     | | | | | |   |   | |   + 6.50% __memcmp_sse4_1
        | |     | | | | | |   |   | |   + 0.10% memcmp@plt

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Kefu Chai over 6 years ago

Assignee set to Kefu Chai

Actions

Copy link

Updated by Matt Benjamin over 6 years ago

Just randomly, is this output just from ceph-osd running under perf?

Matt

Actions

Copy link

Updated by Mark Nelson over 6 years ago

Matt Benjamin wrote:

Just randomly, is this output just from ceph-osd running under perf?

This is output from my gdb wallclock profiler fork. I have to update the documentation, you can run it like:

sudo gdb -ex 'set pagination off' -ex 'attach <ceph-osd pid>' -ex 'source ./gdbprof.py' -ex 'profile begin' -ex 'quit'

It will slow the OSD down pretty significantly and is fairly slow to collect samples, but it's the easiest way to get seemingly good wallclock profiles that I've found so far.

Actions

Copy link

Updated by Kefu Chai over 6 years ago

i set a breakpoint in Fast_CRC32() and Slow_CRC32() when debugging ceph-mon, the breakpoint in Fast_CRC32() is always hit.

and in Choose_Extend(), isSSE42() returns true as expected.

Actions

Copy link

Updated by Matt Benjamin over 6 years ago

@Kefu Chai, that's really elegant work, thanks for the info

Matt

Actions

Copy link

Updated by Mark Nelson over 6 years ago

Kefu Chai wrote:

i set a breakpoint in Fast_CRC32() and Slow_CRC32() when debugging ceph-mon, the breakpoint in Fast_CRC32() is always hit.

and in Choose_Extend(), isSSE42() returns true as expected.

FWIW, after talking to Kefu he tested on debian sid while my run was on:

CentOS Linux release 7.3.1611 (Core)

I'm not sure if that makes a difference here, but I believe radoslaw was also seeing Fast_CRC32 used on Ubuntu 16.04.

Mark

Actions

Copy link

Updated by Kefu Chai over 6 years ago

https://github.com/facebook/rocksdb/pull/2950

Actions

Copy link

Updated by Kefu Chai over 6 years ago

Mark, please let me know if i should update ceph/rocksdb with this fix and pick it up in ceph/ceph if you think we need to fix it ASAP instead of waiting for the fb upstream to merge it.

Actions

Copy link