Project

General

Profile

Actions
Copy link

Feature #61646

open

vectorize MD5 calculations for ETags

Added by Casey Bodley 11 months ago. Updated 10 months ago.

Status:
In Progress
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
md5 performance
Backport:
Reviewed:
Affected Versions:
Pull request ID:

Description

take advantage of SIMD to speed up object etag calculations

there's a c++ library that uses AVX2 instructions that's worth considering: https://github.com/ashbob999/md5-simd

similar work in golang:
https://dzone.com/articles/parallelizing-md5-checksum-computation-to-speed-up
https://github.com/minio/md5-simd

an earlier contribution in https://github.com/ceph/ceph/pull/42435 tried offloading MD5 calculations to background threads, but did not show a consistent improvement

Actions
Copy link
 #1

Updated by Casey Bodley 10 months ago

ceph already uses the ISA-L crypto library for crypto acceleration, which includes multi-buffer md5 support from https://github.com/intel/isa-l_crypto/blob/master/include/md5_mb.h

The MD5 CTX interface functions are available for 4 architectures: SSE, AVX, AVX2 and AVX512. In addition, a multibinary interface is provided, which selects the appropriate architecture-specific function at runtime.

Actions
Copy link
 #2

Updated by Casey Bodley 10 months ago

  • Status changed from New to In Progress

i pushed a draft async library to https://github.com/ceph/ceph/pull/52385

Actions
Copy link

Also available in: Atom PDF