Craton HSM Benchmarks

Craton HSM includes two benchmark suites that measure cryptographic performance at different abstraction levels. This document covers methodology, baseline results, the nine optimizations applied, before/after comparisons, and a three-way head-to-head against SoftHSMv2.

Two reference hosts are used, and the difference between them matters for anything I/O- or SHA-256-bound:

Unless stated otherwise, figures are from L and are single-threaded, --release with LTO and target-cpu=native, medians via Criterion.rs (100 samples; 10 for RSA keygen). SHA-256 through the ABI costs 28-56 us on L and 2.4 us on S for the same 4 KB input -- roughly 15x, almost entirely SHA-NI. Do not compare absolute numbers across hosts.

Benchmark Suites

1. Direct Rust API (benches/crypto_bench.rs)

Measures raw cryptographic throughput by calling Rust functions directly — no FFI overhead, no session management, no PKCS#11 ABI marshalling.

RUSTFLAGS="-C target-cpu=native" cargo bench --bench crypto_bench

Backend comparison benchmarks (RustCrypto vs aws-lc-rs) run automatically when craton_hsm-awslc is available as a dev-dependency.

2. PKCS#11 C ABI (benches/pkcs11_abi_bench.rs)

Measures end-to-end performance through the PKCS#11 C ABI — the same code path that real consumers (OpenSSL, Java SunPKCS11, NSS) use. Loads libcraton_hsm.so/craton_hsm.dll via libloading and calls through C_GetFunctionList.

RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench

Each benchmark iteration includes the full C_*Init + C_* pair (e.g., C_SignInit + C_Sign), which reflects real-world usage patterns. For the SoftHSMv2 comparison, both libraries are loaded in the same process and run identical operations within the same Criterion report, eliminating environmental variance.

---

Phase 1: Baseline (Direct Rust API, RustCrypto)

Initial measurements before any optimization work, using the RustCrypto backend:

---

Phase 2: Optimizations

Nine targeted optimizations were applied across two layers: the cryptographic backend and the PKCS#11 ABI layer.

Crypto-Layer Optimizations

1. RSA Private Key Cache (src/crypto/sign.rs): Parsed RsaPrivateKey structs are cached in a lock-free DashMap keyed by SHA-256(DER). Avoids expensive PKCS#8 DER parsing + bignum reconstruction on every sign operation. Cache holds up to 64 keys with full eviction on overflow.

2. GCM Key ID Fast Path (src/crypto/encrypt.rs): For 32-byte AES-256 keys, the GCM nonce counter lookup uses the raw key bytes directly as the DashMap key instead of computing SHA-256(key). Eliminates a hash computation on every AES-GCM encrypt. AES-GCM encrypt 256B dropped from 1.396 us to 0.600 us (57% faster).

3. Compile-time Tracing Elimination: Added tracing/max_level_info and tracing/release_max_level_info features to eliminate debug! and trace! instrumentation at compile time in release builds.

4. target-cpu=native: Enables hardware-specific instruction selection (AES-NI, AVX2, ADX, MULX). ML-KEM-768 decapsulation improved 25% (179.9 us to 135.3 us) from AVX2 codegen.

5. aws-lc-rs Backend (awslc-backend feature / craton_hsm-awslc crate): FIPS 140-3 validated backend using AWS-LC's assembly-optimized primitives. RSA-2048 verify: 8.3x faster. ECDSA P-256 verify: 4.5x faster. RSA-2048 keygen: 2.3x faster.

Crypto-Layer Results (Direct Rust API)

Backend Comparison: RustCrypto vs aws-lc-rs (Direct Rust API)

Both backends benchmarked with target-cpu=native. The aws-lc-rs backend uses assembly-optimized routines (AES-NI, AVX2, Montgomery multiplication).

Note on AES-GCM Encrypt: The aws-lc-rs encrypt path includes random nonce generation via SystemRandom (OS entropy call per encrypt), while RustCrypto uses a cached deterministic counter nonce. The decrypt path (no nonce generation) shows the true algorithmic difference: aws-lc-rs is 1.8x faster.

ABI-Layer Optimizations

After optimizing the crypto layer, the PKCS#11 ABI layer became the dominant bottleneck. RSA-2048 verify took 26.79 us in direct Rust API calls but 291.3 us through the C ABI — a 10.8x overhead. We traced the hot path and applied four targeted fixes.

6. Async Audit Logging (src/audit/log.rs): Every C_Sign, C_Verify, C_Encrypt call was synchronously computing a SHA-256 hash chain, serializing JSON, and calling fsync() for the audit trail. Moved all expensive work to a background thread via an mpsc channel. The record() method now completes in sub-microsecond. Delivered 48-55% improvement for fast operations like AES-GCM and SHA-256 where audit overhead previously dominated.

7. Cached Object in ActiveOperation (src/session/session.rs): Both C_*Init and the corresponding C_* completion function were re-fetching the key object from a DashMap on every call. C_*Init now caches the Arc<RwLock<StoredObject>> in the ActiveOperation state, so completion functions skip the second DashMap lookup entirely.

8. Thread-Local HSM Reference (src/pkcs11_abi/functions.rs): Every C_* function called get_hsm(), which locked a global mutex and cloned an Arc<HsmCore>. For a typical Init+Operation pair, that was two mutex acquisitions. Added a thread-local cache with a generation counter bumped on C_Initialize/C_Finalize. The fast path now avoids the mutex entirely.

9. parking_lot Mutex for HSM Global (src/pkcs11_abi/functions.rs): Replaced std::sync::Mutex with parking_lot::Mutex for the global HSM singleton. Lower uncontended overhead (spin-then-park vs immediate syscall).

ABI-Layer Results (Before/After, PKCS#11 C ABI)

All three implementations shown. Optimizations affect only Craton HSM; SoftHSMv2 numbers from the same benchmark runs for reference.

Key wins: SHA-256 digest improved 52%, AES-GCM decrypt improved 55% — dominated by eliminating synchronous audit logging overhead (SHA-256 hash chain + file I/O per call). ECDSA P-256 verify improved 39% (RustCrypto) from combined async audit + cached object lookup.

---

Phase 3: SoftHSMv2 Head-to-Head

Three-way PKCS#11 C ABI comparison: Craton HSM (RustCrypto backend), Craton HSM (aws-lc-rs FIPS backend), and SoftHSMv2 2.6.1. All measurements through C_GetFunctionList with dynamically loaded shared libraries, same machine, same Criterion harness. All Craton HSM optimizations applied.

Bold = best result per row. Median values reported.

Analysis

Where Craton HSM wins:

• AES-256 key generation: 5.1x faster — Craton HSM uses direct OS entropy (SystemRandom), while SoftHSMv2 goes through Botan's DRBG layer with additional overhead
• AES-GCM encryption/decryption — SoftHSMv2 does not support null-parameter GCM (requires explicit CK_GCM_PARAMS), so no direct comparison is possible

Where SoftHSMv2 wins:

• RSA-2048 Verify: 6.6x faster — Botan's assembly-optimized Montgomery multiplication with ADX/MULX instructions
• ECDSA P-256 Sign: 4.1x faster — Botan uses precomputed point tables and wNAF scalar multiplication
• EC P-256 keygen: 3.7x faster — Botan's EC scalar generation is heavily optimized
• SHA-256 digest: 1.6x faster — Botan uses platform-specific SHA-256 acceleration

aws-lc-rs vs RustCrypto (same Craton HSM, through PKCS#11 ABI):

• RSA-2048 Sign: 1.4x faster with aws-lc-rs
• ECDSA P-256 Sign: 1.4x faster with aws-lc-rs
• ECDSA P-256 Verify: 1.5x faster with aws-lc-rs
• SHA-256 Digest: 1.7x faster with aws-lc-rs
• AES-GCM Decrypt: 1.4x faster with aws-lc-rs

What This Means

Craton HSM is not the fastest software HSM on every operation. SoftHSMv2's Botan backend has years of assembly optimization behind it. But Craton HSM offers what SoftHSMv2 cannot: memory safety across 19,000 lines of Rust, post-quantum cryptography with nine PQC mechanisms, and a pluggable backend architecture. RSA-2048 sign is within 1.2x of SoftHSMv2. AES-GCM decrypt completes in 4.1 us. AES key generation is over 5x faster. And for post-quantum operations (ML-KEM encapsulation at 51.84 us, ML-DSA signing at 563.3 us), Craton HSM is the only PKCS#11 implementation with numbers to report.

---

---

Phase 4: Audit Trail Throughput

The audit trail sits on the critical path of most PKCS#11 operations — C_Sign, C_Verify, C_Encrypt, C_Decrypt, key generation, and the object and session calls each emit an event (C_Digest notably does not) — so its throughput bounds the rate of audited operations. Optimisation #6 above moved the work off the caller's thread, which fixed latency — but the worker itself still opened the log file, serialised one event, wrote it, and fsync-ed once per event. That capped the whole module at roughly 760 operations per second no matter how fast the cryptography was, and nothing measured it.

Three changes addressed it:

10. Persistent log file handle (src/audit/log.rs): the worker opens the audit file once and keeps it open, rather than calling open() + metadata() per event. On Windows this also removes one ACL-hardening syscall per event.

11. Group commit (src/audit/log.rs): the worker blocks for one command, then drains everything already queued behind it, serialises the whole batch into one buffer, and issues a single write + a single fsync. This is not a durability trade — a record_sync caller is still released only after its own event is on stable storage; the fsync is simply shared with every event that was already waiting.

12. Canonical binary chain encoding (src/audit/log.rs): the chain hash input changed from serde_json to a fixed-width, length-prefixed, injective binary encoding (AUDIT_LOG_FORMAT_VERSION = 1). This shrinks the hashed payload from ~148 bytes to ~40 and removes a per-event allocation. The on-disk NDJSON line is unchanged — it is an interop contract with SIEM consumers — and the verifier dispatches on each record's own format_version, so logs written by earlier builds still verify and a file containing both versions verifies end to end.

Audit Results

Medians of six runs, alternating which binary is measured first and cooling between them. See "Measurement hygiene" below for why that matters.

The first row is the one that matters: it removes a ceiling that sat below every cryptographic operation in the module.

These figures come from a dedicated A/B harness that ran a "before" and an "after" binary alternately against the same workload, which is the only way to compare two builds credibly on this machine.

The same effect through the PKCS#11 ABI

The figures above measure the audit API directly. Reproducing the effect through the C ABI needs care, because Criterion's reported per-iteration time does not show it at all.

record() only enqueues; the worker owes the write. A benchmark that issues N audited operations and stops has not paid for them yet — the queue drains when the module is finalised, outside the measured region. So Criterion reports the same ~4 us per C_Encrypt for both builds while one of them still owes several seconds of fsync.

Measure the whole process instead. It ends when the audit worker has drained, so it accounts for the work actually generated:

AES-256-GCM 4 KB, --measurement-time 5, total wall clock, median of 5, one harness with only the library swapped:

The control — the same build measured as though it were two — is +2.9%, so a 31% difference is comfortably real.

This is strongly host-dependent

The size of the effect tracks fsync latency, so it varies enormously with storage:

On the laptop the worker fell so far behind that even a 30-second Criterion window showed it (per-iteration time degraded 3.55x, versus 1.68x after). On the NVMe server the worker keeps up well enough that the same window shows nothing — 1.04x versus 1.00x, inside the ±3% control. The cost has not disappeared; it has moved into teardown, where the wall-clock method above finds it.

Use the wall-clock method. The window comparison only works on slow storage and will read as a null result on a fast host.

Two ways to measure this wrongly

Both were hit while producing the numbers above, and both look like clean refutations:

• Benchmark an operation that is not audited. C_Digest emits no audit event, so digest benchmarks are flat at every window length and every host.
• Trust Criterion's per-iteration time. It excludes the queued audit work, which on this workload is most of the cost.

---

Backend Comparison on Host S (RustCrypto vs AWS-LC FIPS)

Measured through the PKCS#11 ABI on host S (AMD EPYC, SHA-NI, NVMe), one harness with only the library swapped. RustCrypto is built with insecure-rustcrypto-rsa-private-ops so that its RSA paths can run at all; AWS-LC is --no-default-features --features awslc-backend.

AWS-LC wins everything asymmetric, and RSA verify by 8.2x — closely matching the 8.3x recorded on host L.

SHA-256 is the exception, and it inverts on this host. The Phase 2 table (measured on host L) shows AWS-LC 1.4x faster at SHA-256. On host S, RustCrypto is 1.17x faster. The reason is SHA-NI: the sha2 crate dispatches to the hardware instruction at runtime, and on a CPU that has it there is nothing left for hand-written assembly to win. Host L has no SHA-NI, so the comparison there measured two software implementations.

Choose the backend for the asymmetric and AES numbers, not the digest one, and be aware that any SHA-256 comparison you read is really a statement about the CPU it was measured on.

AWS-LC and RSA availability

The AWS-LC backend is also the answer to the RSA restriction described in troubleshooting.md: CryptoBackend::supports_rsa_private_ops() reports true for it, so C_GenerateKeyPair, C_Sign, and C_Decrypt all work with RSA, as the table above shows.

Note that before the power-on self-test was made capability-aware, an AWS-LC release build failed C_Initialize for the same reason the RustCrypto one did — the POST's RSA known-answer test called the RustCrypto signing path regardless of the configured backend. The POST now runs its algorithm KATs against the configured backend, so an AWS-LC deployment self-tests AWS-LC; it previously tested RustCrypto regardless of what the module was about to use.

Multi-part RSA on this backend has its own scope caveat — signing works and runs on AWS-LC, but through an API outside its FIPS-approved set, and verification still uses RustCrypto. See fips-mode-guide.md before relying on it for a validated deployment.

---

Coverage Added

The suites originally measured single-threaded, fixed-size operations only. Three gaps were closed, each because a whole class of regression was invisible without it.

Object Lookup

Measured against a token holding 257 labelled secret keys:

The two are the same cost, which is the point. ObjectStore::find_objects deliberately scans every object and evaluates the full template against each, including objects that will be filtered out, so the work does not vary with what matches or with whether the caller is logged in — timing cannot reveal how many private objects a token holds. The cost is therefore driven by store population (~125 ns per object here), not by selectivity.

These benchmarks are not a prompt to add an attribute index. A selective search served from an index would be far faster precisely because it stops visiting non-matching objects, which is the property being protected. See performance-tuning.md for why that would leak object existence to an unauthenticated caller.

Payload Scaling

The fixed-size groups cannot distinguish a per-call regression from a per-byte one. These curves can:

At 256 B both are dominated by fixed per-call cost — session lookup, object fetch, audit enqueue, FFI marshalling — which is why throughput is an order of magnitude below the steady state. From 4 KB upward the per-byte cost dominates. A change that moves the small-payload column but not the large one is a per-call regression; the reverse is a primitive regression.

The SHA-256 ceiling of ~110 MiB/s is this host, not the implementation: it has no SHA-NI. Expect several times that on a current server.

Concurrency

pkcs11_concurrent_encrypt and pkcs11_concurrent_sign open one session per thread and run 1, 2, 4, and 8 of them. They exist to catch a scalability regression: every other group is single-threaded, so a new lock on a shared path is invisible to them. A flat or inverted curve from 1 to 4 threads means contention.

Absolute numbers are not published from this reference machine — a thermally throttling 4-core laptop where the control measurement (see below) reaches ±170% on these groups. Run them on a quiesced multi-core host.

A caution these benchmarks taught us

An early comparison of the audit group-commit change measured these groups as part of a full-suite run and reported improvements of 58x and 23x. Re-measuring each group in its own process, against a control, collapsed the difference to 1.0x within a very wide noise band. The first result was an artifact of position in the suite, not a property of concurrent operation.

The cause is worth understanding, because it is a real effect measured wrongly. Before group commit the audit worker sustained roughly 760 events/second. A full benchmark run generates events far faster than that, so by the time the concurrency groups ran — near the end of the suite, after key generation and hundreds of thousands of crypto iterations — the worker was still draining a backlog accumulated by every group before it, and its disk I/O contended with whatever was being measured. Start a fresh process and there is no backlog to contend with, and a single group's short measurement window does not build one.

So the throughput ceiling is real (see Phase 4), and a long-running deployment does hit it. But attributing it to whichever benchmark happened to run last is wrong, and a whole-suite A/B will do exactly that. Measure the audit path directly, or isolate the group.

---

Measurement Hygiene

The numbers above were produced on a laptop-class CPU, and the first attempt at an A/B comparison produced a result that was entirely an artifact of the machine. It is worth recording how, because the same trap applies to anyone reproducing these figures.

Running "before" and "after" binaries in that order, repeatedly, made the "after" binary look 10% slower on EC key generation and 40% slower on RSA key generation — because it was always measured second, on a hotter package. The same comparison with the order alternated and an 8-second cooldown between runs showed both were within noise of each other. A separate measurement taken immediately after a long compile read 3x faster than the identical measurement on a warm machine.

For a comparison to mean anything:

• alternate which binary runs first, and take medians across several pairs;
• insert a cooldown between runs;
• fix the CPU governor to performance and disable turbo where possible;
• never compare a number from one session against a number from another;
• measure a control: run the unchanged build twice, as though it were two different builds, and compare those. That is your noise floor. On this host it is around ±20% for micro-benchmarks, which means a 15% "improvement" is not a result. Every claim in this document either clears its control by a wide margin or is labelled inconclusive.

This machine also lacks the SHA-NI instruction set (Intel added it to mainstream Core parts only with Ice Lake), which makes every SHA-256-bound figure several times worse than it would be on a current server. Absolute values here are not portable; ratios within a single session are.

Post-Quantum Cryptography

PQC algorithms use pure-Rust implementations (ml-kem, ml-dsa crates) — no backend variation.

---

Running Benchmarks

Prerequisites

The PKCS#11 ABI suite loads the built shared library and calls C_Initialize, which runs the power-on self-test. An unsigned local build has no integrity signature, so the POST refuses to start unless the development bypass is set:

export CRATON_HSM_INTEGRITY_BYPASS=unsafe-dev-only

This is a development-only escape hatch — never set it in production. CI sets it globally for the same reason.

RSA groups are skipped on a default release build. Release builds provide no RustCrypto RSA private-key capability (RUSTSEC-2023-0071 / Marvin), so RSA signing and key-pair generation are refused; key-pair generation runs a pairwise consistency test that signs. Both suites detect this, print a note, and run every other group. To measure RSA, use the hardened backend:

cargo bench --bench pkcs11_abi_bench --no-default-features --features awslc-backend

Craton HSM Only

# Direct Rust API
RUSTFLAGS="-C target-cpu=native" cargo bench --bench crypto_bench

# PKCS#11 C ABI (RustCrypto backend)
RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench

# PKCS#11 C ABI (aws-lc-rs backend)
RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench \
    --features awslc-backend --no-default-features

With SoftHSMv2 Comparison

# Linux
SOFTHSM2_LIB=/usr/lib/softhsm/libsofthsm2.so \
    RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench

# macOS
SOFTHSM2_LIB=$(brew --prefix softhsm)/lib/softhsm/libsofthsm2.so \
    RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench

# Windows
SOFTHSM2_LIB=C:/SoftHSM2/SoftHSM2/lib/softhsm2-x64.dll \
    RUSTFLAGS="-C target-cpu=native" cargo bench --bench pkcs11_abi_bench

Installing SoftHSMv2

# Linux (Debian/Ubuntu)
sudo apt-get install softhsm2

# macOS
brew install softhsm

# Windows (portable ZIP)
# Download from https://github.com/nickluck8/SoftHSMv2-x64-MinGW/releases
# Extract to C:\SoftHSM2\

The benchmark harness automatically:

1. Creates a token directory at target/bench-tokens/
2. Generates a softhsm2.conf with absolute paths
3. Initializes a SoftHSMv2 token with the same PINs as Craton HSM
4. Runs identical operations through both libraries

Viewing Reports

open target/criterion/report/index.html    # macOS
xdg-open target/criterion/report/index.html  # Linux
start target/criterion/report/index.html     # Windows

Environment Variables

Known Limitations

• Mostly single-threaded: all groups except pkcs11_concurrent_* run single-threaded. The concurrent groups open one session per thread against the same token; PKCS#11's global singleton state means they cannot be run against two libraries simultaneously
• Warm cache: Keys are pre-generated in setup; measured operations benefit from warm CPU caches
• No AES-GCM comparison: AES-GCM is not compared with SoftHSMv2 due to differing parameter conventions
• RSA keygen variance: RSA key generation time depends on prime number luck; results show high variance
• Release mode only: cargo bench uses --release automatically; debug-mode numbers are not meaningful
• Host-dependent: the reference machine has no SHA-NI, so SHA-256-bound figures are several times worse than on a current server. See "Measurement Hygiene" above before comparing across machines or sessions