CratonVM Benchmarks

Methodology, full notes, and provenance for the numbers in README.md. The headline rule for every table we publish: checksums must match HotSpot on every run — a fast wrong answer is a bug, not a result.

CPU benchmarks

Harness

All seven CPU rows come from one unified harness, bench/CratonBench.java: arithmetic, fib, sieve, matrix, hashmap, stringregex, and bintrees, each runnable in-process or as an isolated phase (CratonBench <phase>). Each phase prints its time and its checksum.

Also in bench/, the per-kernel harnesses, each likewise printing time and checksum — useful when you want one row without the others, and what to fall back on if the unified harness goes missing again:

Whichever harness you use, compare before/after on one host with one binary pair. That is the methodologically sound comparison regardless of harness, and on this shared box it is often the only trustworthy one.

Methodology

• Both VMs run the same flags (notably -Xmx8g for Binary Trees).
• Fresh process per measurement, alternating HotSpot / CratonVM runs.
• Pinned to one logical CPU with taskset on the benchmark host (Azure Linux, EPYC 9V45, SMT).
• Medians over 5–7 runs; no samples discarded.
• Reference JDK: Temurin JDK 25.0.3 (C2).
• Measurements are taken in quiet windows (1-minute load average below 2); the shared host's load otherwise inflates both columns — and CratonVM's memory-heavy rows more — so ratios, not absolute times, are the durable content across host re-provisionings.

Current table: Multi-Configuration Comparison (HotSpot vs CratonVM)

Measured on the Azure EPYC host against OpenJDK JDK 25 LTS (HotSpot C2 with default G1GC). CratonVM release build (d8a690353, dev tip) tested across collectors in default strict mode (--jdk-only):

1. Def-ZGC: Default strict mode (--jdk-only) + default ZGC (-XX:+UseZGC)
2. Def-G1: Default strict mode (--jdk-only) + G1 collector (-XX:+UseG1GC)

All runs executed in isolated fresh processes per phase/variant (-Xmx16g for HashMap 100M; -Xmx8g for other phases). Checksum verification: 100% exact match across all runs and configurations (zero mismatches).

Key observations across configurations:

• Numerical kernels & parity: Sieve beats HotSpot by 26% (0.74x under G1, 0.83x under ZGC). Matrix beats HotSpot under both collectors (0.92x under G1, 0.97x under ZGC). Arithmetic scales linearly and stays within ~6-14% of HotSpot (1.06x at 1.5B, 1.09x at 2B under G1).
• ZGC vs G1: ZGC holds a ~2x advantage on large HashMap in Default mode (15.5s vs 30.3s at 100M) and on String/Regex (15.1s vs 17.2s at 10M). Conversely, G1 is significantly faster on deep object-tree allocation and destruction in Binary Trees (3.0s vs 17.8s at depth 20; 556 ms vs 805 ms at depth 18).

† Sieve: HotSpot is bimodal on this phase

The Sieve row's HotSpot figure is the median of 18 samples pooled from two independent series, and the pooling is a correction rather than a convenience.

HotSpot on this phase lands in one of two modes and nothing between them:

A 9-sample median therefore reports whichever mode won the coin toss: one series read 2,386 ms, the next read 2,734 ms, on an unchanged binary and an unchanged JDK. CratonVM's own 18 samples over the same runs are unimodal (2,276–2,498 ms, CV 2.3%).

Pooled, the two are 2,360 against 2,333 — parity. Quoting the cleanest single series would have given a double-digit swing in either direction depending on which mode the median fell in. Parity is what the data supports. Do not re-derive this row from a single 9-sample run.

‡ Fibonacci: a known shape

Both JIT backends erase the dead shadow-stack thread fetch from the prologue; the single-pass backend jumps over the erased ~46-byte span, and the IR backend previously overwrote it with one-byte NOPs, so every IR method that published nothing retired 46 NOPs on entry, on every invocation. That is fixed. The residual gap to the pre-IR single-pass body (near 2.9x) is precise-root and deopt metadata the IR tier emits and the single-pass backend did not: the safepoint-id slot the collector reads to pick an oop map, and the innermost-RBP mirror the stack walker reads. Nothing today compares an IR body against the C1 body it replaces before keeping it; that is an open policy question, not metadata to be deleted.

The optimizing tier declines a method whose loops the single-pass backend would lower better, and what that backend can do and the IR tier cannot is enumerated in jit/src/x64/single_pass_only.rs rather than discovered one regression at a time.

§ Growth across N: Binary Trees / HashMap / String-Regex under G1 (2026-09-10)

Measured with a one-off sized harness (CratonBenchSized, one phase + one size per argv, not part of the checked-in bench/ tree) so each of these three GC/allocation-heavy kernels could be run at three sizes in one interleaved series instead of the single proxy size the rest of this document uses. CratonVM ran with --XX:UseGc G1; HotSpot needed no flag, since G1 has been its default collector since JDK 9. This is a deliberate collector switch for these three rows only — the rest of this table, and the codebase default, is the Generational collector (see README.md Highlights).

Azure host (dev @ 330773e1, fat-LTO, sha256 bc2bb8b0a044…), CPU 7 pinned via taskset, alternating CratonVM/HotSpot arms, 9 reps per size/kernel combination, no sample discarded, checksum-verified every rep. Load stayed at 2.1–2.3 for the entire series (quietest of any run in this document); memory headroom was 25GB+ available throughout, including the 100M-entry HashMap run given -Xmx24g. Zero checksum mismatches — every rep, every size, both VMs agree exactly.

Binary Trees and String/Regex compound with N; HashMap does not. Binary Trees' ratio rises steadily as depth increases (5.29x → 6.53x → 7.40x, each step roughly quadrupling live-node count). String/Regex compounds far more sharply — the ratio nearly triples at each 10x step in N (4.48x → 16.81x → 50.13x) — consistent with CratonVM's per-match String allocation (Matcher.group(1)) and StringBuilder growth paying an increasing GC tax as the retained string and match count both grow. HashMap's ratio is non-monotonic (12.29x → 5.29x → 10.52x): the 1M run completes in 553 ms, short enough that fixed per-process costs (JVM/VM bring-up, class loading) are still a real share of both columns, which is the more likely explanation than a genuine reversal of the scaling trend — this row would need a finer-grained sweep (e.g. 1M/3M/10M/30M/100M) to separate "small-N fixed overhead" from "true non-monotonic scaling" with confidence, which this series does not attempt to do.

G1 vs the Generational default, same binary, isolated same-day measurement (CratonVM-only, no HotSpot column — this isolates the collector's own effect):

G1 is a large, uneven win, not a uniform one: it more than halves Binary Trees' wall time at every depth tested, helps HashMap and small String/Regex modestly, and measurably regresses String/Regex at 10M — the collector that helps most on one allocation-heavy kernel is the one that hurts on another. (The String/Regex 1M row is omitted from this delta table: the Generational-side sample for that size landed exactly as an unrelated session's build spiked host load to 8.4, so that specific comparison isn't trustworthy — see the raw sample data below.)

Raw samples, manifest, and environment record for both series: /data/cratonvm/regression-suite/perf/results/vs-hotspot-sized-g1-20260910T125919Z/ (G1) and /data/cratonvm/regression-suite/perf/results/vs-hotspot-sized-20260910T121415Z/ (Generational) on the Azure benchmark host.

Sieve was 6.50x yesterday

That was a live regression, not a measurement problem, and it is fixed. CratonBench.sieve([ZI)I had begun receiving a body from the optimizing (C2/IR) tier where it previously fell through to the single-pass backend, which vectorises its boolean[] loops; the IR body was 6.4x slower than the C1 body it replaced (2,462 ms → 15,823 ms, checksums identical). Fixed: the optimizing tier's admission chain now declines a method whose loops the single-pass backend would lower better, and the seven classes of lowering that backend has and the IR tier lacks are enumerated in jit/src/x64/single_pass_only.rs rather than discovered one regression at a time.

Fibonacci's own gap is explained above (see ‡ Fibonacci: a known shape), not by this section's sieve regression.

Row notes, carried over from the table this replaced:

• RETRACTED: the HashMap regression this table used to record (22,077 ms, 21.2x, "CONFIRMED and bounded to a36b9d121..e57f0bc7d") does not reproduce, and the bisect it recommended has nothing in it. Measured before any change, dev @ 9ac1feffe, isolated fresh processes, -Xmx8g, alternating arms pinned to cpu 15 with mpstat -P ALL confirming that core was 90-100% the measuring process:

  n	HotSpot	CratonVM dev	ratio	what this table said
  1,000,000	66 ms	427 ms	6.5x	3,084 ms / 42.7x
  10,000,000	997 ms	3,523 ms	3.53x	22,077 ms / 21.2x

  427 ms at n=1M is within noise of the 434 ms this same note records for the healthy a36b9d121 (07-18) binary, so there is no 7.1x delta left to bisect for. Why the 07-25 readings were ~5x too slow is not established. The obvious candidate — the cpu-13 collision warned about further down this file — was tested directly (same binary, same phase, alternating cpu 13 and cpu 15) and showed no difference: 4131/4004 on cpu 13 against 4063/4042 on cpu 15. Treat any 07-25-era absolute in this document as unverified until re-measured. Full detail: hashmap-half-gap-20260730.md.

  String/Regex's 07-25 row is left as recorded — it has not been re-measured and it was the smaller of the two claims (3.57x → 7.7x), but it came out of the same session, so it deserves the same scepticism.

• HashMap's 1.75x is that corrected baseline plus a real 72.3% gap reduction. From 3,776 ms to 1,780 ms against HotSpot's 1,017 ms, medians of five interleaved fresh-process runs at load 3.4-3.8, every checksum 1549999915000000; the repository's own gate independently reports PASS median 1767ms. A second series at load 10-12 reads 4,065 / 1,870 / 1,062, i.e. 73.1% — the reduction is stable across load levels. Two causes: the Integer-keyed dense overlay was maintaining a second full FxHashMap purely to remember insertion order it could derive from the key (19.7% of the phase in note_fresh_insert alone), and a set of per-object fixed costs — get_header, the ObjectRef provenance bitmap, the single-OS-thread tripwire, a2dbg::record, and re-deriving a monomorphic checkcast's answer — that were each an out-of-line call around a no-op or an idempotent update. The host was at load 10-12, so both columns are inflated; the ratio is the durable content.

• Fibonacci is recursion-bound and its recursive self-call compiles to a guarded direct call. The closeout removed the Linux jit_frame_record helper from the prologue and both post-recursive-call restoration sites, replacing each with one sentinel-probed fs: TLS store. It also permits a metadata-only empty root map when moving-young analysis proves the recursive caller has no live oops. The merged-binary alternating acceptance reduced the HotSpot gap by 71.90%. Full measurements and generated-code evidence are in fibonacci-half-gap-20260730.md.

• Binary Trees was measured at -Xmx8g as seven alternating fresh-process pairs; all fourteen checksums were 68332206. A July 2026 dev regression that temporarily quadrupled this row was root-caused (two independent causes) and fixed. The perf gate's anchored baseline is 1,550 ms, reflecting a deliberate ~2% correctness hardening (explicit header initialization in the inline allocator) accepted after that fix. A follow-up (TLAB zero-elision for the fields that don't need it, plus a GC-inert proof for itemCheck's allocation-free self- recursion) cut a further 14.5% off the wall-time in a clean 10-round interleaved Azure measurement (1,529.5 ms candidate vs 1,789.5 ms origin/dev, 177 ms HotSpot — 16.1% HotSpot-gap reduction). An accompanying attempt to also cap the initial young semispace at 512 MiB measured as a 12-13% regression instead (it multiplied a pre-existing "young GC always falls back to non-moving sweep for this workload" defect by forcing ~6x more young collections) and was reverted. Full measurement history, isolation methodology, and the root-cause writeup are in binarytrees-bt18-half-gap-20260730.md.

• Sieve is three counted boolean[] loops, and single-pass BCE refuses inclusive (<=) loops and non-arr.length bounds, so every element kept a null and bounds check. A later change added three fall-through-only guarded preheaders — a block clear, a strided store, and the whole sieve nest, the last of which scans eight bytes at a time for the next unmarked index. Two independent 9-round interleaved same-binary A/B measurements (CRATONVM_JIT_BULK_BYTE_LOOPS=0 as the dev-equivalent control, all 54 runs checksum 9592) cut the HotSpot gap by 94.35% and 95.30%, moving the ratio from 1.85x to 1.05x. Full measurements, the guard contract, and the differential probe are in cratonbench-sieve-half-gap-20260730.md.

LICM guard attribution: what the blanket refusal cost (2026-09-18)

tools/probes/GuardedDivHoist.java is a counted loop with an idiv, whose divide-by-zero Op::Guard sits in the body, and an invariant this.bias read. ir_optimize::LoopGuards attributes that guard to d, not this, so the read is hoisted. CRATONVM_JIT_IR_LICM_GUARD_ATTRIBUTION=0 restores the blanket refusal in the same binary. Release build, Windows desktop, three runs per arm, same checksum everywhere:

The attribution is the whole gap to HotSpot on this shape (about 14%). String.charAt loops cannot show it: they are pinned to the single-pass tier and never reach LICM. NOTES-opts8.md §4 has the guard-token arms and the negative control.

The performance gate

Perf regressions are guarded by regression-suite/perf/run-cratonbench-gate.sh, which runs every phase isolated (pinned, -Xmx8g, median of 5), verifies the exact checksum on every run, enforces a 5% budget over the per-host anchored baselines in cratonbench-baseline-azure-epyc.tsv, and refuses to measure on a loaded host rather than produce noisy verdicts. Anchored baselines may only be re-anchored with a linked evidence document.

This gate was absent from dev for a period — if you are working on a commit that predates the restore, it will not be there.

Even with the gate available, use the process below for a perf-affecting change whenever you cannot get a quiet host — the gate deliberately refuses to measure under load, and the shared Azure box frequently runs at load 5–8 with several sessions building:

1. Build the pre-change commit and the post-change commit as two uniquely-named binaries on the same host.
2. Run the relevant per-kernel harness from the table above against both, alternating, several runs each.
3. Compare checksums first. A changed checksum means the change is wrong; a faster wrong answer is a bug, not a result.
4. Report absolute times for both binaries and the host's load average, since this host is shared and contended — ratios between your two binaries are meaningful, absolute numbers across hosts and dates are not.
5. perf record -g --call-graph=dwarf -F 199 on both, and report the symbol deltas, not just wall clock.

Optimization history

The journey from the earliest 20–100x gaps to the current table is documented round by round:

• docs/JIT_OPTIMIZATION.md — the current JIT architecture and the optimizations that produced the table above

GPU benchmarks

Setup

Measured on a GeForce RTX 2060 (sm_75, driver 591.86) against HotSpot JDK 25.0.3 (C2) and TornadoVM 4.0.1 (PTX backend, @Parallel/@Reduce + TaskGraph API). CratonVM offload is opt-in and automatic within a narrow envelope: the kernels below are plain static methods over primitive arrays with no annotations and no API at the call site, but they require a GPU build and an explicit flag (cargo build --features gpu-driver, run with --gpu), and anything the analyzer doesn't accept stays on the CPU. All timings are warm and include the full per-call H2D + kernel + D2H round-trip. There is no self-hosted GPU hardware CI, so these are point-in-time measurements rather than a continuously enforced budget.

Ray tracer rows are 6 interleaved rounds each (arm order alternated per round to cancel drift), full H2D+kernel+D2H, checksums bit-identical to HotSpot; the 3840×2160 row is pooled over two independent 6-round passes (12 rounds total, craton faster in all 12) run 30 minutes apart, which agreed within noise. The kernel is the reduced proxy (bench-gpu/RayTracerKernel.java), documented in raytracer-vs-tornadovm-RESOLVED-20260821.md in the internal tree, not the full apps/TornadoVM-Ray-Tracer app (whose real kernel — reflections, soft shadows, a skybox — needs dynamic-length scene loops neither engine's analyzer admits yet). The margin over TornadoVM shrinks with resolution (2.5x → 2.2x → 2.0x → 1.85x, the last at 11520×6480 / 74.6M pixels, 6/6 rounds) as CratonVM's fixed per-launch cost advantage amortises away, leaving a smaller but still consistent per-pixel-throughput edge. Every frame from 160×120 to 11520×6480 is bit-identical to HotSpot's.

Re-verified 2026-09-05 on the same box (RTX 2060, TornadoVM 4.0.1 PTX, N = 2²⁴). The four non-ray-tracer rows still hold and TornadoVM's integer div-chain reproduced exactly at 26 ms. Two things that run did change:

• The dot-product row is now 2 ms, not 12 — the warp-shuffle reduction (one red.global.add per warp rather than per thread) landed after the original measurement.
• The double div-chain row had silently stopped reproducing: it measured 9,276 ms, slower than HotSpot, because runtime::offload_jit_gate asked the constant-pool-free analyzer, which rejects ldc2_w unconditionally, while the dispatcher it gates for asks the pool-aware one. That kernel's + 1.0000001 is an ldc2_w; its integer twin's + 12345 is a sipush with no pool entry, which is why one row worked and the other did not. Fixed the same day; the row now measures 81 ms.

The CPU columns above are the original idle-box measurements and were not re-taken — that re-run shared the host with an unrelated build, which makes a CPU baseline pessimistic and every ratio derived from it flattering. The GPU figures quoted in this note were measured under that same load, so they are conservative rather than optimistic. The ray-tracer row was not re-run: it needs craton-gpu-0.2.0.jar and a cratonvm-gpuray binary that are not present in this tree.

Re-measured 2026-09-07, and this table is now that measurement rather than an accumulation of three vintages. Quiet host (bench-gpu/wait-for-quiet.sh gated it), target-gpu/release/cratonvm.exe, HotSpot Adoptium 25.0.3.9, TornadoVM 4.0.1-jdk25-ptx, N = 2²⁴, best of 5, warm, full host→device→host. Reproduce with bash bench-gpu/rerun-table-rows.sh.

Every arm of a row was run in the same session this time, including the CPU baselines — which had previously been carried over from an older idle-box run. They reproduced almost exactly (2,179 vs 2,146; 1,784 vs 1,780; 1,298 vs 1,300; 1,168 vs 1,172), which is the check that the setup is sound: the ratios moved because CratonVM got faster, not because the baseline drifted.

Three rows moved against the previous table, and the double div-chain row finally agrees with the note that has sat under it since 2026-09-05:

• int div-chain 11 ms → 7 ms, and 128 multiply-adds 8 ms → 7 ms.
• double div-chain 95 ms → 82 ms, TornadoVM 128 ms → 135 ms. The note below said "the row now measures 81 ms against TornadoVM's 135" and the table above it still said 95/128. The table was stale; it is not any more.
• The 128-multiply-add row needed a different harness to measure at all — see the offload-gate defect recorded below.

The pre-2026-09-07 notes follow, kept because they explain how rows got where they are:

• The dot-product row is now 2 ms, not 12 — the warp-shuffle reduction (one red.global.add per warp rather than per thread) landed after the original measurement.
• The double div-chain row had silently stopped reproducing: it measured 9,276 ms, slower than HotSpot, because runtime::offload_jit_gate asked the constant-pool-free analyzer, which rejects ldc2_w unconditionally, while the dispatcher it gates for asks the pool-aware one. That kernel's + 1.0000001 is an ldc2_w; its integer twin's + 12345 is a sipush with no pool entry, which is why one row worked and the other did not. Fixed the same day; the row now measures 81 ms against TornadoVM's 135.

The CPU columns above are the original idle-box measurements and were not re-taken — that re-run shared the host with an unrelated build, which makes a CPU baseline pessimistic and every ratio derived from it flattering. The GPU figures quoted in this note were measured under that same load, so they are conservative rather than optimistic. The ray-tracer row was not re-run: it needs craton-gpu-0.2.0.jar and a cratonvm-gpuray binary that are not present in this tree.

Notes:

• FIXED 2026-09-07: a compiled caller silently stopped offloading a kernel in another class. bench-gpu/GpuComputeWarm.java has main() call GpuCompute.heavy in a second class. Under --gpu --print-gpu-decisions that method never appeared in the decision log at all — not Rejected(...), never asked — and the row read 2,611-2,934 ms on the CPU against 7 ms on the device. The same kernel declared beside main() (GpuComputeWarmSelf.java) offloaded normally, which is what made it look like an analyzer problem. It was not.

  Root cause: the JIT has two one-way doors for a static call site — bind it directly to the callee's entry (jit/src/lib.rs) and inline it (jit/src/x64/bytecode_walk.rs). Both were gated on offload_hook::is_kernel, a lookup in a registry that offload_jit_gate fills as a side effect of scanning callers — and that scan cannot judge a target whose declaring class is not loaded yet. A caller is scanned when it is admitted to the JIT, which happens before it runs, so a callee in another class has typically never been touched at that moment. main here fills two 2²⁴ arrays first, so it is compiled at exactly the wrong time. The site was bound directly, the dispatch helper the offload hook lives behind was gone, and try_compiled_offload's late registration (the 2026-09-06 fix for the same underlying limitation) had no site left to run on.

  The fix is one predicate. A registry miss means either "not a kernel" or "could not have known yet", and these doors treated the two identically while making a decision that is irreversible. offload_hook::keeps_dispatch_helper now falls back on a miss to a descriptor-only test — )V/)I/)J with an array parameter, mirroring target_can_ever_dispatch — which needs no class loading and no locks. Keeping a helper is reversible and cheap; binding directly is neither.

  Verified: GpuComputeWarm went 2,934 ms → 6 ms, and the census now prints registered late, by the compiled site: GpuCompute.heavy([I[I[I)V. Across five runs each on a quiet host the two harnesses are indistinguishable (cross-class 5-6 ms, same-class 5-6 ms), which is the property that was broken. test_classes/gpu/GpuForwardRef.java passes on both arms with equal checksums. GpuHookOverheadBench's base_ns_per_call — a loop calling an ineligible target, which is what a broader predicate would have taxed — reads 7.07 ns with --gpu against 7.58 ns without, so the CPU side pays nothing; for scale, the old caller-refusal approach cost 407.9 ns there.

  bench-gpu/rerun-table-rows.sh now measures this row with the cross-class harness on purpose, and keeps the same-class one as a control arm. If the two ever diverge again, that is what regressed.

• The div-chain rows are the "GPU wins big" cases: division has no competitive CPU-vectorized form, so raw parallelism wins at every size tested (2²⁰–2²⁶; the ratios hold steady across sizes).

• CratonVM's double-division kernel output is bit-exact with HotSpot (div.rn.f64 is IEEE-754 round-to-nearest, same as x86 vdivpd): the 2026-09-07 run prints identical OUT0=45.40459327514974 and OUTN=7.2490508729547125 from both.

  Its harness checksum does not match, and the distinction matters. The checksum is a serial sum of 2²⁴ doubles in the benchmark's own epilogue; floating-point addition is not associative, so summing in a different order gives a different but equally correct total. CratonVM reads 5.928010028745152E7 against HotSpot's 5.92801002867254E7 — a relative difference of 1.2e-11, the size of a reassociated sum.

  TornadoVM's 5.9583712290819384E7 is 5.1e-3 away, eight orders of magnitude larger, and that one is a division difference: its PTX backend does not guarantee bit-exact division. Do not read the two divergences as the same kind of thing.

• Root-caused: TornadoVM's unimplemented is a standing gap in mixed-type reductions, not a version/driver issue. TornadoSnippetReflectionProvider .forBoxed (what the whole stack trace bottoms out in) is an unconditional stub — unimplemented(); return null; — in both the TornadoVM 4.0.1 jar on this box and the current master branch on GitHub, so upgrading would not fix it. Isolated the exact trigger with three minimal repros on this same GPU: a single-array LongArray sum reduction works; a two-array LongArray+LongArray sum reduction (no cast, no multiply) works; a single-array IntArray reduced into a LongArray accumulator with one (long) a.get(i) widening cast fails with the identical stack trace. So the precise gap is a @Reduce kernel whose per-element expression needs a primitive widening conversion (int→long) before accumulating into a differently-typed reduce array — not "long reductions" or "two-array reductions" in general. GpuDotBench.dotReduce's int·int → long shape (needed to avoid int overflow in the product) is exactly that mixed-type case, and there's no workaround that preserves what the row measures, so "unimplemented" here is a real, durable TornadoVM limitation, confirmed by reading the source rather than assumed from the error message. CratonVM's automatic --gpu path handles the same reduction and completes the full dispatch-and-readback (an is_reduction: true kernel), not just a CPU fallback, as an earlier note assumed.

• The multiply-add row's old "honest counter-case" framing was a compiler artifact, not a real result. The previous kernel used a compile-time-constant multiplier (x = x*1103+12345, repeated 96 times); composing an affine map with itself under constant coefficients is itself affine, and HotSpot C2's GVN/reassociation folds the whole chain into a single multiply+add with closed-form coefficients regardless of the unroll count — confirmed by unrolling the old kernel to 6,528 lines and seeing under 2x change in wall time. The benchmark was silently memory-bandwidth-bound on the HotSpot side, not compute-bound, which is why AVX2 looked "competitive." Sourcing the multiplier from a second per-element array (m = b[i], the same trick GpuDivChain already uses for its divisor) removes the closed form: HotSpot now does genuine work and the row flips from "GPU loses" (0.7x) to another 163x GPU win. The dot-product row's swing (0.4x → 98x, and 586x since the warp-shuffle reduction) is different in kind: sum += p was never foldable (both operands are runtime array reads), so scaling its per-element repeat count from 1x to 300x — needed to push HotSpot over 1 second — genuinely shifts the kernel from launch/PCIe-overhead-bound (GPU loses on a single multiply-add) to compute-bound (GPU wins). Both rows' TornadoGpuCompute/GpuWarm/GpuCompute and GpuDotBench sources carry AUDIT comments with the full detail and the measurements that back them up.

• FIXED: dot-product CratonVM-GPU background cudarc panic. Every --gpu run of GpuDotBench used to also print one cudarc panic per completed dispatch to stderr — DriverError(CUDA_ERROR_NOT_PERMITTED, "operation not permitted") out of cudarc's CudaStream::drop. Root cause: dispatch_async's cuLaunchHostFunc completion callback (vm/src/runtime/offload.rs) cloned Arc<StreamSubmission> for its own use and let that clone drop locally at the end of the closure. For a synchronous (non-Future-API) dispatch — exactly GpuDotBench's transparent --gpu path, which never calls register_submission — that clone reliably ends up the last strong reference by the time the driver fires the callback, so its drop (cascading into cuda_bridge::Stream's and cudarc's CudaStream's destructors, which issue a real CUDA driver call) ran from inside the callback — forbidden per CUDA's own cuLaunchHostFunc rules (and per this codebase's own doc comment on Stream::add_host_callback). Fix: the completion reaper's queue (REAPER_QUEUE) now carries the Arc<StreamSubmission> alongside the handle, so the callback moves ownership into the queue instead of dropping it locally — the eventual final drop, if any, now happens on the reaper thread (an ordinary thread, where CUDA driver calls are allowed) rather than inside the CUDA callback. Verified: 3 repeated RUST_BACKTRACE=1 trials of the exact repro command with zero panics, correct DOT_CHECKSUM == DOT_REF and unchanged ~12ms timing every time; cargo test -p cratonvm-vm --features gpu-offload --lib offload (35 tests, including the reaper-specific ones) still passes; the other three kernels (div-chain, float-div-chain, warm MAD) re-verified with matching checksums and no regressions.

Full GPU results — more input sizes, ldc-constant kernels, cold-start numbers up to N = 2²⁸, kernel sources, and eligibility rules — are in docs/gpu/README.md.

Reproducing

⚠ Build with fat LTO for any number you intend to compare against the baselines. [profile.release] is lto = "fat" + codegen-units = 1, and the anchored baselines in cratonbench-baseline-azure-epyc.tsv were measured that way. Building with CARGO_PROFILE_RELEASE_LTO=off — the usual workaround for the OOM SIGKILL below — produces a materially slower binary whose absolute times are not comparable to those baselines. Measured on one commit, two binaries differing only in LTO, interleaved, 3 pairs per phase (host load ~6-7, so absolutes are inflated and the spread is wide — hashmap ranged 23461-31120 within one arm):

phase	LTO=off median	fat-LTO median	fat-LTO effect
bintrees	3019 ms	2262 ms	−25%
hashmap	25484 ms	26196 ms	none
sieve	10262 ms	9552 ms	~−7%

Fat LTO is worth a real amount on bintrees and close to nothing on hashmap and sieve. Use it for anything baseline-comparable — but note what it does NOT explain, below.

LTO=off is fine for relative A/B between two binaries built the same way — that is how the layout-registry win was measured — but say so when reporting, and never quote an LTO=off absolute against a baseline.

# Build. Fat LTO — required for baseline-comparable numbers.
cargo build --release -p cratonvm-cli

# If the final cratonvm-cli rustc dies with "signal: 9, SIGKILL" that is the OOM
# killer during fat-LTO codegen, NOT a compile error. Lower parallelism first:
cargo build --release -p cratonvm-cli -j2
# Only as a last resort, and only for relative A/B (see the warning above):
CARGO_PROFILE_RELEASE_LTO=off cargo build --release -p cratonvm-cli -j6

# CPU, all seven phases in-process:
javac -d bench-classes bench/CratonBench.java
./target/release/cratonvm -Xmx8g -cp bench-classes CratonBench

# One isolated phase (the per-row methodology):
./target/release/cratonvm -Xmx8g -cp bench-classes CratonBench bintrees

# The gated regression check (Linux bench host, refuses to run under load):
bash regression-suite/perf/run-cratonbench-gate.sh -Exe /abs/path/to/cratonvm

# Per-kernel harnesses, for one row without the others.
javac -d bench-classes bench/BinTreesClassic.java bench/HashMapOnly.java
# Binary Trees (depth as argv). Expected checksum at d=18: 68332206.
./target/release/cratonvm -Xmx8g -cp bench-classes BinTreesClassic 18
# HashMap put/get. Expected checksum at n=1,000,000: 15499991500000.
./target/release/cratonvm -Xmx8g -cp bench-classes HashMapOnly

# HotSpot reference on the same host, same flags:
$JAVA_HOME/bin/java -Xmx8g -cp bench-classes BinTreesClassic 18

# Profile rather than guess. This is how the layout-registry lookup was found
# to be ~40% of the Binary Trees run (37.8% in class_layout_for_fields plus
# 4.8% in the SipHash it used for a (u32,u32) key).
perf record -g --call-graph=dwarf -F 199 -o bt.perf -- \
  ./target/release/cratonvm -Xmx8g -cp bench-classes BinTreesClassic 18
perf report -i bt.perf --no-children --percent-limit 1 --stdio