gpu4j
Matrix Maths and Half Precision, in Plain Java
Version 0.4.0 · Java 17+ · Apache-2.0 · io.github.craton-co:gpu4j-core
Your platform is Java. The stage that dominates your wall clock is not.
Write CUDA
Now you have a .cu file, a build that needs nvcc, a JNI layer to marshal across, a native artifact per platform in your release pipeline, and a second language in code review. The GPU work is a few hundred lines. The boundary around it is the rest of the year.
Stand up a Python service
Now your data leaves the JVM on every call, and you have a second deployment, a second on-call rotation, a second auth surface, and a network hop in the middle of a hot loop.
Call a method
gpu4j is the third option. The GPU is a typed method call from the code that already owns your data — and the result is checked bit-for-bit against what the standard JVM would have produced.
The whole integration.
try (GpuExecutor gpu = GpuExecutor.open();
GpuArray<float[]> weights = GpuArray.wrap(w);
GpuArray<float[]> input = GpuArray.wrap(x);
GpuArray<float[]> output = GpuArray.allocateFloat(m * n)) {
GpuBlas.gemm(weights, input, output, m, n, k).get();
float[] result = output.toHost().get();
}No kernel to author, no .cu file in your build, no native library to ship per platform, no serialization boundary between your data and your compute.
Two kernels, chosen by the shape of your output.
A GPU has a fixed number of streaming multiprocessors, and a tiled matrix multiply covers the output in blocks. Choose a tile too large for the problem and you make too few blocks to fill the machine — most of the device sits idle while a handful of blocks do all the work.
128×128 tile · a 256×256 output
4 blocks / 30 SMs
Better arithmetic intensity per block — 64 FMAs per 16 shared loads — and nowhere near enough blocks to use the machine.
64×64 tile · the same output
16 blocks / 30 SMs
Lower intensity per block, four times as many of them, and at this size that is the trade that wins.
| Output size | Winning tile | Faster by |
|---|---|---|
| 64×64 | 64×64 | 18% |
| 128×128 | 64×64 | 35% |
| 256×256 | 64×64 | 67% |
| 512×512 | 64×64 | 13% |
| 1024×1024 | 128×128 | 58% |
Square GEMM, kernel time only, one card. The figure is how much faster the winning kernel is than the losing one at each size — a difference in software, with the hardware held constant.
Neither tile wins everywhere, which is why GpuBlas ships both and picks by output size. Small attention heads and large projection layers want opposite answers. You get the right one without asking for it.
One upload. Many kernels. One read-back.
The cost that decides whether GPU work pays for itself is rarely the arithmetic — it is moving data across the PCIe bus. gpu4j is built so the weights cross once and stay put, and only the values that actually change make the trip.
A decode step multiplies by the same weights on every token. Re-uploading them would cost more than the arithmetic does — which is why GpuArray.allocateFloat exists for intermediates the host never reads, and why copyFromHost writes through a pointer the device already holds instead of allocating a new buffer.
The parts a numerical workload actually needs.
Half precision, first-class
Weights dominate GPU memory and binary16 halves them. Half converts in pure Java — verified against the JDK's own implementation across all 2³² float inputs, a check that runs in CI. gemmHalf keeps a full-precision accumulator, because summing K terms in half precision loses percent-scale accuracy at the K a transformer uses.
Data that stays on the device
Weights uploaded once stay resident across calls, so a decode step multiplying by the same weights every token pays the transfer once rather than per token. Intermediates never need a host mirror at all.
Launch overhead, removed
A launch costs host time whether or not the device is busy, and one decode step of a 1B-parameter transformer issues 453 of them. Record the sequence once and replay it: 3.483 ms of host time to issue, 0.550 ms to replay. Identical device work either way.
Ordering you already understand
GpuStream and GpuFuture are the concurrency model you know. One stream runs in order; independent streams overlap. The stream owns its submission handles and releases them, so the commonest leak in this shape of API is not expressible.
Your own kernels, when you want them
Annotate a static method @GpuKernel and the runtime compiles it for the device — no task-graph API, no @Parallel marker on every loop. Where it cannot guarantee an identical result it runs your method on the CPU instead of guessing, and tells you why.
Testable with no GPU present
A GPU library your CI cannot run is a library your CI does not cover. A stand-in backend installs in one line, so code written against this API is unit-testable on any machine.
Where each option actually wins.
| Approach | Second codebase | Second service | API shape | Backends |
|---|---|---|---|---|
| gpu4j | No | No | Typed Java calls | NVIDIA |
| CUDA + JNI / JCuda | Yes — .cu, nvcc, native artifacts | No | Hand-marshalled | NVIDIA |
| TornadoVM | No | No | @Parallel + TaskGraph | OpenCL, PTX, SPIR-V |
| Python inference service | Yes | Yes | Network call | Anything |
TornadoVM
The mature incumbent, and it covers more hardware than we do. If you need OpenCL or non-NVIDIA devices, use it. The trade is the programming model: it asks you to express work as a task graph with parallelism annotated. gpu4j asks for a method call.
JCuda and friends
Complete control — and complete control is the cost. You are writing, building and shipping CUDA C alongside your Java.
A Python service
The right answer when the model is the product. The wrong answer when the numerics are one stage of a JVM pipeline that owns the data on either side.
gpu4j is the API. CratonVM is the runtime.
A JVM that treats the GPU as a first-class target. Measured 2026-09-07 on a verified-quiet host against HotSpot C2 — the standard production JVM — and TornadoVM 4.0.1, same RTX 2060, N = 2²⁴, best of 5, warm, full host→device→host round-trip. Every arm of a row ran in the same session, CPU baselines included. Same hardware throughout; only the software varies.
| Compute kernel (16.7M+ elements) | CPU (HotSpot C2) | TornadoVM GPU | CratonVM GPU | vs HotSpot | vs TornadoVM |
|---|---|---|---|---|---|
| Integer division chain | 2,179 ms | 27 ms | 7 ms | 311x faster | 3.9x faster |
| Double division chain | 1,784 ms | 135 ms | 82 ms | 21.8x faster | 1.6x faster |
| 128 multiply-adds per element | 1,298 ms | 26 ms | 7 ms | 185x faster | 3.7x faster |
| Dot-product reduction | 1,168 ms | unimplemented | 2 ms | 584x faster | n/a |
| Ray tracer kernel, 7680x4320 | 837.1 ms | 24.29 ms | 12.29 ms | 68x faster | 2.0x faster |
Every kernel output is bit-identical to HotSpot's. Speed that changes your answers is not speed.
One nuance worth stating rather than glossing: on the double div-chain, the two runtimes' kernel outputs match exactly (OUT0=45.40459327514974 from both), while the benchmark's own epilogue — a serial sum of 2²⁴ doubles — differs in the 11th significant figure, because floating-point addition is not associative and the sum is reassociated. TornadoVM's divergence on that row is 5.1×10⁻³, eight orders of magnitude larger, and that one is a real division difference. The two are not the same kind of thing.
Better to find these out here than in week three.
The supported surface
GpuBlas, GpuArray, Half and the stream and graph APIs are hand-tuned and supported. Use them and you get the behaviour described above.
@GpuKernel is narrow
The automatic path covers a deliberately narrow eligibility subset of Java. It is not a promise that arbitrary code runs on the accelerator. Where the runtime cannot guarantee an identical result it falls back to the CPU and says why. Slower and correct beats faster and wrong.
NVIDIA only
The device side needs CratonVM built with GPU support and an NVIDIA GPU. There is no CUDA driver in this library — it is the Java surface, and the hardware work happens underneath.
Pre-1.0, and not yet on Central
The API may break between minor versions until 1.0; pin your version. Namespace verification is still outstanding, so today you build from source or mvn install locally — the published coordinates are the ones it will release under.
Bring us a workload.
A batch that runs too long, a model you cannot serve economically, a pipeline stage that dominates your cluster bill — we will benchmark it honestly against what you run today, and tell you if the answer is no.