gpu4j — GPU compute that stays inside your Java application

Matrix maths, half precision and device-resident data, called from the language your platform is already written in. No CUDA toolchain. No JNI. No second codebase, and no second service.

<dependency>
  <groupId>io.github.craton-co</groupId>
  <artifactId>gpu4j-core</artifactId>
  <version>0.4.0</version>
</dependency>

Apache-2.0. Java 17. Runs on CratonVM, which supplies the engine underneath.

---

The problem this removes

Your platform is Java. The numerical stage that dominates your wall clock is not, and today you have two ways to fix that.

Write CUDA. Now you have a .cu file, a build that needs nvcc, a JNI layer to marshal across, a native artifact per platform in your release pipeline, and a second language in code review. The GPU work is a few hundred lines; the boundary around it is the rest of the year.

Stand up a Python service. Now your data leaves the JVM on every call, and you have a second deployment, a second on-call rotation, a second auth surface, and a network hop in the middle of a hot loop.

gpu4j is the third option: the GPU is a typed method call from the code that already owns your data.

---

Multiply two matrices on the GPU

try (GpuExecutor gpu = GpuExecutor.open();
     GpuArray<float[]> weights = GpuArray.wrap(w);
     GpuArray<float[]> input   = GpuArray.wrap(x);
     GpuArray<float[]> output  = GpuArray.allocateFloat(m * n)) {

    GpuBlas.gemm(weights, input, output, m, n, k).get();

    float[] result = output.toHost().get();
}

That is the whole integration. No kernel to author, no .cu file in your build, no native library to ship per platform, no serialization boundary between your data and your compute.

---

What the library gives you

Matrix multiplication that reads your shapes

GpuBlas is a hand-written GEMM reached through a typed Java call — and it ships two kernels, chosen by the shape of your output, because no single tiling wins everywhere.

A 128×128 tile has better arithmetic intensity, but it covers a 256×256 problem in four blocks and leaves most of a 30-SM device idle; a 64×64 tile makes sixteen and fills it. Measured on one card, kernel time only: at N=1024 the large tile is 58% faster; at N=256 the small tile is 67% faster. The winner reverses.

Small attention heads and large projection layers want opposite answers, and you get the right one without asking for it.

Either operand can be read transposed in place — a different stride pair, not a copied tensor and not an extra pass:

GpuBlas.gemm(a, Transpose.TRANSPOSED, b, Transpose.NONE, c, m, n, k);

Half precision, as a first-class type

Weights dominate GPU memory, and half precision halves them. Half gives you IEEE-754 binary16 conversion in pure Java, verified against the JDK's own implementation across all 2³² float inputs and all 2¹⁶ half bit patterns — a check that runs in CI, not one that ran once.

GpuBlas.gemmHalf runs the multiply with half-precision operands and a full-precision accumulator, so the memory halves and the accuracy where it matters does not. Summing K terms in half precision loses percent-scale accuracy at the K a transformer uses; that is not a detail to optimise away later.

GpuArray<short[]> w = GpuArray.wrapHalf(Half.fromFloats(weights));
GpuArray<short[]> x = GpuArray.wrapHalf(Half.fromFloats(activations));
GpuBlas.gemmHalf(w, x, out, m, n, k).get();   // out is float

Data that stays on the device

GpuArray is a device-resident buffer with a real lifecycle. Weights uploaded once stay resident across calls, so a decode step that multiplies by the same weights on every token pays the transfer once rather than per token — the difference between a demo and a deployment.

Intermediates never need a host mirror at all:

GpuArray<float[]> kv = GpuArray.allocateFloat(cacheSize);  // never touches the host

Launch overhead, removed

A kernel launch costs the host time whether or not the device is busy, and a decode step of a 1B-parameter transformer issues 453 of them. Record the sequence once and replay it:

gpu.beginCapture();
for (Step s : steps) s.dispatch();          // recorded, not run
long graph = gpu.endCapture();

for (int i = 0; i < manyTokens; i++) {
    gpu.awaitSubmission(gpu.replay(graph));  // one submission, all 453
}

Measured on an RTX 2060, those 453 launches cost 3.483 ms of host time to issue and 0.550 ms to replay. The device does identical work either way — this is host-side overhead that stops being your bottleneck.

Ordering you already understand

GpuStream and GpuFuture are the concurrency model you know. Work on one stream runs in order; independent streams overlap. Queue a long chain and wait once, or await each step when you need its outcome individually. The stream owns its submission handles and releases them, so the commonest resource leak in this shape of API is not expressible.

try (GpuStream stream = gpu.newStream()) {
    for (Layer l : layers) {
        stream.dispatch(l.cls, l.method, l.descriptor, l.args);
    }
    stream.synchronize();   // one wait covers the chain
}

Your own kernels, when you want them

Annotate a static method with @GpuKernel and the runtime compiles it for the device — no task-graph API, no @Parallel marker on every loop. Where it cannot guarantee an identical result, it runs your method on the CPU instead of guessing, and tells you why.

Testable with no GPU in the room

A GPU library your CI cannot run is a library your CI does not cover. GpuTestBridge installs a stand-in backend, so code written against this API is unit-testable on any machine, in any pipeline, with no device present.

try (var installed = GpuTestBridge.install(fakeBackend)) {
    assertThat(myPipeline.run()).isEqualTo(expected);
}

The library holds itself to that too: 557 tests, with coverage gates that fail the build below 93% of instructions in the core module, 90% in the sidecar, and 75% in any single class.

---

How it compares

TornadoVM is the mature incumbent and covers more hardware than we do — if you need OpenCL or non-NVIDIA devices, use it. The trade is the programming model: it asks you to express work as a task graph with parallelism annotated. gpu4j asks for a method call.

JCuda and friends give you complete control, and complete control is the cost: you are writing and shipping CUDA C.

A Python service is the right answer when the model is the product. It is the wrong answer when the numerics are one stage of a JVM pipeline that owns the data on either side.

---

The engine underneath

gpu4j-core is the API. The execution engine is CratonVM, a JVM that treats the GPU as a first-class target.

Benchmarked against HotSpot C2 (the standard production JVM) and TornadoVM 4.0.1. Same RTX 2060, N = 2²⁴, warm, full host→device→host round-trip — so these compare software against software on identical hardware:

Every result is bit-identical to HotSpot's output. Speed that changes your answers is not speed.

Full methodology and the engine's own story: the CratonVM site.

---

Where it fits

AI. Matrix multiplication, half precision and resident weights are the operations a transformer spends its life in. This library puts them behind a typed Java call, so inference and training-adjacent work can live inside the JVM application that already owns your data, your auth and your deployment pipeline.

Big Data. Your Spark jobs, Flink operators, ETL transforms and scoring functions are already Java. Numerical stages that dominate your wall clock can move to the accelerator without becoming a separate service with a separate on-call rotation.

Anywhere the second codebase is the real cost. The GPU kernel is rarely the hard part. The build, the marshalling, the platform matrix and the team who can maintain all three usually are.

---

Honest about scope

We would rather you find these out here than in week three.

GpuBlas, GpuArray, Half and the stream and graph APIs are the supported, hand-tuned surface. Use them and you get the behaviour described above.

The automatic @GpuKernel path covers a deliberately narrow eligibility subset of Java. It is not a promise that arbitrary code runs on the accelerator. Where the runtime cannot guarantee an identical result it falls back to the CPU and tells you why. Slower and correct beats faster and wrong.

NVIDIA only. The device side needs CratonVM built with GPU support and an NVIDIA GPU. There is no CUDA driver in this library — it is the Java surface, and the hardware work happens underneath.

Pre-1.0. The API may break between minor versions until 1.0. Pin your version and read the release notes.

Not yet on Maven Central. Namespace verification is outstanding, so today you build from source or mvn install locally. The build is release-ready and the coordinates above are the ones it will publish under.

---

Get started

<dependency>
  <groupId>io.github.craton-co</groupId>
  <artifactId>gpu4j-core</artifactId>
  <version>0.4.0</version>
</dependency>

Apache-2.0 licensed. Java 17 and up. Documentation · Source · Issues

Bring us a workload. A batch that runs too long, a model you cannot serve economically, a pipeline stage that dominates your cluster bill — we will benchmark it honestly against what you run today, and tell you if the answer is no.

---

Also in this repository

gpu4j-sidecar supervises a local inference server (llama.cpp, stable-diffusion.cpp) as a child process: resolves a model by name, waits until the server is genuinely ready rather than merely listening, evicts cleanly on a model switch, and guarantees the process never outlives your JVM and strands GPU memory. Point any HTTP client at it. Auxiliary to the main event — a reference to measure against while inference moves onto gpu4j itself.

---

Performance figures measured on an NVIDIA RTX 2060; results vary with hardware, problem size and data shape. Kernel-selection and graph-replay figures are host-side and device-side timings respectively, not end-to-end call latency, which additionally includes host↔device transfer. Comparative benchmarks hold the hardware constant and vary only the software. Benchmark methodology and per-row footnotes are published with CratonVM.