gpu4j

gpu4j

CI Maven Central License: Apache 2.0

Java API for GPU offload under CratonVM.

This is the standalone Java library — formerly bundled inside the Rust crate craton-gpu/ in the CratonVM workspace, moved here so the Java API can evolve, be published, and be depended on independently from the Rust JVM build.

Renamed in 0.4.0. The repository was craton-gpu-java, the offload module was craton-gpu and the sidecar was craton-sidecar. The artifacts are now gpu4j-core and gpu4j-sidecar under the unchanged io.github.craton-co group. The Java packages are unchanged (craton.gpu, craton.sidecar): CratonVM's native registry binds them by exact name, so renaming them is a coordinated two-repository change and is deferred to 1.0. Imports do not change.

Installation

Not on Maven Central yet. Nothing has been published under io.github.craton-co (the group directory returns 404 on repo1.maven.org), so the coordinates below do not resolve today and the Maven Central badge above will render as "not found". The build is release-ready; what is outstanding is namespace verification and signing credentials — see docs/MAVEN_CENTRAL.md. Until then, build from source (see Build) or install locally with mvn install.

Maven:

<dependency>
    <groupId>io.github.craton-co</groupId>
    <artifactId>gpu4j-core</artifactId>
    <version>0.4.0</version>
</dependency>

Gradle (Kotlin DSL):

dependencies {
    implementation("io.github.craton-co:gpu4j-core:0.4.0")
}

Use

import craton.gpu.GpuExecutor;
import craton.gpu.GpuFuture;
import craton.gpu.GpuException;

public class Example {
    public static void main(String[] args) throws InterruptedException {
        try (GpuExecutor exec = GpuExecutor.open()) {
            int[] a = new int[1 << 20];
            int[] b = new int[1 << 20];
            int[] out = new int[1 << 20];
            // ... fill a, b ...

            // className uses the JVM-internal slash-delimited form, as
            // documented on GpuExecutor#submit. The bare-name form
            // (e.g. "EligibleVectorAdd") is also accepted by current
            // implementations.
            GpuFuture<Void> f = exec.submit(
                "com/example/EligibleVectorAdd",
                "vectorAdd",
                "([I[I[I)V",
                a, b, out);

            f.get();
            // out[] is now filled by the GPU kernel.
        } catch (GpuException e) {
            // Wraps any device-side failure surfaced from submit/get.
            throw new RuntimeException(e);
        }
    }
}

Matrix multiply, through the built-in kernel:

try (GpuExecutor gpu = GpuExecutor.open();
     GpuArray<float[]> weights = GpuArray.wrap(w);
     GpuArray<float[]> input   = GpuArray.wrap(x);
     GpuArray<float[]> output  = GpuArray.allocateFloat(m * n)) {

    GpuBlas.gemm(weights, input, output, m, n, k).get();

    float[] result = output.toHost().get();
}

Runtime requirement

This API is meaningful only when the running JVM is CratonVM compiled with --features gpu-driver. On a stock JVM there is no library loaded for the craton.gpu.internal.Native methods, so any call into the bridge (including GpuExecutor.open()) fails with java.lang.UnsatisfiedLinkError. This library does not attempt to load a native shared object on its own — the bridge is installed by CratonVM's native-builtin registry at VM startup.

Surface

Submitting work

  • GpuExecutor — open() / submit() / launch() / newStream() / prepare() / close(). Its class javadoc has a table of which submission path suits which shape of work; in short:
    • submit(cls, method, descriptor, args...) for one kernel whose outcome you need individually.
    • the typed convenience overloads submitV() (int[]/long[]/float[]/double[] element-wise kernels) and submitScale() (int/float scalar-times-array). These exist for compile-time arity and element-type checking. They are not an allocation optimization — they delegate to the varargs submit and build the same Object[]. An earlier revision of this README and of their javadoc claimed otherwise.
    • the fire-and-forget trio dispatchNamedHandle() / awaitSubmission() / releaseSubmission(), for queueing a long chain of kernels on one stream and waiting only on the last. See "Fire-and-forget dispatch" below.
  • GpuStream — an ordered submission queue, and the recommended way to run a chain. dispatch() submits onto it, synchronize() waits for the whole chain, and close() releases every submission handle it issued. Prefer this over the bare-handle trio: a stream owns its handles, so the per-kernel leak that path allows cannot happen.
  • PreparedKernel — a kernel target resolved once, from GpuExecutor.prepare(...), for repeated dispatch.
  • GpuFuture<T> — get() / get(long, TimeUnit) / isDone() / getNow() / cancel() / isCancelled() / thenApplyGpu() / toCompletableFuture() / toCompletableFuture(Executor); the sync point.

Graph capture and replay

  • GpuExecutor.beginCapture() / endCapture() / replay(long) — record a sequence of dispatches once and run it as one submission, for a loop whose shape does not change between iterations. Measured on an RTX 2060, one decode step of a 1B-parameter transformer (453 launches) costs 3.483 ms of host time to issue and 0.550 ms to replay.
  • beginReplay(long) / endReplay() — re-supply a captured graph's arguments without changing the kernels, for a caller whose values change every iteration.
  • graphNodeCount(long) / releaseGraph(long) — check what a graph holds against the number of dispatches you made, and free it.

Data

  • GpuArray<T> — a handle to a primitive buffer that is, or may become, device-resident. Typed wrap() factories over a host array, allocateInt/Long/Float/Double/Half() for device-only buffers with no host mirror, wrapHalf() for fp16, toHost(), the buffer-reusing toHost(dest) overloads, copyFromHost(src) to overwrite in place (the write path a graph replay needs), isResident(), length(), elementType(), close().
  • GpuDevice — count(), get(int), all(), queryable(), and compute capability. Ask what is on the machine before picking an ordinal for GpuExecutor.open(int).

Built-in kernels

  • GpuBlas — gemm(...) and gemmHalf(...): C[M×N] = A[M×K] · B[K×N], row-major and unpadded, over GpuArray handles so weights stay resident between calls. Either operand can be read Transpose.TRANSPOSED in place. gemmReference(...) is a plain triple loop for tests to check against.
  • Transpose — NONE / TRANSPOSED, an enum rather than a boolean because a swapped pair of bare booleans in the middle of a matrix multiply still produces numbers.
  • Half — IEEE-754 binary16 conversion in pure Java (the JDK's Float.floatToFloat16 needs Java 20; this library targets 17). Verified against the JDK's own implementation exhaustively.

Kernels and errors

  • GpuCallable<T>, GpuRunnable, GpuFunction<T,R> — SAM-style task interfaces the runtime can dispatch.
  • Annotations: @EnableGpuAsync, @GpuExclude, @GpuKernel.
  • Enums (used as @GpuKernel parameter types): AdmissionHint, GridShape.
  • Exceptions: GpuException, and the three subtypes a failing kernel is classified into — GpuCompileException, GpuOutOfMemoryException, GpuLaunchException.

Internal (craton.gpu.internal)

Exported by the module descriptor, because NativeBridge is a type consumers must be able to implement — see module-info.java for why.

  • Native / NativeBridge / NativeImpl — the host-side bridge CratonVM intercepts, behind a swappable seam. Probe memoises which optional entry points a given VM actually registers.
  • GpuExecutorImpl, GpuFutureImpl, GpuStreamImpl — reference implementations used by open().
  • StreamCleaner — cleanup for every native handle this library owns (executors, streams, futures and arrays), on both the explicit-close() and the unreachable-object paths.
  • GpuErrors — maps a native failure string onto the GpuException subtype that fits it.
  • KernelArgs, ImmediateFutures, Submissions — argument validation, the single already-resolved GpuFuture, and the seam that lets GpuBlas hand back a future from outside this package.

Fire-and-forget dispatch

A GpuFuture costs a ReentrantReadWriteLock, a Cleaner registration and the phantom reference behind it. A caller that queues a chain of dependent kernels on one stream and waits only for the last pays that per kernel and uses one of them. dispatchNamedHandle skips the wrapper and hands back the bare submission handle. Measured with craton.gpu.bench.DispatchBench on an RTX 2060, building the future is 7.2 µs against 5.8 µs for the entire rest of the Java-side path.

The handle is yours to free. Nothing collects it — there is no Java object behind it for a Cleaner to watch — so every handle must reach releaseSubmission, including the intermediate ones you never await:

long[] chain = new long[n];
long last = 0L;
try {
    for (int i = 0; i < n; i++) {
        chain[i] = last = exec.dispatchNamedHandle(cls, mth, desc, args[i]);
    }
    exec.awaitSubmission(last);   // stream order: one wait covers the chain
} finally {
    for (long h : chain) exec.releaseSubmission(h);
}

What you give up is per-kernel observability: there is nothing to poll or cancel, and a failure in kernel k surfaces at whichever handle you do await. Use submit() for any kernel whose outcome you need individually, and prefer GpuStream.dispatch where a stream will do — it owns the handles for you.

Relationship with CratonVM

CratonVM's Rust workspace contains the craton-gpu4j crate (a build shim that compiles these Java sources during cargo build) and jit-cuda (which lowers Java bytecode to PTX). The Rust crate's build.rs reads these sources from a path configured in the CratonVM tree — see craton-gpu4j/build.rs there for how it resolves. $CRATON_GPU_JAVA_SRC overrides it; otherwise it probes a sibling checkout (named gpu4j, then the legacy craton-gpu-java) and then a couple of absolute Windows defaults, trying each root in every layout this repository has had: <repo>/gpu4j-core/src/main/java, the 2026-08-28 aggregator <repo>/craton-gpu/src/main/java, and the pre-0.3.0 flat <repo>/src/main/java.

If you move these sources, update that build script. It does not fail when it cannot find them: it emits an empty annotations directory and a cargo:warning, leaving a VM that silently recognises no @GpuKernel at all.

The Java package names are the other half of that contract: CratonVM binds craton/gpu/internal/Native, the three *Impl handle wrappers, and descriptors naming Lcraton/gpu/GpuExecutor; and the rest, by exact string. The 0.4.0 rename therefore changed the repository, the modules and the artifacts but not the packages.

For local development without Maven you can keep editing these files in place and run cargo build -p cratonvm-gpu; the build.rs will pick them up and produce target/.../out/classes/craton/gpu/*.class.

Build

mvn -q verify                       # both modules: tests and coverage gates
# Output: gpu4j-core/target/gpu4j-core-<version>.jar
#         gpu4j-sidecar/target/gpu4j-sidecar-<version>.jar

mvn -q -pl gpu4j-core package       # just one module

Requires JDK 17 or newer (only because of the compiler plugin defaults — the source itself is plain Java with no Project-Loom or post-17 language features).

Both POMs accept -Dbuild.qualifier=-something, which is appended to the artifact file name. It is empty by default, so a release build produces gpu4j-core-<version>.jar exactly as before; set it when building from a review worktree alongside the main checkout so the two jars cannot be confused.

gpu4j-core/examples/Demo.java is not under a Maven source root. CI compiles and runs it separately (it exercises the whole public surface against a simulated bridge):

javac -d gpu4j-core/target/example-classes -cp gpu4j-core/target/classes \
    gpu4j-core/examples/Demo.java
java -cp "gpu4j-core/target/classes:gpu4j-core/target/example-classes" Demo

(On Windows the classpath separator is ;.)

Limitations

The surface is intentionally pre-1.0 and several features are still in progress:

  • GpuFuture.cancel() returns false. It reaches the device now (the request runs under the read lock, so it is no longer queued behind an in-flight get(), and CratonVM registers the native at all — before, cancel() raised UnsatisfiedLinkError), but there is no device-side cancellation primitive yet, so the request is always rejected. isCancelled() follows.
  • GpuFuture.get(long, TimeUnit) is backed by a real timed wait in GpuFutureImpl, which uses the bridge's blocking futureAwait where one exists and polls where it does not. The interface default throws TimeoutException immediately when the future is not done, so a third-party implementation should override it.
  • GpuArray.toHost() is synchronous: it returns an already-completed future. There is no device-side async copy to wire it to. The toHost(dest) overloads use a direct read-back entry point where the bridge has one, which removes the intermediate allocation.
  • GpuBlas takes no device or executor, so on a multi-GPU box a GEMM targets whatever the built-in stream is bound to. The overloads taking a GpuStream are the way to order a multiply against other work; without one it uses a shared internal stream that is ordered against other built-ins and nothing else.
  • GridShape.ROW_PER_THREAD and BLOCK_REDUCTION are declared but have no lowering. A kernel asking for one is rejected by the analyzer and runs on the CPU — correct, and slower.
  • GpuExecutor.handleForDispatch() is deprecated for removal in 1.0.0. Nothing in the library reads it any more, and it hands out a raw native pointer that outlives the closed-state check that guarded it.
  • Off-CratonVM execution: every native bridge method throws UnsatisfiedLinkError on a stock JVM. The library must be run on a CratonVM build configured with --features gpu-driver. See the Runtime requirement section above.

Versioning

This project follows Semantic Versioning. It is currently pre-1.0, so the public API may break between minor versions until 1.0.0 is cut. Pin the exact version you depend on and read the release notes before upgrading.

License

Licensed under the Apache License, Version 2.0. See the LICENSE file at the repository root for the full text.

Local model sidecars (gpu4j-sidecar)

A second module that supervises a local inference server as a child process. It does not speak the inference protocol:

try (LocalModelServer llm = LocalModelServer.llm()) {
    URI endpoint = llm.baseUri().resolve("/v1/chat/completions");
    // ... any HTTP client. llama-server is OpenAI-compatible, so an existing
    // one works unchanged.
}   // process stopped, VRAM released

This is auxiliary to CratonVM, not an inference path. Running a model on CratonVM is the goal; a supervised llama.cpp or stable-diffusion.cpp server beside that work is the reference to measure against and the oracle to check outputs against. Nothing here touches craton.gpu — the sidecar owns the device.

What it does that a chat client does not, because it comes from owning the process:

  • Readiness that means it. llama-server binds its listener and answers /health with 503 "Loading model" long before it can serve, so "the connection was accepted" is not ready.
  • The process does not outlive the JVM. A shutdown hook force-kills the child, so a crash or a Ctrl+C does not strand a server holding several GB of VRAM.
  • One model resident at a time. ensureModel evicts before it loads, and resolves before it evicts — so naming a model that does not exist leaves the working one serving.
  • Model names are validated. A logical name becomes a filename and then a child-process argument; ModelResolver rejects anything that is not a plain file-name component. See docs/SIDECAR_SETUP.md.

Earlier versions also carried an inference client: @LlmPrompt / @DiffusionPrompt annotations over a proxy, then a hand-rolled OpenAI-compatible client with its own JSON parser. Both are gone. The annotations never approached in-JVM GPU inference — that belongs to gpu4j-core — and the HTTP half duplicated what LangChain4j, Spring AI and the OpenAI Java SDK already do against the same endpoint.

See docs/SIDECAR_SETUP.md for CUDA prerequisites, where to get the server binaries, model placement, and the environment reference.

Testing against a stand-in GPU

craton.gpu.testing.GpuTestBridge installs a substitute backend, so code that uses this library can be tested with no CratonVM and no device:

NativeBridge fake = mock(NativeBridge.class);
try (var installed = GpuTestBridge.install(fake)) {
    // craton.gpu now routes every native call to `fake`
}   // previous backend restored, even if the test failed

Note that a Mockito mock does not run NativeBridge's default methods — it answers the return type's zero value. That is deliberate: every optional entry point is specified so the zero value means "not supported, use the fallback", so a plain mock exercises the fallback path. Stub a method to take the other one.

gpu4j-core/examples/Demo.java is a worked example of the whole surface against such a bridge.

Reviews

  • docs/REVIEW-2026-09-06.md — the current one: architecture, performance, documentation, tests, direction, the gpu4j rename, and eleven defects, all of which are fixed in 0.4.0.
  • docs/AUDIT-2026-08-28.md — the previous one, kept as a dated record. It describes the pre-0.3.0 layout and the older module names.

Contributing / Reporting bugs

Contributions are welcome. See CONTRIBUTING.md for the development workflow, coding conventions, and how to run the tests. Bug reports and feature requests should be filed on the GitHub issue tracker at https://github.com/craton-co/gpu4j/issues.