Local model sidecars (gpu4j-sidecar)

Prerequisites for gpu4j-sidecar, which supervises a local inference server as a child process.

This module is auxiliary to CratonVM, not an inference path. Running a model on CratonVM is the goal; a supervised llama.cpp or stable-diffusion.cpp server beside that work is the reference to measure against and the oracle to check outputs against. It runs on a stock JVM, touches nothing in craton.gpu, and the sidecar owns the device. See the root README.md for gpu4j-core itself.

It does not speak the inference protocol. LocalModelServer gives you a base URI; point any HTTP client at it. llama-server is OpenAI-compatible, so LangChain4j, Spring AI or the OpenAI Java SDK work unchanged.

How it runs the backends

LocalModelServer launches a prebuilt server binary as a child process. There is no JNI or JNA binding and no native library on java.library.path.

That is a deliberate choice, not a placeholder. Neither llama.cpp nor stable-diffusion.cpp publishes a prebuilt Windows+CUDA binding library, but both publish prebuilt Windows+CUDA server binaries on their GitHub Releases pages — so this is the path that works without installing a C++ toolchain. craton.sidecar.internal.SidecarProcess owns the child process lifecycle: it launches the binary, polls a readiness URL, redirects the child's output to a log file, and registers a shutdown hook so the process (and the VRAM its model holds) does not outlive the JVM. Readiness is the subtle part: llama-server binds its listener and answers /health with 503 "Loading model" well before it can serve, so a first request sent on "the connection was accepted" fails for reasons that look like anything but a race.

Consequences worth knowing:

• Each engine binds a local port. Defaults are 8081 for llama-server and 8082 for sd-server; see the environment table below.
• Only one model per engine is resident. A call whose resolved model name differs from the running server's stops that server and starts a new one.
• The first call after a model switch pays the full model load. On an RTX 2060 that was roughly 2.5 s for a 1B GGUF, 5 s for an 8B, measured disk-cold.
• The child's stdout and stderr go to craton-ai-llama-server.log / craton-ai-sd-server.log in java.io.tmpdir. When a launch fails, that file is where the reason is.

Hardware / driver prerequisites

• A CUDA-capable NVIDIA GPU. Developed against a single RTX 2060 (12 GB VRAM, Turing architecture, compute capability 7.5).
• An NVIDIA driver supporting CUDA 12.x (matches gpu4j-core's own CUDA expectation — see REQUIREMENTS.md).

Getting the server binaries

Download the CUDA build for your platform from each project's releases, or build from source with CUDA enabled:

• llama.cpp — releases; you want llama-server. From source: cmake -B build -DGGML_CUDA=ON.
• stable-diffusion.cpp — releases; you want sd-server. From source: cmake -B build -DSD_CUDA=ON, then cmake --build build --config Release.

Point the library at them with CRATON_AI_LLAMA_SERVER_BIN and CRATON_AI_SD_SERVER_BIN (below). The defaults assume llama-server.exe and sd-server.exe are on PATH.

Environment

One system property tunes the LLM server: craton.sidecar.llm.gpuLayers, the -ngl count, default 999 (offload every layer — the point of running this at all is that the model is on the device). A non-numeric value there, or in either port variable, logs a warning and falls back rather than failing class initialisation. The token cap and request timeout are gone with the client: they belong to whoever makes the request now.

Models

Models are identified by a short logical name, resolved against the models directory — never by raw file path.

• LLM models: <name>.gguf — e.g. models/llama3-8b.gguf.
• Diffusion models: <name>.safetensors, falling back to <name>.gguf.

A logical name must be a single plain file-name component: letters, digits, ., - and _, not starting with .. Anything else is rejected — see "Model names are not trusted input" below.

The environment variables set the initial value only. A running process switches models through ModelConfig.setActiveLlmModel(String) / setActiveDiffusionModel(String), since a real OS environment variable cannot change under a running JVM. Read ModelConfig.activeLlmModel() fresh on each call rather than caching it, or a switch will not be observed.

You are responsible for downloading model weights yourself and for complying with their license terms (e.g. Llama model weights carry usage restrictions separate from this library's Apache-2.0 license).

Model names are not trusted input

Whatever name reaches ModelConfig becomes a filename under CRATON_AI_MODELS_DIR and then an argument to a child process. If your application lets its own users set the active model — an admin endpoint is the obvious way — you are handing them influence over which file the sidecar opens.

ModelResolver rejects anything that is not a plain file-name component, re-checks that the resolved path is inside the models directory, and re-checks after following symlinks. That closes the traversal, but it is not authorisation: authenticate any request that carries a model name.

Usage

try (LocalModelServer llm = LocalModelServer.llm()) {       // ModelConfig.activeLlmModel()
    URI endpoint = llm.baseUri().resolve("/v1/chat/completions");
    // ... your HTTP client
    llm.ensureModel("some-other-model");                    // evicts, reloads
}   // process stopped, VRAM released

try (LocalModelServer sd = LocalModelServer.diffusion()) {
    URI endpoint = sd.baseUri().resolve("/sdapi/v1/txt2img");
}

LocalModelServer is safe to share across threads. Closing it stops the child process; so does JVM exit, via the shutdown hook.

Note. Earlier versions carried an inference client too: first @LlmPrompt / @DiffusionPrompt annotations over a proxy, then a hand-rolled OpenAI-compatible client with its own JSON parser. Both have been removed. The annotations were never going to reach in-JVM GPU inference — that is gpu4j-core's job — and the HTTP half duplicated existing clients against the same endpoint.

VRAM budget on a 12GB card

Running an LLM and a diffusion model concurrently can be tight: an 8B LLM at Q4 quantization (~5GB) plus a loaded diffusion checkpoint (~2–4GB) plus CUDA context overhead adds up quickly. Each server evicts its previously-loaded model whenever ensureModel names a different one, so only one model per server is ever resident — but an LLM and a diffusion model are still concurrently resident by design. If you hit out-of-memory errors, prefer smaller or more aggressively quantized checkpoints over running both at full size.

Build

mvn -q -f gpu4j-sidecar/pom.xml package