relm is not tied to Qwen. Load a compatible local
GGUF file with llm(); the generation call
stays the same when you change the file. Compatibility depends on the
architecture supported by relm’s pinned engine, the model’s metadata and
the operation you want to run. A successful text generation does not
establish support for tracing, embeddings, projection or vision.
Start with installation. The examples below are instructions for an explicit user run, not new model-validation or performance results.
Choose a pinned download
The bundled registry records exact URLs, revisions, SHA-256 digests, byte sizes and licenses. These are the five current aliases:
| Alias | File download, decimal units | Intended use / limit |
|---|---|---|
qwen2.5-0.5b-instruct-q8_0 |
676 MB | Small text model used for the bounded research examples |
qwen2.5-1.5b-instruct-q4_k_m |
1.12 GB | Larger text model used in the demos; results from 0.5B do not transfer automatically |
spark-x2.5-4b-q8_0 |
4.38 GB | Optional text model with a separate validation record; activation tracing is unsupported |
qwen2-vl-2b-instruct-q4_k_m |
986 MB | Vision-language model; image input needs the matching projector below |
qwen2-vl-2b-instruct-mmproj-f16 |
1.33 GB | Vision projector, not a standalone language model |
All five entries record Apache-2.0. Download size is not required RAM/VRAM: weights, context/KV, compute buffers and captured R objects add separate costs. The vision pair downloads about 2.32 GB in total. Quantization labels describe the weight format; they do not guarantee a particular speed or answer quality.
Inspect the registry shipped with your actual installation without downloading:
registry <- read.csv(
system.file("models.csv", package = "relm"),
colClasses = "character", na.strings = character(), check.names = FALSE
)
registry[c("alias", "size_bytes", "license")]To switch the small Qwen example to Qwen 1.5B, change the alias:
# Explicit download on first use; a matching cached file is reused.
path <- relm::llm_download("qwen2.5-1.5b-instruct-q4_k_m")
generate_text <- function(path, prompt) {
m <- relm::llm(path, backend = "cpu", context_length = 512L)
on.exit(close(m), add = TRUE)
relm::llm_generate(m, prompt, max_tokens = 32L,
temperature = 0, seed = 17L)
}
generate_text(path, "Explain what a confidence interval describes in two sentences.")llm_download() verifies registry files against their
pinned digest. An unknown alias produces
relm_error_download listing the available aliases. A newer
registry can differ from the one in an older installation; inspect the
installed registry rather than assuming an alias exists.
chat = TRUE, the generation default, applies the model’s
chat template. Use chat = FALSE for raw continuation. A
model needs an appropriate template for chat; changing that flag is a
deliberate change to the input format.
Modern chat templates
Chat support depends on the embedded template as well as the
architecture. The D-054 formatter recognizes four exact source
templates: Qwen3.5-9B’s original author template and pinned Unsloth GGUF
spelling, Qwen3.8-27B’s pinned Unsloth GGUF spelling, and
Gemma4-12B-it’s original template. Provenance and independent reference
bytes are in
rebirth/src/rust/rebirth-llm/tests/fixtures/modern-chat/README.md.
For the single user prompt accepted by llm_generate(),
ordinary Qwen chat preserves the template’s declared default: thinking
for the author spelling, non-thinking for the supported Qwen3.5 GGUF
spelling. Qwen3.8 preserves its default xhigh reasoning
system turn and thinking opener. Schema-constrained output uses the
template’s non-thinking mode, including omission of that Qwen3.8
reasoning system turn. Gemma4 uses its modern turn/channel markers and
non-thinking default. No response text is stripped or repaired. Modified
modern templates are refused rather than replaced with a guessed legacy
format. This is not arbitrary Jinja, tool calling or multi-turn
conversation support.
Correct formatting alone does not establish load compatibility, factual quality, speed or support for every quantization. Record the exact GGUF and template when reporting a result; older Gemma/Qwen measurements retain their original scope.
Bring a different local GGUF
Obtain a GGUF from a source you trust, follow its license/access requirements, and verify its SHA-256 against an independently recorded expected digest. Then pass that file to the same function:
# Reuses generate_text() above; this call performs no download.
generate_text("models/my-model.gguf", "Summarize the purpose of a control group.")This is a path example, not a certified model name. A
.gguf extension alone does not prove compatibility. relm
uses a pinned, patched llama.cpp revision; support added to a newer
upstream engine is not automatically present here. GGUF models and
projectors are parsed by native code, so a checksum identifies bytes but
does not make an untrusted file safe.
llm_download("https://...") also accepts a direct HTTPS
URL. Unlike a registry alias, that route has no trusted expected digest:
it reports the computed hash, does not claim verified provenance, and
fetches the URL again on another call. For a reproducible experiment,
retain the verified local file, its digest, source revision and license.
The single-file registry is not a general model catalog or an
authenticated/gated-download client.
Match the model to the research operation
| Operation | What to check when changing models |
|---|---|
| Text / constrained JSON | Supported architecture, tokenizer and chat template; JSON shape does not establish factual correctness |
| Embeddings | Supported embedding route/pooling; vectors from different models are not a shared coordinate system |
| Prompt trace / live observation | Supported architecture, component, layer and position; Spark tracing is explicitly refused |
| Additive steering / ablation | Supported intervention site and the new model’s dimensions; loading alone does not validate the intervention |
| Static projection | Recorded scope is dense Llama
attn_out/mlp_out and dense Qwen2
mlp_out; no arbitrary-architecture or MoE promise |
| Images | A supported vision-language model plus its own matching projector; no vision-encoder tracing claim |
See the supported workflow and Spark validation record for the actual tested operations and limits. The historical model matrix also names its producing engine/build; old results do not certify a newer build.
Start a new model with a short generation and the
specific observation you need, keeping warnings and
classed refusals visible. Do not retry unsupported operations blindly.
Reduce capture scope before scaling up: one layer,
positions = "last" and a bounded token count. A direction
recorded for another model/checkpoint must not be reused just because
the vectors have the same length. Construct and evaluate a new direction
with declared, disjoint data.
Use a matched vision pair
# Two explicit downloads; later calls reuse matching cached files.
model_path <- relm::llm_download("qwen2-vl-2b-instruct-q4_k_m")
projector_path <- relm::llm_download("qwen2-vl-2b-instruct-mmproj-f16")
describe_image <- function(model_path, projector_path, image_path) {
m <- relm::llm(model_path, projector = projector_path,
backend = "cpu", context_length = 4096L)
on.exit(close(m), add = TRUE)
relm::llm_generate(m, "Describe what is visible.",
images = list(image_path), max_tokens = 32L,
temperature = 0, seed = 17L)
}
# Supply your actual local image:
# describe_image(model_path, projector_path, "images/example.png")Do not combine a model with an unrelated projector. Matching dimensions alone are not evidence that the pair belongs together. See the vision guide for other local pairs and limits.
Record the experiment, including the backend
CPU is the explicit starting point above.
backend = "auto" selects available compiled backends in
priority order: Metal, CUDA, CPU. It does not benchmark them. Requesting
"cuda" needs an actual CUDA-enabled installation; see Linux
CUDA. Close the current handle before loading another large model if
you do not need both resident.
Record the installed relm version/source, engine revision, model/projector digests, backend, context, prompt/template, generation settings and returned seed. Different models, quantizations and backends can change both answers and timings; the measured Qwen CUDA result is not a benchmark of other models.
The agent examples already accept a local
model_path and an independently known
expected_sha256; changing models does not require editing
their code. Their required capabilities still apply (for example two
layers and supported tracing for the direction example). Keep the
expected digest independent of the file being checked.
CPU threads and mixed CPU/GPU inference
llm(path, backend = "cuda", gpu_layers = 12, threads = 8)
requests twelve GPU layers and eight CPU threads for both prompt
processing and token generation. Choose these counts for the available
hardware and workload; they are not automatic recommendations.
threads = NULL retains the previous defaults. R check-time
core caps still apply. Derived text contexts inherit the request; the
separate vision projector keeps its own defaults. More threads or GPU
layers can be slower or leave too little working memory. Include the
context, KV/recurrent state, compute buffers and other applications in
the budget.
The exact supported Qwen3.5-35B-A3B converter template has a distinct MoE architecture and thinking default. Support applies to verified single-user text formatting; structured output explicitly disables thinking. Model loading, quality, memory fit and speed each require actual evidence. Active parameters per token do not describe the total resident weights of an MoE model.
Bounded dense/MoE comparison
A Linux CUDA experiment on a 16 GB RAM / 6 GB RTX 2060 host compared
two pinned quantizations using the same production candidate and
threads = 8. Each model answered the same six previously
unexecuted Italian extraction cases: four selected official-page
excerpts and two synthetic adversarial cases. The models ran
sequentially, with context 4096, mmap enabled, schema-constrained chat,
seed 21021, temperature 0, top-p 0.95 and at most 384 generated
tokens.
| Setting or result | Qwen3.8-27B UD-IQ3_XXS | Qwen3.5-35B-A3B UD-Q2_K_XL |
|---|---|---|
| Requested GPU layers | 12 | 8 |
| Passed frozen field/evidence criteria | 36/36 | 32/36 |
| Official excerpts / synthetic criteria | 24/24 and 12/12 | 21/24 and 11/12 |
| Median decode tokens/second | 1.31 | 8.06 |
| Median complete response time | 101.7 s | 20.7 s |
| Complete response range | 71.0–115.2 s | 16.2–30.9 s |
| First-token range | 5.32–13.08 s | 3.97–9.26 s |
| CPU-mapped weight buffers | 7609 MiB | 9406 MiB |
| GPU weight buffers | 2474 MiB | 2177 MiB |
| Smallest observed free VRAM | 1815 MiB | 2383 MiB |
Both models returned the correct dates, amounts and percentages. The MoE returned two beneficiary descriptions outside the frozen accepted alternatives, a truncated quote that did not establish the deadline, and a quote where the contract required null. These remain failures; valid JSON does not establish correct extraction. The dense model met every criterion in this small experiment, which does not establish reliability on arbitrary grants or complete PDFs/OCR.
Calibration separately compared four/eight threads and adjacent GPU-layer counts on two repeated synthetic prompts. Eight threads improved dense median decode throughput by 36.4% and 36.8% relative to the start/end four-thread controls. The MoE gain was 11.3% against its predeclared combined baseline (13.4% at the start, 9.2% at the end); long-prompt first-token latency worsened in calibration. Adding one GPU layer did not meet the decode improvement criterion and reduced the reserve. Calibration prompts are not fresh quality validation.
These are observations of different models, quantizations and GPU allocations, with sequential order and cache conditions; the speed difference cannot be attributed solely to parameter count or MoE architecture. The final experiment used the production development binary before its version metadata changed to 0.6.0. It does not establish other hardware/backend performance, a generic chat-template interpreter, a CUDA-defect fix, or an automatic eight-thread default.
Memory mappings and MemAvailable overlap reclaimable
file cache: do not add mapped RSS to available memory.
Context/KV/recurrent and compute buffers are additional to weights. Zram
activity was observed, with no disk swap used; global swap activity
cannot be attributed exclusively to the model. Peak memory PSI was
13.56%, below the declared sustained stop condition. Both processes
closed and released their GPU allocations. Keep headroom for the desktop
and other programs.
Model identities: Unsloth Qwen3.8-27B-GGUF revision
4ca720788d1e01f1bff70c033e0d0028fd02e502, SHA-256
c0b7c3038681ed2e3040456c1dd45f9858b6c2290bed172c70388a94874f3eee;
and Qwen3.5-35B-A3B-GGUF revision
bc014a17be43adabd7066b7a86075ff935c6a4e2, SHA-256
855b8d11f4b52e7ea4b9eeea2fbe3555c4d96d5b7b74702b6144810db3e77270.
These files are local-model inputs, not new registry aliases.