Autoregressively continues each prompt. With chat = TRUE (the default) the
prompt is wrapped as a user turn using the model's own chat template, so the
formatting matches what the model was trained on; with chat = FALSE the
prompt is completed verbatim.
Usage
llm_generate(
m,
prompt,
max_tokens = 256,
temperature = 0.8,
top_p = 0.95,
seed = NULL,
chat = TRUE,
stop = NULL,
images = NULL,
schema = NULL,
async = FALSE,
on_progress = NULL,
on_token = NULL,
on_state = NULL,
layers = integer(),
components = "residual",
top = 20L,
spill = TRUE,
spill_dir = NULL
)Arguments
- m
An
llmhandle fromllm().- prompt
A character vector of prompts; the result has one element per prompt and preserves
names(prompt).- max_tokens
Single positive integer: the maximum number of tokens to generate per prompt.
- temperature
Single non-negative number.
0is greedy (deterministic); higher values sample more diversely.- top_p
Single number in
(0, 1]: nucleus sampling keeps the most probable tokens whose cumulative probability reachestop_p.- seed
NULL(draw and record a seed) or a single non-negative whole number for a reproducible run.- chat
Single logical.
TRUEapplies the model's chat template;FALSEcompletes the raw prompt. If the model's embedded template cannot be detected by the engine (e.g. some Gemma models), a built-in template for the model's architecture is used instead; if neither applies, a classed error is raised rather than mis-formatting the prompt.- stop
NULL, or a character vector of stop sequences that end generation.- images
NULL(default: text-only, unchanged) or the image file paths to attach to each prompt — a list parallel toprompt, or a bare character vector for a single prompt. Accepted formats: JPEG, PNG, BMP. Requires a handle loaded withllm(projector = ); see the Image input section.- schema
NULL(default) or one non-NA UTF-8 string containing JSON Schema text in the supported profile. This is JSON text, not a file path or R list; one schema applies to every prompt. See Structured output.- async
One nonmissing logical.
FALSEreturns synchronously;TRUEreturns a promises promise backed by native background generation.- on_progress
NULLor a function accepting one progress data frame. Requiresasync = TRUE; see Background generation.- on_token
NULL, a function accepting one event data frame, or an already-open writable binary basefile()connection. Requiresasync = TRUE; see Token streaming.- on_state
NULLor a function receiving one live state; requires background text-only generation. Return NULL to continue or a coefficient reply for existing steering entries; see Live state observation.- layers
Live capture block indices: integer() for logits only, NULL for all blocks, or unique valid 1-based indices.
- components
Live capture components, using
llm_trace()'s meanings.- top
Number of raw logit summaries per live state, from 0 through min(128, vocabulary size).
- spill
Whether over-budget live activations may spill to Arrow files.
- spill_dir
NULL for managed session storage, or a custom directory. Capture arguments require a non-NULL
on_state.
Value
A character vector the same length as prompt (names preserved), each
element the generated continuation. The seed used is attached as
attr(result, "seed"). With async = TRUE, a promise resolving to that vector.
Details
Decoding is greedy when temperature = 0 — the exact, reproducible path — and
temperature + nucleus (top-p) sampling otherwise. Sampling is drawn on the CPU
from a seeded generator, so a run is fully reproducible: the same seed and
arguments produce the same text across runs and sessions. When seed = NULL
a seed is drawn (from R's RNG, so set.seed() makes even that reproducible)
and recorded — the seed actually used is always returned as
attr(result, "seed"), so any generation can be replayed.
Without a schema, generation stops at max_tokens, at the model's
end-of-generation token, or
as soon as one of the stop strings appears (the output is truncated just
before it). A prompt longer than the model's context window raises
relm_error_context_overflow, whose message states by how much.
With the supported Spark-X2.5 chat template, ordinary chat opens the response
with the author's <think> marker. Generated reasoning text and closing
markers are returned as generated and count toward max_tokens. With a
schema, chat instead uses the author's </think> non-thinking opener, so
the JSON constraint starts at the first generated token. This does not strip
or repair model output. Spark chat supports the pinned official template and
the existing single user turn; custom templates require chat = FALSE and
caller-supplied formatting. Other models keep their existing chat behavior.
Background generation
async = TRUE requires optional packages later (>= 1.4.8) and promises
(>= 1.5.0), and returns a promise resolving to the same named character vector
and seed attribute. Generation runs on a native worker using the loaded model.
Only one native job may run per R process; competing native operations raise
relm_error_busy. Ordinary R calculations and metadata printing remain usable.
R validation and admission precede the omitted-seed draw. Execution errors
reject the promise with their classed conditions. Use llm_cancel() to request
cooperative cancellation, or close() to close the handle.
Optional on_progress(state) runs on R's thread with a fresh one-row
data.frame: integer columns prompt_id, prompts_completed, prompts_total,
generated_tokens, max_tokens, then character phase ("prefill",
"generate", "complete"). Snapshots are coalesced and may skip intermediate
states; generated tokens count the current prompt, including removed stop
suffixes. Success delivers one final complete snapshot before resolving.
A callback error requests cancellation and rejects with relm_error_callback,
retaining the original condition in parent. Callback return values are ignored.
One timer checks progress at a 50 ms target interval on the global event loop.
Callbacks normally run at the interactive prompt. In scripts, attach handlers
with promises::then() and pump later::run_now() until your handler records
completion. A long R expression delays delivery; a slow callback can block R.
Dropping the promise or handle does not cancel a pending job. Namespace
shutdown requests cancellation and joins the worker. The native library stays
mapped so retained external-pointer finalizers can run safely. Forced
dyn.unload() while external pointers remain alive is unsupported.
Async accepts at most 128 prompts, 1 MiB UTF-8 per prompt, 16 MiB total copied
text (including names, stop strings, schema and image paths), and 8192 tokens
per prompt. A separate 16 MiB descriptor budget bounds retained string/vector
overhead, including empty strings. Output is limited to 8 MiB UTF-8 per call (relm_error_oom);
existing stricter schema and image limits also apply. Nothing is truncated
or retried to satisfy these bounds. Native cancellation is cooperative and
may wait for an in-flight decode or image encoder call to finish.
The 8 MiB output cap measures UTF-8 text bytes, not resident memory: copied
inputs, templates, token vectors, transient output buffers and R character
objects need additional storage. Existing model/context, sampler/grammar and
vision working memory are separate from these transport limits.
Token streaming
Supply on_token = function(batch) ... to receive nonempty plain data frames
on R's thread. Each dispatch contains at most 64 rows and 64 KiB of text.
The columns, in order, are integer event_id, character event, integer
prompt_id, integer token_pos, integer token_id, character text, double
elapsed, character finish_reason, and logical validated. Event IDs are
contiguous across the call; prompt IDs and vocabulary token IDs are 1-based.
event = "token" records each sampled non-EOG token, including tokens later
removed with a stop suffix or sampled at context exhaustion. token_pos is
its generated position. event = "text" carries a nonempty committed UTF-8
delta; concatenating these per prompt reproduces the successful returned
string exactly. Tokens and text chunks need not correspond one-to-one.
event = "prompt_end" records finish_reason ("length", "stop",
"stop_string", or "context_full"). Non-token rows have missing token
fields; non-text rows have empty text; non-end rows have empty
finish_reason. elapsed is nonnegative monotonic production time in seconds
since submission. validated is missing for ordinary generation, FALSE
for structured token/text, and TRUE for a successful structured prompt end.
Structured text is provisional until that prompt's independent schema
validation succeeds; this validates format, not factual accuracy.
Alternatively, supply a writable binary base file() connection to an empty
regular local file at position zero. Append, text, compressed, raw, socket,
pipe and device connections are rejected. relm neither opens nor closes the
connection. Do not independently seek, write, truncate or replace its file
until settlement. CSV contains one header, UTF-8 bytes, LF delimiters, comma
separators and no row names. Character fields are always quoted; embedded
quotes are doubled. Missing numeric/logical fields are empty; elapsed uses
round-trip numeric precision independently of print options. Read it with:
read.csv(path, fileEncoding = "UTF-8", check.names = FALSE,
na.strings = character(),
colClasses = c("integer", "character", "integer", "integer", "integer",
"character", "numeric", "character", "logical"))The connection is flushed after batches and before success; no atomic commit or disk-durability guarantee is made. Failed calls may leave a delivered prefix or an incomplete record. Successful settlement establishes completion.
Streaming retains the execution reservation until all batches are delivered.
Native operations from every on_token, including the last, see busy; final
on_progress runs after release. Closing the model before release abandons
delivery and rejects an otherwise successful result with
relm_error_cancelled, reason stream_closed. A token/progress callback error
or interrupt rejects with relm_error_callback, identifying callback and
preserving the original parent. Connection and representation failures use
relm_error_stream, with reason, prompt_id, event_id and parent when
available. The first consumer failure takes precedence over subsequent native
errors; once a native failure is observed, remaining events are discarded.
A slow consumer applies backpressure to a queue bounded at 256 rows and
256 KiB text, with text chunks at most 16 KiB. Callbacks and file writes run
synchronously on R and can block it. Batch boundaries and elapsed times vary
across runs. Consumer side effects remain on failure, and caller-retained
batches are outside relm's memory estimate. on_token = NULL retains the
ordinary async behavior without a stream queue.
Live state observation
on_state = function(state) ... observes each sampled non-EOG token on the
same generation context. It requires async = TRUE, one text prompt,
max_tokens <= 1024, no images and no schema. The named list contains step,
logits and trace; return invisible(NULL) to continue. Call llm_cancel()
from the callback to stop before another token is sampled. Cancellation still
rejects the promise, and callbacks can block R while they run.
step is one plain data-frame row: integer state_id, prompt_id,
token_pos, token_id, context_pos, source_pos, character source,
double elapsed, followed by integer steering_revision,
applied_after_state and effective_source_pos. Let P be the actual
templated prompt token count and k the
generated position. The sampled token belongs at context_pos = P + k;
the forward pass selecting it came from source_pos = P + k - 1.
source is "prompt" for the first state and "generated" thereafter.
Positions and vocabulary IDs are 1-based. A last sampled token may remain
undecoded at a context/stop boundary; the state describes its valid source.
logits has llm_logits()'s six columns, raw logits and full-vocabulary
softmax probabilities before temperature/top-p sampling. top = 0 disables
this table. trace is a bounded relm_trace for one source position, with
position_space = "model_context", prompt_token_count and state_id
attributes. Its token positions identify the source, not the sampled token.
layers = integer() captures no activations; NULL explicitly selects all
blocks. Selected residuals include interventions already applied to the handle.
A callback may instead return exactly
list(steer = data.frame(intervention = 1L, coef = 0)) to change coefficients
of existing steering entries. Indices address m$interventions (including
intervening ablation entries); only steering entries may be updated. Partial
replies retain other coefficients; an empty table or unchanged values do not
advance the revision. Replies are validated atomically, including finite
float32-range coefficients and finite summed layer vectors. Invalid replies
reject with relm_error_callback, callback = "on_state",
reason = "state_reply", preserving the original condition as parent.
A changed reply to state k affects the decode of token k and sampling of
token k+1, never the already sampled token k or historical KV entries.
attr(state, "steering") is a worker-produced data frame with integer
intervention, integer layer and double coef, sorted by original index.
Its coefficients and the three audit columns describe the adapters actually
used for the reported state. Baseline audit values are 0, 0 and 1. A change
from state k advances the revision and becomes effective at source position
P+k; a terminal reply has no promised later-state receipt. Zero removes that
entry's contribution. Ablation still follows steering.
The original adapters are restored before model ownership returns, including cancellation and failure. The R handle and other handles are unchanged; a subsequent fresh generation uses the original coefficients. Changing back during a running generation does not undo its earlier effects on KV history. Direction, layer and ablation changes are unavailable in this reply protocol.
One state waits for acknowledgement. All earlier token events are delivered
before on_state; the current token/text events and the next decode follow
the callback. No state is emitted for EOG. Removed stop-suffix tokens still
have states. Inside the callback the handle remains busy; other native model
operations are unavailable, while pure R calculations and lazy trace reads
remain possible. A 5 ms polling target replaces the ordinary 50 ms target
only for live calls; delivery depends on R's event loop.
State materialization is bounded by the smaller of relm.trace_budget and
32 MiB, including metadata and conversion accounting. Oversized activations
spill to completed Arrow files when spill = TRUE; otherwise admission
raises relm_error_oom. A single activation vector is limited to 1 MiB f32,
capture/writer transport to 8 MiB plus accounted scratch, and conservative
whole-call spill output to 2 GiB / 1024 files. Bounds do not cover model/KV
memory, allocator overhead or objects retained/copied by user code. Managed
files survive job/model closure until session cleanup; custom directories
remain caller-managed. Callback failures retain their original condition
in relm_error_callback, with callback = "on_state".
Structured output
Supply schema as JSON text to constrain text generation to a bounded subset
of JSON Schema 2020-12. The root must be a non-nullable object with explicit
properties, required listing every property exactly once, and
additionalProperties: false. Nested objects follow the same rules. Supported
values are bounded strings (maxLength, optional minLength), non-nullable
string enums, integers with explicit minimum and maximum in the signed
32-bit range, booleans, null, and nullable versions of these types. Nullable
enums, arrays, fractional numbers and all other keywords (including
annotations) are rejected. Optional root $schema must be
"https://json-schema.org/draft/2020-12/schema".
The schema is compiled once per call; each prompt uses fresh grammar state and the same scalar seed. Every successful element is complete, independently validated JSON text. Keys are generated in UTF-8 lexicographic order. String lengths count decoded Unicode scalar values; duplicate keys and invalid Unicode are rejected. Describe the task and fields in the prompt: a schema constrains output structure, not factual correctness.
Nonempty stop or image collections cannot be combined with a schema. Empty
collections count as absent, and vision handles may make text-only structured
requests. Generation returns when the root object is complete, including on
the last allowed token. It does not retry, repair or return incomplete JSON.
Malformed, unsupported or oversized schemas raise relm_error_schema with
reason and schema_path (a JSON pointer). Generation failure raises
relm_error_structured_output with reason, 1-based prompt_id, seed,
generated_tokens and bounded partial_bytes (a raw vector). The first
failed prompt aborts the call; no partial result vector is returned. Input
context overflow retains relm_error_context_overflow.
Hard limits: schema text 64 KiB, nesting 8, schema nodes 128, 16 properties
per object and 64 total; 32 enum members of at most 128 Unicode scalar values;
string maxLength 2048; compiled grammar 512 KiB and 65536 elements. A call
accepts at most 128 prompts, 1 MiB per prompt and 16 MiB total UTF-8 input,
with max_tokens <= 8192. Temperature must be finite and fit a 32-bit float;
an explicit seed must be finite and less than 2^64 (subject to R's numeric
precision). Output is limited to 64 KiB per prompt and 8 MiB
per call. schema = NULL preserves ordinary text and vision generation.
Image input (vision models)
On a handle loaded with llm()'s projector argument, images attaches
image files to each prompt: a list parallel to prompt, where
images[[i]] is a character vector of image file paths for prompt i
(character(0) for none). A bare character vector is treated as
list(images) — one image set — and pairs with a single prompt; with
several prompts it is recycled across all of them with a warning (an empty
vector is the same as NULL: no images, no warning). Each
prompt's images are inserted before its text. Exactly three file
formats are accepted: JPEG, PNG, BMP (anything else — GIF and audio
included — is rejected before any decode with relm_error_image). Size
limits, enforced before decoding: at most 64 MB per file by default
(override with options(relm.image_max_bytes = ); hard ceiling
2147483647 bytes), each dimension between 1 and 16384 pixels, and at most
33554432 total pixels. Images on a handle loaded without a projector raise
relm_error_image; the combined text+image token count must fit
context_length (relm_error_context_overflow states by how much).
One content restriction applies to an image-bearing prompt: the literal
string "<__media__>" (the engine's internal media marker) is reserved —
relm inserts one marker per image before the text, so a literal marker in
the prompt would corrupt the image placement, and the call raises
relm_error_argument naming prompt. Prompts without images may contain
the string freely (it is ordinary text there).
Examples
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_QWEN"))
m <- llm(Sys.getenv("RELM_TEST_MODEL_QWEN"))
llm_generate(m, "In one sentence, what is R?", max_tokens = 40, seed = 1)
schema <- paste0('{"type":"object","properties":{"answer":',
'{"type":"string","enum":["yes","no"]}},',
'"required":["answer"],"additionalProperties":false}')
llm_generate(m, 'Is R a programming language? Return an object with "answer".',
schema = schema, max_tokens = 64, temperature = 0, seed = 1)
close(m)
}
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_VLM")) && nzchar(Sys.getenv("RELM_TEST_MMPROJ_VLM"))
# Image input (requires a projector -- see llm()).
m <- llm(Sys.getenv("RELM_TEST_MODEL_VLM"),
projector = Sys.getenv("RELM_TEST_MMPROJ_VLM")
)
img <- file.path(tempdir(), "square.png")
png(img, width = 224, height = 224)
par(mar = c(0, 0, 0, 0))
plot.new()
rect(0.3, 0.3, 0.7, 0.7, col = "red", border = NA)
dev.off()
llm_generate(m, "What color is the square?",
images = img,
max_tokens = 16, temperature = 0
)
close(m)
}