Encodes each string in x into a fixed-length numeric vector, returning a base
matrix with one row per input and one column per embedding dimension (so the
matrix is length(x) by the model's hidden_size).
Usage
llm_embed(
m,
x,
pooling = c("mean", "last", "model"),
normalize = TRUE,
images = NULL
)Arguments
- m
An
llmhandle fromllm().- x
A character vector of one or more non-empty strings to embed;
NAand empty strings ("") are rejected (an empty string is allowed only for an input that carries at least one image).names(x)become the row names.- pooling
How to reduce each input's per-token vectors to one vector:
"mean"(average),"last"(final token), or"model"(the model's own pooling when the GGUF defines one; otherwise an error asking for"mean"/"last").- normalize
Single logical.
TRUE(default) L2-normalizes each row so rows are unit vectors and dot products are cosine similarities.- images
NULL(default: text-only, unchanged) or the image file paths to pair with each input — a list parallel tox, or a bare character vector for a single input. Accepted formats: JPEG, PNG, BMP. Requires a handle loaded withllm(projector = ); see the Image input section.
Value
A numeric matrix, length(x) rows by the model's embedding size
(columns), with row names names(x) when set, else the input positions as
characters.
Details
Each input is tokenized (with the model's begin/end-of-sequence markers) and run
through a forward pass in a dedicated embeddings-mode context; the per-token
hidden states are then pooled into one vector per input by pooling. "mean"
averages the token vectors, "last" takes the final token's vector, and
"model" uses the model's own pooling when the GGUF defines one (a purely
generative model such as Qwen2.5 defines none, so "model" raises
relm_error_embed asking for "mean" or "last").
With normalize = TRUE (the default) each row is L2-normalized to a unit
vector, so the dot product of two rows is their cosine similarity. A zero vector
is returned unchanged (never NaN). normalize is validated and its effect is
explicit — there is no silent normalization.
The model must carry a tokenizer; a no_vocab model raises
relm_error_tokenize. No model ships inside the package yet (the in-repo
synthetic model has no tokenizer), so the runnable example is guarded by the
RELM_TEST_MODEL_QWEN environment variable — point it at a local Qwen2.5
GGUF to run it.
Embedding an intervened handle (from llm_steer()/llm_ablate()) raises
relm_error_embed: interventions currently apply to generation and logits
only, and the embedding context does not inherit them, so returning base vectors
labeled as intervened would be silent mislabeling. Embed the original handle.
Image input (vision models)
On a handle loaded with llm()'s projector argument, images embeds each
(text, image) pair into one row: a list parallel to x, where
images[[i]] is a character vector of image file paths for input i
(character(0) for none), or a bare character vector for a single input
(recycled across several inputs with a warning; an empty vector is the same
as NULL) — the same pairing contract
as llm_generate()'s images. Images are inserted before the text.
The accepted formats and pre-decode limits are exactly llm_generate()'s:
JPEG, PNG, BMP only, at most 64 MB per file by default
(options(relm.image_max_bytes = )), dimensions 1–16384 px per side, at
most 33554432 total pixels; the literal marker "<__media__>" is reserved
in an image-bearing input (relm_error_argument). An input that carries an
image may have empty text (x = "") — the image alone is embedded; empty
text without an image is still rejected.
Pooling semantics with images: the per-token vectors reduced by pooling
are those of the text positions (including the model's own
image-delimiter tokens); the image conditions those vectors through
attention. This matches the reference llama.cpp behavior at the pinned
engine version — image patch positions expose no per-token hidden states —
and text-only inputs are completely unchanged. Images on a handle loaded
without a projector raise relm_error_image.
Examples
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_QWEN"))
m <- llm(Sys.getenv("RELM_TEST_MODEL_QWEN"))
e <- llm_embed(m, c(a = "cats and dogs", b = "domestic pets"))
dim(e)
close(m)
}
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_VLM")) && nzchar(Sys.getenv("RELM_TEST_MMPROJ_VLM"))
# Image embeddings (requires a projector -- see llm()).
m <- llm(Sys.getenv("RELM_TEST_MODEL_VLM"),
projector = Sys.getenv("RELM_TEST_MMPROJ_VLM")
)
img <- file.path(tempdir(), "square.png")
png(img, width = 224, height = 224)
par(mar = c(0, 0, 0, 0))
plot.new()
rect(0.3, 0.3, 0.7, 0.7, col = "red", border = NA)
dev.off()
e <- llm_embed(m, "", images = img) # the image alone, one row
dim(e)
close(m)
}