Skip to contents

Encodes each string in x into a fixed-length numeric vector, returning a base matrix with one row per input and one column per embedding dimension (so the matrix is length(x) by the model's hidden_size).

Usage

llm_embed(
  m,
  x,
  pooling = c("mean", "last", "model"),
  normalize = TRUE,
  images = NULL
)

Arguments

m

An llm handle from llm().

x

A character vector of one or more non-empty strings to embed; NA and empty strings ("") are rejected (an empty string is allowed only for an input that carries at least one image). names(x) become the row names.

pooling

How to reduce each input's per-token vectors to one vector: "mean" (average), "last" (final token), or "model" (the model's own pooling when the GGUF defines one; otherwise an error asking for "mean"/"last").

normalize

Single logical. TRUE (default) L2-normalizes each row so rows are unit vectors and dot products are cosine similarities.

images

NULL (default: text-only, unchanged) or the image file paths to pair with each input — a list parallel to x, or a bare character vector for a single input. Accepted formats: JPEG, PNG, BMP. Requires a handle loaded with llm(projector = ); see the Image input section.

Value

A numeric matrix, length(x) rows by the model's embedding size (columns), with row names names(x) when set, else the input positions as characters.

Details

Each input is tokenized (with the model's begin/end-of-sequence markers) and run through a forward pass in a dedicated embeddings-mode context; the per-token hidden states are then pooled into one vector per input by pooling. "mean" averages the token vectors, "last" takes the final token's vector, and "model" uses the model's own pooling when the GGUF defines one (a purely generative model such as Qwen2.5 defines none, so "model" raises relm_error_embed asking for "mean" or "last").

With normalize = TRUE (the default) each row is L2-normalized to a unit vector, so the dot product of two rows is their cosine similarity. A zero vector is returned unchanged (never NaN). normalize is validated and its effect is explicit — there is no silent normalization.

The model must carry a tokenizer; a no_vocab model raises relm_error_tokenize. No model ships inside the package yet (the in-repo synthetic model has no tokenizer), so the runnable example is guarded by the RELM_TEST_MODEL_QWEN environment variable — point it at a local Qwen2.5 GGUF to run it.

Embedding an intervened handle (from llm_steer()/llm_ablate()) raises relm_error_embed: interventions currently apply to generation and logits only, and the embedding context does not inherit them, so returning base vectors labeled as intervened would be silent mislabeling. Embed the original handle.

Image input (vision models)

On a handle loaded with llm()'s projector argument, images embeds each (text, image) pair into one row: a list parallel to x, where images[[i]] is a character vector of image file paths for input i (character(0) for none), or a bare character vector for a single input (recycled across several inputs with a warning; an empty vector is the same as NULL) — the same pairing contract as llm_generate()'s images. Images are inserted before the text. The accepted formats and pre-decode limits are exactly llm_generate()'s: JPEG, PNG, BMP only, at most 64 MB per file by default (options(relm.image_max_bytes = )), dimensions 1–16384 px per side, at most 33554432 total pixels; the literal marker "<__media__>" is reserved in an image-bearing input (relm_error_argument). An input that carries an image may have empty text (x = "") — the image alone is embedded; empty text without an image is still rejected.

Pooling semantics with images: the per-token vectors reduced by pooling are those of the text positions (including the model's own image-delimiter tokens); the image conditions those vectors through attention. This matches the reference llama.cpp behavior at the pinned engine version — image patch positions expose no per-token hidden states — and text-only inputs are completely unchanged. Images on a handle loaded without a projector raise relm_error_image.

Examples

if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_QWEN"))
m <- llm(Sys.getenv("RELM_TEST_MODEL_QWEN"))
e <- llm_embed(m, c(a = "cats and dogs", b = "domestic pets"))
dim(e)
close(m)
}
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_VLM")) && nzchar(Sys.getenv("RELM_TEST_MMPROJ_VLM"))
# Image embeddings (requires a projector -- see llm()).
m <- llm(Sys.getenv("RELM_TEST_MODEL_VLM"),
  projector = Sys.getenv("RELM_TEST_MMPROJ_VLM")
)
img <- file.path(tempdir(), "square.png")
png(img, width = 224, height = 224)
par(mar = c(0, 0, 0, 0))
plot.new()
rect(0.3, 0.3, 0.7, 0.7, col = "red", border = NA)
dev.off()
e <- llm_embed(m, "", images = img) # the image alone, one row
dim(e)
close(m)
}