Skip to contents

Converts between text and the model's tokens. With decode = FALSE (the default) x is text and the result is the token ids; with decode = TRUE x is token ids and the result is the reconstructed string.

Usage

llm_tokens(m, x, decode = FALSE)

Arguments

m

An llm handle.

x

For encoding, a character vector; for decoding, an integer vector of 1-based token ids.

decode

Single logical. FALSE (default) encodes text to ids; TRUE decodes ids to text.

Value

Encoding: a named integer vector (single input) or a list of them (several inputs). Decoding: a single string.

Details

Encoding (decode = FALSE). x is a character vector. Each element is tokenized into a named integer vector: the values are the token ids and the names are the token pieces (the text each id renders as). For a single string a named integer vector is returned; for several strings a list of such vectors is returned, preserving names(x). No beginning/end-of-sequence markers are added (you get exactly the tokens of the text); chat formatting is llm_generate's job, not this function's.

Decoding (decode = TRUE). x is an integer vector of token ids and the result is a single string. Decoding is UTF-8 correct even when a multi-byte character spans two tokens — always decode the whole id vector rather than concatenating the piece names, which can split a character.

Token ids are 1-based in the R API (like every other index in relm); subtract 1 to compare them with a raw vocabulary index (llama.cpp / Hugging Face). Encoding and decoding are exact inverses: llm_tokens(m, llm_tokens(m, txt), decode = TRUE) reproduces txt.

The model must carry a tokenizer; a no_vocab model raises relm_error_tokenize.

Examples

if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_QWEN"))
m <- llm(Sys.getenv("RELM_TEST_MODEL_QWEN"))
ids <- llm_tokens(m, "The quick brown fox")
ids
llm_tokens(m, ids, decode = TRUE)
close(m)
}