Converts between text and the model's tokens. With decode = FALSE (the
default) x is text and the result is the token ids; with decode = TRUE
x is token ids and the result is the reconstructed string.
Value
Encoding: a named integer vector (single input) or a list of them (several inputs). Decoding: a single string.
Details
Encoding (decode = FALSE). x is a character vector. Each element is
tokenized into a named integer vector: the values are the token ids and
the names are the token pieces (the text each id renders as). For a single
string a named integer vector is returned; for several strings a list of such
vectors is returned, preserving names(x). No beginning/end-of-sequence
markers are added (you get exactly the tokens of the text); chat formatting is
llm_generate's job, not this function's.
Decoding (decode = TRUE). x is an integer vector of token ids and the
result is a single string. Decoding is UTF-8 correct even when a multi-byte
character spans two tokens — always decode the whole id vector rather than
concatenating the piece names, which can split a character.
Token ids are 1-based in the R API (like every other index in relm);
subtract 1 to compare them with a raw vocabulary index (llama.cpp / Hugging
Face). Encoding and decoding are exact inverses:
llm_tokens(m, llm_tokens(m, txt), decode = TRUE) reproduces txt.
The model must carry a tokenizer; a no_vocab model raises
relm_error_tokenize.
Examples
if (FALSE) { # nzchar(Sys.getenv("RELM_TEST_MODEL_QWEN"))
m <- llm(Sys.getenv("RELM_TEST_MODEL_QWEN"))
ids <- llm_tokens(m, "The quick brown fox")
ids
llm_tokens(m, ids, decode = TRUE)
close(m)
}