Compare actual observed states under recorded experimental settings
Source:R/graphics-compare.R, R/graphics-plots.R
llm_compare.RdBuilds an ordinary data frame from two delivered on_state observations.
No inference is performed. Differences are withheld when the complete sampled
input histories diverge. Caller-recorded context is validated, not authenticated.
Usage
llm_compare(
reference,
intervention,
context,
layer,
component = "residual",
neurons = NULL,
max_bytes = 64 * 1024^2
)
# S3 method for class 'relm_comparison'
plot(x, ...)Arguments
- reference, intervention
Single observations from
llm_generate(on_state=).- context
A list with
referenceandinterventionentries, each holdingmodel_sha256,settings, and integergenerated_tokensthrough its state. Settings contain, in order,seed,chat,temperature,top_p,max_tokens,stop,context_length,backend,relm_version,engine_revision.- layer
One 1-based captured transformer layer.
- component
One of
residual,attn_out,mlp_out.- neurons
NULL for all captured coordinates, or unique 1-based indices.
- max_bytes
Working materialization budget, whole bytes from 64 KiB to 256 MiB.
- x
A relm_comparison returned by llm_compare().
- ...
Named main, cex or col (one or three colors).
Value
A relm_comparison data frame with layer, component, neuron, reference,
intervention and difference. Attributes retain bounded provenance, alignment,
applied steering and a truncated logit comparison, never the original traces.
The plot method returns the comparison invisibly.
Details
The input for state k excludes its just-sampled token. Equal model/settings/ prompt records establish prompt identity; every earlier sampled token must also match. A divergent history keeps the observations but returns NA differences. Missing top-k entries are NA, not zero probabilities. A sampled token may never appear in committed output after cancellation or stop removal. Returning a coefficient to zero does not erase prior effects in KV history.
Examples
# Synthetic schema example: no model, inference or download.
source(system.file("examples", "graphics-states.R", package = "relm"), local = TRUE)
comparison <- llm_compare(reference, intervention, context, layer = 2)
comparison$difference
#> [1] 0.5 -1.0 1.0 0.0
figure <- tempfile(fileext = ".pdf")
grDevices::pdf(figure, width = 10, height = 7)
tryCatch(plot(comparison), finally = grDevices::dev.off())
unlink(figure)