Fits one binary ridge logistic probe per layer using the optional glmnet package. Group-disjoint development cross-validation selects regularization and the default layer. Scaling and fitted coefficients use development data only. A declared holdout evaluates frozen fits; absent a holdout, results are exploratory CV diagnostics without inferential confidence intervals.
Usage
llm_probe(
formula,
data,
method = "glmnet",
cv = 10,
metric = c("auc", "accuracy"),
seed = NULL,
groups = NULL,
test_groups = NULL
)Arguments
- formula
A formula such as
label ~ activations(layer = 1:3). Labels are logical, numeric 0/1, or a two-level factor (second level positive), from a prompt-constant trace column or an external vector in the formula environment.- data
A
relm_trace, with one captured position per prompt, matching prompt/position/neuron coordinates across requested layers. Spilled traces require an external label vector.- method
Currently only
"glmnet"(binary ridge logistic regression).- cv
Number of group-disjoint development folds, an integer at least 3. Infeasible class support raises an error rather than reducing this value.
- metric
Selection and reporting metric:
"auc"or"accuracy". Accuracy uses a fixed probability threshold of 0.5.- seed
Optional nonnegative integer. Seeded calls restore the caller's RNG state, including its prior absence;
NULLuses the caller's RNG.- groups
Character source-group IDs, one per captured prompt in increasing prompt-ID order, or named by exact prompt IDs.
NULLassumes each prompt is independent. Related documents, revisions, paraphrases and matched contrasts must share a caller-supplied group.- test_groups
Character IDs of whole groups reserved before analysis, or
NULLfor exploratory development CV without held-out estimates.
Value
An llm_probe classed list: development-only fits, per-layer
metrics, fold-level cv_scores, prompt-level split audit, held-out
predictions, selected_layer, counts and statistical provenance. Fits store
compact coefficients and preprocessing, not every CV path or the source trace.
Details
The fixed ridge grid is 10^seq(4, -4, length.out = 41). Each fitting
partition supplies its own means and RMS scales. Constant features are
dropped. Selection uses the mean of fold metrics, with stronger regularization
and then the lower layer breaking ties. Errors inherit relm_error_probe;
an estimated materialized-memory excess raises relm_error_oom before
densification, using options(relm.trace_budget) as a ceiling.
Held-out intervals use 2,000 whole-group bootstrap draws and type-7 percentile quantiles. They are approximate 95% pointwise intervals for sampling variability conditional on the frozen fits, development data, selection and split, not training/selection uncertainty or simultaneous coverage. Intervals are withheld with a reason below 20 evaluation groups, below five groups containing either class, on undefined draws, or for a degenerate interval. Larger groups contribute more observations to metrics. Decodability is association, not evidence of causal use. Repeatedly changing an analysis after inspecting the holdout consumes its independent status.
Examples
# Model-free trace for demonstrating the API, not scientific evidence.
label <- rep(0:1, 12)
tr <- expand.grid(neuron = 1:2, layer = 1:2, prompt_id = seq_along(label))
tr$token_pos <- 1L
tr$token <- "example"
tr$component <- "residual"
tr$value <- sin(tr$prompt_id * tr$neuron) + label[tr$prompt_id]
class(tr) <- c("relm_trace", "data.frame")
fit <- llm_probe(label ~ activations(layer = 1:2), tr, cv = 3, seed = 42)
summary(fit)
#> <llm_probe summary> auc; positive class: 1
#> layer selected on development CV: 1
#> CV scores are selection diagnostics; held-out intervals are conditional and pointwise.
#> layer lambda cv_score test_score lower upper interval_status valid_draws
#> 1 10000 0.8541667 NA NA NA not_evaluated 0
#> 2 10000 0.8541667 NA NA NA not_evaluated 0
#> undefined_draws boundary
#> 0 TRUE
#> 0 TRUE
#> Split audit, class/group counts and preprocessing are available in this summary.
predict(fit, tr)
#> 1 2 3 4 5 6 7 8
#> 0.5000058 0.5000091 0.4999911 0.5000094 0.4999803 0.5000014 0.5000050 0.5000134
#> 9 10 11 12 13 14 15 16
#> 0.4999896 0.5000105 0.4999841 0.4999965 0.5000013 0.5000177 0.4999896 0.5000097
#> 17 18 19 20 21 22 23 24
#> 0.4999885 0.4999941 0.4999956 0.5000208 0.4999917 0.5000078 0.4999923 0.4999946