Skip to contents

Fits one binary ridge logistic probe per layer using the optional glmnet package. Group-disjoint development cross-validation selects regularization and the default layer. Scaling and fitted coefficients use development data only. A declared holdout evaluates frozen fits; absent a holdout, results are exploratory CV diagnostics without inferential confidence intervals.

Usage

llm_probe(
  formula,
  data,
  method = "glmnet",
  cv = 10,
  metric = c("auc", "accuracy"),
  seed = NULL,
  groups = NULL,
  test_groups = NULL
)

Arguments

formula

A formula such as label ~ activations(layer = 1:3). Labels are logical, numeric 0/1, or a two-level factor (second level positive), from a prompt-constant trace column or an external vector in the formula environment.

data

A relm_trace, with one captured position per prompt, matching prompt/position/neuron coordinates across requested layers. Spilled traces require an external label vector.

method

Currently only "glmnet" (binary ridge logistic regression).

cv

Number of group-disjoint development folds, an integer at least 3. Infeasible class support raises an error rather than reducing this value.

metric

Selection and reporting metric: "auc" or "accuracy". Accuracy uses a fixed probability threshold of 0.5.

seed

Optional nonnegative integer. Seeded calls restore the caller's RNG state, including its prior absence; NULL uses the caller's RNG.

groups

Character source-group IDs, one per captured prompt in increasing prompt-ID order, or named by exact prompt IDs. NULL assumes each prompt is independent. Related documents, revisions, paraphrases and matched contrasts must share a caller-supplied group.

test_groups

Character IDs of whole groups reserved before analysis, or NULL for exploratory development CV without held-out estimates.

Value

An llm_probe classed list: development-only fits, per-layer metrics, fold-level cv_scores, prompt-level split audit, held-out predictions, selected_layer, counts and statistical provenance. Fits store compact coefficients and preprocessing, not every CV path or the source trace.

Details

The fixed ridge grid is 10^seq(4, -4, length.out = 41). Each fitting partition supplies its own means and RMS scales. Constant features are dropped. Selection uses the mean of fold metrics, with stronger regularization and then the lower layer breaking ties. Errors inherit relm_error_probe; an estimated materialized-memory excess raises relm_error_oom before densification, using options(relm.trace_budget) as a ceiling.

Held-out intervals use 2,000 whole-group bootstrap draws and type-7 percentile quantiles. They are approximate 95% pointwise intervals for sampling variability conditional on the frozen fits, development data, selection and split, not training/selection uncertainty or simultaneous coverage. Intervals are withheld with a reason below 20 evaluation groups, below five groups containing either class, on undefined draws, or for a degenerate interval. Larger groups contribute more observations to metrics. Decodability is association, not evidence of causal use. Repeatedly changing an analysis after inspecting the holdout consumes its independent status.

Examples

# Model-free trace for demonstrating the API, not scientific evidence.
label <- rep(0:1, 12)
tr <- expand.grid(neuron = 1:2, layer = 1:2, prompt_id = seq_along(label))
tr$token_pos <- 1L
tr$token <- "example"
tr$component <- "residual"
tr$value <- sin(tr$prompt_id * tr$neuron) + label[tr$prompt_id]
class(tr) <- c("relm_trace", "data.frame")
fit <- llm_probe(label ~ activations(layer = 1:2), tr, cv = 3, seed = 42)
summary(fit)
#> <llm_probe summary> auc; positive class: 1
#>   layer selected on development CV: 1
#>   CV scores are selection diagnostics; held-out intervals are conditional and pointwise.
#>  layer lambda  cv_score test_score lower upper interval_status valid_draws
#>      1  10000 0.8541667         NA    NA    NA   not_evaluated           0
#>      2  10000 0.8541667         NA    NA    NA   not_evaluated           0
#>  undefined_draws boundary
#>                0     TRUE
#>                0     TRUE
#>   Split audit, class/group counts and preprocessing are available in this summary.
predict(fit, tr)
#>         1         2         3         4         5         6         7         8 
#> 0.5000058 0.5000091 0.4999911 0.5000094 0.4999803 0.5000014 0.5000050 0.5000134 
#>         9        10        11        12        13        14        15        16 
#> 0.4999896 0.5000105 0.4999841 0.4999965 0.5000013 0.5000177 0.4999896 0.5000097 
#>        17        18        19        20        21        22        23        24 
#> 0.4999885 0.4999941 0.4999956 0.5000208 0.4999917 0.5000078 0.4999923 0.4999946