rulvar API reference / @rulvar/evals
@rulvar/evals
Classes
| Class | Description |
|---|---|
| EvalJudgeError | Thrown when a judge run does not settle ok. |
Interfaces
| Interface | Description |
|---|---|
| CanaryDriftReport | - |
| CanaryProbeSet | - |
| CheckpointArm | - |
| CheckpointCell | - |
| CheckpointLadder | One declared checkpoint ladder: rungs are concrete pool members. |
| CheckpointPool | - |
| CheckpointReport | - |
| CriterionOneReport | - |
| CriterionTwoReport | - |
| EvalCase | One quality-measurement case. The shape is the documented interface verbatim; display names derive from the workflow name (the suite runner disambiguates duplicates by ordinal). |
| EvalCaseResult | The measured result of one EvalCase. |
| EvalCommitterOptions | - |
| EvalMatrixReport | - |
| EvalSuiteResult | Aggregate view of a suite run. |
| GoldenGraderOptions | - |
| Grader | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| GraderContext | What a grader sees; judge() is the only channel back into the engine. |
| GraderVerdict | One grader's outcome for one case. |
| JudgeGraderOptions | - |
| JudgeSpec | A judge invocation specification. The judge runs through the engine as an ordinary journaled, budgeted invocation; model selection is subject to the router quality floors, and @rulvar/evals ships NO default judge model: weak defaults for judging are forbidden, so the model is always explicit. |
| MatrixCell | One configuration under comparison. |
| MatrixCellReport | - |
| MeasuredClaimInput | - |
| OrchestratedCase | - |
| RubricCriterion | - |
| RubricGraderOptions | - |
| RunCheckpointOptions | - |
| RunEvalCaseOptions | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| RunEvalSuiteOptions | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| RunSweepOptions | - |
| SweepCase | An eval case bound to the taskClass axis of the matrix. |
| SweepCellReport | - |
| SweepModel | One fixed pool member; effort is part of the claim subject identity. |
| SweepPool | - |
| SweepReport | - |
| SweepThresholds | - |
Variables
| Variable | Description |
|---|---|
| JUDGE_VERDICT_SCHEMA | The default judge verdict shape. |
| SWEEP_THRESHOLD_DEFAULTS | - |
Functions
| Function | Description |
|---|---|
| canaryFingerprint | Runs the fixed probe set through the ordinary engine and returns the fingerprint. Probes run sequentially in declaration order, one run per probe, so recordings replay deterministically. |
| commitEvalMeasured | Commits measured claims through the eval-committer gate with the documented rebase recipe: on a CAS rejection, re-read current() and retry against the fresh version. Returns the committed version. |
| evalMeasuredClaim | One measured claim; claimExpiry applies the TTL from the decay table. |
| flipStaleOnCanaryDrift | Flips the model's ACTIVE eval-measured claims to stale when their recorded canary fingerprint differs from the fresh one. Claims without a recorded fingerprint have no baseline and stay untouched (the documented no-probe posture); a second run is an idempotent noop. CAS-rebased like every maintenance commit. |
| goldenGrader | - |
| judgeGrader | - |
| normalizeCanaryOutput | The committed v1 normalization (OQ-06): NFC, trim, collapse whitespace. |
| renderCheckpointReport | The deterministic render for the M12 gate docs amendment. |
| rubricGrader | - |
| runEvalCase | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| runEvalMatrix | Runs the same case list against every cell's engine, sequentially and in declaration order (deterministic cassette consumption), and reports per-cell aggregates for side-by-side comparison. |
| runEvalSuite | @rulvar/evals: quality measurement strictly on the public APIs (L6). EvalCase with golden, rubric, and LLM-judge graders; judge calls run through the engine (journaled, budgeted, VCR-recordable), so eval CI is deterministic; config-matrix comparison reports pass-rate, cost, and latency per cell. Matrix sweeps feeding ModelKnowledge, the eval-committer identity, and canary fingerprints are the M11 round-3 extensions. |
| rungRuleHolds | The OQ-09 cell rule (shared by the per-cell and pooled verdicts). |
| runSweepMatrix | Runs the fixed matrix sequentially in declaration order (deterministic cassette consumption), aggregates per (model, taskClass) cell, emits threshold-crossing claims, and commits them through the eval-committer identity when a store is given. |
| runValueCheckpoint | Runs the checkpoint over the fixed pool. Sequential in declaration order (deterministic cassette consumption when recorded); every cell runs baseline then treatment. |