A Practical Guide to AI Evals
Generative and agentic AI systems are moving into production faster than the methods used to govern them. To keep pace, organizations increasingly lean on automated risk evaluations ("evals") to surface legal, regulatory, reputational, and operational risk before it reaches a customer.
Three approaches dominate today:
- LLM-as-a-judge, which points a single model and a single prompt at a system's output and scores whether a given risk is present.
- Runtime Safeguards, which apply lightweight filtering to content as it is generated.
- Red Teaming, which simulates adversaries trying to break a model's safety protections.
This paper shows how each of these approaches, on their own or in combination, fail to capture the risks that pose the biggest threat to companies adopting AI. The approaches miss real risks and over-report risks that aren't there, all while offering too little detail to act on either.
The shared cause is Low Dimensionality: a risk posed as a single general question can only be answered in a single dimension, and one dimension is rarely enough to describe how a probabilistic system actually behaves.
We lay out the failure modes of low-dimensionality evaluation and explain how an alternative method we devised — High-dimensionality Evaluation — succeeds where other methods fail. High-dimensionality evals decompose each risk into many sub-components, require the evaluator to show its analysis rather than just its verdict, run across multiple models, and anchor their standards in the rigor of legal and compliance review.
This approach doesn't replace runtime safeguards or red teaming. But for anyone serious about managing AI risk in production, it is a requirement.
