Article URL: https://blog.mozilla.ai/from-evaluation-to-guardrails-what-we-brought-to-acm-facct-2026/ Comments URL: https://news.ycombinator.com/item?id=49026059 Points: 8 # Commen…

At ACM FAccT, we demonstrated why AI guardrails need the same scrutiny as models. Moving from static policies to context- and language-specific evaluations, our hands-on session proved that agentic guardrails equipped with tools like web search are vital for reliable, real-world deployment. In June, we traveled to Montreal for the ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), the premier venue for safe and responsible AI development; a conference where computer scientists, social scientists, policymakers, and lawyers share a room to ask not just how to build AI systems, but whether, when, and for whom. It was the right audience for our tutorial, Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems. Evaluation has been one of the main conversations in AI safety and a key step toward responsible deployment. The field is shifting from celebrating general capabilities to measuring real-world, domain- and language-specific performance, and cross-sector efforts like the EvalEval Coalition are now building the science and infrastructure for evaluating evaluations themselves. Despite the money spent on evals, benchmark saturation, and evaluation data going stale, one practical function endures: evaluation shapes guardrails. When an eval shows a model produces harmful or toxic content in a given context or language, one response is to design a guardrail around that exact failure. Guardrails – the mechanisms that filter, flag, or constrain LLM inputs and outputs – shape what users actually see in chatbots, platforms, and public-facing services. Yet they get far less scrutiny than the models they govern. Historically, they were proprietary classifiers, visible only through unexplained refusals. Open-source guardrail models and policy-prompt guardrails makes independent evaluation possible. Our argument at FAccT was simple: evaluating guardrails is as important as evaluating the LLMs they protect, and context- and language-specific evaluation results should inform guardrails that move beyond static taxonomies of harm toward dynamic policies. That path is what brought us to FAccT, because connecting the two is rarely straightforward. Our route: a community and language-informed evaluation of 120 refugee and asylum-focused scenario pairs across English, Farsi, Arabic, Kurdish-Sorani, and Pashto, scored by native-speaker evaluators from Respond Crisis Translation on six rights-based criteria. We published the results as the open MHRE evaluation data on Mozilla Data Collective, on terms set with the evaluators, and turned the recurring failures such as unsafe referrals, missing disclaimers, and stereotyped assumptions into concrete guardrail policies in English and Farsi. Testing those policies exposed a structural gap: criteria like factuality and actionability cannot be judged from text alone. Does this NGO exist? Is this law current in this jurisdiction? Often text-only guardrails rubber-stamped responses they had no way to verify, hallucinated terms, and scored identical English and Farsi policies differently (check out our blogpost). So we formed a hypothesis: LLM-enabled guardrails need tools such as search, retrieval, and fact-checking to judge in a more reliable and trustworthy manner. That was the agentic guardrail we brought to Montreal to test. The overall methodology and the case for contextual guardrails, with language- and context-dependent policies and tools, resonated with participants. Thirty-five participants ran our demo by selecting their own scenarios and policies and comparing agentic and non-agentic judges on the same responses.