Your AI Agent Just Made a Decision Nobody Can Explain. That Is About to Become Your Problem.
Somewhere in your stack, an AI agent made a decision today.
Maybe it ranked a loan application. Maybe it filtered a job candidate. Maybe it flagged a transaction as fraudulent, denied a support request, or routed a customer to a lower service tier.
The decision happened in milliseconds. It was logged as a status code. Nobody reviewed it. Nobody can explain it. And as of this year, that is no longer just an engineering problem.
It is a legal one.
What just changed
The EU AI Act’s high-risk system provisions are now fully enforceable.
High-risk AI systems, which include any automated system making consequential decisions about people in credit, employment, access to services, and law enforcement, are now required to maintain human oversight, produce explainable outputs, and keep auditable records of automated decisions.
This applies to systems deployed to EU residents. It applies whether you are an EU company or not. It applies whether you knew about the regulation or not.
The fines are not token amounts. They scale to 3% of global annual revenue for non-compliance with the technical requirements. For a company doing $50 million in revenue, that is $1.5 million per violation category.
Most AI agent deployments in production today fail at least two of the three technical requirements.
The explainability gap
Ask yourself a direct question: if a user asked why your AI agent made the specific decision it made about them, what would you say?
Not in general terms. Not “the model evaluated multiple factors.” Specifically. Which inputs drove the output? What weight did each one carry? What would have changed the decision?
Most teams cannot answer this. Not because the engineering is sloppy. Because the architecture was never built to answer it.
The model makes a call. The call gets executed. The call gets logged as a result, not as a reasoning chain.
# What most agent logs look like
{
"timestamp": "2026-06-12T09:14:22Z",
"agent": "loan_review_agent",
"input_id": "application_44821",
"output": "declined",
"confidence": 0.87,
"latency_ms": 340
}
# What an auditable agent log needs to look like
{
"timestamp": "2026-06-12T09:14:22Z",
"agent": "loan_review_agent",
"agent_version": "2.4.1",
"model": "claude-sonnet-4-6",
"input_id": "application_44821",
"input_hash": "sha256:a3f9...",
"output": "declined",
"decision_factors": [
{
"factor": "debt_to_income_ratio",
"value": 0.67,
"threshold": 0.45,
"contribution": "primary_negative"
},
{
"factor": "employment_tenure_months",
"value": 3,
"threshold": 6,
"contribution": "secondary_negative"
},
{
"factor": "credit_score",
"value": 710,
"threshold": 650,
"contribution": "positive"
}
],
"reasoning_trace": "Application declined primarily due to...",
"counterfactual": "Decision would change to approved if DTI were below 0.45",
"human_review_required": false,
"human_review_triggered": false,
"confidence": 0.87,
"latency_ms": 340
}The second log is not significantly harder to produce than the first. It requires the agent to be designed to produce it from the start. Retrofitting it onto an existing agent is painful. Retrofitting it while the compliance clock is running is worse.
The three requirements most teams are failing
Requirement one: Human oversight for high-stakes decisions.
The regulation requires that consequential decisions have a human in the loop, or at minimum a mechanism by which a human can review and override the automated decision before it takes effect.
Most agent implementations have no override mechanism. The decision executes immediately. By the time anyone could review it, it has already affected the user.
The fix is a decision queue for high-confidence negative outcomes. Not every decision. The ones that matter.
from dataclasses import dataclass
from enum import Enum
from datetime import datetime, timezone, timedelta
from typing import Optional
class DecisionRisk(Enum):
LOW = "low" # Auto-execute, log only
MEDIUM = "medium" # Auto-execute, flag for review
HIGH = "high" # Queue for human review before execution
@dataclass
class AgentDecision:
decision_id: str
agent_id: str
subject_id: str
outcome: str
confidence: float
risk_level: DecisionRisk
reasoning: str
decision_factors: list[dict]
counterfactual: Optional[str]
created_at: datetime
execute_after: datetime
executed: bool = False
human_reviewed: bool = False
human_override: Optional[str] = None
def classify_decision_risk(
outcome: str,
confidence: float,
decision_type: str,
) -> DecisionRisk:
HIGH_RISK_OUTCOMES = {"declined", "suspended", "flagged", "rejected"}
HIGH_RISK_TYPES = {"credit", "employment", "benefits", "insurance"}
if decision_type in HIGH_RISK_TYPES and outcome in HIGH_RISK_OUTCOMES:
return DecisionRisk.HIGH
if outcome in HIGH_RISK_OUTCOMES and confidence < 0.75:
return DecisionRisk.HIGH
if outcome in HIGH_RISK_OUTCOMES:
return DecisionRisk.MEDIUM
return DecisionRisk.LOW
async def process_agent_decision(decision: AgentDecision) -> None:
if decision.risk_level == DecisionRisk.HIGH:
await queue_for_human_review(decision)
await notify_reviewer(decision, deadline_hours=24)
return
if decision.risk_level == DecisionRisk.MEDIUM:
await execute_decision(decision)
await flag_for_retrospective_review(decision)
return
await execute_decision(decision)Requirement two: Auditable records.
The regulation requires that records of automated decisions be retained and retrievable. Not just the outcome. The input, the model version, the reasoning, and the context at the time of the decision.
“The time of the decision” is the part that makes this hard. Context changes. Feature values drift. A record that says “the model evaluated the application” is not auditable. A record that captures what the feature values were at the exact moment the model evaluated the application is.
Most systems do not store this. They store the outcome and a pointer to the input that may have since changed.
Requirement three: Explainability on request.
Any individual subject to an automated decision in a high-risk category has the right to request an explanation of that decision.
That request might arrive six months after the decision was made. The team needs to be able to reconstruct not just what was decided but why, in terms a non-technical person can understand, against the specific context that existed at the moment of the decision.
This is not currently possible for most deployed agents. The logs do not contain it. The model does not retain it. The feature values may have changed. The agent version may have been updated.
What a compliant agent architecture looks like
The teams who have solved this have built three things that most have not.
Immutable decision records. Every decision writes an immutable record containing the agent version, the model version, a snapshot of every feature value used in the decision, the reasoning trace, the outcome, and the confidence. Not a pointer to the input. A snapshot of the input. The record must be readable six months later even if everything else has changed.
Confidence-gated human queues. Decisions above a risk threshold do not auto-execute. They go into a queue where a human can approve, override, or escalate within a defined window. The queue has SLAs. Decisions that are not reviewed within the SLA get escalated or auto-rejected conservatively. This is the human oversight mechanism the regulation requires.
Explanation generation. The agent is prompted to produce a plain-language explanation of its reasoning alongside its decision. Not the reasoning chain from the model. A cleaned, structured summary that could be delivered to a user. This gets stored with the decision record.
DECISION_SYSTEM_PROMPT = """
You are a loan review agent. For each application, you will:
1. Evaluate the application against the provided criteria
2. Reach a decision: approved, declined, or referred
3. Provide your reasoning in two formats:
TECHNICAL: A structured breakdown of which factors influenced
your decision and in what direction.
PLAIN_LANGUAGE: A 2-3 sentence explanation that could be sent
directly to the applicant, using no jargon, explaining the
primary reason for the decision.
4. If declining, provide a COUNTERFACTUAL: the single most
impactful change that would have changed the outcome.
Your plain language explanation will be stored and may be
provided to the applicant on request. Write it accordingly.
"""This is not additional work for the model. It is a prompting decision that produces the compliance artifact as a byproduct of the decision itself.
The companies that are already exposed
The regulation applies retroactively to systems in operation. You do not get a grace period for systems that were already deployed when the enforcement provisions took effect.
If you are running automated decisions in high-risk categories against EU residents, you are already subject to the requirements. The clock is not starting. It has started.
The companies most exposed are not the ones who deployed carelessly. They are the ones who deployed carefully, in 2024 and early 2025, when the technical requirements seemed distant, and built systems that work well but were never designed to be auditable.
Good systems, wrong architecture, bad timing.
The audit that is coming
The enforcement posture of EU data protection authorities has changed in the past twelve months. They are no longer waiting for complaints. They are conducting proactive audits of high-risk AI deployments.
The audit request asks for three things.
A record of every automated decision made about EU residents in the past 12 months. The technical documentation showing how the system meets the human oversight requirement. Evidence that explanations can be produced on request.
Most teams could not respond to this request today. Not because they have been negligent. Because the systems were not designed to respond to it.
The teams that can respond will produce the records, walk through the oversight mechanism, generate an example explanation in five minutes, and move on.
The teams that cannot will spend three months trying to reconstruct records from logs that were not designed to support reconstruction, document an oversight mechanism that does not exist, and explain why the explainability function returns a stack trace.
The difference between those two outcomes is mostly architectural. It is not a large amount of engineering work to do it right from the start. It is an enormous amount of engineering work to retrofit it under regulatory pressure.
Build the audit trail now.
The auditor is coming. And unlike your users, they will not accept a spinner and a vague message about currently processing.