Not Every Problem Needs an Agent: Determinism as an Architectural Decision

An agent for everything

#

The agent has become that hammer. I keep seeing teams reach for one whatever the problem: a multi-step loop that plans, calls tools, reflects on the result, and plans again. The trouble is that this reflex reaches tasks a local rule resolves with no inference tokens and, depending on its implementation, with much lower latency than a model call. Normalizing a ticket’s format, checking whether an identifier exists in a table, counting how many times an alert has repeated in the last hour: none of that needs to reason.

In Before Building Your First AI Agent, on the Keepler blog, I wrote about the data an agent needs. This post is about where you don’t need one.

The symptom shows up early: systems that are expensive, that cannot be reproduced or audited, and that hallucinate on binary decisions where an if always returns the same answer.

The question almost nobody asks before writing the first prompt is the only one that matters: which part of this actually needs to reason?

The thesis: reason where it is needed, and nowhere else

#

Not every problem needs an agent. Deciding where to reason is an architectural decision, and AI has its place in it. There is work a generative model does well that a rule cannot: interpreting free text, correlating a new incident with historical root cause analyses (RCAs) written by different people, extracting the root cause from an ambiguous description. That is unstructured evidence, and reasoning adds real value there.

And there is work determinism does better: classifying against a closed catalog, deciding whether a threshold is crossed, acting on an exact piece of data. Rules, state machines, and lookups are repeatable, cheap, and auditable by construction. So what decides where to reason and where not is the kind of data each step handles.

I am not the only one who sees it this way: decision models point in the same direction. Models like Strands Decider (Strands Labs) or Jev (TypeSafe) exist precisely so they do not generate text: they receive evidence and a set of options, and return a typed decision with a confidence score. This is AI designed to run inside a deterministic control flow. The principle comes down to this: the decision model decides, the LLM writes the words, and the control flow stays in your code.

Why “an agent for everything” is expensive

#

Putting an agent where a rule would have sufficed is not a cosmetic detail. You pay for it in four ways:

• Economic cost. Every trivial decision costs tokens, multiplied by volume. With thousands of tickets a day, reasoning over every classification that a rule table resolved for free is a permanent tax.
• Uncertainty. The same ticket can receive two different answers. In a summary that is tolerable; for deciding whether to wake up someone on call, it is not.
• Reproducibility and auditability. “Why was this ticket escalated?” has to be answerable months later. A deterministic branch lets you record which rule was applied and with what data; “the model decided that one time” does not pass an audit in a regulated environment.
• Failure surface. Every call to an LLM adds the risk of prompt injection, of hallucination, and of depending on an endpoint. A local rule is not exposed to prompt injection and does not fall over because an endpoint is saturated. It still has to validate its inputs.

The pattern: the flow is deterministic, the reasoning is a service

#

The four costs share the same origin: inference where it was not needed. The solution is not to remove the model, but to bound it. The pattern splits the flow into three zones. Let’s look at the diagram to see how a level 1 (L1) incident triage ticket crosses them, with AWS Step Functions orchestrating the whole path:

Deterministic zone. This is the skeleton of the system: rules, a state machine, and lookups. This zone holds the keyword-based base classification, the CMDB (configuration management database) query, recurrence detection by count and threshold, and the escalation decision. Everything that is structured data, a closed catalog, or a numeric threshold stays in this zone. It is cheap, repeatable, and leaves a trail.

Bounded reasoning zone. This is where GenAI comes in, but with a well-defined job: root cause analysis over free-text descriptions and semantic correlation with historical RCAs. “Bounded” is the key word. The model receives a concrete question about unstructured evidence and returns an interpretation; the flow is driven by the state machine.

Gray zone: the decision models. This is the hinge between the two previous ones. The base classification is deterministic, but it is adjusted if RAG (retrieval-augmented generation) finds precedents of similar incidents. That adjustment is decided by a decision model that returns a typed option plus a confidence score, not free-form LLM output. The control flow is still yours. If the confidence does not reach the threshold, the rule-based classification is kept. If it reaches it in a high-risk category, a person decides. The decider informs; your code decides what to do with that information.

In AWS Step Functions that split is a Choice state with a Default branch. This fragment is an excerpt (the state and field names are mine, and the threshold travels in the input so it is not hardcoded in the state machine); I validated it with aws stepfunctions validate-state-machine-definition inside a minimal state machine:

"AdjustClassification": {
  "Type": "Choice",
  "Choices": [
    {
      "And": [
        { "Variable": "$.decision.confidence", "NumericGreaterThanEqualsPath": "$.config.confidenceThreshold" },
        { "Variable": "$.classification.highRisk", "BooleanEquals": true }
      ],
      "Next": "HumanReview"
    },
    {
      "Variable": "$.decision.confidence",
      "NumericGreaterThanEqualsPath": "$.config.confidenceThreshold",
      "Next": "ApplyAdjustedClassification"
    }
  ],
  "Default": "KeepBaseClassification"
}

It has three exits. If the confidence reaches the threshold and the category is low-risk, the adjustment is applied. If it reaches the threshold but the category is high-risk, the ticket goes to human review (the orange branch in the diagram). If it does not reach the threshold, execution falls into Default and the rule-based classification is kept. Order matters: Choice evaluates its rules in order and takes the first match, so the human path comes first.

The decider’s own failure is a separate case. The intended design is to put a Catch on the Task state that invokes it and redirect to KeepBaseClassification. That Task and its Catch are not in the fragment above and I have not validated them; the mechanism is described in the AWS Step Functions error-handling documentation.

Seen as a state machine, each transition of the adjustment is an explicit condition: the model supplies the score and your code picks the branch.

stateDiagram-v2
    [*] --> InvokeDecider
    InvokeDecider --> AdjustClassification: decision + score
    InvokeDecider --> KeepBaseClassification: Catch (decider failure)
    AdjustClassification --> HumanReview: confidence ≥ threshold and high risk
    AdjustClassification --> ApplyAdjustedClassification: confidence ≥ threshold and low risk
    AdjustClassification --> KeepBaseClassification: below threshold (Default)
    ApplyAdjustedClassification --> [*]
    KeepBaseClassification --> [*]
    HumanReview --> [*]

And where does the decider run? Strands Decider is open source and can run on your own compute; its deployment guide documents self-hosting the server and an Amazon SageMaker endpoint. The state machine invokes it from the InvokeDecider Task state. I am not including a reference implementation, nor do I assume Jev deploys the same way.

The worked example: L1 incident triage, step by step

#

The three-zone diagram already shows the full path of an L1 triage ticket in an ITSM (IT service management) system. Now let’s lay it out as a table: a ticket arrives and has to be classified, enriched with context, and routed to resolution, grouping, or escalation. At each step, notice which part reasons and which part does not.

Reading the table top to bottom, the split is clear: of eight steps, two actually reason (root cause analysis and RCA synthesis), one is the gray zone of the decision model, and the rest is pure structured data.

Look at the last row. The model can reason all it wants, but the decision that wakes a person at 3 in the morning is made by a rule. Reasoning gives escalation context; it never triggers it.

AWS Step Functions orchestrates the flow, but its trace is not an audit log: the history of a Standard Workflow can only be queried by API for 90 days, and Express Workflows do not record it. To answer “why did this happen?” months later, use a Standard Workflow and enable logging to Amazon CloudWatch Logs. In addition, persist each decision’s input data, the rule and its version, the threshold applied, the outcome, and a correlation identifier to a durable store (for example Amazon DynamoDB or Amazon S3), with retention that meets your audit requirements.

Limitations, costs, and failure modes

#

Before committing to this split, it is worth measuring where the pattern wins and where it loses.

What it costs

#

Deterministic does not mean correct: a badly written rule gets it wrong the same way every time. These are the real costs for the team that adopts it:

• The rule table is one more system to maintain. Keyword rules break with new phrasings, with other languages, or with services not yet in the catalog, and someone has to review and version them.
• Calibrating the threshold requires labeled data. You need historical tickets classified by people to measure how many decisions it accepts in error and how many it rejects in error (false acceptances and false rejections). How many tickets you need and who labels them depends on your catalog and your error tolerance; I can’t give a number that holds in general.
• The self-hosted decider is one more component. It has to be deployed, versioned, monitored, and sized for the compute it runs on.
• Three zones and two retrieval stores add operational complexity compared with a single agent, and also more points to test independently.

If your volume is low and the decisions are of little consequence, that cost may not be worth it.

What it saves

#

Inference cost. Let’s put numbers to it. An L1 triage that receives 10,000 tickets a day, and two designs: the “agent for everything,” which reasons across all eight steps inside an agentic loop, and the zoned design, which calls the LLM in two steps (root cause and RCA synthesis) and the decision model in the third.

On top of the zoned design’s 20,000 calls you have to add 10,000 to the decider, which does not bill per token: Strands Decider is open source and its cost is that of the machine where you deploy it. Under these assumptions the zoned design makes twelve times fewer LLM calls and costs about eight times less in inference. Even so, do not take it as a fixed figure. Complex tickets lengthen the agent’s loop and its context grows on each turn, which widens the gap; prompt caching of the repeated system prompt or a smaller model on the easy steps narrows it. Run the numbers with your volume and your prompts, but the shape does not change: the zoned design pays for inference where there is unstructured evidence, and nowhere else.

The cheapest alternative is a single call per ticket that classifies everything. It costs less, but the classification remains probabilistic. When enabled for supported bedrock-runtime calls, Amazon Bedrock model invocation logging can store the input, output, and invocation metadata. That is a trace, not a decision you can reproduce. To get closer to the latter you would also have to store the model version, the prompt, and the output schema, and even then the model is not bound to return the same thing. The pattern’s argument is not only about price.

Latency. A decision model like Strands Decider resolves in around 200 ms: a single forward pass, local, with no round trip to a heavy inference endpoint. A multi-turn agentic loop, in contrast, chains LLM calls, each waiting on the previous one. In the scenario above the agent makes 24 sequential calls per ticket; the zoned design makes 2 plus the decider’s, not counting the tools. I have not measured per-call latency, so I am not giving an absolute figure. Whether that difference in calls turns into queuing depends on the volume and the concurrency you allow.

Failure modes

#

Three failure modes, each with its mitigation.

The decision model proposes a wrong adjustment. The confidence threshold reduces the risk, it does not remove it: if the score does not reach it, the adjustment is not applied and the base rule is kept. But a model can be confident and wrong, so an adjustment that reaches the threshold can still be mistaken. That is why the higher-risk categories should go through human review even when confidence is high.

RAG returns an irrelevant precedent. Nothing guarantees that the decider lowers the confidence of that option. Check it with irrelevant-retrieval test cases and add retrieval-quality checks (for example, a minimum similarity distance) before the precedent reaches the decider.

The decider or Amazon Bedrock is unavailable. Without the decider you lose the precedent-based adjustment; without Amazon Bedrock, the root cause context. The intended design in both cases is to degrade to deterministic classification rather than fall over, with a Catch on each Task that calls one of those dependencies: triage continues and escalations keep going out, albeit with less context for whoever receives them.

When more autonomy does make sense

#

None of this is anti-agent dogma. There are scenarios where raising autonomy does make sense: exploratory tasks, low-consequence ones, or ones where the action space is too open for a rule. An example from the same domain would be the L2 investigation of an incident with no precedent, where an agent can explore logs and metrics, while escalation remains a rule. What matters is that raising autonomy be a conscious, measured decision, and not the default you fall into because everyone was building agents.

Wrapping up

#

The boundary is drawn by the nature of the data. Structured data, a closed catalog, or a numeric threshold: a rule. Free text, semantic correlation, or ambiguous evidence: reasoning. The question is not “can I put an agent here?” but “what am I touching in this step?”.

Escalation, and any decision with consequences, is deterministic and auditable: AI informs it, it does not make it. If you cannot record why it happened and reproduce it, a model should not have decided it. That said, deterministic does not mean correct: the pattern brings its own costs, such as the rules you have to maintain, the labeled data to calibrate the threshold, and one more component to operate.

If you are designing a system like this, walk the steps one by one and ask yourself what data each one touches. Where the data is free text, use the model. Where it is a catalog, a threshold, or an identifier, use a rule.

References

#

• Act When Certain, Ask When Not: Building Safer AI Agents with Local Decision Models
• strands-labs/strands-decider — GitHub
• Decision Time: Jev vs. a Fine-Tuned Classifier — Anaconda
• Search and Multi-Teacher Distillation with Jev — arXiv:2610.09188
• Amazon Bedrock pricing
• AWS Step Functions: Standard vs. Express Workflows
• AWS Step Functions: Choice state
• AWS Step Functions: Error handling
• Amazon Bedrock: Monitor model invocation using CloudWatch Logs and Amazon S3