RAG, agents, or ordinary software: choosing the right architecture for a workflow

Start from the workflow and its failure cost

Write the workflow as observable steps: trigger, inputs, decisions, systems consulted, actions, human approvals, outputs, and exceptions. Then mark which steps are fully specified and which require interpretation of unstructured information.

A calculation, permission check, routing rule, state transition, or compliance threshold should normally remain deterministic. Natural-language interpretation may help summarize documents, classify intent, draft material, or explain evidence. The cost of a wrong answer determines how much validation and human control the architecture needs.

Use ordinary software when the rule is knowable

Deterministic code is the strongest choice when inputs and outcomes can be specified, exact repeatability matters, or an incorrect result creates material harm. It is easier to test, monitor, reason about, and audit.

“Ordinary” does not mean unsophisticated. Good search, filters, structured forms, workflow engines, validation, and integrations often solve the real problem more reliably than a language model. AI can still support an unstructured edge without controlling the entire process.

• Calculations and eligibility rules
• Identity, permission, and tenant enforcement
• Schema and business-rule validation
• Known state machines and approval paths
• Exact data transformation and financial operations

Use search when people need sources, not synthesis

Keyword, metadata, and semantic search may be sufficient when the user can inspect the source directly. Search preserves the original material and avoids a generated answer that might omit nuance.

Improve content structure, permissions, metadata, indexing, and the results interface before assuming a chatbot is necessary. Retrieval quality remains important even if generation is added later.

Use RAG when answers must depend on controlled, changing knowledge

Retrieval-augmented generation is justified when a user needs a synthesized answer grounded in an approved body of content that changes more often than a model should be retrained. The system retrieves relevant sources and supplies them as context to generation.

RAG does not guarantee truth. Access controls, chunking, indexing, freshness, ranking, context assembly, citations, and evaluation determine whether the answer is trustworthy. Test retrieval and generation separately. When no adequate source is found, refusing or returning search results may be better than generating.

Use tool calls when the system must interact with external state

A model can select among tightly defined tools to read a CRM record, draft a ticket, query a service, or prepare an action. The application—not the model—should enforce authentication, authorization, input schemas, business rules, idempotency, and logging.

Separate preparation from execution. Let the system propose a consequential action, display what will change, and require human confirmation where risk warrants it. Grant the narrowest tool and data scope needed for the task.

Use agents only when planning across variable steps creates enough value

An agent chooses or repeats actions based on intermediate results. That flexibility can help with open-ended research, multi-system coordination, or tasks whose sequence cannot be fully known in advance. It also multiplies possible states, costs, latency, and failure paths.

Before using an agent, test whether a fixed workflow with one or two model decisions would solve the job. If autonomy remains justified, limit the goal, actions, iteration count, time, budget, and data scope. Make progress inspectable and define when the system stops or escalates.

Most production systems combine patterns

A dependable architecture might use deterministic permissions and workflow state, search for source discovery, RAG for grounded explanation, one model call to classify an exception, a tool to prepare a change, and a human to approve it. The question is not which label wins. It is which component should be variable and which must remain exact.

Keep each boundary testable. Evaluate the retrieval set, model output, tool selection, action validation, and end-to-end task separately. This turns “the AI failed” into a diagnosable engineering problem.

A compact decision sequence

Choose the simplest pattern that meets the workflow and risk. Complexity is justified only when it creates an observable advantage.

• Can explicit rules solve the step? Use deterministic software.
• Does the person need to find original material? Improve search.
• Must the answer synthesize controlled knowledge? Consider RAG.
• Must the system read or modify another system? Add narrow, validated tools.
• Must it choose a variable sequence of actions? Consider bounded agency.
• Would an error cause material harm? Add stronger validation, human approval, or remove autonomy.