Chapter Objective: Understand the fundamental architecture and core mechanisms of an LLM application—not an Agent, but a conventional LLM-powered application. This chapter covers the essential building blocks: models, prompts, context, tools, memory, state, and runtime. It also introduces the engineering foundations required for production LLM applications, including Chat Completion, Streaming, Structured Output, Tool Calling, error handling, retries, timeouts, model routing, and cost management.
2.1 The Core Components of an LLM Application
Before discussing Agents, we first need to understand what a conventional LLM Application actually looks like.
An Agent is not an entirely new species of software. It is a more sophisticated form of an LLM application. Understanding the foundation of LLM applications is therefore essential for understanding Agent architecture later.
A typical LLM application consists of seven fundamental components:
- Model
- Prompt
- Context
- Tool
- Memory
- State
- Runtime
2.1.1 Model
The Model is the core computational component. At a conceptual level, it transforms an input sequence of tokens into an output sequence of tokens.
In production systems, however, application developers rarely interact directly with the neural network. Instead, the model is exposed as a callable API endpoint.
Typical inputs include:
Input:
messages + parameters
(temperature, max_tokens, tools, etc.)
Output:
generated content
(text and/or tool-call requests)
Common model providers include Anthropic Claude, OpenAI GPT, Google Gemini, and self-hosted open-source models such as Llama, Qwen, and DeepSeek.
A critical architectural insight is that an LLM should generally be treated as a stateless computation.
The model does not inherently remember the previous API call, maintain application state, or create persistent side effects. Each invocation is based only on the input provided for that invocation.
Given the same input and sampling configuration, the model produces an output drawn from the same underlying probability distribution; therefore, the result is not necessarily deterministic.
This statelessness is one of the most important starting points for LLM application architecture.
Because the model itself does not maintain application-level memory, concepts such as:
- conversation history,
- memory,
- sessions,
- task state,
- checkpoints,
must be implemented by the application layer.
In other words:
The model generates intelligence; the application provides continuity.
2.1.2 Prompt
A Prompt provides instructions and task context to the model.
It tells the model:
- who it is,
- what it should do,
- what constraints it must follow,
- what information it should use,
- and what output format is expected.
A typical conversational prompt contains several logical layers.
System Prompt
The System Prompt defines the model’s role, global rules, behavioral constraints, and other high-level instructions.
It is usually relatively stable compared with user input.
User Message
The User Message contains the user’s actual request or task.
Assistant Message
The Assistant Message represents previous model responses and is used as part of the conversation context.
Prompt Engineering was once treated as the primary skill for building LLM applications. Its importance has evolved as models have become more capable.
The key challenge is increasingly not simply:
“How do I write a clever prompt?”
but:
“How do I construct the right context for the model?”
This is where Context Engineering becomes important.
A useful distinction is:
The prompt defines what the model is being asked to do; context determines what the model knows when doing it.
Many LLM applications perform poorly not because their prompts are insufficiently sophisticated, but because the context provided to the model is incomplete, inaccurate, or poorly organized.
2.1.3 Context
Context is everything the model can see when generating a response.
Depending on the application, this may include:
- system instructions,
- user messages,
- conversation history,
- previous assistant responses,
- tool results,
- retrieved documents,
- file contents,
- structured application state.
Context can therefore be viewed as the model’s working memory.
The quality of the context has a direct impact on the quality of the model’s output.
However, context has two fundamental constraints:
- Finite capacity — models have a limited context window.
- Increasing cost — more input tokens generally mean higher inference cost.
As a result, context management becomes one of the most important engineering problems in LLM applications.
A production system must continuously answer questions such as:
- What information should enter the context?
- What information should be removed?
- What information should be summarized?
- What information should be retrieved dynamically?
- How should context size and cost be controlled?
These questions become even more important for long-running Agents.
2.1.4 Tool
A Tool is an external function that the model can request the application to execute.
Without tools, an LLM primarily generates information.
With tools, the model can interact with external systems and perform actions.
Examples include:
- searching files,
- querying databases,
- calling APIs,
- executing commands,
- retrieving weather information,
- sending messages,
- creating or modifying resources.
Technically, a tool is typically exposed through a function definition with a JSON Schema.
The model receives the tool definition and determines whether and how the tool should be used.
Tools therefore form the bridge between the model and the external world:
Without tools, the model can primarily talk. With tools, the model can act.
This capability is fundamental to Agent architecture.
2.1.5 Memory
Memory refers to information that an application persists across sessions.
This distinction is important because Context and Memory are not the same thing.
Conversation history and tool results that are relevant to the current interaction belong to the current context, or working memory.
Memory refers specifically to information that can be persisted and retrieved in future sessions.
Examples include:
- user preferences,
- project knowledge,
- long-term instructions,
- historical decisions,
- frequently used information.
Memory is usually implemented through application-level storage, often combined with retrieval mechanisms such as database queries or vector search.
The LLM itself does not automatically remember previous conversations.
What appears to be “memory” is usually implemented as:
Persist information
↓
Retrieve relevant information
↓
Inject it into context
↓
Generate the next response
This leads to an important architectural principle:
LLM memory is externalized memory.
Its reliability therefore depends on application-layer design, including storage quality, retrieval accuracy, relevance ranking, and context construction.
2.1.6 State
State represents the current runtime condition of an application.
Examples include:
- current execution step,
- session ID,
- task queue,
- workflow progress,
- checkpoints,
- intermediate results.
State is conceptually different from Memory.
A useful distinction is:
State represents where the application is. Memory represents what the application knows.
In practice, however, the two often overlap.
For example, information such as:
Task progress: Step 3 of 5
is runtime state while the task is executing. But if the application needs to recover after a failure or restart, that state may also need to be persisted.
2.1.7 Runtime
The Runtime is the orchestration layer that connects all the components together.
It is responsible for tasks such as:
- constructing requests,
- calling model APIs,
- executing tools,
- managing state,
- handling errors,
- controlling execution flow,
- recording usage and telemetry.
A useful analogy is:
The first six components are the parts; the Runtime is the assembly line that turns those parts into a working machine.
In Agent systems, this Runtime is closely related to what we call the Harness.
A Harness can be understood as the Agent’s Runtime, but with additional responsibilities such as:
- permission management,
- session management,
- sandboxing,
- observability,
- execution control,
- evaluation,
- delivery.
This distinction becomes increasingly important as we move from simple LLM applications toward production-grade Agents.
2.2 Chat Completion Architecture
Chat Completion is the most fundamental LLM interaction pattern:
Provide a sequence of messages and receive the next model response.
A simplified example using the Anthropic Messages API looks like this:
const response = await anthropic.messages.create({
model: "claude-sonnet-5",
max_tokens: 1024,
system: "You are a helpful assistant.",
messages: [
{ role: "user", content: "Hello. Please introduce yourself." },
{ role: "assistant", content: "Hello, I am an AI assistant." },
{ role: "user", content: "What can you help me with?" }
]
});
The exact model identifier should always be replaced with a currently supported model in the target environment.
Although the API appears simple, several architectural concepts are embedded in this structure.
2.2.1 Messages Are the Core Input
The model does not automatically see an application’s previous interactions.
The application must construct the message sequence that represents the conversation or task context.
The model then generates the next assistant response based on that input.
2.2.2 System Instructions Provide Global Guidance
The system parameter provides relatively stable, high-level instructions.
It typically defines behavioral rules, role, constraints, or application-level policies.
2.2.3 max_tokens Controls Output Length
max_tokens limits the maximum amount of output the model can generate.
It therefore affects both response behavior and potential inference cost.
Message Roles
Message roles provide a structured representation of the conversation.
For the Anthropic Messages API, the messages array primarily contains:
userassistant
The system instruction is provided separately as a top-level parameter.
The assistant role represents model-generated content, which may include both natural-language responses and tool-call requests.
When tools are introduced, additional content blocks appear.
For example:
tool_useis carried within anassistantmessage.tool_resultis carried within ausermessage.
These are content blocks rather than independent message roles.
The relationship between a tool request and its result is particularly important.
A tool_use block contains an identifier such as:
tool_use.id
The corresponding tool_result must reference that identifier using:
tool_use_id
This creates a strict request/result relationship.
Conceptually:
assistant
│
└── tool_use(id = X)
│
▼
Tool Execution
│
▼
user
└── tool_result(tool_use_id = X)
If this relationship is broken—for example, if a tool result is missing—the model may no longer have a coherent representation of the interaction.
This type of consistency issue becomes especially important in Agent runtimes. In later source-code analysis, mechanisms such as yieldMissingToolResultBlocks exist specifically to address missing tool-result blocks.
The Limitation of Chat Completion
Chat Completion is fundamentally request-response oriented.
The model generates a response, and the interaction ends.
It does not automatically continue working on the task.
This is the key limitation that Agent architectures attempt to overcome.
Agents combine:
- Tool Calling,
- execution state,
- iterative model invocation,
- and control loops
to turn a single model response into a longer-running execution process.
This leads naturally to the Agent Loop, which we will examine in the next chapter.
2.3 Streaming
Streaming is one of the most important mechanisms for improving the user experience of LLM applications.
Without streaming, an application typically waits until the model has generated the entire response before displaying anything.
For a long response, this can leave the user staring at a loading indicator for several seconds—or even minutes.
With streaming, generated content is delivered incrementally as it becomes available.
The experience becomes similar to watching someone type the response in real time.
A simplified example:
const stream = await anthropic.messages.stream({
model: "claude-sonnet-5",
max_tokens: 1024,
messages: [{ role: "user", content: "Write a poem." }]
});
stream.on("text", (text) => {
process.stdout.write(text);
});
However, production streaming is considerably more complicated than simply displaying text incrementally.
2.3.1 State Management
During streaming, the application must maintain the accumulated response.
It may need the complete response after streaming finishes for tasks such as:
- persisting the response,
- generating a summary,
- updating conversation history,
- triggering downstream processing,
- calculating usage metrics.
Therefore:
Streaming delivery and response persistence are two separate responsibilities.
An application cannot simply send tokens to the client and discard them.
2.3.2 Tool Calls and Streaming
Tool calling makes streaming significantly more complicated.
When a model generates text, the application can usually render the partial text immediately.
When the model generates a tool call, however, it may produce structured JSON incrementally.
For example:
{
"city": "Beijing",
"unit": "celsius"
}
The JSON itself may arrive in multiple fragments.
The application therefore needs to accumulate the partial JSON before parsing it.
A simplified implementation might look like:
let toolCallJson = "";
for await (const event of stream) {
if (event.type === "input_json_delta") {
toolCallJson += event.partial_json;
}
if (event.type === "content_block_stop") {
const toolCall = JSON.parse(toolCallJson);
}
}
This example assumes a single tool call.
Production implementations need to account for multiple tool calls and independently accumulate partial input based on the corresponding content-block index.
The result is effectively a stream-processing state machine that distinguishes between different types of streamed content.
2.3.3 Error Handling
Streaming introduces another challenge: partial responses.
A network connection may fail after the model has generated only part of the response.
The application then has to decide whether to:
- retry the entire request,
- preserve the partial response,
- inform the user that generation was interrupted,
- or attempt recovery.
A practical production pattern is:
Stream text to the user, but treat tool calls as structured, stateful events.
The application collects the complete tool-call arguments, executes the tool, and then uses the result as input for the next model invocation.
This creates the foundation for the iterative execution model used by Agents.
2.4 Structured Output
Structured Output allows an LLM to produce machine-readable data that conforms to a defined structure, typically represented by JSON Schema.
There are three common implementation approaches:
- Prompt-based constraints
- JSON Mode
- Constrained Decoding
2.4.1 Prompt-Based Constraints
The simplest approach is to instruct the model:
Return only valid JSON.
This is easy to implement but relatively unreliable.
The model may:
- add explanatory text before or after the JSON,
- produce invalid JSON,
- omit quotation marks,
- use incorrect field names,
- return incorrect data types.
For experimentation this may be sufficient.
For production systems, it is generally not enough.
2.4.2 JSON Mode
Some model APIs provide a JSON-oriented generation mode.
This is more reliable than relying entirely on natural-language instructions because the API provides additional structural constraints.
However, producing valid JSON does not necessarily mean producing the correct schema.
For example:
{
"temperature": "twenty-five"
}
may be valid JSON while still violating the expected schema:
{
"temperature": 25
}
Therefore JSON validity and schema validity are different concerns.
2.4.3 Constrained Decoding
Constrained Decoding applies structural constraints during generation.
Instead of asking the model to “try” to produce valid JSON, the decoding process restricts the possible outputs so that they conform to a defined schema.
This provides a much stronger guarantee.
The trade-off is greater implementation complexity and, depending on the implementation, potential performance overhead.
Why Prompt-Based JSON Generation Is Not Enough
Consider a prompt that says:
“Return only JSON.”
The model may usually follow the instruction—but “usually” is not a sufficient reliability guarantee for a production API.
Failure cases include:
- Extra explanatory text around the JSON.
- Invalid JSON syntax.
- Valid JSON with incorrect field names.
- Incorrect data types.
- Missing required fields.
A common fallback pattern is:
Generate → Parse → Validate → Return or Retry
For example:
async function getStructured<T>(
prompt: string,
schema: ZodSchema<T>
): Promise<T> {
for (let attempt = 0; attempt < 3; attempt++) {
const text = await callModel(prompt);
try {
const json = extractJson(text);
return schema.parse(json);
} catch (err) {
prompt += `
The previous output did not conform to the required format.
Please return valid JSON that matches the schema exactly.
Validation error:
${err.message}
`;
}
}
throw new Error("Structured output generation failed");
}
This pattern is extremely common in LLM applications:
Generate
↓
Parse
↓
Validate
↓
Success ──────────────→ Return
│
└── Failure
↓
Provide error feedback
↓
Retry
The underlying idea is important:
Use machine validation to detect model errors, then feed structured error information back into the generation process.
This is an early form of the broader Self-Correction pattern discussed later in this book.
For production systems, native structured-output capabilities or constrained decoding should generally be preferred where supported.
A particularly useful design technique is to represent structured output through a tool call.
Because tool calls already have a structured name and JSON input schema, many frameworks can implement structured output by forcing the model to invoke a dedicated “output tool.”
This allows an existing Tool Calling mechanism to be reused for structured generation.
2.5 Function Calling / Tool Calling
Tool Calling and Function Calling generally refer to the same fundamental capability: allowing a model to request the execution of an external function.
The terminology varies by model provider.
Tool Calling is one of the fundamental building blocks of Agent architecture, so its complete lifecycle is worth understanding.
The lifecycle consists of five steps:
1. Define the Tool
↓
2. Model Decides to Call
↓
3. Execute the Tool
↓
4. Return the Tool Result
↓
5. Model Continues Reasoning
Step 1: Define the Tool
The developer declares the tools available to the model.
Each tool typically includes:
- a name,
- a description,
- an input schema.
For example:
const tools = [
{
name: "get_weather",
description: "Get the current weather for a specified city.",
input_schema: {
type: "object",
properties: {
city: {
type: "string",
description: "The city name, such as Beijing."
}
},
required: ["city"]
}
}
];
Step 2: Model Decision
During generation, the model determines whether a tool is required.
Instead of returning natural language, it may return a structured tool-call request:
{
"type": "tool_use",
"id": "toolu_01AbCdEf",
"name": "get_weather",
"input": {
"city": "Beijing"
}
}
The important point is that the model is requesting an action.
It is not necessarily executing the function itself.
Step 3: Tool Execution
The Runtime—or Harness in an Agent architecture—receives the request.
It resolves the tool name to an actual implementation and executes it.
function getWeather(city: string) {
return fetchWeatherApi(city);
}
const result = getWeather("Beijing");
<em>// { weather: "Clear", temp: 25 }</em>
The Runtime is responsible for execution.
This separation is architecturally important:
Model
│
│ tool request
▼
Runtime / Harness
│
│ actual execution
▼
External System
Step 4: Return the Tool Result
The Runtime then returns the execution result to the model as part of the next message sequence.
messages.push({
role: "user",
content: [
{
type: "tool_result",
tool_use_id: "toolu_01AbCdEf",
content: JSON.stringify({
weather: "Clear",
temp: 25
})
}
]
});
The tool_use_id establishes the relationship between the request and its result.
Step 5: Continue Reasoning
The model receives the tool result and uses it as additional context.
It can then:
- produce the final answer,
- call another tool,
- request additional information,
- or continue working on the task.
These five steps form one fundamental tool-call turn.
An Agent Loop is essentially this interaction repeated:
Model → Tool → Result → Model → Tool → Result → …
This is the basic execution unit of an Agent.
Parallel Tool Calling
Some models can request multiple independent tools in a single response.
For example:
Model
├── get_weather(Beijing)
├── get_weather(Shanghai)
└── get_weather(Shenzhen)
If these operations are independent, the Harness can execute them concurrently.
However, operations with dependencies or side effects may need to execute sequentially.
This distinction becomes important in sophisticated Agent runtimes.
Forced Tool Calling
Some APIs provide a tool_choice mechanism that allows the application to influence or force tool usage.
For example, the application may require the model to:
- call a specific tool,
- call any available tool,
- or avoid tools.
This capability is particularly useful for structured-output workflows where the application wants the model to return data through a predefined tool schema.
2.6 Tool Schema
A Tool Schema is the interface contract exposed by a tool to the model.
Its quality has a direct impact on tool-selection accuracy and tool-call correctness.
Tool Schema design is therefore one of the most underestimated aspects of Tool Systems.
A good Tool Schema should follow several principles.
1. Use Precise, Self-Describing Names
Prefer:
get_weather
over:
tool1
Consistent naming conventions, such as snake_case, also improve readability.
2. Write Useful Descriptions
The description should explain:
- what the tool does,
- when it should be used,
- what limitations apply,
- what the tool returns.
The model uses this information when deciding whether to invoke the tool.
3. Describe Every Parameter
Each parameter should explain:
- its meaning,
- expected format,
- valid values,
- units,
- timezone assumptions,
- or other constraints where relevant.
4. Define required Correctly
Required parameters should be explicitly identified.
This reduces missing or invalid arguments.
5. Avoid Over-Design
More schema does not necessarily mean better schema.
Too many parameters or unnecessarily verbose descriptions consume context tokens and can make tool selection harder.
A Poor Tool Definition
{
name: "f",
description: "do something",
input_schema: {
type: "object"
}
}
The model has almost no useful information with which to decide how or when to use this tool.
A better design would be:
{
name: "search_files",
description:
"Search project files for a specified text or regular expression. " +
"Returns matching file paths and line numbers. " +
"Use it to locate function definitions, variable references, " +
"or error messages.",
input_schema: {
type: "object",
properties: {
pattern: {
type: "string",
description: "The text or regular expression to search for."
},
directory: {
type: "string",
description:
"Directory to search. Defaults to the project root."
},
file_glob: {
type: "string",
description:
"Optional file-name filter, such as '*.ts'."
}
},
required: ["pattern"]
}
}
The deeper principle is:
Tool Schema design is essentially API contract design for an LLM consumer.
Good API documentation helps human developers use an API correctly.
Good Tool Schema documentation helps models use tools correctly.
There is another important distinction:
Describe the tool from the model’s perspective, not the implementation’s perspective.
The implementation developer may know that search_files internally uses ripgrep, recursively traverses directories, and maintains an index.
The model generally does not need those implementation details.
It needs to know:
- when to use the tool,
- what parameters to provide,
- what those parameters mean,
- and what result to expect.
2.7 Tool Result
A Tool Result is the information returned to the model after a tool has executed.
Its design is just as important as the Tool Schema because the model must use the result to determine what to do next.
Several principles are important.
1. Return Structured Data
Structured JSON is generally easier for models to interpret reliably than large blocks of unstructured text.
2. Control Result Size
Tool results become part of the model’s context.
If a tool returns 100,000 tokens of content, it can quickly consume the available context window.
Therefore, tool results often need to be:
- truncated,
- filtered,
- summarized,
- paginated,
- or selectively retrieved.
This becomes particularly important for Context Compaction and long-running Agents.
3. Distinguish Success from Failure
A successful result should contain useful data.
A failure should provide enough structured information for the model to understand:
- what failed,
- why it failed,
- whether the problem is recoverable,
- and potentially what should be tried next.
4. Return Sufficient Context
A tool should provide enough information for the model to make its next decision.
For example, a file-search tool returning only:
3 matches found
is often insufficient.
The model may need:
- file paths,
- line numbers,
- matching content,
- surrounding context.
Otherwise, it may need to invoke another tool simply to understand the result.
This leads to a critical principle:
Tool-result quality directly influences Agent decision quality.
Poor tool results can cause:
- incorrect decisions,
- unnecessary tool calls,
- repeated searches,
- wasted tokens,
- or execution loops.
One particularly undesirable behavior is tool-call thrashing: the model repeatedly invokes tools but receives results that do not provide enough information to make meaningful progress.
Good Tool Result design helps prevent this behavior.
2.8 Error Handling
Error handling in LLM applications is more complex than in traditional applications because failures can occur at multiple layers.
Common categories include:
- Network errors — timeouts, connection failures, gateway errors.
- API errors — rate limiting, overload, authentication failures, invalid parameters.
- Model errors — invalid output structure, incorrect tool arguments, hallucinated information.
- Tool errors — missing files, failed commands, unavailable external services.
The key strategy is:
Classify errors before deciding how to handle them.
For example:
async function callModelWithRetry(request) {
try {
return await callModel(request);
} catch (err) {
if (isRateLimitError(err)) {
await sleep(err.retryAfterMs ?? 2000);
return callModelWithRetry(request);
}
if (isOverloadError(err)) {
await sleep(exponentialBackoff());
return callModelWithRetry(request);
}
if (isAuthError(err)) {
throw new AuthError("Invalid API credentials");
}
throw err;
}
}
The most important distinction is between retryable and non-retryable failures.
| Error Category | Retryable? | Typical Handling |
|---|---|---|
| Network timeout/interruption | Yes | Retry with backoff |
| Rate limit (429) | Yes | Respect Retry-After and retry |
| Service overload | Yes | Exponential backoff |
| Authentication failure (401) | No | Fail immediately |
| Invalid request (400) | Usually no | Correct the request |
| Context too long | Special case | Compress or reduce context |
| Output limit reached | Special case | Continue, truncate, or degrade gracefully |
This classification becomes a fundamental design principle for robust Agent Loops.
2.9 Retry
Retry is one of the basic mechanisms for improving LLM application reliability.
However:
Retry does not mean blindly sending the same request again.
A production retry strategy should consider several factors.
1. Exponential Backoff
Increase the delay between attempts:
1s → 2s → 4s → 8s
This prevents clients from continuously hitting an overloaded service.
2. Jitter
Add a random component to the delay.
Without jitter, many clients may retry simultaneously and create a thundering herd effect.
3. Maximum Retry Attempts
Set an explicit upper bound, typically a small number such as 3–5 depending on the workload.
4. Idempotency
Retries can be dangerous for operations with side effects.
For example:
send_message()
create_order()
write_file()
delete_resource()
If the first request succeeds but the response is lost, retrying may execute the operation twice.
Therefore, side-effecting tools require appropriate idempotency mechanisms.
A simplified implementation:
async function withRetry<T>(
fn: () => Promise<T>,
maxRetries = 3
): Promise<T> {
for (let i = 0; i < maxRetries; i++) {
try {
return await fn();
} catch (err) {
if (i === maxRetries - 1 || !isRetryable(err)) {
throw err;
}
const delay =
1000 * 2 ** i +
Math.random() * 500;
await sleep(delay);
}
}
throw new Error("Unreachable");
}
Circuit Breakers
A more advanced mechanism is the Circuit Breaker.
When failures exceed a defined threshold, the circuit opens temporarily and subsequent requests fail fast instead of continuing to retry.
This protects the system from cascading failures.
Conceptually:
Healthy
│
│ repeated failures
▼
Open Circuit
│
│ cooldown
▼
Half Open
│
├── success → Healthy
│
└── failure → Open
This pattern is particularly useful when an LLM application depends on unstable external services.
2.10 Timeout
Timeouts prevent LLM applications from waiting indefinitely.
Model inference can be slow, especially for long-context or reasoning-intensive workloads.
A robust timeout design should therefore be layered.
1. Connection Timeout
Maximum time allowed to establish the connection.
2. Time-to-First-Token Timeout
Maximum time allowed before the first streamed output arrives.
This is especially important for user-facing streaming applications.
3. Overall Request Timeout
Maximum allowed duration for the complete operation.
However, a single aggressive total timeout can be problematic for long-running tasks.
A better approach for streaming workloads is often:
Time-to-first-token timeout + idle timeout
The application can allow a long-running generation to continue as long as output continues to arrive.
For example:
let lastTokenTime = Date.now();
stream.on("text", () => {
lastTokenTime = Date.now();
});
const idleTimer = setInterval(() => {
if (Date.now() - lastTokenTime > IDLE_TIMEOUT) {
controller.abort();
}
}, 1000);
The fundamental trade-off is:
A timeout that is too short kills legitimate long-running work; a timeout that is too long wastes resources.
Layered timeouts provide a better balance.
2.11 Model Router
A Model Router selects an appropriate model for a given request.
Its existence is based on a simple reality:
There is no universally optimal model.
Different models offer different combinations of:
- reasoning capability,
- latency,
- context capacity,
- reliability,
- availability,
- and cost.
Common routing strategies include:
1. Task-Based Routing
Use a fast and inexpensive model for simple tasks and a more capable model for complex reasoning.
2. Cost-Based Routing
Choose models based on the application’s budget or cost constraints.
3. Availability-Based Routing
Automatically fail over to another model when the primary provider or model is unavailable.
4. Cascade Routing
Start with a smaller or less expensive model.
If the result is inadequate—for example, if validation fails or confidence is low—escalate the task to a more capable model.
A simple router might look like:
interface ModelConfig {
name: string;
costPerInputToken: number;
costPerOutputToken: number;
capability: "basic" | "advanced";
}
class ModelRouter {
route(task: Task): ModelConfig {
if (task.requiresAdvancedReasoning) {
return advancedModel;
}
if (task.isSimple) {
return cheapModel;
}
return defaultModel;
}
}
Cascade Routing is particularly attractive from a cost perspective.
The strategy is:
┌── Success ──→ Return
Simple Task ──→ Cheap Model
│
└── Failure
↓
Stronger Model
The assumption is that most simple requests do not require the most expensive model.
Only difficult cases are escalated.
This can significantly reduce average inference cost while preserving quality for complex tasks.
In an enterprise Agent Platform, Model Routing eventually becomes a central component for:
- cost control,
- reliability,
- model governance,
- workload optimization,
- and capability management.
2.12 Token and Cost Management
Token and cost management become unavoidable when moving an LLM application from a prototype to production.
There are three major areas to consider.
2.12.1 Token Accounting
The first requirement for cost control is measurement.
For every model request, the application should record information such as:
- input tokens,
- output tokens,
- model identifier,
- pricing information,
- cached input tokens where applicable.
A typical usage record may look like:
{
"usage": {
"input_tokens": 1500,
"output_tokens": 300,
"cache_read_input_tokens": 0,
"cache_creation_input_tokens": 0
}
}
Exact usage fields vary by provider and API version, so production systems should normalize provider-specific usage information into an internal accounting model.
2.12.2 Cost Estimation
At a simplified level:
Cost =
Input Tokens × Input Price
+
Output Tokens × Output Price
Pricing varies significantly between models and providers.
In many systems, output tokens are more expensive than input tokens, although exact pricing depends on the model and provider.
A particularly important cost factor for Agents is execution loops.
A single user request may result in:
Model Call 1
↓
Tool Call
↓
Model Call 2
↓
Tool Call
↓
Model Call 3
↓
...
An Agent that executes dozens of iterations can therefore consume dramatically more tokens than a simple request-response interaction.
This is why loop control is also cost control.
2.12.3 Cost Optimization
Common optimization techniques include:
1. Remove Redundant Context
Do not send irrelevant information to the model.
This is often one of the most effective ways to reduce cost.
2. Use Prompt Caching
Stable prefixes such as:
- system instructions,
- tool definitions,
- reusable context,
may benefit from provider-specific prompt caching mechanisms.
Caching can substantially reduce the effective cost of repeated input tokens where supported.
3. Use Model Tiering
Use less expensive models for simple workloads and stronger models only when necessary.
This is where Model Routing becomes valuable.
4. Compress Long Context
Summarize or compact long conversations and intermediate results before continuing execution.
This becomes increasingly important for long-running Agents.
5. Limit Output Length
Use appropriate max_tokens limits to prevent unnecessary generation.
Cost management is not simply about minimizing spending.
It is about making the system economically sustainable.
An Agent with unrestricted loops, excessive context, and no model-routing strategy can generate unexpectedly high API costs—particularly in long-running or autonomous workloads.
This is why cost should be treated as an architectural concern rather than an afterthought.
Chapter Summary
This chapter established the architectural foundation of an LLM application.
The key concepts are:
1. Seven Core Components
A production-oriented LLM application can be understood through seven fundamental components:
Model, Prompt, Context, Tool, Memory, State, and Runtime.
The most fundamental architectural insight is that the model itself is stateless.
Continuity, memory, state, and orchestration must therefore be implemented by the application.
2. Chat Completion Is Request-Response Oriented
A basic Chat Completion produces one response and stops.
Agents extend this model through:
Tool Calling + State + Iterative Execution
which transforms a single response into a continuous execution process.
3. Streaming Improves User Experience
Streaming reduces perceived latency, but introduces additional complexity around:
- state management,
- partial responses,
- tool-call parsing,
- network interruptions,
- and recovery.
4. Structured Output Enables Reliable Machine Consumption
Prompt-only JSON generation is probabilistic and therefore insufficient for many production scenarios.
Where supported, applications should prefer stronger schema enforcement mechanisms such as native structured outputs or constrained decoding.
5. Tool Calling Is the Basic Unit of Agent Execution
The five-step lifecycle is:
Define Tool → Model Decision → Tool Execution → Tool Result → Continue Reasoning
An Agent Loop is essentially this lifecycle repeated over time.
6. Production Engineering Matters
A production LLM application must also address:
- Error Handling
- Retry
- Timeout
- Model Routing
- Token Accounting
- Cost Management
These are not secondary implementation details.
They are part of the fundamental architecture required to operate LLM applications reliably at scale.
With this foundation in place, the next chapter focuses on one of the most important pieces of Agent architecture:
The Agent Loop.
It may only require a few dozen lines of code, but understanding those lines is the key to understanding the architectural boundary between an LLM Application, an Agent, and the Harness that makes the Agent reliable and operationally useful.
Related posts:
Chapter 1: Rethinking AI Agents
Building a Python-based AI Agent with LangGraph and OpenRouter: A Hands-On Guide
Clone a WordPress with ASP.NET and React Part 1: Initialize Project Structure with AI
AI Agent Architecture and Engineering Practice Guide
A Comprehensive Guide to AI Agents: Definition, Role, Examples, and Future Prospects
The Key Milestones in the History of Artificial Intelligence (2026.08)
Artificial Intelligence (AI) Learning Roadmap for Beginners in 2025
Unity in Practice 0007 – Very First Unity C# Code to Move and Jump a 2D Ball
For the full guide, see [AI Agent Architecture and Engineering Practice Guide – AI & Software Engineering](https://wcsee.com/ai-agent-architecture-and-engineering-practice-guide/).