LLM Apps Are Unpredictable by Design

Unlike a normal function, an LLM call is probabilistic, slow, expensive, and can be steered by its input. Productionizing it means adding the layers ordinary software doesn’t need: evaluation, security, caching, and observability. The prompt is not the product; the system around the prompt is.

Evals: The Unit Tests of LLM Apps

You cannot ship what you can’t measure. Evals are scored test suites that gate every prompt or model change:

Every prompt edit is a candidate release: run the evals, compare against the previous prompt, and gate on regression. This is the single highest-leverage practice in LLM engineering.

Prompt Injection and Jailbreaks

The input to an LLM is untrusted code. A user can write “ignore all previous instructions and …” directly in their message. Defenses:

• Separate and label instructions from data in the prompt.
• Guardrails that classify inputs and outputs for injection or policy violations.
• Least privilege — give the model no tool access it doesn’t need.
• Don’t trust model output with system privileges — validate and authorize any tool call.

Assume injection is possible and design so a successful injection cannot escalate.

Guardrail Layers

Guardrails sit before and after the model, as testable components:

1. Input layer — detect injection, PII, blocked topics; enforce rate limits.
2. Model layer — constrained decoding, allowed domain, RAG grounding.
3. Output layer — validate format, redact PII, flag policy violations.

Guardrails are cheap classifiers and rules, not more prompting — they run every request and are independently tested.

Caching and Cost Control

LLM tokens cost money per request. Repeat questions (FAQ, common queries, cached summaries) should never hit the model twice:

• Semantic caching — hash the embedding of the query; on a near match, return the stored completion.
• KV-cache reuse — shared system prompts avoid re-processing common prefixes.
• Batching and provider choice — larger batches and the cheapest sufficient model.

Set budgets per feature, and add a latency/cost budget check to the eval suite.

Observability and Tracing

LLM failures are non-exceptions — nothing throws. You need traces: prompt sent, context retrieved, tokens generated, latency, cost, and the final output, per request. Standard practice is OpenTelemetry spans with the prompt/response payload attached, plus a session-scoped trace ID so a user complaint can be replayed exactly. Without traces, debugging a bad answer is archaeology.

Versioning Prompts, Models, and Data

Prompts, model versions, and RAG data change independently. Production systems version all three together per release, so a deployed feature is the tuple (prompt v7, model gpt-x-v2, index 2026-03-01). Rollback restores the tuple. Untracked prompts are undeployable software.

Governance and Compliance

Model outputs may expose private data, produce copyrighted text, or make decisions that require human review. Practical posture:

• Log inputs/outputs with retention and access controls.
• Route PII-bearing prompts to approved providers/regions.
• Add human-in-the-loop for high-stakes decisions.
• Document the model, training-data provenance, and evaluation results.

Practice Trajectory

1. Write three golden eval cases for a support chatbot and score them.
2. Design an injection-resistant prompt plus a guardrail check.
3. Sketch a semantic cache and explain the near-match threshold.
4. List the spans a production trace must capture for replay.
5. Define the release tuple that a production LLM feature must version.