LLM integration patterns for production apps

Most teams don’t fail at LLMs because the model “isn’t smart enough.” They fail because they treat an LLM like a deterministic library call—then are surprised when latency spikes, costs balloon, or the model confidently invents facts. Production success is about architecture: choosing integration patterns that make uncertainty manageable.

Below are the patterns we most often recommend in AI consulting engagements—what they’re good for, where they break, and how to ship them with the operational discipline you’d apply to any critical system.

Pattern 1: Prompt-as-API (the baseline)

What it is: Your app sends user input + a prompt template to an LLM and returns the response.

Use it for: Low-risk text generation (drafting, tone rewrites, summarization of user-provided text). Things where hallucinations are tolerable or easily user-verified.

Production advice (opinionated):

Keep prompts versioned like code. Store them in your repo, review diffs, and attach a semantic version to each prompt.
Separate system instructions (policy, style, constraints) from task instructions (what to do now).
Add “response contracts”: require JSON output with a schema when downstream code depends on it.

Common failure mode: People shove business logic into prompts. Don’t. If it’s logic, enforce it in code.

Pattern 2: Structured extraction with constrained outputs

What it is: Use an LLM to transform unstructured text into structured data: entities, classifications, routing decisions, or normalized fields.

Use it for: Support ticket triage, contract clause extraction, lead qualification, compliance tagging.

How to make it robust:

Use a JSON schema (or typed models) and validate responses. If validation fails, retry with a “fix-to-schema” prompt.
Prefer “extract only from provided text” instructions and require evidence fields (e.g., source spans).
Add a confidence score and route low-confidence outputs to a human or fallback.

Example: A fintech app extracts “merchant name,” “recurring flag,” and “category” from transaction memos. The LLM returns {merchant, category, evidence}; your code rejects outputs missing evidence.

Pattern 3: Retrieval-Augmented Generation (RAG)

What it is: Instead of asking the LLM to “know” your domain, you retrieve relevant documents (from a vector DB or hybrid search) and provide them as context.

Use it for: Internal knowledge assistants, customer support bots, product documentation Q&A, legal/policy guidance—any scenario where answers must be grounded in company truth.

Key design choices:

Chunking strategy: Split documents in a way that preserves meaning (sections, headings), not arbitrary token counts.
Hybrid retrieval: Combine vector similarity with keyword/BM25 to improve precision on exact terms (SKUs, error codes).
Citations: Require the model to cite document IDs/URLs. If it can’t cite, it should say it doesn’t know.

Hard truth: Most “RAG doesn’t work” complaints are actually retrieval quality issues (bad chunking, stale docs, weak filtering). Treat retrieval as a first-class system with metrics.

Pattern 4: Tool/function calling (LLM as a router)

What it is: The LLM decides when to call backend tools (search, CRM lookup, calculator, ticket creation) and your system executes those calls.

Use it for: Workflows where the model shouldn’t invent facts: “What’s my order status?”, “Create a Jira ticket,” “Summarize last quarter revenue by region.”

Implementation guidelines:

Define a small, explicit toolset. More tools ≠ better; it increases confusion and attack surface.
Put authorization in the tool layer, not the prompt. The LLM should never be the security boundary.
Make tools idempotent where possible and include request IDs to prevent double-execution.

Example: A customer support assistant uses tools: get_order(order_id), list_refunds(user_id), create_return(order_id). The model can’t “guess” status—it must call.

Pattern 5: Agentic workflows (multi-step planning)

What it is: The model plans and executes multiple steps, potentially looping: gather info, call tools, reflect, produce final output.

Use it for: Complex tasks with open-ended steps: incident response assistants, research synthesis, multi-system operations (CRM + docs + analytics).

When not to use it: If you can express the workflow deterministically. Agents are expensive, slower, and harder to debug.

Safer agent design:

Use a state machine (explicit steps) rather than unconstrained “keep going.”
Enforce budgets: max tool calls, max tokens, max wall-clock time.
Log every tool call and intermediate reasoning artifacts (without storing sensitive chain-of-thought—store decisions and inputs/outputs).

Pattern 6: Human-in-the-loop (HITL) approvals

What it is: The LLM drafts, a human approves, and then actions are executed.

Use it for: High-stakes operations: sending customer emails, publishing content, updating database records, financial actions.

Why it’s underrated: HITL is not a cop-out; it’s a product pattern. It turns probabilistic output into a reliable workflow.

Make it efficient:

Provide a diff view (“before/after”).
Ask the LLM to generate a rationale and highlight uncertain points.
Capture feedback signals to improve prompts and routing.

Pattern 7: Guardrails and policy enforcement

What it is: A combination of input validation, content moderation, output constraints, and runtime checks.

Use it for: Any public-facing LLM feature.

Practical guardrails that actually work:

Prompt injection defense: isolate retrieved content, label it as untrusted, and prohibit it from altering instructions.
PII handling: redact sensitive fields before sending to third-party LLMs; use allowlists for what can leave your boundary.
Output checks: schema validation, profanity filters, “must cite sources” rules, and post-generation classifiers.

Opinionated take: Don’t rely on “the model will follow policy.” Assume it won’t, and build enforcement around it.

Pattern 8: Fallbacks, caching, and cost controls

What it is: Reliability patterns that make LLM features predictable.

Use it for: Everything in production.

Tactics:

Model routing: use cheaper models for low-risk tasks; escalate to stronger models only when needed.
Semantic caching: cache answers to repeated questions (especially for RAG with stable docs).
Graceful degradation: if the LLM fails, return a deterministic experience: search results, a form, or a human handoff.

Observability: treat prompts like a production system

You need visibility into:

Latency (p50/p95), token usage, cost per request
Retrieval metrics (hit rate, top-k quality, “no answer” rate)
Tool-call success rates and failure reasons
Quality metrics: user ratings, escalation rate, groundedness checks

Set up offline evals with a fixed test set before every major prompt/model change. If you don’t measure regressions, you’ll ship them.

Conclusion: pick patterns that bound uncertainty

LLMs are powerful precisely because they’re flexible—and that flexibility is what makes them risky in production. The winning strategy is to bound uncertainty with architecture: ground answers with RAG, enforce truth with tools, constrain outputs with schemas, protect users with guardrails, and keep humans in the loop when the stakes demand it.

If you’re building an LLM feature and your plan is “we’ll just prompt it better,” you’re not designing a system—you’re hoping. Production apps deserve better patterns than hope.