LLM Integration Patterns for Production-Grade Apps

LLMs are easy to demo and notoriously hard to productionize. The gap isn’t model quality—it’s integration design: how you retrieve knowledge, constrain behavior, manage latency and cost, and build feedback loops that don’t rot.

Below are integration patterns we use in AI consulting when moving from “prompt in a notebook” to “LLM inside a real product.” Each pattern includes what it’s good for, where it breaks, and what to instrument.

Pattern 1: Prompted workflow (LLM as a stateless function)

When to use: copywriting, summarization, classification, extraction, and “assistive” features where correctness is probabilistic and human review exists.

Architecture: Your app calls the LLM with a structured prompt template and returns output. No external context beyond the user input.

Production tips:

• Treat prompts as versioned artifacts (store in Git; tie to feature flags).
• Use structured outputs (JSON schema / tool calls) instead of free-form text.
• Add deterministic post-processing and validation (schema validation, regex checks, allowlists).

Failure modes: prompt drift as product changes; hidden coupling between prompt and UI; silent format breakage.

Instrument: output validity rate, token usage, latency distribution, user correction rate.

Pattern 2: Retrieval-Augmented Generation (RAG)

When to use: question answering over proprietary docs, customer support, internal copilots, policy lookup—any time you need grounded answers.

Architecture: User query → embed → retrieve top-k chunks (vector DB + optionally keyword search) → LLM answers using retrieved context.

What works in practice:

• Hybrid retrieval (BM25 + vectors) beats “vectors-only” for many enterprise corpora.
• Chunking is a product decision, not a backend detail. Chunk by semantics (headings, sections) and preserve source metadata.
• Citations are non-negotiable for trust: return links/ids and show them in UI.

Failure modes: irrelevant retrieval, missing critical context, hallucinated synthesis, stale embeddings.

Instrument: retrieval hit rate (did we fetch the doc that humans expect?), groundedness checks, citation click-through, answer deflection rate (did the bot reduce tickets?).

Pattern 3: Tool calling (LLM as a router to deterministic functions)

When to use: anything involving real-world state—pricing, inventory, account data, scheduling, analytics queries, on-chain reads.

Architecture: LLM decides whether to call tools (functions/APIs) with structured arguments; your system executes tools; LLM produces a final response.

Why it’s a production favorite: it moves correctness to code. The model’s job becomes “decide which function to call,” not “invent the answer.”

Practical guardrails:

• Keep tool surface area small; prefer fewer, well-designed tools.
• Validate tool args server-side (types, ranges, access control).
• Separate “read tools” (safe) from “write tools” (risky) and require explicit user confirmation for writes.

Example: A fintech app: the LLM calls get_transactions(user_id, date_range) and categorize_transaction(description); the final UI shows computed totals with provenance.

Pattern 4: Structured generation (LLM output as data, not prose)

When to use: extraction (contracts, invoices), form filling, lead qualification, generating configs, or any output that feeds downstream systems.

Architecture: Constrain outputs to JSON schema or a DSL; reject/repair malformed outputs.

Key techniques:

• Use a schema-first approach: design your target structure, then prompt to fill it.
• Add an “output repair” pass only when needed (and cap retries).
• Prefer smaller/faster models for extraction; save premium models for reasoning-heavy tasks.

Failure modes: partial fields, wrong enums, subtle numeric errors.

Instrument: schema pass rate, field-level accuracy via labeled samples, retry rate.

Pattern 5: Agentic orchestration (multi-step planning + tools)

When to use: open-ended tasks that require decomposition (research briefs, multi-step troubleshooting, “do X across systems”).

Opinionated take: agents are powerful, but most teams adopt them too early. If you can express the workflow as a finite state machine, do that first.

Production approach:

• Use bounded agents: hard limits on steps, tools, budget, and time.
• Prefer plan-then-execute with checkpoints over free-form looping.
• Persist state and intermediate artifacts so runs are debuggable.

Failure modes: infinite loops, tool thrashing, runaway costs, nondeterministic behavior.

Instrument: step count distribution, tool call frequency, abandonment rate, cost per successful task.

Pattern 6: Human-in-the-loop (HITL) for risk and quality

When to use: legal, medical, finance, customer communications at scale, or any feature where a mistake is expensive.

Architecture: LLM drafts → reviewer approves/edits → system sends/publishes. Feedback is captured for evals/fine-tuning.

Make it efficient:

• Provide diffs, highlight risky spans, and show sources.
• Build “approval queues” and role-based permissions.

Failure modes: reviewers rubber-stamp; feedback not captured; throughput bottlenecks.

Instrument: review time, edit distance, escalation rate, incident rate.

Pattern 7: Model routing and fallback

When to use: you need predictable latency/cost, or you serve multiple task types.

Architecture: classify task → route to best model (cheap for extraction, premium for complex reasoning). Add fallbacks for outages/timeouts.

Practical routing rules:

• Route by required context length, need for tools, risk level, and expected ROI.
• Implement timeouts and “degraded mode” responses (e.g., retrieval-only answers without synthesis).

Failure modes: inconsistent outputs across models; routing bugs; hidden cost spikes.

Instrument: per-route success rate, p95 latency, cost per endpoint, fallback activation rate.

Pattern 8: Observability + evaluations as first-class infrastructure

If you can’t measure it, it will regress.

What to log (with privacy discipline):

• Prompt version, model id, temperature, tool calls, retrieved doc ids
• Token counts, latency, retries, errors
• User feedback (thumbs up/down), corrections, and downstream outcomes

Evals to run continuously:

• Regression suites on golden prompts
• RAG evals (retrieval relevance + groundedness)
• Safety/red-team tests (prompt injection, data exfiltration)

Non-negotiable security basics:

• Prompt injection is an app-layer vulnerability: isolate tools, least privilege, and never let retrieved text directly instruct tool calls.
• Protect PII: redact before logging; encrypt at rest; define retention.

Conclusion: choose the simplest pattern that meets the risk

Most production apps succeed with a small set of primitives: RAG for grounding, tool calling for correctness, structured outputs for reliability, and strong observability/evals to prevent drift. Agents and complex orchestration can be valuable, but they’re rarely the first milestone.

If you’re designing an LLM feature roadmap, start by answering three questions: (1) What must be correct vs helpful? (2) What can be deterministic via tools/data? (3) How will we measure success weekly? Nail those, and the model choice becomes an implementation detail—not a business risk.