Why Most Agent Projects Fail Before the Model Becomes the Bottleneck
Agent projects fail from missing coordination, not weak models — nothing tracks state, retries, or handoffs. Five orchestration patterns, one shared loop.
Most agent projects fail before the model ever becomes the bottleneck.
The problem isn’t intelligence: it’s coordination. A single LLM call can draft a report, write code, answer a question. It cannot manage state across steps, recover from a failed tool call, route a complex task to the right model, or pause for a human decision without losing its place. That’s what orchestration does.
Agent orchestration is the layer that sits between your application and the model. It decides what runs, in what order, with what inputs, and what to do when something breaks. Without it, you have a chatbot. With it, you have a system that can execute multi-step tasks reliably at scale. Here’s the mental model and the five patterns that underlie almost every production system.
What “Orchestration” Actually Means
The underlying mechanics are simpler than they look. Every orchestration system, regardless of framework or pattern, follows the same loop:
- Create a unit of work.
- Run it.
- Persist the result.
- Continue, or stop if done.
On failure, if the unit was persisted correctly, you pick up from there. That’s it. Urmzd’s engineering post on durable workflows puts this precisely:
“Non-deterministic computation does not require non-deterministic infrastructure. The model can be chaotic. The orchestration layer shouldn’t be.”
https://twitter.com/urmzd_/status/2065581826928783764
The log is what makes this work. Every intent, every tool call, every result (and every failure) gets written to a persistent store before the next step runs. The model reasons over what’s in that store; the store doesn’t rely on the model to remember what happened. That distinction is the difference between an orchestration system you can trust and one that randomly loses its place.
The Five Patterns and When Each One Applies
Anthropic’s “Building Effective Agents” describes five workflow patterns that cover almost every real use case. The difference between them is where the judgment lives: in the sequence, in a router, in parallel branches, in a supervisor, or in a feedback loop.
Most multi-agent failures come down to missing structure, not model capability. Choosing a pattern deliberately is the structure. (A finding consistent across every production post-mortem I’ve read.)
- Prompt chaining (sequential pipeline): Each LLM call feeds the next. You use this when the steps are known in advance and each one depends on the previous output: draft a report, then summarize it, then extract action items. Simple, cheap, easy to debug. The failure mode is brittleness: if step two’s output format changes, step three breaks.
- Routing: A classifier decides which agent or model handles the request. Coinbase uses a Business Procedure Classifier that routes incoming queries to specialized sub-agents, with simpler requests going to cheaper open-source models and complex ones reserved for frontier models. In June 2026, Coinbase CEO Brian Armstrong posted that their custom harnesses preprocess prompts and route to the best model for the job: frontier models for planning, cheaper models for execution where they would otherwise be overkill. Result: AI spend cut nearly in half while token usage continues to grow. Intercom’s Fin AI runs a fine-tuned ModernBERT classifier that decides in real time whether to keep answering, offer escalation, or escalate immediately, replacing a heavier LLM call and achieving greater than 98% accuracy at 0.5 seconds lower latency. The pattern generalizes: send easy, common queries to a smaller, faster model and hard or unusual ones to a more capable model. You get cost efficiency without sacrificing quality on the edge cases.
- Parallelization: Multiple agents work on independent subtasks at the same time, and you aggregate their outputs afterward. There are two variants: sectioning (divide a task into non-overlapping parts) and voting (run the same task N times and pick the majority answer). Voting is useful when you need higher confidence than a single model call gives you: code review, contract analysis, sensitive classification. Cost scales linearly with the number of parallel branches, so this is a deliberate choice, not a default.
- Orchestrator-worker: A higher-level agent decomposes the goal, delegates subtasks to specialized workers, and synthesizes the results. This is the dominant pattern for complex, multi-step tasks that don’t fit in a single context window. Shopify’s River agent runs on this pattern: a central orchestrator receives a task, delegates to specialized workers that read code, run tests, open pull requests, and query the data warehouse independently, then synthesizes the results. As of May 2026, 1 in 8 merged PRs at Shopify is co-authored by River. The orchestrator holds the plan; the workers do the execution.
- Evaluator-optimizer: A worker generates output; an evaluator scores it; the worker iterates. Use this when quality matters more than latency: content generation that has to hit a specific tone, code that has to pass a test suite, analysis that has to cover a checklist. The evaluator can be another LLM, a deterministic check, or a human. The loop runs until the score is good enough or a max iteration count is hit.
The One Thing Every Pattern Shares
Those are the five. They look different at the top, where the judgment lives. At the bottom, they’re the same loop.
Under different names and topologies, all five patterns reduce to the same structure: define the work, run it, record what happened, decide whether to continue.
The “record what happened” part is where most early implementations cut corners, and it’s what makes systems fragile. In a production-grade orchestration system:
- A tool call (the intent to do something) is written to the log before the tool runs.
- A failure is persisted just like a success: it’s an output, not an exception to be swallowed.
- A human-in-the-loop checkpoint is just a unit where the result comes from a person instead of an API. The workflow pauses, no process needs to stay alive, and when the human responds the workflow resumes from exactly where it stopped.
Resuming a workflow and running it fresh are the same operation. That’s the property you want.
LangGraph’s persistence model makes this concrete. After every step in a graph, a checkpointer writes the full graph state to an external database (Postgres, Redis, or SQLite). The execution process and the state store are completely decoupled: if the process dies, any other process can resume the graph from the last checkpoint with no in-memory dependency. The graph never owns its own state. The store does.
What Production Looks Like (It’s Not a Single Pattern)
The five patterns are a vocabulary, not a menu. Real systems combine them. The interesting engineering is in the handoffs between patterns, not inside any single one.
Uber’s internal developer tooling uses a pipeline + orchestrator-worker hybrid across two flagship tools: Validator (a security and best-practice checker) and AutoCover (automated test generation). A pipeline stage classifies and scopes the work; an orchestrator-worker stage fans out to domain-specific agents that handle each task in parallel. The result: approximately 21,000 developer hours saved.
LinkedIn’s SQL Bot starts with routing (a natural language query gets classified and dispatched to the right query-building agent) and then uses orchestrator-worker to construct and validate the SQL in multiple steps. Two patterns, one workflow, one user-facing feature (non-technical employees querying internal data without writing SQL).
More recent deployments follow the same logic. Airbnb’s Automation Platform v2 (April 2026) replaced a static-workflow-driven conversational system with an LLM-native agentic design where the orchestrator dynamically selects tools and their execution order at runtime, shifting from hard-coded pipelines to adaptive orchestration. Meta’s capacity efficiency system (April 2026) runs 50+ specialized agents behind a unified tool interface, combining parallelization and orchestrator-worker to optimize infrastructure performance at hyperscale.
A few things are almost always true by the time a system hits production: 68% of agent deployments require human intervention within 10 steps (per Measuring Agents in Production, a study of 306 practitioners across 26 domains), which means the orchestrator-worker and evaluator-optimizer patterns almost always end up with a human-in-the-loop checkpoint grafted on. Pure patterns exist in demos. Production ships hybrids.
Start Simple. Add a Pattern When You Hit a Wall.
If you’re building your first orchestration system, start with the simplest pattern your actual problem requires:
- Fixed, sequential steps where each builds on the last → prompt chaining.
- Input type determines who handles it → routing.
- Independent subtasks that can run in parallel → parallelization.
- Goal is too complex for one context window → orchestrator-worker.
- Output quality is what matters, latency is secondary → evaluator-optimizer.
Add complexity only when you hit a concrete wall, not before. Engineers who’ve built and shipped production agent systems consistently report the same lesson: premature decomposition creates coordination overhead without benefit. One well-structured agent is better than three loosely coupled ones.
Once you know which pattern fits, you need a framework to build it in. The right choice depends on where you are in the build.
CrewAI is the fastest path to a working prototype. You define roles, assign tasks, wire up a crew, and something runs. The role/task/crew abstractions are opinionated enough to eliminate early-stage decisions, which is the point. The limitation is the same: it’s role-based by design, so topologies that don’t map cleanly onto that model (complex routing, dynamic fan-out, multi-stage evaluation loops) start to fight the framework.
LangGraph is where you go when the prototype needs to become a system. It models your workflow as a graph of nodes and edges, each node a discrete unit of work, edges defining what runs next and under what condition. That structure gives you stateful, non-linear flows, built-in persistence, and precise control over branching and loops. The learning curve is steeper, but it pays off once you need human-in-the-loop checkpoints, conditional retries, or workflows that span multiple steps without holding everything in memory. Uber, LinkedIn, Shopify, and JPMorgan all run it in production.
Mastra is the TypeScript-native equivalent: a workflow DSL with built-in memory, evals, and agent primitives. If your stack is TypeScript, it’s the only serious option at this level of completeness.
One level below all of these sits the durability layer. LangGraph and Mastra have built-in persistence, which is enough for most workloads. When it isn’t (when you need guaranteed exactly-once execution, automatic retries with state preserved across failures, and workflows that can pause for days waiting on a human or an external system), teams reach for Temporal. It’s not agent-specific; it’s a general durable execution engine that your agent workflows run on top of. This is what the “record what happened before the next step runs” principle looks like as hardened infrastructure.
The right tool is always the least infrastructure your problem actually requires.
The log remembers. The model reasons. Keep those two responsibilities clearly separated and the rest follows.