From AI Roulette to Orchestrated Success

Why 85% of AI Projects Fail (And How to Join the 15% That Succeed)

Every day, millions of people play the game of AI roulette. For trivial tasks like summarizing a document or drafting an email, these tools shine. But climb the complexity curve, and the odds plummet. You get articles that miss your brand voice, applications that won’t compile, and images that bear no resemblance to your vision.

How do you increase your chances of success?

At first, the answer seemed to be prompt engineering—learn the magic incantation that makes the genie grant your wish. While that’s part of the solution, it’s far from the whole story.

The Architect’s Lesson

Imagine trying to build a house by immediately laying down bricks and slathering mortar. How well would that structure turn out? My late father, an architect, would have been horrified. Where are the blueprints? How will this align with building codes? That building won’t survive an encounter with a huffing wolf, let alone a hurricane.

Yet that’s precisely how we treat our interactions with AI models. We input a single prompt and expect perfection to emerge. Sometimes it does, but often it doesn’t.

Why isn’t this working?

The answer lies in understanding that AI is a stochastic engine, and we're playing a probability game. While single attempts might suffice for simple tasks, these probabilistic limitations compound dramatically when we try to scale to more ambitious efforts. Fortunately, there are proven methods to enhance our AI yield rates.

Five Ways to Stack the Odds in Your Favor

1. Develop a Theory of AI Mind

The skills that make someone an effective parent, teacher, coach, or manager are precisely what you need for AI success. Consider any role where you guide or mentor others. You instinctively assess: What knowledge do they currently possess? What tools are available to them? What’s their current state and capacity? Are the goals crystal clear?

This same approach of empathetic bridging helps transform AI interactions. You need to understand what information the AI has access to, its capabilities and limitations, and how it’s likely to process data to arrive at an ideal output. Once you put yourself in the LLM’s shoes by understanding its knowledge boundaries, processing style, and inherent limitations, it becomes second nature to guide the outputs.

2. Context Engineering: Build the Complete Picture

Most AI disappointments aren’t model failures—they’re context failures. The AI lacked the necessary components to succeed. Context engineering manages the entire information ecosystem:

• Prompt that clearly communicates needs, along with optional assets like images
• System instructions defining persona and rules
• Conversation history, maintaining continuity
• Retrieved external knowledge (RAG), providing facts
• Web search for real-time information and current events
• Tool definitions making capabilities explicit
• Output structure specifying what success looks like

As Andrej Karpathy elegantly puts it:

“Context engineering is the delicate art and science of filling the context window with just the right information for the next step.”

Advanced implementations show dramatic results: Combining Contextual Embeddings and Contextual BM25 reduced retrieval failure rates by 49%, according to Anthropic’s research on contextual retrieval.

3. Multiple Attempts: Embrace the Probability Game

AI is more like a slot machine than a calculator. Instead of betting everything on one pull, try 10, 20, even 100 variations. Each attempt compounds your probability of success. This includes:

• Agentic loops that iterate until goals are met
• A/B testing different approaches systematically
• Human-in-the-loop refinement for complex tasks

4. Multi-Model Strategy: Diversify Your Portfolio

Different models have varying strengths that fluctuate over time. Like a diversified investment portfolio:

• Leverage models from a variety of providers
• Use specialized models for specific domains
• Combine outputs from multiple models for substantive improvements
• Build in fallback patterns for resilience
• Route simple tasks to faster, cheaper models (70% cost reduction possible)

This strategy also protects against “LLM drift”—the phenomenon where model performance degrades over time as providers update or throttle their services. What works brilliantly today might be nerfed tomorrow. Multiple models ensure you’re never at the mercy of a single provider’s decisions.

5. Iterative Refinement: Critique, Validate, and Rebuild

Complex outputs require cycles of improvement:

• Break work into smaller, validated steps
• Implement critique mechanisms at each stage
• Regenerate based on feedback until quality thresholds are met
• Use automated evaluation frameworks that score AI outputs against predefined quality metrics, allowing for rapid, data-driven refinement

The challenge with doing everything correctly—multiple attempts, multiple models, well-thought-out context—is that it’s time-consuming when handled manually in traditional interfaces like ChatGPT. Fortunately, an emerging set of tools is automating much of this work, such as the one below that I've been developing.

Three Modes of AI Interactions

It is imperative to understand these three fundamental modes of AI interaction:

Exploratory Mode: "I don’t know exactly what I want"

Perfect for examining the possibility space. Dialogue-based, iterative, unpredictable—like brainstorming with a knowledgeable colleague. Great for discovery but inherently variable in outcomes, as any Vibe Coder can attest at the end of a long session where the application doesn’t quite work.

Planning Mode: "Now I know what I want, I need a comprehensive plan"

The critical bridge between ideas and execution. Transform vague goals into detailed, actionable blueprints. To pull from recent headlines, this is where you go from “concept of a plan” to an actual, detailed one. The good news is that AI can be a powerful partner in this planning phase. By providing a high-level goal, you can use an AI model to decompose the task into a detailed series of steps, identify potential roadblocks, and even suggest the best tools or models for each sub-task.

For coders, tools are already being built to explicitly address this space. In fact, Amazon’s new Kiro IDE makes this a central part of the development experience:

For content creators, tools like Writer start the blog creation process with a generated outline that can be tweaked before the actual post is created. Without this decomposition into discrete, verifiable sub-tasks, execution of larger artifacts often fails to meet expectations.

Execution Mode: "I have a solid plan, now make it happen"

With proper planning, AI can work autonomously or semi-autonomously. But skip the planning phase, and you get “AI slop”—incoherent results that waste time and resources.

These modes aren’t just for individual interactions—they’re foundational for sophisticated agentic workflows.

The Two Critical Mindset Shifts

First: Master the Mode Transitions

The first mindset shift is tactical but crucial: recognizing that each interaction mode serves a distinct purpose and requires a different approach. Many people get stuck in exploratory mode—the comfortable back-and-forth of vibe coding or haphazard image prompting, where they tinker and iterate without structure. While this feels creative and engaging, it’s precisely why skilled professionals often dismiss AI as incapable of producing highly complex work matching their standards.

The breakthrough comes when you learn to transition between modes deliberately:

• Use exploration to understand the problem space
• Shift to planning to create detailed, verifiable blueprints
• Move to execution only after you have a solid plan

This isn’t about abandoning the creative exploration that makes AI engaging. It’s about recognizing that exploration alone won’t produce complex, professional-grade outputs. You need all three modes, applied in sequence, to match or exceed traditional craft quality with AI.

Second: From Craftsman to Conductor

The deeper transformation is in how we relate to our work itself. As highlighted in a recent Forbes analysis, roles are evolving rapidly—for instance, “Content Project Manager” is transforming into “AI Orchestrator”, a role focused on managing prompts, models, and workflows for both AI and human creators.

This shift is profound because it challenges our professional identity:

• The writer becomes the editor-in-chief
• The coder becomes a systems architect
• The analyst becomes a strategic advisor

It’s analogous to the historical transition from artisanal craftsmen producing individual goods to factory systems producing at scale. This isn’t about being replaced—it’s about being elevated. But it requires letting go of the craftsman’s need to personally touch every detail and embracing the conductor’s role of orchestrating many performers.

The resistance is understandable. Many of us derive satisfaction from the craft itself—the physical act of writing, coding, or analyzing. However, those who make this transition discover something powerful: orchestrating allows you to achieve outcomes on a scale and with a sophistication that individual craftsmanship alone could never attain.

From Foundation to Symphony: The Next Level

Once you’ve mastered these elements, you’re ready for the game-changer: coordinating multiple AI agents working together.

Think of it this way: You’ve learned to play your instrument—now it’s time to conduct the ensemble.

This approach shifts the focus from individual interactions to directing multiple agents and models that work in concert, each contributing its unique strengths and building on the foundational elements above.

The business case for mastering the fundamentals before moving to this stage is compelling:

• 80-85% of AI projects fail (twice the rate of traditional IT projects)
• According to RAND Corporation research, this staggering failure rate isn’t due to the AI’s shortcomings, but rather to a combination of messy data (affecting 43% of failures), vague project goals, and weak integration and deployment plans.
• But the subset that do succeed generate $12.50 for every $1 invested
• Organizations with comprehensive orchestration frameworks report up to 60% greater ROI

What separates winners from losers? Not model performance—orchestration quality and effective management of the data surround tend to have larger impact.

Orchestration in Action

Morgan Stanley's AI system succeeds through sophisticated orchestration that empowers financial advisors with instant access to 100,000+ internal documents, achieving 98% adoption among advisor teams. Their success comes from systematically applying all five probability-boosting elements at scale:

• RAG-powered context management across the firm's entire intellectual capital
• Multi-tool architecture with AI @ Morgan Stanley Assistant for research and Debrief for meeting summaries
• Iterative refinement with human validation at every step
• Robust governance ensuring responses come only from vetted internal sources

Meanwhile, the $62 million failure of IBM Watson for Oncology at MD Anderson Cancer Center wasn’t due to a lack of powerful AI. Instead, it was an orchestration failure. The system struggled to integrate with the hospital’s electronic health records, had difficulty interpreting doctors’ notes, and was ultimately not flexible enough to be practically applied in a real-world clinical setting. They had powerful AI but lacked the systematic application of these foundational elements.

Your Path Forward

The era of single-prompt miracles is over. Success requires:

Master the Fundamentals:

1. Develop a theory of AI mind to understand the system’s capabilities
2. Engineer comprehensive context, not just prompts
3. Design for multiple attempts, not single shots
4. Leverage multiple models strategically
5. Build in cycles of critique and refinement

Adopt the Strategic Mindset:

• Know when you’re exploring vs. planning vs. executing
• Match your approach to your current mode
• Transition deliberately between modes

Scale Through Orchestration:

• Combine all elements into systematic workflows
• Build evaluation frameworks that measure the whole process
• Maintain human ownership while enabling AI autonomy

The Bottom Line

We’re at an inflection point. The model-centric era has ended. The future belongs to those who can orchestrate—who understand that success isn’t about access to the best model but about systematically stacking the odds in your favor.

Architects will tell you, great buildings don’t emerge from randomly laying bricks. They require blueprints, systems, and coordination among multiple specialists.

The same is true for AI success. The question isn’t whether you have the best model. It’s whether you can systematically apply these probability-boosting elements to conduct the entire symphony.

The tools and frameworks to implement these strategies are emerging rapidly. Your next step? Pick one element—perhaps context engineering or start leveraging multiple models—and experiment with it this week. The compound effect of mastering each element will transform your AI outcomes from random chance to reliable results.

Ready to stop playing AI roulette? The orchestra is waiting for its conductor.