Two identical consulting analyses. Same frontier AI model. Wildly different results. What changed? Everything invisible.

The difference between accurate AI analysis and expensive mistakes in professional services often comes down to configuration choices that end users never see. Here's what continuous experimentation reveals about the levers that actually matter.

The Commoditisation Reality

Everyone can subscribe to Claude or GPT-4. That's not a competitive advantage. It's a utility bill.

The differentiation reality is different: custom tooling, domain-specific workflows, and continuous optimisation create competitive moats that model access alone never will. The two structural moats in professional services AI are private data and custom tooling. Tuning is where the custom tooling moat gets built.

The Levers That Matter

Temperature settings. Creative scenario analysis needs different temperature than precise numerical outputs. A financial model that's "creative" is dangerous. A strategic brainstorm that's "precise" is useless. Knowing when to use which requires domain expertise, not model expertise.

Retrieval strategies. When your AI searches its knowledge base, what it finds first determines what it outputs. Regulatory guidance needs different weighting than case precedent. Current standards need priority over historical practice. Getting this wrong doesn't produce errors — it produces plausible-sounding outputs that are subtly outdated or misweighted.

Prompt architecture. The structure of how you ask AI to analyse information changes the output dramatically. Approaches that excel at structured data analysis collapse when handling unstructured interview transcripts. One size fits nothing.

Context window management. Processing a 10-page brief requires different strategies than a 1,000-page document review. Token limits aren't just technical constraints — they're analytical constraints. What gets included and excluded from the AI's working memory determines what patterns it can identify.

Same Input, Three Configurations

We tested the same set of financial due diligence documents through three different configurations of the same model:

Configuration A (default settings): Identified 60% of material findings. Missed several critical items that would have changed the deal assessment. Produced confident-sounding analysis that was incomplete.

Configuration B (temperature-optimised only): Identified 78% of material findings. Better, but still missed context-dependent items that required domain-specific retrieval weighting.

Configuration C (fully tuned — temperature, retrieval, prompt architecture, domain context): Identified 94% of material findings. The remaining 6% were edge cases that required expert judgement — exactly the division of labour you want.

The difference between Configuration A and Configuration C isn't the model. It's months of domain-specific refinement.

Why "Set and Forget" Fails

Tuning isn't a one-time project. It's continuous refinement.

Regulatory frameworks evolve. Client expectations shift. New document types emerge. The AI configuration that was accurate six months ago may be subtly wrong today — not because the model changed, but because the context changed.

This is where why most AI projects fail becomes relevant. The 95% failure rate isn't about bad technology. It's about implementations that assume AI is plug-and-play rather than an ongoing investment in domain-specific optimisation.

The Compounding Advantage

Every tuning iteration teaches you something. Temperature ranges that work for your specific document types. Retrieval weightings that match your professional domain. Prompt architectures calibrated against your actual outcomes.

This knowledge compounds. A firm that's been tuning for 18 months has a body of implementation intelligence that can't be replicated by subscribing to the same model. As base models improve, this tuning sophistication multiplies — better foundation models make refined configurations even more powerful.

The Measurement Challenge

How do you know if tuning is working? Not by asking AI to evaluate itself.

The only reliable measurement is expert review of outputs against known-good benchmarks. Domain specialists assessing whether the AI's analysis matches what they would have concluded — and flagging where it diverges. This creates a feedback loop: expert review informs tuning, tuning improves outputs, improved outputs refine the benchmarks.

Check your organisation's readiness for this level of AI sophistication — the firms that treat tuning as infrastructure rather than a project are the ones building lasting competitive advantage.

What to Ask Your AI Provider

Three questions that reveal whether an AI implementation is genuinely sophisticated or just using default settings: