What Would Make Us Change Our Minds?
A proposal is easier to judge when it explains, in advance, what would count against it.
Consider a hypothetical proposal to improve a website's enquiry page. If enquiries rise, the change worked. If they remain flat, it needs more time. If they fall, the market has been difficult.
Each explanation might be true. Together, though, they leave the business without a result that could alter the decision. Whatever happens, the recommendation survives.
Before approving the work, ask what evidence would lead to a different recommendation. It is a useful question for a page edit, a marketing programme or an AI project.
Write down the decision first
Begin with what the intervention is supposed to improve and the decision the measurement will inform.
For an enquiry page, that might be whether a clearer explanation attracts more suitable enquiries. For an internal AI assistant, it might be whether the system answers a defined set of questions accurately enough to justify a wider trial.
Those are proposed examples, not claims about a particular project. Each needs a baseline, a method of observation and a definition of what would be unacceptable. Otherwise, it is too easy to choose the most flattering measurement afterwards.
The person authorising the work should understand those choices. A successful experiment on a minor intermediate measure may still leave the original business question unanswered.
Include the result you do not hope for
Corluka's SEO Meta Study makes a useful structural choice: its replication framework asks when to revert a change, and the author calls that the part that matters most, because "without a rule for when to revert, every outcome reads as confirmation". That question deserves attention before implementation, while the team is still able to consider failure without defending work it has already done.
The answer need not always be "revert". It could be to revise the approach, stop expanding it or collect more evidence. An inconclusive result should also have a place in the plan.
An illustrative enquiry-page trial might set out three decisions. Continue if the evidence supports improved enquiry quality without an unacceptable loss of suitable contacts. Revise if customers are confused by the new conditions. Defer a conclusion if there are too few enquiries to distinguish a pattern.
The detail will depend on the business. Agreeing it in advance limits the temptation to reinterpret every outcome as success.
Keep a comparison where it can teach you something
SearchPilot's account of a year of SEO testing, given in 2019, explains its use of changed and unchanged groups of pages. It also describes a site whose low traffic made its model unsuitable. The method's ability to distinguish an effect is part of the question, not a detail to assume away.
For a small site, a tidy before-and-after chart may remain ambiguous. Seasonality, promotion and changes in demand can occur alongside the edit. Record them. If a comparison group is possible, decide how it will be chosen before inspecting the result.
Where several changes are necessary together, evaluate the package honestly. Fixing a broken process may matter more than isolating the contribution of each part. The report should then describe the package, rather than awarding its entire effect to the most marketable component.
A factual correction has a different test
Some changes should not wait for an experiment. An inaccurate statement needs correcting. A broken form needs repairing. Essential access to information should not depend on a favourable traffic result.
Give those changes appropriate checks: is the statement now true, does the form work, can the person complete the task? Traffic can be observed separately. A flat graph is not a reason to restore an error.
This distinction prevents an experimental mindset becoming an excuse to withhold necessary work. It also makes the performance claims cleaner: the team can explain which changes were repairs and which were tests of a growth hypothesis.
Leave a result someone else can use
A useful record states what changed, what was measured, what else happened and what decision followed. Preserve the result when the intervention did little or when the available evidence could not settle the question.
Make that record part of completion, rather than an optional case study produced only for the successes. Another team can then see why an approach was continued, abandoned or left unresolved.
Before the next project begins, add a paragraph to its proposal: what would make us change our minds? If the answer is clear, the work has a better chance of leaving the organisation wiser, whatever the graph does.
Related Articles
Acceptance Criteria for AI Projects: Examples That Actually Protect You
Most AI projects fail at the end, in the argument about whether the delivered thing is the bought thing. Acceptance criteria are how you have that argument at the start, when it costs nothing - with examples you can copy.
The Missing Context Behind the Impressive Percentage
A measured improvement can be real and still tell us surprisingly little about what another business should do. The useful work begins with the conditions behind the number.