One A/B test on the New Zealand Trails homepage lifted new-user conversion rate by 15%. No extra ad spend. No extra traffic. Just making more of what was already there. That is the entire argument for conversion rate optimisation, and it is why I look at the site before I look at the budget.
It is also the discipline most likely to sit outside your agency’s scope, which is a strange place for your biggest lever to live.
What is conversion rate optimisation?
Conversion rate optimisation – CRO, spelled optimization in the US – is the practice of increasing the percentage of your existing visitors who complete a defined action, such as an enquiry, a booking or a sign-up. It is evidence-led rather than opinion-led: you find where visitors drop out, form a hypothesis about why, run a controlled test, and let the result decide. The action is defined by your business, not by your analytics tool.
That definition rules a few things out. A redesign is not CRO, because nobody measured what the old design was doing. A checklist of best practices is not CRO, because a practice that worked on somebody else’s audience is a hypothesis about yours, not a finding. And an agency telling you your conversion rate improved is not CRO either, unless they can show you what it would have done if they had changed nothing.
How do you calculate a conversion rate?
Conversions divided by visitors, expressed as a percentage. 50 enquiries from 2,000 sessions is 2.5%.
The argument is always about the denominator. Sessions, users, or new users? Each gives a different number from the same underlying reality, and each is defensible. What is not defensible is comparing your figure against a published benchmark without knowing which denominator that benchmark used – which is most of them. An unqualified industry average is worse than no benchmark at all, because it gives you false confidence in a direction. Pick a denominator, write it down, and compare yourself against yourself.
Why is it usually a bigger lever than more spend?
Arithmetic, and it is worse than it looks for the traffic side. Buying 15% more traffic costs at least 15% more every month, and usually more than that, because the next click is always dearer than the average one – you have already bought the cheap demand. Lifting conversion by 15% applies to every visit you have already paid for, and to every visit you buy afterwards, at no recurring cost.
“More traffic is a bill you pay every month. A better page is closer to a bill you pay once.”
There is a trap in that arithmetic worth naming, because I have watched agencies walk into it. On the same account, restructuring Google Ads off Performance Max and onto specific search campaigns lifted conversion rate 27%. Nothing on the website changed. What changed was who was arriving – better-matched intent, fewer accidental clicks. The conversion rate moved, but that was media work, not CRO. Two different levers. If you cannot tell them apart in your reporting, you cannot tell which one to pull next time.
How much traffic do you need before a test means anything?
More than almost anyone selling you CRO will admit. The number of visitors a test needs rises with the square of how small a change you want to detect, which gets punishing fast.
Work it through on Kohavi's rule of thumb, at 95% confidence and 80% power – an exact calculation lands a few per cent higher still. On a 2% baseline conversion rate, detecting a 20% relative lift needs roughly 19,600 visitors per variant – about 39,000 in total. Halve the lift you are chasing to 10% and it becomes roughly 78,400 per variant, or 157,000 altogether. At 5,000 sessions a month, that second test takes over two years. It will never finish, and the version of it that does finish, because somebody stopped it when the line looked good, is not a result.
Run tests for a minimum of two full weeks even when volume allows less, so you cover both weekend and weekday behaviour. And do not check daily and stop when you are ahead. Repeated peeking pushes the false-positive rate far above the 5% the tool reports, which is how sites end up with a folder full of wins and a flat revenue line.
Apply that to the test I opened with and you can see the problem. A homepage change is a big change, which is what made it detectable at all, and it still ran on an account doing tens of thousands of sessions a month. On a smaller site the same test would have produced a number and no information. If a supplier shows you a 15% lift, ask how many visitors it took and how long it ran before you believe it – including when the supplier is me.
What if you don’t have that much traffic?
Most businesses do not, and it does not mean the work stops. It means A/B testing is the wrong instrument and something else is the right one.
- Test bigger changes.
Sample size falls with the square of the effect you are looking for, so doubling the size of the change quarters the traffic you need. Test a different offer or a restructured page, not a button colour.
- Move the measurement upstream.
Test against a higher-frequency step – enquiry form started, dates selected – rather than the booking itself. You get a usable signal sooner, provided you have checked that step actually correlates with revenue.
- Do the work that needs no statistics.
Session replay, form drop-off analysis, checkout friction audits, five-second tests, talking to people who nearly booked. None of it needs significance, and all of it finds things that are simply broken.
- Fix the known-broken before testing the arguable.
If your enquiry form fails on mobile Safari, that is not a hypothesis. Ship the fix. Save the tests for the decisions where reasonable people disagree.
Why do most tests fail?
Because most ideas are wrong, including good ones. When Microsoft published a decade of experimentation data, roughly a third of well-designed experiments improved the target metric, a third did nothing, and a third made things worse. That is not an indictment of testing – it is the reason testing exists. The alternative is shipping all three thirds and calling the result a redesign.
It also explains why “our conversion rate went up” is such a weak claim. Conversion rate moves on its own, constantly, because your traffic mix changes, because it is a different month, because a channel shifted. Without a control running at the same time, you are reading the weather and calling it a decision.
Where should CRO sit?
Next to the media, run by the same people. The 27% example is the argument: the account change and the site change move the same number for different reasons, and you can only tell them apart if one team sees both. When CRO is a separate retainer with a separate scope, it gets sold as an upsell, reported in isolation, and quietly dropped in the first budget review. That is one of the reasons we run it as one connected system rather than a menu – the same reason we think you should expect more from a performance marketing agency than a channel and an invoice.
The honest summary is this. CRO is not a growth hack and it is not a redesign. It is the discipline of finding out what your site actually does, changing one thing on purpose, and being willing to learn that you were wrong. Done properly it is the cheapest growth you will buy. Done as a checklist, it is decoration.