Did your redesign work? How to measure before and after

To know whether a redesign worked, measure the same thing before and after, in the same way, with enough votes on both sides. That sounds obvious, but most redesigns are judged on a gut feeling, the opinion of the person who paid for it, or analytics that changed for a dozen other reasons. This guide shows how to set up a fair comparison of the first impression.

Why "it looks better" is not enough

Everyone who worked on a redesign has seen the new version many times. They know what it's meant to say, and they want it to be better. Clients and colleagues are in the same position. What you want to know is whether a stranger, seeing it for the first time, understands it better, trusts it more or would stay longer. That means asking strangers, twice.

Step 1: measure before you change anything

Run a campaign on the current version first. This is your baseline. Without it, you have nothing to compare with, and you can't go back once the old site is gone. Aim for the same number of votes you plan to collect for the new version.

Step 2: change one thing, if you can

If you change the headline, the photo and the colours at once and the score goes up, you don't know which change did it. For a full redesign that's fine: you just want to know whether the new version as a whole is better. For tuning, change one element per round. The headline is usually the best place to start, see writing a homepage headline.

Step 3: keep everything else the same

A comparison is only fair when the only difference is the change itself:

• The same audience. If the first campaign used reviewers from everywhere and the second only entrepreneurs from the Netherlands, the difference may come from the people, not the page.
• The same page. Compare the homepage with the homepage, not with a new landing page.
• The same number of votes, roughly. A result with 10 votes is far less certain than one with 50. See how many votes you need.

On StayOrBounce, the four questions never change, and each campaign keeps its own screenshots. That way, the old result always shows the version people actually judged.

Step 4: read the difference, not just the numbers

A stay score that goes from 52% to 60% looks like progress. With 10 votes on each side, it is well within the noise: the same page tested twice could easily give that difference. Even with 50 votes on each side, the stay score has to move by about 20 percentage points before the difference is clear.

When you compare two campaigns, StayOrBounce marks each question as clearly better, clearly worse or within the noise. A difference counts as clear when it is larger than the margin of error of the difference at 95% confidence. Both campaigns need at least 10 votes before any verdict is shown.

"Within the noise" doesn't mean nothing changed. It means these votes can't tell. If the difference matters to you, collect more votes on both versions.

Step 5: look at segments, carefully

If you targeted several countries, age groups or roles, you can also compare results per group. Groups are small, so their margins are large: treat a difference between segments as a hint, not a conclusion. StayOrBounce only shows a segment from 3 votes, and that is a minimum, not a guarantee.

What a better first impression doesn't prove

A higher score on Would I stay? means strangers say they would stay. It doesn't prove more sales or sign-ups. After the new version has been live for a while, check what visitors actually do: your engagement rate, sign-ups or enquiries. If you have enough traffic, an A/B test gives the most direct answer. A first-impression test is the faster first check, and works before you have that traffic.