Sometimes a new ad gets back slightly higher response rates than your old one. so you aren’t completely sure your new ad is superior. Deciding whether to replace the old ad with the new one is crucial now. Here’s a way to be more certain that your new ad is superior, or isn’t…
This requires some statistics. If you have the average (or overall) response response rate for your old ad, and at least one test campaign’s worth of response rate for the new ad, then you can do a one-tailed hypothesis test using proportions with a 5% level of significance (examples here)
Here’s an example. Suppose your old ad has been getting an overall conversion rate of 3.1% (.031). Then you send out 1000 fliers of your new ad, which gets back a conversion rate of 3.3% (.033). With a 5% level of significance, is your new ad superior to your old ad?
The hypothesis is:
H0: p* = .031 H1: p* > .031
p* = .033 p = .031 n = 1000 and Zc =1.645
Z = (p*-p)/√(p(1-p)/n) = (.033-.031)/√(.031(.969)/1000) = .365.
Since .365 is less than your Zc of 1.645, there’s not enough evidence to prove that your new ad is superior to your old ad. It’s plausible that their response rates are fairly equal.
But don’t forget to look at your profit from two increased sales. From our example, are those 2 extra sales’ profit margins so significant to say that your new ad is that much more profitable than your old ad? What if each sale brought a profit of $50? how about $1000? Should you keep trying out this new ad, or try for something better and keep only your old ad until then?
Given these figures, you can come to perspective by statistical significance and profitability increases.
The most you can conclude from this example is that the new ad should be kept side to side with the old ad, and try another hypothesis test after your new ad has been used for a few months or so.
Or make more ads and do test campaigns on them until you find one that has statistically significantly higher conversion rates (a Z score higher than 1.65), and then replace your old ad with that winner.
Don’t replace old ads that work with new ones just for the sake of change. Let the numbers help you jump wisely.