Testing, 123: how many multivariate and a/b tests you should run

When you run a lot of tests on a website, you start to see patterns emerge. Some things that should generate results (changing colors, button placement, stock photography) will consistently yield bigger deltas than other things (shuffling the deck chairs on the titanic, the footer text color, the company logo color, etc). Even though a lot of emotion is invested into the company logo color, look and feel, it’s not the logo that closes a sale. It’s the content, the beating heart of the page that sells, or fails, with the end user. After running enough multivariate tests, most smart marketers get a decent handle on what does and does not drive performance.

As a result, it is very worth asking at what point do you stop testing?

For most marketers the answer will be simple, you stop testing when you achieve the desired result. However, I firmly believe that we are all chasing moving targets. In that sense, we are never really “done” with testing. The end result could always be improved and as long as we have the time, energy, budget and resources available the next test should be queued, measured and analyzed. If we purse this never ending horizin with the mindset of Lexus, “the relentless pursuit of perfection” then we are far, far better off as a company than if we decide to iterate through to some short term goal, hit it and call it done.

Testing is a tool, one that when used properly, can drive the kind of exponential growth that most startups are looking for and larger companies dream about yet fail to achieve. If you look at the big dogs, Google, Facebook or Twitter, they all test endlessly, relentlessly driving incremental improvement that when looked at macroscopically, has had a profound impact on their businesses.

The fair question for you, dear reader, to ask me is: What are *you* testing? Well, I just published my fourth (less than a week after my third) story on Amazon and Barnes & Noble. This was a new test, based around a concept that my wife (thank you) piloted and proved successful. As a result, I’m blending the results of her marketing approach with that of my third story, which was my most successful so far. I’ll be done testing after I’m the new J.K. Rowlings and I get a multi volume blockbuster movie deal, thank you. :)