If you haven't been living under a rock for the past few weeks, certainly by now you've heard about the bickering back and forth between A/B testing proponents and the Multi-armed Bandit crew.

Visual Website Optimizer started the whole thing with a blog they posted titled Why multi-armed bandit algorithm is not "better" than A/B testing.

Chris Stucchio was among those who took exception and responded with a blog of his own just a couple days later: Why Multi-armed Bandit algorithms are superior to A/B testing

Guess what guys, you're both wrong.

This isn't a math debate.

Record screeching sound goes here

Both algorithms work great. Which is better has little to do with calculations and everything to do with what exactly you're testing.

The mult-armed bandit approach means that your testing never ends. Even your losers are being 'explored' just in case something has changed that would alter the winner of a test.

The problem is that approach doesn't work very well when you're trying to build a recognizable brand.

Last summer 37Signals performed an A/B test on the landing page for Highrise. This wasn't your everyday A/B test where you change some text or the color of a button, they completely reinvented the site multiple times over and got significant gains many times.

When you're working to significantly increase conversions, you have to accept that for some period of time users are going to see potentially vastly different versions of your site. Eventually, however, building a brand necessitates consistency so your users aren't shocked by different versions each time they visit.

Sometimes you just want to start and finish an A/B test.

Sometimes you don't. Continuing with our 37Signals example, here's a good situation where multi-armed bandit fits great.

As you'd expect, these changes resulted in smaller variations. But as a brand, you're fine with giving John and Jocelyn a small percentage of your traffic to see if things change. Your users aren't going to be totally thrown off by a picture of a different person int he background and maybe eventually people will notice John's broken arm and feel sorry for him.

Okay, that probably wouldn't happen but you never know.

That's why we test.

So to summarize...

A/B - Better at determining a final winner. e.g. - Which brand/theme users prefer and then committing to it.

Multi-arm bandit - Better at continuous testing. e.g. - Minor changes to a landing page where you it is acceptable to have continuous changes looking for the ever increasing conversion rates.