What To Do When The Result Of A/B Testing Is None Of The Above

It seems to make sense on its face: Asking people to choose from two or three pre-selected choices is a lot easier than asking them to show you what they want on a blank page.

White boarding leads to whimsical flights of fancy, but choice puts customers in the role of buyers showing preferences between two or more options. That’s better, right?

But sometimes the constraints of A/B testing don’t give you a decisive result, and you have to try again with a modified approach.

The story of the creation and adoption of the Philadelphia 76’ers latest mascot, Franklin The Dog, is a great example.

New management took over the team in 2011, and soon de-commissioned their then-reigning mascot, a bunny named Hip Hop. They quickly partnered with Jim Henson’s Creature Shop to develop three new mascot choices — B. Franklin Dogg, Phil E. Moose and Big Ben — and put it to a fan vote.

It looked like they did everything right:

1. Explained why the selection is important, why they’re putting it to a vote and why they are choosing the particular population — the fans, the people who buy the tickets — to do the voting.
2. Made sure the options were fully formed. Not just drafts and proposals. The choices given to the fans by the Sixers weren’t just the team’s “concept” of the character. Each of the three proposed mascots were in ready-to-go shape before they were presented for the fan vote.
3. Gave them more than just the image. When the Sixers reached out to the fans, they didn’t just show them pictures. They gave them a little back story on each character so they could have more to go on.

After all this — what? Management didn’t feel they got the decisive groundswell of opinion they were looking for from the fan vote. Several years went by with no decision on a new mascot.

Then, a couple of years later, they reinvigorated the project with three significant modifications to the typical A/B testing process:

1. Redefined their sample base. Team research showed fans first become familiar with a team as kids. So, this time they went out to 1,000 kids, ages 6 -10.
2. Instead of starting with two or three choices, they first showed them lots of designs — 40 in all — to get a general sense of their tastes.
3. Asked specific character questions to narrow it down: Should the mascot have fur? What color should it be? Should it dunk? Should it be tall? Fat? Short?

This time they came up a fluffy blue dog named Franklin introduced in early 2015. And so-far, so-good with fan acceptance.

There is some wisdom in the theory behind A/B testing: Putting constraints in the creative process by forcing decisions from among a limited range of options. And it makes sense to start with that kind of methodology. But sometimes it doesn’t give you the resounding weight of opinion you’re looking for, and you have to take a second look.

This often involves going back with more options, and even providing a limited tool-kit for a higher level of engagement and participation by the respondents you really want to hear from.