Testing, it will free your mind: Amelia Showalter at TEDxAthens 2013


Translator: Denise RQ
Reviewer: Chryssa Takahashi In 1971, my mother was a college student
at Harvard, right as it was going co-ed. Women were now equal in the classroom, but women still weren’t allowed
to use the school’s swimming pool, and my mother wanted to swim. So she and her friends marched
into the athletic director’s office and demanded that the pool be integrated. “Impossible!” said the athletic director. “Women couldn’t use the pool,” he said,
“because the men swim naked.” “The men swim naked,” he said,
“and there was no women’s locker room.” “Pool had always been men only, and it would simply
have to remain that way.” Well then, my mother and her friends
tried a different approach. They said, “If women
were allowed to use the pool, not in any kind of permanent way,
but as an experiment for a semester, can you think of ways
to make that experiment happen?” Well, the athletic director thought
for a little while and he said, “Well, I guess we could ask the men
to wear swimsuits.” And he thought a little more and said, “You know, we do have a second locker room
for the visiting swim teams when they came to compete against Harvard. I guess the women
could use that locker room.” So they tried this crazy experiment,
and the sky didn’t fall. And men and women have been using
the pool together ever since. And 30 years later,
when I myself went to Harvard, I did not see a single naked man
in that swimming pool. I am sure this made
my mother happy on many levels. Framing things as an experiment
freed the athletic director’s mind, allowing him to come up
with a novel solution without feeling like his world
was crashing down around him. Experiments are not just a great way
of getting new information, they are also a way to open ourselves up
to new ideas and new experiences. So today, I would like to talk to you
about the beauty of testing. I want to tell you about how testing
helped me free my mind, and how it can free your mind, as well. On the 2012 Barack Obama campaign,
I was the director of Digital Analytics, and I saw first hand, how our organization’s strong culture
of testing really helped us, especially with our online
fundraising operation. Nearly all of our digital communications
went through a rigorous gauntlet of randomized controlled experiments. And in the online world,
we usually call these “AB tests”. And let me give you an example
of what this means. Pretend that all of you in this room
want to donate to the Barack Obama campaign. You come to our website, and you click on the button
that will take you to our donation page. We can use a little snippet of code
to divide you into two groups. This side of the audience, we are going to call you group A,
and you are going to see this page. You will have to fill out
a bunch of information in all the fields, and then click the “Donate” button
at the bottom of the page. This side of the room,
you are going to be group B, and you will still have to fill in
all the same information, but only a little bit at a time. On this new donation page, you have to say
how much you want to donate, then it flips you to another tab where you enter in
a little more information, and so on, until finally, you hit the “Donation” button
at the end of that sequence. Whether you were assigned page A or B, not all of you would actually complete
the donation process. Some of you would get bored, some of you might not have
your credit card handy. Neither one of these two pages
is going to have a 100% completion rate. But if you, in this room,
behave the way our website visitors did when we first ran this test, about 5% more of you
on this side of the room would actually complete
the donation process versus those of you
on this side of the room. In other words, the experiment told us that our new donation page
was better that our old donation page, because we had shown each one
to an equivalent group of people, and gotten significantly better results
from one versus the other. Now if you’re thinking
that 5% doesn’t sound like any big deal, I understand, it’s kind
of a small effect size. But it is important to remember
that when we implemented this new page, after that point, we raised
few hundred million dollars in donations, and 5% of a few hundred million
starts to be real money. I should also mention
that after we implemented this new page, we weren’t done with the testing process. Our website development team
continued to test, and tweak, and find new ways to improve this page. And if that sounds daunting, I understand,
because it was actually a lot of work. But, you know, on the other hand,
it was sort of liberating to know that we didn’t have to get things
perfectly right, right from the start. A nice thing about testing is you can start with a new product
or a new web page and just continue improving
through testing. On the Obama campaign, we had the opinion
that everything should be tested. We tested our web pages,
our text messages, our online ads, even all our offline
communication, as well. But in our email program,
that’s where we got particularly complex. You see, millions of people had signed up for the Barack Obama
campaign’s email list, and that meant we had the sample size to test out lots of different messages
before we sent each full list email. Here is an example. So, at the end of August of 2012, we were
about to send out a fundraising email. We had six different possible drafts
that our email team had written that we could send out to the list. All of these six drafts mentioned
an upcoming fundraising deadline, but each draft had
a slightly different flavor. The first one mentions
the party conventions that were happening around that time. The second one talks about
our opponent, Mitt Romney. The third one is really long, but it talks about how the other side is
going to spend more money than we would. The fourth one is very personal, talking about how it was
Barack Obama’s last campaign. The fifth one mentions
that the election is getting closer, and finally, the sixth one
talks about collective action, how we all needed to come together
to re-elect the President. None of these messages
contradicted the others, they were just different flavors. Then, in addition
to all of these different drafts, we would also test out
different subject lines. Usually, about
three subject lines per draft. So in this case, we ended up
with 18 different versions of this email. And we sent each one of those 18 versions
out to about 1% of our list. We were looking for the email
that would get us the most people to click on the link
and finally donate to the campaign. And this is what the results looked like. There was considerable variation
across all 18 versions of the email. In fact, the email that won, which happened to be the one mentioning
Barack Obama’s last campaign, that got over twice as many donations as the worst performing email
among this bunch of 18. Now, this was really good news for us
because at this point, we’ve only used up about 20%
of our list running these tests. That means we have 80% of our list
left over to actually send that one really good winning email to. And once we did, we raised about 3.7 million dollars
just from this one email. If we had sent the worst version
among these 18 different variations, we would’ve raised about 2 million dollars
less for the campaign. It was really good that we did
all that testing because without it, we wouldn’t have known
what was the best and what was the worst. Believe me, we actually had
a betting pool within the office to see if anyone could accurately predict
what would win and what would lose, because obviously,
if we had perfect foresight, we wouldn’t have to do all this testing. I mean, it was extra work.
It would have been nice to avoid that. But it turns out we were terrible
at predicting the winners and losers. I think we were actually worse
than flipping a coin. What that told us is if even experts can’t accurately predict
what messages will win and lose, it’s just that much more important
to do a lot of testing and let the data tell us the answer. I found this testing process
to be very freeing, even comforting in a weird way. Because I knew that if I had an idea
that seemed a little crazy, it wasn’t a hard sell
to my colleagues to say, “Hey guys, let’s just try this out
on a small piece of our list.” Venturing into uncharted
waters is a lot easier if you only have to dip a toe in first. Of course, sometimes, our crazy ideas
turned out to be brilliant, and that was always a very nice feeling. There is nothing so awesome as being right
and having the data to prove it. Of course, sometimes we were wrong. For instance, one time, I had this idea that we should make
our emails look prettier. I had noticed that our emails
were looking kind of plain, and we had this beautiful website. So I thought, why don’t we make
our emails look more like our website? So we tried it. We put
some nice looking buttons into the email. Generally, it made the email look more
like those donation pages I showed you. And that failed. Making our emails prettier
did not increase donation rates, and my instincts were proven wrong. Really, testing is an incredibly good cure
for egotism I found. But the great thing about testing is now we were free to think
of new ideas and new theories. If pretty doesn’t work, how about ugly?
Can we make our emails uglier? So we tried it. We did a test where we added
ugly yellow highlighting to certain parts of the email, and that actually got us
significantly better results. From the failure of our first experiment, we achieved success
with our second experiment. And we found this over and over, that testing really allows you
to fail upwards. Of course, before you all run out and start sending ugly emails
to all of your friends, I should mention that the novelty of this ugly yellow highlighting
eventually wore off. And this is also something
that we found throughout the campaign. Tactics that would test well at one point
wouldn’t necessarily test well later, and so we always had to be coming up
with new ideas and new innovations. We are here today in Athens, and certainly, the scientific
method owes a lot to the great thinkers of Ancient Greece. But you don’t have to be
an ancient philosopher or even a modern scientist to use
this scientific method in your own life. Look around you and free your mind. Is there anything in your life
that you could turn into an experiment? Say for instance that you are someone
who uses online dating services, and you got a pretty standard pick up line
that you like to use. Maybe something like this guy
who messaged me recently, “You are a very beautiful woman.” Flattering, yes,
but it’s not very original. It’s pretty boring and impersonal. So maybe you decide you need something
a little bit better than this. Or at least something different. So maybe you want to try something like this other guy
who also messaged me recently, “Do you own a crock pot?” If it is something that doesn’t translate,
a crock pot is an electric cooking device. It is not something that comes up
in romantic conversations very much, which actually makes
“Do you own a crock pot?” a pretty unique and original pick up line. But it’s also pretty weird, and certainly, if you are trying to decide
between these two pick up lines, it’s not actually clear
which one is better. So you could test them. Every time you were online and you saw someone
that you wanted to flirt with, you could flip a coin and send that person either your original pick up line,
or your new crock pot pick up line. And I think before too long, you would probably start
to see one of these lines get more positive responses
than the other. Or maybe another possibility here is that
neither of these is any good whatsoever, and you really need to free your mind
to come up with better pick up lines. If you are looking
for more serious examples, certainly the business world
has a lot of examples to offer. Testing happens all the time
in the business world, and there was a great example
a few years ago, when some researchers worked
with a major hotel chain to test out different ways of asking
the hotel’s guests to re-use their towels rather than having the hotel staff
wash them everyday. You might have seen these little cards
in hotel bathrooms where they ask you to save the environment
by re-using your towels. Well, the researchers tested that
and randomly divided guests to receive either that environmental message
or a different message that simply said that re-using towels
was the popular thing to do. They were just told that most other guests in their hotel
had chosen to reuse their towels. And that second message,
the one which a social pressure, that actually worked significantly better
than the environmental message. So, if you own a business
or you are thinking about starting one, think about ways that you can adopt
the testing mentality into your business. Think about ways that you can use testing to improve your profits
or improve your costumer service. Let testing be something
that inspires you to innovate. As you look around your life, and think about ways
to incorporate experiments, let me leave you
with these five suggestions. These are my five ways
to use testing to free your mind. Number 1: start somewhere. If you force yourself to be perfect
right from the start whatever the project, you are probably going to be
more risk averse and less creative. The great thing about testing
is it frees you from this pressure because you know
that no matter where you start, you are always going to be able
to keep improving through testing. Number 2: fail upwards. On the Obama campaign, we had to learn
to be humble because a lot of times, testing would prove our instincts wrong. But on the other hand,
we also had to learn that failure is just another opportunity
to consider new ideas. Number 3: test all the time. If you make it a priority to do testing
anytime the opportunity presents itself, you will start coming up
with more creative ideas just to meet those opportunities. On the Obama campaign,
we had kept a testing calendar, and anytime we saw
a blank space in that calendar, we knew it was time
for a brainstorm session. Testing all the time forced us
to be more innovative. Number 4: surround yourself
with diverse perspectives. The people on my digital
analytics team on the campaign came from all sorts
of different backgrounds. We had people who had worked in politics, but also people from computer science,
financial analysis, and marketing and even, no joke, particle physics. What this meant is that we all had different approaches
to the testing process, and we ended up learning
a lot from each other. So as you seek to incorporate
testing into your life, make sure you are surrounding yourself with people who have
different perspectives. Maybe even different agendas. Certainly, the athletic director
at Harvard found that out when my mother came into his office. It’s good to get different perspectives. And then finally: trust in testing. After the campaign ended,
people would always come up to me and say, “Amelia,
can you tell us, you know, what did you learn on the campaign
that we can use in our campaign?” Well, the truth is there are
no universal best practices. Positive test results don’t last forever, and what works in one context
may not work in another context. The only true best practice
is testing itself. So, get out there and do some testing.
It will free your mind. Thank you very much.
(Greek) Thank you. (Applause)

, , , , , , , , , , , , ,

Post navigation

3 thoughts on “Testing, it will free your mind: Amelia Showalter at TEDxAthens 2013

  1. This lady helped Obama get elected?

    I have her test results, She is a horrible person. I'm serious, that's what it says: A horrible person. We weren't even testing for that.

    But if this lady is reading this comment, I have to say; "Don't let that 'horrible person' thing discourage you. It's just a data point. If it makes you feel any better, science has now validated your birth mother's decision to abandon you on a doorstep"

    But anyways, people do horrible things for money. Such as sexual favors for money and helping liars get elected.

    And if this lady helped the Mormon get elected she would still be a horrible person.

    But it's ok, I have some good news. I figured out what to do with all the money the world will save when she stops breathing.  When she dies, I'm going to laminate her skeleton and pose her in the lobby at Harvard. That way future generations can learn from her and how not to have her unfortunate bone structure.

    ~Sincerely,
    GLaDOS

  2. This is brilliant Amelia.  So cool to see you do this, and to hear about your experiences. As a teacher in today's world, more and more I am faced with collecting data, determining what the data means, and then how to use it. I feel like the points you make at the end really help to put "data" into perspective.  I find this so relevant to what I do in so many ways, and I think its neat to see the connections even though we do totally different things for our jobs. 

Leave a Reply

Your email address will not be published. Required fields are marked *