These famous psychologists would fail their IAs!

Travis DixonInternal Assessment (IB)

Can you see what's wrong with the graphs in these famous TED Talks?

I can barely get through a psychology TED™ Talk by a psychologist these days without sighing at the blatant manipulation being presented. In class, I can’t help but pause the video and comment. My students know what’s coming and I’m sure I annoy the heck out of them. But I think it’s important. Here’s why these psychologists would fail their IAs and why it matters.

Psychology is in somewhat of a crisis. It’s called the replication crisis (aka. the reproducibility crisis). Famous studies are failing to be replicated. This means the psychological phenomena we thought we understood, or even thought existed, might not be so. Why? Well, I think it’s because of our innate desire to prove our point, to show people that we’re right.

Read more:

If I’m completely honest, I think the desire to be famous might also play a part. It’s not hard to imagine that a lot of young Psychologists want to become the next Milgram, Zimbardo or Loftus.

You can download a step-by-step guide to the IB Psych IA.

But psychologists don’t get famous or even get their studies published, unless they are showing an effect. If the participants in the Stanford Prison Experiment sat around all day and did nothing, we’d never have heard of the experiment. If Asch’s subjects in his conformity studies didn’t conform, we’d never see those line cards again.

When psychologists are explaining their studies in peer-reviewed journals or in academic conferences, they can talk about p and z values and all manner of complicated statistics. But these mean little to the general public. They need visuals, metaphors and graphs. This is why you won’t see many TED talks about studies that don’t show some kind of graph.

So why do they annoy me? Simple. They lie.

Take this example from Lisa Welling’s TEDx Talk, “It’s not you, it’s my hormones.”

Welling’s graph supposedly showing the link between testosterone levels and face preferences.

See anything wrong? What about this even more egregious example?

I hope you can see it now. If not, focus on the y axis. The graph on the left would get marked down even in a middle school math class. If Welling submitted this for the IA, she could forget about getting top marks in the “Analysis” section. The graph is misleading, pure and simple. It exaggerates the effects that Welling is trying to explain. Interestingly, the study is labelled “unpublished.” I wonder why that is? Perhaps the results weren’t statistically significant? Or maybe it’s just waiting to be published.

The results, if we study the graph carefully, just aren’t that impressive. It’s about 67% compared to 65%. Barely an effect and yet this horrendous graph is desperately trying to falsely convince us otherwise.

What about this example from Larry Rosen’s TED talk on “Technology and the Brain, the Latest Research and Findings?” Notice anything similar?

And again, another even more preposterous example a couple of minutes later in the same talk.

It looks like a massive trend, but really the values only vary between about 2.8 windows open to 3.1. Barely a difference.

This is another reason why psychologists are reevaluating the use of inferential tests to determine the “significance” of data in psychological studies. This can also be misleading. Some results that have barely a difference between conditions reported as “statistically significant.” While technically this may be true, it still makes it sound better than it actually is.

My final example comes from one of the most famous, or now perhaps infamous, explanations of psychological research in a TED Talk – Amy Cuddy’s talk “Your body language may shape who you are.” I’m please she at least put my favourite word in the title – “may.” But why oh why does she try to persuade the audience of the greatness of the effect of powerposes with this graph…

In case you quite can’t see it, the y axis begins at 0.5 and goes to 1.0. The effect looks pretty impressive – do a high power pose and your tolerance for risk drastically improves. But does it? Actually, it’s about 0.85 in one condition, and 0.6 in the other. Not a massive difference.

Or is it? It depends on what those figures mean. It could be a massive effect, but Cuddy doesn’t label her y axis, so we have no idea what these scales mean. Nor does she explain the values in the talk to give the reader some context. Not even a little bit.

For IB Psychology students, this is important to remember. You’ll be desperate to prove an effect of some sort. But the aim of any study is not to prove anything – it’s to test to see if there is an effect in the first place. Also, don’t exaggerate your results. Analyze them carefully and draw rational and logical conclusions based on the data. Finally, definitely don’t manipulate your graph to try to show a big effect when it’s actually quite small.

Next time you’re watching a TED Talk and they show a graph, analyze it. I bet they probably do the same thing. Once you notice it, it’s impossible to ignore. But I think there’s a lesson here for all psychologists – our job is not to show impressive effects and discover new psychological phenomena – it is to learn the truth about the human experiences of thought and action. Let’s not forget that in the quest for fame.

OK, so maybe they wouldn’t “fail,” but they’d definitely lose marks in their IAs for their graphs. Maybe my title is guilty of the same thing I’m accusing these psychologists of – distorting my presentation of information to prove a point.