How to evaluate any study in 3 simple steps

Travis DixonResearch Methodology, Revision and Exam Preparation, Studies and Theories

Evaluating studies can be difficult, but this step-by-step guide will help get you started.

Being able to critically evaluate a study is a key skill for any budding psychologist. However, like anything, when you’re first learning how to do this it can be very difficult. In this post, we look at 3 simple steps you can take to evaluate any study.

Read More

Teaching Tip: You could use this as a class activity by having students choose one study and following these steps by themselves.

Step 1: Identify the Method

Read the study you want to evaluate carefully and try to identify the method that is used. How was the data gathered and analyzed? Was it a true experiment, a correlational study, a case study, a survey, an observation, etc.? Perhaps it was a twin study, a meta-analysis, or a longitudinal study. Before you can evaluate the study, you must know the methodology.

The IB has very strict definitions of “research method.” When evaluating a study, feel free to focus on any methodology used by the researchers.

Tricky Bits: Sometimes studies combine multiple methods. For example, it might have been a survey that also calculated correlations. Or a twin-study that was longitudinal in nature. This is fine. Identifying more than one method or technique that was used in the study is OK. In fact, it might make the next steps a little easier.

 

Step 2: Identify General Strengths + Limitations

After you’ve identified the method, it’s important that you can recall the general strengths and limitations of this method. The table below is a brief summary.

MethodStrengthsLimitations
True ExperimentCan control variables to establish a cause-effect relationshipGeneralizability to real-life settings could be an issue.
Quasi or Natural ExperimentCan study the effect of a naturally occuring IV that can’t be manipulated in the lab.Difficult to control all extraneous variables.
Field ExperimentCan study the effects of an IV on a DV in a real-life setting.Difficult to control all extraneous variables.
Correlational StudyCan test the strength of a relationship between two naturally occurring variables.Correlation does not mean causation.
ObservationsCan see how people act in real-life, naturalistic settings.Depending on the type (covert v overt), ethical issues or the Hawthorne effect might occur.
Survey/QuestionnaireCan gather data relatively quickly and easily from a wide range of people.Gathers self-report data, which may not be 100% reliable.
Case StudyCan use a range of methods to get an in-depth understanding of behaviour.Limited sample sizes, so generalizability could be an issue.

Exam Tip: For IB Psychology exams, I recommend becoming an expert on true experiments and correlational studies. This makes all questions regarding research methods and ethics more manageable. It will also help you develop your ability to evaluate these two methodologies.

99% of IB Psychology students only do the first two, so learning how to complete the third step will separate your answers from the rest.

Step 3: Apply to YOUR Study (+ -)

Examiners are not impressed by one sentence, generic evaluations.

Nearly every student in psychology only ever gets as far as these two steps. The classic example of a student evaluating an experiment goes like this: “But this was a laboratory experiment so it lacks ecological validity.” This is not critical thinking. This third step is essential for you to show your critical thinking. You need to take one of the strengths and/or a limitation and explain how it might apply to your study. This is difficult and takes time and careful consideration. The following table gives you some pointers for a few common methods you may be evaluating.

MethodTip for Evaluation
True ExperimentExplain WHY the results might not apply to a real-life situation or scenario.
Quasi Experiment .Explain how a specific extraneous variable might affect the results.
Case studyExplain WHY the results might not apply to another group or population.
Correlational StudyExplain possible bidirectional ambiguity, or how a third factor might explain the relationship.
Survey/ QuestionnaireExplain WHY people may not be 100% honest or accurate in their responses. Look at the nature of the topic they’re being asked about or the types of questions asked.

Example: Loftus and Palmer’s Car Crash Study (Read more)

This study was a true experiment because the researchers manipulated the independent variable (verbs in the question) in a controlled environment. One strength of this is the ability to control variables. In their second experiment, they had a control condition who didn’t watch any video. This helps to isolate the verb in the question as the single variable that’s increasing speed estimates and the memory of seeing broken glass. It can help control for demand characteristics because people may say “yes” to seeing broken glass because they may feel they’re supposed to.

However, it also means that we can question the extent to which this study may apply to real-life situations. To explain this properly you need to explain why it might not apply to a real-life situation by focusing on specific details (this video on our YouTube channel will help explain how to do this).

So a student’s evaluation might look like this:

One strength of this this experiment is they could control variables to establish a cause-effect relationship. In their second experiment, they had a control condition who didn’t watch any video. This helps to make sure the verb in the question is the only factor affecting speed estimates and the memory of seeing broken glass. It can help control for demand characteristics because people may say “yes” to seeing broken glass because they may feel they’re supposed to. Using all students also helps to control for participant variability, because people who are older may have more driving experience and be better at estimating speeds and so less likely to be influenced by a leading question.

However, it also means that we can question the generalizability of this study and the extent to which the results may apply to real-life situations. For example, could we expect the same effect in a court room or interrogation room involving a real life crime? In these situations there are massive consequences for false memory and misremembering key details, so people may focus more and try harder to remember. Perhaps even the stress could improve their memory of the event (because acute release of cortisol improves memory consolidation) so their memories will be more reliable.

Read More

You can see from the example above that a true evaluation of a study is difficult and takes very careful consideration. But if you can write a fully-developed explanation of the strengths and limitations of a study, you are on track to do very well in your exams.

Exam Tip: If you are asked in an essay question to specifically evaluate a study, you should include strengths and limitations. However, if you’re evaluating a study as one of your counter-arguments in a more general question, then just focusing on the limitations is OK. 

Watch more here…

Question or comment? Leave it below. 


Find more exam tips in our Revision Textbook designed for IB Psychology students.