Deciding on what scientific studies to replicate
Peder Mortvedt Isager defended his thesis on May 24th at department of Industrial Engineering and Innovation Sciences.
To ensure that scientific claims are valid, researchers replicate or repeat each other’s studies to verify that original observations can be confirmed by an independent source. However, more and more scientific studies are being published every year, and there are not enough resources available to replicate and check all studies. Therefore, researchers need to know which studies are most important to replicate. PhD researcher Peder Mortvedt Isager has created a method for quantifying the potential value of replicating a study. The method could be used to create strategies for study selection that would greatly increase the return on investment for resources assigned to replication research.
Replication is an important part of quality control in science. Replication ensures that scientific observations are independent of the research leading to the observations, and it helps identify dishonesty, honest mistakes, and flukes in the scientific process.
Large scale replication efforts in social science suggest that many published findings may not be reproducible, making replication all the more important in this field. However, replication research is often not prioritized by researchers in practice. Studies suggest that replication studies make up less than 1% of empirical research in the social sciences. This has led to the so-called replication crisis in social science – replication studies are needed but are rarely carried out, which makes the validity of published research claims highly uncertain.
Recognized by stakeholders
Fortunately, the need for replication research is increasingly recognized by scientists and other stakeholders. Replication studies are slowly becoming easier to fund, conduct, and publish.
However, the number of non-replicated studies far exceed the resources available for replication. Therefore, researchers need to know which studies are most important to replicate. In other words, which replication studies are most valuable?
Assisting replication in social sciences and beyond
The goal of Peder Mortvedt Isager’s PhD research was to quantify the replication value of published studies, and to help create indicators of replication value that researchers can use to decide which studies in their field to prioritize for replication. As part of his research, Isager makes three important contributions towards achieving this goal.
First, he provides a general theory of what makes a replication study valuable, which did not previously exist until now. Second, Isager constructs a quantitative indicator of replication value from the theory. Third, he demonstrates how a quantitative indicator could be used to select studies for replication in practice. Further research is needed to determine if the indicator works well enough to be used for study selection in practice. If validated, the indicator could be a useful asset both to researchers who replicate studies and to funders of replication research. In addition, by demonstrating how a quantitative indicator of replication value can be built and justified, Isager’s research will hopefully make it easier to develop new and improved indicators in the future.
This research thus provides a roadmap for increasing the value generated by replication research in social science and beyond. In addition, this research is an important step towards better coordination in science more generally. By directing more time and resources towards those research questions that are most important and most uncertain, scientists can increase the value of science for the taxpayer who funds it.
You can also learn more about the research of Peder Mortvedt Isager in a recent Cursor article.
Title of PhD thesis: ‘Deciding what to replicate’. Supervisors: Chris C. P. Snijders, Daniël Lakens, and Anna Elizabeth van ‘t Veer (external co-promotor).
Latest news
