Research reproducibility ensures that scientific results are valid and reliable. Reproducibility is essential so you and other scientists can build upon existing research and advance scientific knowledge. It also builds trust in the scientific method. Proper labeling, controls, and publishing detailed methodology sections and negative data make your work more reproducible and maximize its impact. Avoiding expired reagents, assigning work appropriately, and being mindful of external factors such as mood and fatigue also help.

You’ve just completed an experiment and obtained your results. Congratulations! You’re one step closer to expanding the knowledge base and bringing us to a closer approximation of the truth.

But how can you be sure that your findings are correct?

Performing reproducible research is critical in science, and there are steps you can take at each stage of your research to help ensure that it is reproducible. This article discusses eight steps you can take to improve your research reproducibility.

What Is Research Reproducibility?

Research reproducibility, repeatability, or other similar terms, depending on whom you ask, is the ability for your findings to be obtained again by yourself and other researchers.

Why Does Reproducibility Matter?

Science is a toolbox for seeking objective truth, and a toolbox is much more useful if the tools are reliable and the person using them is consistent and knows what they’re doing. Research is similarly more helpful if it is reproducible, and for several reasons:

  1. Lab mates come and go, and new people will continue your work.
  2. Other scientists will likely expand upon your work.
  3. Irreproducible findings prevent or hinder our pursuit of truth.
  4. Reproducible results further our quest for truth.
  5. Reproducible findings build trust in our toolbox.
  6. Attempting to replicate findings identifies broken or outdated tools in our toolbox.

How Do I Perform Reproducible Research?

At each stage of your research, from beginning to end, there are steps you can take to help ensure the reproducibility of your findings.

1. Properly Label Everything

Before starting your experiment, it’s crucial to ensure that all your reagents, materials, and tools are correctly labeled. What constitutes proper labeling will vary depending on what is being labeled, but essential information  typically added to labels includes:

  • Full substance name, chemical, reagent, buffer, etc.
  • Key properties such as concentration, dilution, weight, volume, etc.
  • Dates on which the substance was received and first opened
  • Name and/or initials of the person who prepared or received it

If your laboratory materials are improperly labeled, your research may be doomed to be irreproducible before it even begins because you cannot source or make identical reagents and solutions. To return to the toolbox analogy, imagine you’ve been told to finish screwing in flathead screws. If you accidentally use a Phillips head screwdriver because it was labeled as a flathead, you most likely won’t reproduce the quality of the work done with the proper tool.

It’s much easier to tell apart a flathead and Phillips head screwdriver than to tell apart two scientific reagents that look identical yet differ in unobservable ways. If you want to set the stage for reproducible research, properly label everything.

2. Avoid Expired Reagents or Perform Quality Control Tests

You’ve ensured you’re using the right tools for the job, but do you know how long those tools last? Do the chemicals in your lab degrade over time? Have you checked whether expired reagents change the results you get?

Replacing expired reagents with unexpired ones will massively improve your research reproducibility. Manufacturers often employ rigorous testing on their supplies or reagents. These tests inform expiration dates and should not be ignored entirely. Expiration dates are also informed by (among other things) regulatory requirements and may not perfectly reflect the reagent’s stability or usability.

Buying new materials also costs money—money that may not be there if a grant has yet to be funded.

Whether for this reason or others, you may wind up in a situation where you are using expired reagents.

The good news is you can ensure greater research reproducibility if you perform quality control tests to confirm your materials’ continued stability or usability.

These tests vary in design, and what they investigate. However, showing that unexpired and expired reagents generate the same results when everything else is kept constant can prevent confounding factors from impairing the reproducibility of your work.

When performing these quality control tests, there will likely be times when somebody in your laboratory discovers that a reagent or substance works perfectly past its expiration date. It’s important to consider that you may need additional quality control tests as expired reagents continue to age further.

Quality control tests themselves take time and resources. Ideally, avoid using expired reagents altogether. But if you have no other option, perform quality control tests.

3. Design Your Experimental Protocols in Detail

You have the right tools, and they’re all working properly. Now it is time to design your experiments. When writing your protocols, pay immense attention to detail.

Researchers find several reasons to avoid including an appropriate amount of detail. I’m sure you’ve heard some of the following:

  1. “I have a really good memory, and I’ll remember what I did or how to do the experiment.”
  2. “We need to cut down on the methods section to meet the word limit.”
  3. “This step doesn’t need to be done exactly the same each time.”
  4. “Certain details are unnecessary.”

We often give ourselves and our memory too much credit. It is easy to forget simple steps or mix up numbers in our heads. It’s also not the case that you will be the only one to perform a particular experiment. A cornerstone of research is the ability of other researchers to replicate our findings. And other researchers cannot rely on our memory alone.

Concision vs. Detail

Now, there are examples of times when concise writing is important, and, at least in their current form, journal article submissions are among these examples. Keep in mind though, that concision often opposes detail.

That’s not to say that, despite submitting a more concise methodology as part of an article, you can’t also maintain a more detailed and precise protocol elsewhere. Good examples of places you can save a more detailed protocol are:

  1. Your laboratory notebook.
  2. A dedicated binder in the lab.
  3. A network drive.

These more detailed protocols can then be referred to by other members of your lab, or even sent to other researchers should they reach out about an article you’ve published. When writing these detailed protocols, it’s important to avoid leaving out the information you take for granted and deem unnecessary.

Here’s an example of information that’s easy to take for granted.

A protocol may state, “perform the experiment using ten million cells, counted using a hemacytometer”.

This may seem to give plenty of detail, as the required number of cells and the technique used to count the cells are listed.

But were there any clumped cells when you counted? Did you count both living and dead cells, or only living cells? How much of your cell suspension did you load into the hemacytometer? Did you dilute your cell suspension before counting? What calculations did you perform to convert from your hemacytometer counts to your final cell count?

It’s imperative that details such as these are communicated and shared. Steps such as these are often assumed to be performed equivalently between researchers. Especially researchers working within the same lab. Or they’re wrongly assumed not to impact the results.

However, it could be the case that using 9 or 11 million cells, due to different counting methods, results in different outcomes. Even if steps and details seem trivial, it is better to be safe and design your experimental protocols in detail than to risk irreproducible research.

4. Include Appropriate Controls and Practices

Part of designing your experiment is choosing which controls to include. Without the appropriate positive and negative controls, it is much more difficult to determine whether the results you observe are genuinely due to your experimental variable or confounding factors.

Which controls are appropriate for your experiment will vary, but it is very rare for it to be the case that no appropriate controls exist. Some important types of controls include (but are not limited to):

  1. Positive Controls. Known to produce the expected effect.
  2. Negative Controls. Known not to produce the expected effect.
  3. Procedural Controls. Such as taking measurements twice to reduce potential error.

In addition to controls, several practices can (and when possible, should) be incorporated into experiments to limit error, prevent bias, and increase reproducibility. These include (but are not limited to):

  1. Randomization of group assignments or other factors when appropriate. [1]
  2. Blinding, or not providing certain information about samples, participants, or data either to researchers or participants to prevent biased collection or interpretation of results. [2]
  3. Consistency in otherwise overlooked experimental elements such as time of day. Even something such as the sex of the researcher could impact animal study results.

Confirmation biases, confounding variables, and other things that will otherwise cause your research to be irreproducible are much more easily identified if you include appropriate controls and practices in your experimental design and protocols.

5. Assign Work Appropriately

You’ve written your protocol, you have the right controls, and it is time to perform the experiment. Who in your lab will perform each step? Do you have collaborators in other labs that will be performing parts of the experiment?

Collaboration in a lab is often a gateway to success and can bring together different expertise and skill sets to tackle a research question. However, it is important to allocate experimental responsibilities appropriately.

In any lab, individuals will have different strengths and weaknesses, and various levels of experience. This is especially true if your lab includes trainees such as students and fellows (at any stage). While it may seem obvious, lab members should be fully trained to handle any experimental steps that they are assigned.

For example, imagine you have a lab member who is an expert at performing Western blots, and another lab member who is an expert at performing PCR. This provides an excellent opportunity for cross-training and sharing knowledge. However, until cross-training is complete and both lab members are fully equipped to perform both techniques, Western blot, and PCR experiments are more appropriately handled by the respective experts in the lab.

Who is fully equipped to perform different techniques will change over time as cross-training occurs, and lab members come and go. For this reason, it is vital to keep this step in mind and assign work appropriately for every experiment.

6. Publish Honest Methodology Sections

Now that you’ve completed your experiments, you’re ready to publish them in a journal. Congratulations! As I’ve said, you may have to shorten your protocols and methods to meet the journal’s word limit and help the paper flow or read better.

You’ve probably read many papers where the authors will say one of the following:

  1. “According to manufacturer recommendations.”
  2. “As per Smith et al.”
  3. “As previously described elsewhere.”

These strategies cut down the length of the article to meet word limits and remove details that are perhaps unnecessary. However, it also introduces the potential to leave out critical methodological details or misrepresent the work performed.

If you intend to describe methods as being done “according to manufacturer recommendations”, be sure that the manufacturer’s recommendations were followed exactly.

Often, manufacturers provide a range of recommendations and encourage researchers to begin within that range and optimize as needed. Other times, manufacturers will provide recommendations for one context, but not the exact context in which your experiment was performed. These are both examples of instances where it is impossible to give all the details by describing methods in this manner.

Similar problems can arise when describing methods as being done according to another article, whether your own or another. The cited article may not provide enough detail to replicate the methods, or it may similarly cite an earlier article from which the methods were taken, leading to a reproducibility rabbit hole.

This is not to say that credit should not be given when methods are taken from another study, but where possible, those methods should be fully and exactly described. If it isn’t appropriate to include this information in the main publication, including it in supplemental materials or publicly reporting it elsewhere should be prioritized.

As a last resort, if publicly reporting the full methodology is not possible, researchers should be able to reach out to you to obtain the full methodology (remember Tip #3!). However you handle the final publication, be sure to publish honest methodology sections so that your work can help us reach a closer approximation of the truth.

7. Include Negative Data and Correct Publication Errors

Researchers often only publish positive findings without publishing associated negative findings. Publishing negative data can help other researchers not only know what to do to replicate your results but also what not to do.

As the negative data may not be the sexiest part of your results, it may make more sense to include them in supplemental materials—but they should still be included.

After publishing your research, including both your positive and negative findings, you may discover an error in your published article. Whether the error is as simple as a typo or misplaced decimal point, or as drastic as incorrectly analyzed or presented data, prioritize its correction to avoid the irreproducibility of the findings. 

Even if this means retracting your article, it is better to have the error corrected than to have it continue to misguide other researchers into performing irreproducible research. No one enjoys being wrong or explaining how they’ve failed. But to improve your research reproducibility, and the reproducibility of researchers who build upon your findings, be sure to include negative data and correct publication errors.

8. Be a Mindful Researcher

Even if everything else sets the stage for a successful and reproducible experiment, differences in researcher behavior could impact results or the ability to perform experiments reproducibly.

For example, are you too hopped up on caffeine to pipette accurately? Conversely, are you too tired to focus and avoid mistakes? Or maybe you’re distracted by things going on outside of work.

Finding the right mental space to perform your experimental steps accurately and precisely is crucial, and not something to be ignored or taken for granted. Even a perfect toolbox is most helpful in the hands of a mindful worker. If you want your research to be reproducible, it’s important to not only perfect your toolbox but also be a mindful researcher while using that toolbox.

Research Replication in Summary

There you have it! Eight actionable tips that will immediately improve the reproducibility of your research, and that will make you a better scientist. To help you remember these key points, we’ve summarized them in a simple table (see Table 1).

Table 1:

How Do I Perform Reproducible Research?

Description

Properly label everything

Avoid using the wrong reagent by keeping them well-labeled.

Avoid using expired reagents or perform quality control tests

Remove reagent degradation as a possible confounding factor.

Design your experimental protocols in detail

Properly balance concision and detail to ensure that your work can be replicated exactly.

Include appropriate controls and practices

Ensure that your findings are not due to confounding factors or biases.

Assign work appropriately

Rule out technical inexperience as a possible confounding factor.

Publish honest methodology sections

Avoid unintentionally misguiding other researchers.

Include negative data and correct publication errors

Tell the truth, the whole truth, and nothing but the truth.

Be a mindful researcher

Set yourself up for success and avoid mistakes.

Whether you’re at the experimental design stage, in the middle of performing your experiments, or have already collected your results and are seeking to publish your findings, these tips can help ensure that you continue to use, build, and improve upon the truth-seeking toolbox of science.

Got any advice on making science more reproducible? Share them in the comments section!

References

  1. Suresh K (2011) An overview of randomization techniques: An unbiased assessment of outcome in clinical research. J Hum Reprod Sci 4(1):8–11
  2. Bespalov A, Wicke K, and Castagné V (2019) Blinding and Randomization. In: Bespalov A, Michel M, and Steckler T (eds) Good Research Practice in Non-Clinical Pharmacology and Biomedicine. Handbook of Experimental Pharmacology, vol 257. Springer, Cham

More 'Basic Lab Skills and Know-how' articles

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.