Better Inference in Neuroscience: Test Less, Estimate More

Estimation for Exploration

As in other sciences, neuroscientists pursue projects where the data takes them, with each new result spawning new questions. Forging a trail of discovery requires exploration. New manipulations and new assays may have to be brought into the lab, and the exact parameters for a sensitive experiment may not be immediately obvious. Moreover, we are often screening for important factors rather than testing well-formed hypotheses, cycling through a range of possibilities of about equal promise.

Exploration is essential for fruitful science, but we recognize that the hypotheses that emerge are especially tenuous because of the numerous opportunities exploration provides for capitalizing on chance. For example, a lab may screen inhibitors of five different signaling molecules before obtaining a statistically significant effect on a behavior of interest. While the significant result is intriguing, the multiple tests conducted provide multiple chances for spurious findings, and this risk would be further increased if the researcher has used the achievement of statistical significance to guide decisions about adding samples, selecting an analysis, or refining exclusion criteria.

Exploration needs to be capped off with strong tests or reported in ways that makes the tentative nature of the conclusions clear. This is not always the practice in neuroscience. Instead, the first significant p value found during exploration is often what ends up published, presented as confirmatory and usually without mention of other factors that were screened but found nonsignificant. When exploration regularly masquerades as confirmation, the research literature produced will be unreliable: the practical significance of true findings will be exaggerated, and spurious claims will be unacceptably prevalent. It is hard to gauge the extent of this problem in neuroscience, but the clear excess in significant findings across the published literature is troubling (Button et al., 2013; Szucs and Ioannidis, 2017), suggesting that relevant nonsignificant results are either discarded or unduly coaxed under the threshold for statistical significance. More informally, most researchers in the field can share stories of frustration encountered trying to build from ostensibly solid findings from the published literature.

The solution is not to universally impose requirements for rigorous testing. At early stages of the research process, asking a researcher to preregister their hypothesis and sampling plan would only lead to frustration, evasion, or pro forma efforts that provide a veneer of rigor. Instead, the solution is to use estimation to guide exploration, and then to conduct strong tests as the proper capstone of the research hypotheses that emerge.

Given that current practices often obscure the distinction between exploratory and confirmatory research, would it be possible to adopt reforms that draw a more clear line between them? Yes. Confirmatory research is research that puts your hypothesis at risk, where a negative result is interpretable and would change your thinking in a meaningful way. Other sources discuss at length what is required to conduct what has been called a severe test (Popper, 1959; Mayo, 2018; Scheel et al., 2020), but for NHST some key requirements would be:

• A clear and quantitative prediction derived from your research hypothesis.

• A sample-size plan that will provide a high-powered test of your prediction.

• An analysis plan with limited flexibility, with a priori specification of what data will be counted as valid, what analysis strategy will be used, and what standards will indicate confirmation and disconfirmation of the prediction.

A researcher about to conduct a confirmatory hypothesis test should find it straightforward to preregister their study and/or undergo preregistered review.

Properly understood, much of what neuroscientists currently report via NHST as though it was confirmatory should be reported via estimation as exploratory. For example, a researcher may read that PKM inhibitors extend fear memory in rats, and from that may hypothesize that she will observe the same effect in eye-blink conditioning in rabbits. Is this confirmatory research ready for NHST? Probably not. First, the researcher probably does not have a clear prediction yet (Would the effect be just as strong as what was reported for fear memory? Perhaps a smaller effect should be predicted based uncertainty in the original finding or ceiling effects in the eye-blink assay?). Without a clear prediction, a power analysis cannot be conducted to determine an adequate sample size. Moreover, important procedural details will start off as informed hunches (What dose? What time point? What criteria for exclusion because of side effects?). Even if the researcher's guesswork does yield p < 0.05 on the first batch of animals, this is still an exploratory and tentative finding that should be reported via estimation. Why? Because this initial stage of “throwing stuff at the wall” does not put the researcher's (vague) hypothesis at risk: any nonsignificant result would be sensibly dismissed as a sign of bad luck in adapting the protocol rather than as evidence that some substantive aspect of the researcher's thinking is flawed. If a negative result would not convince you that you have got something wrong, then you are not testing your hypothesis.

Replacing our illiberal use of testing with estimation could have several benefits. First, estimation more clearly focuses on uncertainty. Rather than categorical claims (Here we showed protein X enhances LTP, p = 0.04), researchers would emphasize the range of effect sizes compatible with their results (We estimate protein X enhances LTP, but it is not yet clear whether this is an infinitesimal, small, or moderate facilitation, Mean increase = 20% 95% CI [1, 40]). Although this is just a different way of summarizing the same data, the estimation frame more clearly indicates the need for additional research to confirm a meaningful effect. Under our current practices, we often create the illusion that this confirmatory step has already succeeded.

Another benefit of estimation is that estimates are not categorized as significant and nonsignificant. Estimation might therefore support more fully reporting screening activity (We also screened protein Y, finding a mean increase of 5 ± 19%, and protein Z, finding a mean 0 ± 23%; we did not further follow-up on these observations). Our current practice of underreporting nonsignificant results leads to waste and makes it impossible to extract accurate knowledge from the published literature.

A third benefit of estimation is that it provides sample-size planning that is suitable for exploration. Under current practices, sample-size planning for NHST seems to be rare (Szucs and Ioannidis, 2020) or poor (Goodhill, 2017), and sampling-to-significance may be common (John et al., 2012; Buchanan and Lohse, 2016). This is a sign that NHST is often being applied too early, before the researcher has developed a quantitative prediction and other inputs that are required to plan a strong hypothesis test. Estimation is more suitable for exploration, allowing researchers to plan for the precision of their estimates rather than the power to detect a predicted effect (Cumming and Calin-Jageman, 2017). With planning for precision, the researcher specifies a desired level of accuracy (say ±30% of the observed effect) and then either predetermines the sample size likely to be needed or collects data until the desired precision is obtained (Kelley et al., 2018). Planning for precision lets the researcher efficiently and iteratively invest resources suited to the value of an accurate answer.

Another benefit of estimation is that it can help us better calibrate our expectations for future research. It turns out the p values are surprisingly erratic (Cumming, 2008), and they elide much of the information needed to judge how surprising a new result is relative to a previous one. For example, imagine you recently discovered a statistically significant effect of an antioxidant on long-term memory (t₍₈₎ = 3.34, p = 0.01). You then repeat the experiment with some extension to probe mechanism (e.g., while also inactivating a signaling pathway you think might mediate the effect). In this follow-up study, however, the basic effect of the antioxidant on memory turns out nonsignificant (t₍₈₎ = 1.2, p = 0.25). Based on significance status alone, you might be tempted to judge the follow-up finding as surprising, potentially even as an indication that something in your protocol has gone wrong. The reality, however, is that this is an unexceptional sequence of results. From an initial finding of p = 0.01, we should expect exact replications to yield p values mostly in the range of 0.00001 to 0.41, with only 33% also turning out statistically significant at the 0.05 level (Cumming, 2012). If that seems surprising, you are not alone; researchers can underestimate the expected variation in p values (Lai et al., 2012). Thus, judging by p values alone risks conflating normal sampling variation with substantive differences in results. Estimation can help. Consider the same set of results expressed as estimates: d = 1.9 95% CI [0.65, 4.38] for the first study; d = 0.70 95% CI [–0.58, 2.46] for the follow-up. This way of expressing the results makes clear that the difference between studies is fairly small and unsurprising relative to expected sampling variation.

Estimation for Planning Strong Tests

In addition to expanding the use of estimation, we should improve the use of testing. A meaningful test is a well-planned test: one where the researcher's hypothesis has been clearly articulated, a sample that will provide an interpretable result has been planned, and there is limited room for post hoc flexibility in the filtering and analysis of the data collected.

Estimation is helpful for planning strong tests. First, estimation yields effect sizes and uncertainty, the key inputs a researcher needs to ponder when determining a reasonable sample for NHST. Second, estimation makes it possible to adopt a fair testing procedure, one that puts both the null and the researcher's hypothesis at risk (Fig. 1). This requires one extra step before data collection: the researcher must define the range of effects they consider negligible for their research purposes. Then, when the data are collected, two tests are conducted: one to test the researcher's hypothesis that the effect is substantive (called a minimal-effect test), the other to test the alternative hypothesis that the effect is negligible (called an equivalence test). If both tests are nonsignificant, the test is deemed uninformative. This has been called “inference by interval” (Dienes, 2014), because it is easy to understand and communicate this mode of testing through estimation (see Fig. 1): for an α of 0.05, the result is deemed substantive if the entire 95% CI is outside the range of negligible effects, negligible if the 90% CI is entirely within that range, and ambiguous otherwise (those confidence levels are not a typo; a 90% CI is used to see whether an effect is negligible because both ends must be inside the null interval). There are excellent tutorials on estimation by interval (Lakens et al., 2018) and accessible software resources for the actual computations (Lakens and Caldwell, 2022). Estimation by interval might be especially useful for neuroscience projects that achieve large sample sizes, as this mode of testing is not overwhelmed by sample size the way using NHST with a point null is.

• Download figure
• Open in new tab
• Download powerpoint

Figure 1.

Strong testing via inference by interval. In this approach to testing, the researcher defines the range of effect sizes that are negligible relative to the research question at hand (gray box with dashed border). Significance at α = 0.05 can then be seen graphically by plotting the effect size in the sample (triangle) with the 95% confidence interval (thin bar) and 90% confidence interval (thick bar). The effect is deemed substantive if the entire 95% confidence interval is outside the zone of negligible effects, negligible if the entire 90% confidence interval is inside this zone, and ambiguous otherwise.

Inference by interval is not the only way to conduct fair testing. Another option is the use of Bayesian statistics. The Bayesian approach quantifies the degree of support for the researcher's hypothesis and the degree of support for the null hypothesis. Results can be ambiguous between the two, in which case the study is deemed uninformative. There are multiple useful approaches to Bayesian hypothesis testing (Kruschke and Liddell, 2018), and the approach is increasingly accessible, with excellent tutorials and software resources available (Love et al., 2019; Keysers et al., 2020).

There are costs to these solutions. First, strong testing requires the researcher to make more judgements when designing the test. In Bayesian statistics, the researcher must specify their priors (the probabilities they assign to the range of possible effect sizes). For inference by interval, the researcher must decide what range of effect sizes they consider negligible. A second cost is in sample size: compared with our current practice of using NHST against a point null of exactly 0, Bayesian statistics tends to require larger sample sizes to reach a clear decision, and inference by interval even more so.

These costs are not trivial, but there is no scientific alternative. A testing procedure that can both confirm and disconfirm hypotheses is required for a science that is parsimonious, self-correcting, and credible.

Estimation for Interpreting Tests

When we have conducted a strong test, it is important to then interpret that test well. Current practices suggest this can be difficult. For example, here is a lightly-adapted and anonymized set of results from a recent issue of the Journal of Neuroscience: There was not a significant difference between treated mice and their controls in distance covered in an open-field test (t₍₂₀₎ = 1.8, p = 0.08), suggesting that Treatment X does not affect general locomotion. There was, however, a significant increase in freezing 24-h after associative fear-conditioning (t₍₂₂₎ = 3.8, p = 0.001). Taken together, these results show a selective effect on memory function.…Treatment X enhances memory.

While the details have been elided, this approach to using NHST should feel familiar; it is a fairly typical example of how NHST results are interpreted in the neuroscience literature. Unfortunately, what is typical is also quite dubious:

• A nonsignificant result is not a reliable indicator that there is no effect; instead, we need to conduct inference by interval or use Bayesian analysis to quantify support for the null (Lakens et al., 2018; Keysers et al., 2020).

• Comparing significance status is not a reliable way to determine specificity (Gelman and Stern, 2006; Nieuwenhuis et al., 2011); instead, this requires a formal comparison of results (a test for an interaction)

• A single significant finding cannot support sweeping generalizations about an effect; instead, we need to calibrate claims to the remaining uncertainty in the finding and rely on the regular occurrence of statistical significance across multiple tests for the clear rejection of the null hypothesis

Reporting estimates along with tests could help with each of these issues. Here is the same data reported and interpreted through an estimation lens: The difference between treated and control mice in an open-field test was compatible with treatment X producing a small decrease in locomotion, no difference, and up to a very large increase in locomotion (d = 0.73 95% CI [–0.12, 1.62]). For fear-conditioning, results were compatible with anywhere between a moderate to very large increase in memory (d = 1.52 95% CI [0.67, 2.43]). In our samples, Treatment X produced a larger effect on memory, but the estimated difference in effect is uncertain (d = 0.79 95% CI [–0.39, 2.00]).…We are uncertain about the effect of Treatment X on locomotion, but conclude that it produces at least moderate to potentially startling levels of memory enhancement. Collecting data to refine this estimated effect will be important, as if the true level of memory enhancement is only moderate (d = 0.67) it will require substantial resources (n = 48/group for 90% power) or a change in design to replicate and extend this finding.

With the estimation lens, these results provide a more clear-eyed appraisal of the knowledge that has been gained and what is left to still nail down. That is of critical importance for fruitful science: we cannot leverage previous findings if the tenuous often masquerades as the confirmed.