Neuroscience Needs to Test Both Statistical and Scientific Hypotheses

Alger’s defense of testing with p values is informed by a falsificationist approach to science, one in which scientists derive predictions from their theories, collect data, and then judge their predictions as corroborated or refuted.

Alger argues that estimation is not suitable for a falsificationist approach to science.  Are p values more suitable?  Under current practice: no.  Neuroscientists currently test the null hypothesis, not their own (often unspecified) predictions.  In fact, the current approach to using p values does not allow the researcher’s predictions to be falsified.  Non-significant results, which should be “valuable ‘negative information’ for scientific knowledge” are published at implausibly low rates. 

What would it take to use p values according to falsificationist ideals of science?  Estimation thinking.  That is, researchers would need to think more about effect sizes and uncertainty, enough to derive quantitative predictions from their hypotheses (If our theory is correct, CREB1 enhancement will boost memory by >30%).  Then, researchers would need to conduct tests that put those predictions are real risk.  That would involve, at a minimum: a) planning for an adequate sample size b) specifying an interval null hypotheses to provide a clear standard for falsifying their prediction (Our prediction will be falsified if memory retention changes by less than 20%).  Current use of p values is simply not falsificationist; estimation thinking could help reform it to the ideals Alger convincingly extolls.

Although Alger acknowledges that hypothesis testing is not the only mode of scientific inquiry, he approves of the fact that it is now “a principal method and is pervasive”.  Perhaps we should be troubled by the current hegemony of testing.  For example, Hodgkin and Huxley developed their model of the action potential (1952) without p values.  Moreover, their work does not reflect a falsificationist mode of inquiry, but rather a pragmatic epistemology best summarized by Box’s dictum that “all models are wrong, but some are useful” (1979, p. 202).  Hodgkin and Huxley gladly made use of wrong but useful simplifications; they noted discrepancies between model and data not as refutations but as topics for additional research.  This mode of inquiry seems fruitful and valuable for neuroscience.  Perhaps hypothesis testing, even if improved, should become less pervasive.

Finally, Alger presents a dour assessment of pre-registration, claiming it is not feasible at the cutting edge, would allow labs to be scooped, and lets researchers game the system.  It is true that pre-registration is impossible for researchers pursuing projects so novel that “too little is known to be able to specify in advance all of the relevant variables or the likely outcomes of manipulations”.   That is as it should be: by Alger’s standards, researchers so far out on the cutting edge are not yet ready to put their still-nascent theories to the test.  When researchers have derived clear statistical predictions from their scientific hypotheses, they will not find pre-registration difficult or time-consuming.  When that time comes, there is no need to worry about being scooped; the Open Science Framework allows an embargo period during which pre-registrations are kept private (but not forever!).  And yes, pre-registration can be gamed.  What’s notable is that the pre-registration process allows us to detect, quantify, and scold poor practices.  Under current norms we can only guess at the degree to which questionable practices underlie our sparkling published literature.  If we want improved testing with accountability, pre-registration is a useful tool. 

References

Box GEP (1979) Robustness in the Strategy of Scientific Model Building. In: Robustness in Statistics (Launer RL, Wilkinson GN, eds), pp 201–236. Academic Press.

Hodgkin AL, Huxley AF (1952) A quantitative description of membrane current and its application to conduction and excitation in nerve. The Journal of Physiology 117:500–544.