Replication Crisis

The Replication (Reproducibility) Crisis has been discussed a lot lately:

1. https://www.vox.com/future-perfect/21504366/science-replication-crisis-peer-review-statistics
2. https://www.nature.com/articles/533452a
3. https://www.ipr.northwestern.edu/news/2024/an-existential-crisis-for-science.html
4. https://www.nature.com/articles/s44271-023-00003-2
5. https://www.psychologytoday.com/us/basics/replication-crisis
6. https://www.vox.com/science-and-health/2018/8/27/17761466/psychology-replication-crisis-nature-social-science
7. https://pmc.ncbi.nlm.nih.gov/articles/PMC5579390/
8. https://www.newscientist.com/article/mg25433810-400-the-replication-crisis-has-spread-through-science-can-it-be-fixed/
9. https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/
10. https://plato.stanford.edu/entries/scientific-reproducibility/
11. https://www.npr.org/2016/05/24/477921050/when-great-minds-think-unlike-inside-sciences-replication-crisis
12. https://www.annualreviews.org/content/journals/10.1146/annurev-psych-122216-011845

DARPA Score

A long time ago a proposal of mine advanced through the first rounds of a DARPA RFI (Request for Information) Challenge (I subsequently proposed withdrawing after learning that the Center for Open Science got involved - they were among the first to point out the problem in the first place - submissions were blind so I didn't know if any major institutions were already involved):

1. https://www.wired.com/story/darpa-wants-to-solve-sciences-replication-crisis-with-robots/
2. https://www.vox.com/science-and-health/2019/2/25/18211125/darpa-score-center-for-open-science-ai
3. https://www.wired.com/story/darpa-bs-detector-science/

I'm not sure if very much has happened since. It seems like the Replication Crisis has just deepened with no end in sight.

This is an interesting question that involves many areas of traditional Philosophy of Science:

1. The Scientific Method - championed and first articulated by the Logical Positivists, Logicial Empiricists, and Rudolph Carnap.
2. The Objectivity of Science (e.g. - that Peer Review is an essential part of Natural Science and that anyone should be able to replicate experimental results provided follow the same steps/setup).
3. What is a Scientific Theory?
4. How do we distinguish Science from Pseudo-Science?

Some Thoughts

It seems that lines of critique concern Methodology - (which mirrors traditional challenges from academic philosophy that have gone mostly ignored).

1. Such challenges involve issues Term Ambiguity, Multiple Realizability, Eliminativism, Social Constructivism, etc. as being hard limits on the objectivity of the Social Sciences.
2. Problematically, the Replication Crisis has started to "spread" to "infect" other Sciences (and ones not traditionally considered to be "Soft Sciences" - Geoscience, for example).
3. That certain standard Mathematical practices are improper, incorrect, are compatible with high levels of inaccuracy, or have serious limitations that have been understood or acknowledged (compatibility with Sum of Squares, Null Hypothesis testing, etc.).

A few possibly unoriginal hypotheses to throw into the ring:

1. Journals - that the Replication Crisis originates from or with the broken Publication process. That although Peer Review is and should be an essential part of the Scientific Method it, should be tweaked or refined as it currently exists. Specifically, that the monopoly on academic publications and lowered standards for Peer Review have created a culture of lessened Scientific objectivity.
2. Science isn't as Objective as we all think (and never has been) - this view that the startling findings from the Replication Crisis stem mostly from the first systematic look at the Objectivity by Scientists. So, following thinkers like Thomas Kuhn there really is a difference between Normal Science and Breakthrough Paradigm Shift Science (and one of those differences is the variation of levels of Objectivity). Like most human activities, Quantity exceeds Quality (and Quality is rare).
3. Academic Bureaucracy and Publish or Perish - contemporary Academic standards have resulted in poorer Quality of scientific output stemming from unreasonable emphasis on Publications. Given the way that most institutions calculate the supposed "impact" of a Paper (typically measured solely on the number of citations it receives) and recent findings that (a) Nobel Prize level work is often the very most cited but (b) so too is the mostly abysmal work, this shouldn't come as a surprise.

I don't think I've seen much said about the above. If correct, the following might be helpful to resolve the crisis:

1. Shakeup Academic Publication - change the way that Peer Review operates, emphasize Quality over Citations (alone) through improved Metrics - most universities emphasize Publish or Perish (and so it seems that the Quality of Publications have simply declined while all kinds of Publications have proliferated). Why does it matter to Publish so much? I think one or two really impactful Research Papers that profoundly and Objectively change a field would be far valuable.
2. Carefully rethink Foundations - consider the advice of over a century of Academic Philosophy.