Cover
Title Page
List of Contributors
Introduction
1. Psychological Science Under Scrutiny
2. The Replication Crisis
3. Other Challenges to Psychological Science
4. Media Attention
5. Scrutiny of Psychological Science as Healthy Self‐Criticism
6. This Book
7. Format and Table of Contents
8. References
Part I: Cross‐Cutting Challenges to Psychological Science
1. 1 Maximizing the Reproducibility of Your Research
  1. Project Planning
  2. Project Implementation
  3. Data Analysis
  4. Reporting
  5. Programmatic Strategies
  6. Implementing These Practices: An Illustration with the Open Science Framework
  7. Conclusion
  8. References
2. 2 Powering Reproducible Research
  1. Background
  2. False Positives and False Negatives
  3. Type S and Type M Errors
  4. Consequences of Low Power
  5. Why Does Low Power Persist?
  6. Possible Solutions
  7. References
3. 3 Psychological Science’s Aversion to the Null, and Why Many of the Things You Think Are True, Aren’t
  1. Falsification and Null Results
  2. Statistical Power
  3. The File‐Drawer Problem
  4. Moving From Protoscience to Science
  5. References
4. 4 False Negatives
  1. Introduction
  2. Dialectics of Allegedly Correct and Wrong Decisions in Statistical Hypothesis Testing
  3. Concluding Remarks
  4. Acknowledgment
  5. References
5. 5 Toward Transparent Reporting of Psychological Science
  1. Historical Context
  2. Disclosure Problem in Psychology
  3. Why We Seek Transparent Reporting
  4. Predicted Effects of (Non)compliance
  5. Psychological and Institutional Obstacles to Transparent Reporting
  6. Potential Remedies
  7. Relation to Other Extant Challenges
  8. Final Thoughts
  9. References
6. 6 Decline Effects
  1. Four Types of Decline Effects
  2. Separate Reflections on Unconventional Sources of Decline Effects by Schooler and Protzko
  3. Meta‐Science and the Empirical Unpacking of the Decline Effect
  4. Acknowledgments
  5. References
7. 7 Reverse Inference
  1. Thinking is Inferring
  2. The Basics of Reasoning
  3. Inferences in Science
  4. Reverse Inference Unleashed
  5. Conclusion
  6. References
8. 8 The Need for Bayesian Hypothesis Testing in Psychological Science
  1. The Logic of p‐values: Fisher’s Disjunction
  2. The Illogic of p‐values
  3. Generalizing Logic: The Bayesian Perspective
  4. A Concrete Example: Results from AUFP Re‐examined
  5. The Bayesian Remedy
  6. Concluding Comments
  7. Acknowledgement
  8. References
Part II: Domain‐Specific Challenges to Psychological Science
1. 9 The (Partial but) Real Crisis in Social Psychology
  1. The (Partial but) Real Crisis in Social Psychology Defined and Illustrated
  2. Why the (Partial but) Real Crisis in Social Psychology?
  3. Five Recommendations for Ending the Partial but Real Crisis
  4. References
2. 10 Popularity as a Poor Proxy for Utility
  1. Introduction
  2. Creating the Implicit Prejudice Meme
  3. Deconstructing the Implicit Prejudice Meme
  4. What Is Implicit Prejudice, and Why Don’t Its Measures Agree?
  5. Conclusion
  6. References
3. 11 Suspiciously High Correlations in Brain Imaging Research
  1. Challenges of fMRI Analysis
  2. Non‐independence/Circularity
  3. Associated Problems that Emerged in the Fallout
  4. Conclusions
  5. Appendix A: A Quick Tour of fMRI Analysis
  6. Appendix B: Power Calculations with False Discovery Rate Correction
  7. References
4. 12 Critical Issues in Genetic Association Studies
  1. Introduction
  2. Molecular Genetic Considerations for Genetic Association Studies
  3. Genetic Association Study Designs
  4. Leveraging Strength in Numbers: Multi‐site Studies of Genetic Association
  5. Consortium‐based Mega‐analysis
  6. The Issue of Heterogeneity in Multi‐site Studies
  7. Genetic Association Studies in Research and Application: Hyperbole and Hope
  8. The Implication of Issues in Genetic Association in the Study of Gene–Environment Interaction
  9. The Future of Genetic Association Studies
  10. Short‐term Benefits of Genetic Association Studies
  11. Long‐Term Goals and Considerations for Genetic Association Studies
  12. References
5. 13 Is the Efficacy of “Antidepressant” Medications Overrated?
  1. Fluoxetine: Creation of a Blockbuster “Antidepressant”
  2. The Efficacy of “Antidepressant” Medications
  3. Conclusion
  4. References
6. 14 Pitfalls in Parapsychological Research
  1. Introduction
  2. The Origins of Parapsychology
  3. From Psychical Research to Parapsychology
  4. Elusiveness of the Results
  5. Parapsychology: The Elusiveness Continues
  6. The Contemporary Scene: Conflicting Claims about Replicability
  7. Retrospective vs. Prospective Replication
  8. Bem’s Precognition Experiments: Confounding Exploratory with Confirmatory Research
  9. The Overarching Pitfalls of Parapsychological Research
  10. References
Part III: Psychological and Institutional Obstacles to High‐Quality Psychological Science
1. 15 Blind Analysis as a Correction for Confirmatory Bias in Physics and in Psychology
  1. Biases in the Research Process
  2. Corrective Practices in Psychology
  3. Blind Analysis in Physics
  4. Applying Blind Analysis to Psychology
  5. Discussion
  6. Appendix
  7. References
2. 16 Allegiance Effects in Clinical Psychology Research and Practice
  1. The Allegiance Effect
  2. Allegiance Effects in Psychotherapy Research
  3. Allegiance Effects in Assessment Research
  4. Allegiance Effects in Allied Fields
  5. Allegiance Effects among Practicing Clinicians
  6. Proposed Explanations for Allegiance Effects
  7. Proposed Remedies
  8. Conclusion
  9. References
3. 17 We Can Do Better than Fads
  1. Recognizing Fads
  2. Intense Peer Pressure
  3. Restriction of Range of Questions
  4. Costs of Fads
  5. Quality of Work
  6. Encouraging Career Choices on the Basis of Fashion Rather than Passion
  7. Fueling What May Be a Foolish Fad
  8. Choking Off Important Areas of, or Approaches to, Research
  9. Methods Rather than Substance
  10. Fundamental Values
  11. An Alternative Approach
  12. References
AfterwordCrisis? What Crisis?
1. Whose Crisis?
2. The Limits of Replication
3. Our Subjects Are WEIRD
4. References
Index
End User License Agreement

List of Tables

Chapter 01
1. Table 1.1 Increasing the reproducibility of psychological research across the research lifecycle.
Chapter 02
1. Table 2.1 Beliefs about sample size and statistical power.
Chapter 03
1. Table 3.1 Probabilities of correct and incorrect decisions.
Chapter 08
1. Table 8.1 Bayes factors for different priors. BF = 1/BF₁₀.
Chapter 12
1. Table 12.1 Practical considerations for conducting or evaluating a genetic association study.
Chapter 13
1. Table 13.1 Problematic design features in antidepressant trials conducted by pharmaceutical companies.
2. Table 13.2 Mean improvement (weighted for sample size) for fluoxetine and placebo in trials submitted to the FDA and published versions of the FDA trials.
3. Table 13.3 Problematic reporting features in antidepressant trials conducted by pharmaceutical companies.
Chapter 15
1. Table 15.1 Simulation 1: F Statistics.
2. Table 15.2 Simulation 2: F Statistics.

List of Illustrations

Chapter 02
1. Figure 2.1 The winner’s curse. Note: Suppose the true population effect is μ, but the study testing for that effect is small and lacks power. Only those findings that by chance happen to be much greater than the true effect will pass the threshold for statistical significance (α = 0.05). This is often referred to as winner’s curse; the scientists whose study yields results that pass the threshold for statistical significance have “won” by finding evidence of an effect, but are also “cursed” as their results are a gross overestimation of the true population effect. This effect inflation is also referred to as a Type M (magnitude) error.
Chapter 03
1. Figure 3.1 Illustration of parameters affecting power. (a) Power for population effect size δ = 0.7 and sample sizes of n₁ = n₂ = 20. (b) Effect of greater population effect size (δ = 1) on power. (c) Effect of increasing sample size to n₁ = n₂ = 50 on power.
Chapter 04
1. Figure 4.1 Terminological conventions borrowed from signal detection theory.
2. Figure 4.2 Graphical illustration of the signal‐detection framework from which the terms “false positive” and “false negative” are borrowed.
3. Figure 4.3 Impressive hits concerning a specific rule (inner circle) can prevent researchers from discovering that the data reflect a more general rule (outer circles).
Chapter 06
1. Figure 6.1 Changes in the heritability of intelligence for yearly cohorts of almost every male in Norway when they are 18 years old.
Chapter 08
1. Figure 8.1 A trio of p‐values, showing that the diagnosticity of a significant result hinges on the specification of the alternative hypothesis. Top panels: a significant result that is ambiguous; middle panels: a significant result that is moderately informative; bottom panels: a significant result that is evidence in favor of the null hypothesis. The left column shows the population distribution under H₁, and the right column shows the two relevant sampling distributions (i.e., one under H₀, the other under H₁) of the test statistic for the difference between 25 participants viewing AUFP stimuli and 25 participants viewing non‐AUFP stimuli.
2. Figure 8.2 A boxing analogy of the p‐value. By considering only the state of boxer H₀, the Fisherian referee makes an irrational decision.
Chapter 11
1. Figure 11.1 The set of correlations surveyed by Vul et al. (2009a), showing how the absolute correlation (y) varies with sample size of the study (x), along with the marginal histograms of both sample size and absolute correlation. Individual observations are color‐coded by whether a request for information from the authors revealed the analysis to be independent (black), non‐independent (red), or if no response was obtained (blue). The vast majority of the surprisingly high correlations (r > 0.7) were obtained by a non‐independent analysis procedure that is guaranteed to inflate effect size, especially when sample sizes are small.
2. Figure 11.2 Illustration of the non‐independence error. The sampling distribution of the sample correlation varies with the sample size (columns) and the true underlying population correlation (rows; illustrated as a solid black dot on each histogram). A statistical significance threshold (here we use the common cluster‐height threshold of p < 0.001), however, yields a constant critical correlation value for every sample size (black lines). The average sample correlations that pass the significance threshold (open circles) are much higher than their true population correlations unless the statistical power of the threshold is high (meaning that most of the sampling distribution is larger than the threshold, as in the case of n = 64, r = 0.75). Consequently, the selected sample correlations are very likely to be much higher than the true populations correlations.
3. Figure 11.3 The mechanism of bias in non‐independent analyses. Even in the presence of non‐zero true correlations (x axis), the sample correlations (y) selected as exceeding a particular threshold are systematically overestimated. With a sample size of 16, the minimum sample correlation to pass a common p < 0.001 whole‐brain threshold is quite large, ensuring that all observed correlations will be large, even if their true population correlations are small.
4. Figure 11.4 What is the expected value of the peak correlation reported from an analysis? The expected maximum correlation (y) increases with the number of independent brain regions it is chosen from (x), yielding large overestimates of the true correlation, regardless of its value (colors). Lines reflect the expectation, while shaded regions show the 90% interval. These calculations used a sample size of 16 subjects.
5. Figure 11.5 How does multiple comparisons correction influence the bias from non‐independent analyses? We simulated how the absolute selected sample correlation (y) relates to the absolute true underlying correlation (x) for different numbers of subjects (8, 16, and 32), as we varied the statistical threshold (between p < 10⁰, to p < 10⁻⁵; with larger circles indicating more stringent thresholds). For each threshold, we show both the average, and the 90% interval of selected and true correlations. Bias (discrepancy between selected and true correlation – y‐distance above the diagonal identity line) is smaller under larger sample sizes, but increases systematically as the statistical threshold becomes more conservative. (The distribution of population correlations is pictured above in gray; this distribution captures the common assumption that there are many small correlations, and few large ones, in the brain; formally, this is obtained via a truncated normal distribution with a mean of 0 and a standard deviation of 1/3 on the Fisher z’ transforms of the population correlations.)
6. Figure 11.6 The influence of statistical power on overestimation from non‐independent analyses. (Left) Average selected correlation (x) under different true population correlations (y); each point represents a particular sample size, with the color corresponding to the statistical power of a p < 0.001 threshold with that sample size and true population correlation. Although the relationship is not numerically uniform across population correlations, in all cases, less power means greater overestimation. (Middle) Magnitude of overestimation of the coefficient of determination (r²): the difference between the selected sample r² and the population ρ² decreases with the power of the test. (Right) Collapsing over true population correlations, statistical power (x) seems to impose an upper bound on the magnitude of overestimation, such that the maximum observed overestimate decreases as power increases.
7. Figure 11.7 The importance of adequate multiple comparisons correction. As the number of independent brain regions in a whole‐brain analysis increases (x), the probability of falsely detecting a correlation (or any other signal) increases if the statistical threshold is held constant. The common p < 0.001 threshold is sufficient to correct for 50 multiple comparisons to the α = 0.05 level, but will yield more than 60% false positives if there are 1000 voxels in the whole‐brain analysis.
8. Figure 11.8 The correlations surveyed in Vul et al. (2009a), plotted as a function of the number of subjects, and the (absolute) reported correlation. Color corresponds to the (uncorrected) p‐value of the correlation, and lines indicate the critical correlation values at different α levels. While the reported correlations are large, they are not very significant, especially when considering that many of them arose from whole‐brain analyses that would require multiple comparisons correction.
9. Figure 11.9 Statistical power (y) for Bonferroni‐corrected correlation tests as a function of population correlation (panels), sample size (lines), and the number of independent correlations in the analysis (x). A small population correlation (ρ = 0.25; left) yields low power even with few independent correlations. In contrast, large correlations (ρ = 0.75; right) can be tested with high power with just 16 subjects, provided that the analysis considers only one correlation; however, a whole‐brain analysis with 1000 correlations requires twice as many subjects to achieve the same level of power. A test for an optimistic but plausible population correlation (ρ = 0.5; middle) requires nearly 100 subjects to achieve a high level of power in a whole‐brain analysis.
10. Figure 11.10 (a) The histogram of sample sizes from the studies surveyed in Vul et al. (2009a), color coded to match the colors in Figure 11.9. (b) Histograms of the power these studies will have to detect a population correlation of 0.5 or 0.75, either with a single measured correlation, or with a 1000‐voxel whole‐brain analysis. The sample sizes used in these studies offer a lot of power for detecting an implausibly large population correlation in a univariate analysis (ρ = 0.75, one region), but all have less than 20% power to detect a plausible (ρ = 0.5) correlation in a whole‐brain analysis.
11. Figure 11.11 Sample size required (y) to achieve a certain level of power (x) as a function of the population correlation (panels), and the number of Bonferroni‐corrected comparisons (brain regions). A realistically small population correlation (ρ = 0.25) will require hundreds of subjects in a whole‐brain analysis (e.g., 1000 voxels) to achieve adequate power. However, even optimistic but plausible population correlations (ρ = 0.5) require many more subjects than are commonly run in whole‐brain across‐subject correlation studies.
12. Figure 11.12 Statistical power (y) for FDR‐corrected correlation tests as a function of population correlation (panels), sample size (lines), and the proportion of voxels in the whole brain that contain the effect (x). A small population correlation (ρ = 0.25; left) yields low power even when nearly 30% of brain voxels have this signal. In contrast, large correlations (ρ = 0.75; right) can be tested with high power with just 16 subjects, provided that 30% of the voxels contain the effect; however, if only 1/1000 voxels carry the signal, then twice as many subjects are needed to achieve the same level of power. A test for an optimistic, but plausible, population correlation (ρ = 0.5; middle) that is highly localized (occurring in 1/1000 voxels of the brain) requires nearly 100 subjects to achieve a high level of power.
13. Figure 11.13 Histograms of the power the studies surveyed by Vul et al. (2009a) will have to detect different population correlations using FDR correction (for ρ = 0.5 and ρ = 0.75, under different prevalence rates of the effect among tested voxels). 36% of the sample sizes used in these studies offer a lot of power for detecting an implausibly large and dense population correlation (ρ = 0.75, prevalence = 10%); but all have less than 30% power to detect a plausible (ρ = 0.5) correlation with a prevalence of 1%; and less than 10% power if the prevalence is 1/1000.
14. Figure 11.14 Sample size required (y) to achieve a certain level of power (x) as a function of the population correlation (panels), and the proportion of signal‐carrying voxels in the FDR‐corrected analysis. A realistically small population correlation (ρ = 0.25) will require hundreds of subjects to achieve adequate power. However, even optimistic but plausible population correlations (ρ = 0.5) will require many more subjects than are commonly run in whole‐brain across‐subject correlation studies, if true effects are as sparse as reported results suggest.
Chapter 13
1. Figure 13.1 Selective and multiple publication of fluoxetine trials submitted to the FDA.
Chapter 15
1. Figure 15.1 Reported estimates of various physical parameters by year of publication.
2. Figure 15.2 Results of Conley et al. (2006, Figure 6). Grey contours represent the 68.3%, 95.4%, and 99.7% confidence regions.
3. Figure 15.3 Six sets of blinded means perturbed by “cell scrambling.” (By chance, Set 1 is identical to the unblinded raw data.)
4. Figure 15.4 Raw means from Simulation 1 (left panel) and raw means perturbed by noise (right panel).
5. Figure 15.5 Raw means from Simulation 1 (left panel) and raw means perturbed by cell‐specific bias (right panel).
6. Figure 15.6 Raw means from Simulation 1 (left panel) and raw means perturbed by both noise and cell‐specific bias (right panel).
7. Figure 15.7 Raw means from Simulation 1 (left panel) and blinded means perturbed by “row scrambling” (right panel).
8. Figure 15.8 Raw means from Simulation 2 (left panel) and blinded means perturbed by cell‐specific bias (right panel).

List of Contributors

Amy Adkins
Virginia Commonwealth University

Paul Bloom
Yale University

Marcus T. Boccaccini
Sam Houston State University

Katherine S. Button
University of Bath

Brett J. Deacon
University of Wollongong, School of Psychology, New South Wales, Australia

Danielle Dick
Virginia Commonwealth Univesity

Christopher J. Ferguson
Stetson University

Klaus Fiedler
University of Heidelberg

Moritz Heene
Ludwig Maximillian University Munich

Ray Hyman
University of Oregon

Leslie K. John
Harvard University

Joachim I. Krueger
Brown University

Etienne P. LeBel
Montclair State University

Scott O. Lilienfeld
Emory University

Alexander Ly
University of Amsterdam

Robert J. MacCoun
Stanford University

David Marcus
Washington State University

Dora Matzke
University of Amsterdam

Gregory Mitchell
University of Virginia

Richard D. Morey
University of Groningen

Marcus R. Munafò
MRC Integrative Epidemiology Unit, UK Centre for Tobacco and Alcohol Studies, and School of Experimental Psychology, University of Bristol, United Kingdom

Daniel C. Murrie
University of Virginia

Harold Pashler
UCSD Psychology

Saul Perlmutter
University of California at Berkeley

Anthony R. Pratkanis
University of California, Santa Cruz

Elizabeth Prom‐Wormley
Virginia Commonwealth University

John Protzko
University of California, Santa Barbara

Jeffrey N. Rouder
University of Missouri

Jonathan W. Schooler
University of California, Santa Barbara

Malte Schott
University of Heidelberg

Glen I. Spielmans
Metropolitan State University, Department of Psychology, Minnesota
University of Wisconsin, Department of Counseling Psychology, Wisconsin

Helen Steingroever
University of Amsterdam

Robert J. Sternberg
Cornell University, Ithaca, New York

Philip E. Tetlock
University of Pennsylvania

Josine Verhagen
University of Amsterdam

Edward Vul
UCSD Psychology

Eric‐Jan Wagenmakers
University of Amsterdam

Irwin D. Waldman
Emory University

Recent Challenges and Proposed Solutions