Introduction. Metascience studies research practice with research methods. Three findings matter here. Many-analysts studies give the same data and question to independent teams and measure the spread of conclusions: Silberzahn et al. (2018) had 29 teams test whether referees give more red cards to dark-skinned players and obtained odds ratios from 0.89 to 2.93, with 20 teams finding a significant effect and 9 not, unexplained by expertise or peer-rated quality; Breznau et al. (2022) had 73 teams test one hypothesis on one dataset and found that coded analytic decisions explained only a few percent of the variance in results. Preregistration fixes hypotheses and analyses before data are seen, but adherence is poor: Claesen et al. (2021) found that of 27 badged preregistered studies in Psychological Science, two had no deviations and nine disclosed none. Registered Reports move peer review and the decision to publish before results are known; Scheel, Schijen and Lakens (2021) found 96 percent positive first results in 152 standard articles against 44 percent in 71 Registered Reports. Cox, Arnold and Villamayor-Tomás (2010) is a different kind of precedent: they coded 91 studies against Ostrom’s eight design principles, found them supported, reformulated three, and found no study moderately or strongly negative.
Important authors. Brian Nosek (University of Virginia; co-founder and executive director of the Center for Open Science) leads the reproducibility and preregistration programme in which Silberzahn et al. sits. Nate Breznau (Bremen) coordinated the 73-team study. Daniël Lakens (Eindhoven University of Technology) works on preregistration, power and Registered Reports; Anne Scheel is now at Utrecht. Wolf Vanpaemel and Francis Tuerlinckx lead quantitative psychology groups at KU Leuven. Chris Chambers (Cardiff) launched the Registered Reports format at Cortex in 2013. Michael Cox (Dartmouth) and Sergio Villamayor-Tomás (ICTA-UAB) trained in Ostrom’s Workshop at Indiana.
Importance for cybernetics and the VSM. The VSM tradition has never measured its own analytic variability, has no preregistration practice, has no journal with a Registered Reports track, and has a coded applications literature with no negative cases. Metascience says what to expect from each of those conditions. Many-analysts results set the base rate for how much trained VSM diagnosticians will disagree; the preregistration-adherence results say that a voluntary register will drift; the Registered Reports comparison says what enforcement at the point of publication achieves; Cox et al. shows that a supportive coded literature is itself weak evidence.
Importance for the article. §8 (Demonstration III) preregisters low unconditional agreement for IIIa on the many-analysts base rate and moves the discriminator to the conditional question (C1 §5). §10.6 replaces v01’s standalone register with a Registered Reports track at named journals, on the strength of Claesen et al. and Scheel et al.; the contribution statement in §1 makes this element (iii), because “a voluntary, unenforced register is the configuration with the worst record” (C1 §7). §3.4 and §10.4 use Cox et al. twice: as the executed precedent for testing a set of context-sensitive design principles, and as the reason the negative-results provision is enforced rather than encouraged (C1 §10: “steal the coding protocol”). §3.5 disclaims priority over all of these instruments. Reviewer pressure: the numbers come from psychology and survey data, and a reviewer will ask whether their transfer to qualitative organisational diagnosis is more than analogy; Claesen is Tier B and †; and the Registered Reports mechanism needs a journal to agree, which §12.8 names as a failure condition.
Sources in the reading list.
- the 29-team spread and the finding that expertise and quality ratings do not explain it
- the 73-team design and how little of the variance coded decisions explain
- 2 of 27 clean, 9 with undisclosed deviations; pin the citation before deposit
- 96 versus 44 percent; the effect size of gatekeeping
- the coding protocol and the zero-negative-studies finding
Other important sources and authors.
- Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. — the case for preregistration and the distinction between prediction and postdiction that Field 5 depends on
- Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29–42. — the format’s history, uptake and evidence; the reference for negotiating a track
- Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. — researcher degrees of freedom, the concept behind §4.4’s demand to name estimators in advance
- Botvinik-Nezer, R., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84–88. — a third many-analysts study, in neuroimaging; strengthens the base-rate argument beyond social science
- Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. — the method for reporting all defensible analytic paths; a candidate instrument for IIIc’s time-series analysis
- Mellers, B., Hertwig, R., & Kahneman, D. (2001). Do frequency representations eliminate conflict? An adversarial collaboration. Psychological Science, 12(4), 269–275. — the model adversarial collaboration §10.3 refers to
- Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. — the replication result that made the reform programme necessary