Cheat sheet — Philosophy of Science and Metascience

This theme supplies the paper’s method rather than its object. The receipt protocol is a piece of applied philosophy of science (what a claim must specify to be capable of losing, and how a programme rather than a claim is appraised) fitted with instruments borrowed from metascience (preregistration, Registered Reports, many-analysts designs, reliability statistics, coded evidence reviews) and checked against one precedent of a neighbouring field correcting itself without any of that apparatus. A co-author must be able to say why the paper cites Lakatos and not just Popper, what the metascience numbers actually show, what Krippendorff’s alpha does and does not settle, and why the scale-free episode both supports and threatens the paper’s case. Take the sub-themes in spec order: 2.1 gives the appraisal logic, 2.2 the machinery, 2.3 the one statistic the protocol names, and 2.4 the precedent the paper must confront last.

Prerequisites. T1.5 (the Schwaninger and Scheef test) before 2.1, so that “protective belt” and “absorbed adverse result” have a concrete referent; T1.2 for what is being appraised.

Falsification, research programmes and the Duhem–Quine problem

Introduction. Popper’s criterion was that a scientific claim must forbid some observation, and that a theory is tested by attempts to produce the forbidden case. Duhem and later Quine showed why single refutations rarely bite: a prediction follows from a theory only together with auxiliary hypotheses about instruments, initial conditions and measurement, so an adverse result can always be blamed on an auxiliary. Lakatos accepted this and changed the unit of appraisal. A research programme has a hard core of commitments that its adherents decide not to give up, a protective belt of auxiliary hypotheses that absorbs anomalies, and heuristics that guide how the belt is modified. Programmes are not refuted; they are appraised over time. A problemshift is theoretically progressive when each modification predicts some new fact, empirically progressive when some of those predictions are corroborated, and degenerating when modifications only accommodate what is already known. The criterion is comparative and historical: a degenerating programme is abandoned when a rival is more progressive. The practical consequence is that “what would count against this claim” must be asked of the belt, one modification at a time, and “is this tradition learning” must be asked of the record of novel corroborations.

Important authors. Imre Lakatos (1922–1974) taught philosophy at the London School of Economics; his 1970 essay in the volume he edited with Musgrave is the standard statement of the methodology of scientific research programmes, and his Proofs and Refutations applied the same historical method to mathematics. Karl Popper (1902–1994), also at LSE, was his teacher and target. Pierre Duhem (1861–1916) was a French physicist and historian of science; W. V. O. Quine (1908–2000) at Harvard generalised Duhem’s holism to all of knowledge. Paul Meehl at Minnesota applied Lakatosian appraisal to psychology and is the nearest precedent for applying it to a soft field.

Importance for cybernetics and the VSM. The VSM has a hard core (five necessary functions, recursion, requisite variety) and a large protective belt (functions realised informally, levels moved after the fact, structures that have “evolved”). Nobody in the tradition has written down which is which, so every adverse result can be absorbed into the belt and no one can say whether the modifications have ever predicted anything new. Lakatos gives the tradition the question it has never asked of itself. It also explains why van der Zouwen’s 1996 call for testability had no effect: naming the problem in Popperian terms does not tell a field what to do with a refutation it can always deflect.

Importance for the article. The receipt architecture (§4) forces belt modifications into the open one claim at a time; Field 5 fixes the revision before the result, and §4.5 now also records the favourable case, because a novel corroboration is “the unit by which a programme is judged progressive.” §4.7 adds programme-level appraisal, §10.1 gives the register a programme-appraisal field with a running tally, and §12.8 turns the criterion on the paper itself. All of this comes from C1 §6: “You cite Lakatos and then behave like Popper… the register can record fifty narrowings and never say whether the tradition is progressing or degenerating.” §5.4 identifies Schwaninger and Scheef’s reinterpretation of H3 as “the protective-belt manoeuvre in print.” Reviewer pressure: Lakatos’s novelty criterion is contested (temporal versus use-novelty), and a philosopher will ask which version the register uses when it decides a claim “was a novel prediction.” A co-author should be able to name the three candidate novel predictions in §4.7 and say why each counts.

Sources in the reading list.

hard core, protective belt, progressive and degenerating problemshifts; the passages on novel facts that §4.7 relies on

Other important sources and authors.

Popper, K. R. (1959). The Logic of Scientific Discovery. Hutchinson, London. — falsifiability as demarcation; what “Popperian” commits Schwaninger and Scheef to
Popper, K. R. (1963). Conjectures and Refutations: The Growth of Scientific Knowledge. Routledge & Kegan Paul, London. — the essays on risky prediction and on ad hoc rescues, the source of the “immunising” vocabulary used in §5.4
Duhem, P. (1954). The Aim and Structure of Physical Theory (P. P. Wiener, Trans.). Princeton University Press, Princeton. (Original work published 1906.) — the original statement that a physicist can never test an isolated hypothesis
Quine, W. V. O. (1951). Two dogmas of empiricism. The Philosophical Review, 60(1), 20–43. — holism generalised; the “Duhem–Quine” half the paper concedes in §12.1
Lakatos, I. (1978). The Methodology of Scientific Research Programmes: Philosophical Papers, Volume 1 (J. Worrall & G. Currie, Eds.). Cambridge University Press, Cambridge. — the collected papers, including the replies to critics on novelty
Meehl, P. E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46(4), 806–834. — the classic account of why a soft field’s tests never bite; the nearest analogue to the VSM’s situation
Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141. — Lakatosian defence made operational for a soft science; directly relevant to §4.7 and §12.8

Metascience: many-analysts studies, preregistration, Registered Reports

Introduction. Metascience studies research practice with research methods. Three findings matter here. Many-analysts studies give the same data and question to independent teams and measure the spread of conclusions: Silberzahn et al. (2018) had 29 teams test whether referees give more red cards to dark-skinned players and obtained odds ratios from 0.89 to 2.93, with 20 teams finding a significant effect and 9 not, unexplained by expertise or peer-rated quality; Breznau et al. (2022) had 73 teams test one hypothesis on one dataset and found that coded analytic decisions explained only a few percent of the variance in results. Preregistration fixes hypotheses and analyses before data are seen, but adherence is poor: Claesen et al. (2021) found that of 27 badged preregistered studies in Psychological Science, two had no deviations and nine disclosed none. Registered Reports move peer review and the decision to publish before results are known; Scheel, Schijen and Lakens (2021) found 96 percent positive first results in 152 standard articles against 44 percent in 71 Registered Reports. Cox, Arnold and Villamayor-Tomás (2010) is a different kind of precedent: they coded 91 studies against Ostrom’s eight design principles, found them supported, reformulated three, and found no study moderately or strongly negative.

Important authors. Brian Nosek (University of Virginia; co-founder and executive director of the Center for Open Science) leads the reproducibility and preregistration programme in which Silberzahn et al. sits. Nate Breznau (Bremen) coordinated the 73-team study. Daniël Lakens (Eindhoven University of Technology) works on preregistration, power and Registered Reports; Anne Scheel is now at Utrecht. Wolf Vanpaemel and Francis Tuerlinckx lead quantitative psychology groups at KU Leuven. Chris Chambers (Cardiff) launched the Registered Reports format at Cortex in 2013. Michael Cox (Dartmouth) and Sergio Villamayor-Tomás (ICTA-UAB) trained in Ostrom’s Workshop at Indiana.

Importance for cybernetics and the VSM. The VSM tradition has never measured its own analytic variability, has no preregistration practice, has no journal with a Registered Reports track, and has a coded applications literature with no negative cases. Metascience says what to expect from each of those conditions. Many-analysts results set the base rate for how much trained VSM diagnosticians will disagree; the preregistration-adherence results say that a voluntary register will drift; the Registered Reports comparison says what enforcement at the point of publication achieves; Cox et al. shows that a supportive coded literature is itself weak evidence.

Importance for the article. §8 (Demonstration III) preregisters low unconditional agreement for IIIa on the many-analysts base rate and moves the discriminator to the conditional question (C1 §5). §10.6 replaces v01’s standalone register with a Registered Reports track at named journals, on the strength of Claesen et al. and Scheel et al.; the contribution statement in §1 makes this element (iii), because “a voluntary, unenforced register is the configuration with the worst record” (C1 §7). §3.4 and §10.4 use Cox et al. twice: as the executed precedent for testing a set of context-sensitive design principles, and as the reason the negative-results provision is enforced rather than encouraged (C1 §10: “steal the coding protocol”). §3.5 disclaims priority over all of these instruments. Reviewer pressure: the numbers come from psychology and survey data, and a reviewer will ask whether their transfer to qualitative organisational diagnosis is more than analogy; Claesen is Tier B and †; and the Registered Reports mechanism needs a journal to agree, which §12.8 names as a failure condition.

Sources in the reading list.

the 29-team spread and the finding that expertise and quality ratings do not explain it
the 73-team design and how little of the variance coded decisions explain
2 of 27 clean, 9 with undisclosed deviations; pin the citation before deposit
96 versus 44 percent; the effect size of gatekeeping
the coding protocol and the zero-negative-studies finding

Other important sources and authors.

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. — the case for preregistration and the distinction between prediction and postdiction that Field 5 depends on
Chambers, C. D., & Tzavella, L. (2022). The past, present and future of Registered Reports. Nature Human Behaviour, 6(1), 29–42. — the format’s history, uptake and evidence; the reference for negotiating a track
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. — researcher degrees of freedom, the concept behind §4.4’s demand to name estimators in advance
Botvinik-Nezer, R., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84–88. — a third many-analysts study, in neuroimaging; strengthens the base-rate argument beyond social science
Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. — the method for reporting all defensible analytic paths; a candidate instrument for IIIc’s time-series analysis
Mellers, B., Hertwig, R., & Kahneman, D. (2001). Do frequency representations eliminate conflict? An adversarial collaboration. Psychological Science, 12(4), 269–275. — the model adversarial collaboration §10.3 refers to
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. — the replication result that made the reform programme necessary

Measurement, reliability and analyst agreement

Introduction. Content analysis is the technique of making replicable, valid inferences from texts to their contexts; Krippendorff’s textbook gives its design logic (unitising, sampling, coding, reducing, inferring) and its treatment of reliability. Reliability among coders is the degree to which independent coders, working from the same instructions, assign the same categories to the same units. Percent agreement overstates it because coders agree by chance. Krippendorff’s alpha corrects for chance agreement, handles any number of coders, any metric (nominal, ordinal, interval, ratio), missing data and small samples, and is therefore the general-purpose coefficient; Cohen’s kappa is the two-coder nominal special case and intraclass correlation the interval case. Alpha is a point estimate and its sampling distribution is not simple, so confidence intervals are obtained by bootstrapping, and the width of the interval depends on the number of units coded. A reliability study is therefore designed backwards from the precision required: state the smallest alpha that would be acceptable, state the interval width needed to distinguish it from alternatives, and derive the sample. Reliability is necessary and not sufficient for validity: coders can agree reliably on a category that tracks nothing real.

Important authors. Klaus Krippendorff (1932–2022) was the Gregory Bateson Professor of Communication at the Annenberg School, University of Pennsylvania, a past president of the International Communication Association and of the American Society for Cybernetics; trained at Ulm and at Illinois in von Foerster’s circle, he was a second-order cybernetician as well as a methodologist. Andrew Hayes (Ohio State, later Calgary) co-authored the standard argument for alpha as the default coefficient and supplied its software. Jacob Cohen introduced kappa in 1960.

Importance for cybernetics and the VSM. VSM diagnosis is a coding task: an analyst assigns units of an organisation to Systems One to Five and to recursion levels. No reliability study of that assignment exists. Krippendorff’s authorship is a small irony: the coefficient the VSM tradition needs was developed by a cybernetician, and the tradition never used it. The distinction between reliability and validity is also the distinction between the instrumental and constitutive readings in §2.2: two trained analysts agreeing does not show the levels are there.

Importance for the article. §4.4 requires that a reliability receipt “names the reliability statistic (Krippendorff’s alpha with reported confidence intervals†), a target precision, and the sample size that precision requires.” §8.3 applies it to IIIa: alpha estimated separately for named units, functional relations and diagnostic predictions, “at a preregistered target precision that determines the sample; six to ten cases and three to five analysts will not reach it, and the receipt says so.” The import is from C1 §5 (“you also don’t name a reliability statistic”). IIIa rival (3), analysts disagreeing on boundaries but converging on predicted vulnerabilities, requires reliability to be estimated on predictions as well as partitions; that is the informative endpoint. Reviewer pressure: what alpha threshold counts as “stated reliability” and why; whether units, relations and predictions are themselves codable; and the reliability–validity gap, which is why IIIb and IIIc exist. Krippendorff (2004) is Tier C and † in v02; the 2018 fourth edition is current.

Sources in the reading list.

the reliability chapters: alpha’s definition, its confidence intervals, and the argument that reliability does not establish validity

Other important sources and authors.

Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. — the case for alpha over kappa and the bootstrapped confidence interval procedure §8.3 presupposes
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. — the reply to critics of alpha; useful when a reviewer proposes a different coefficient
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. — kappa, the coefficient most reviewers will expect; know why alpha is preferred
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. — the interval-scale case and the forms of ICC, relevant if diagnostic predictions are rated on scales
Gwet, K. L. (2014). Handbook of Inter-Rater Reliability (4th ed.). Advanced Analytics, Gaithersburg. — the comprehensive reference on coefficients, their paradoxes and sample-size planning
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. — the metascience of measurement; the frame for §5.2’s criticism of the viability proxy

Self-correction in a neighbouring field: the scale-free network episode

Introduction. Barabási and Albert (1999) proposed that many real networks have power-law degree distributions generated by growth with preferential attachment, and “scale-free” became the headline result of network science, extended to metabolic, protein, social and Internet topologies with claims about robustness to random failure and vulnerability to targeted attack. Within a decade the claim was taken apart from inside the field. Keller (2005) showed that power laws had been reported and over-interpreted since Pareto and Zipf, that many mechanisms produce them, and that the fits were often weak. Lima-Mendez and van Helden (2009) subjected the biological network claims to proper statistical tests and found most did not fit, with good fits often sampling artefacts. Willinger, Alderson and Doyle (2009) showed that the Internet claim rested on traceroute data with known artefacts and on a model that ignored engineering constraints, that real router networks have high-degree nodes at the edge, and that the “Achilles’ heel” prediction was false. Later work (Clauset, Shalizi and Newman 2009; Broido and Clauset 2019) supplied the statistical tests and the census. The method survived; the universality claim did not.

Important authors. Evelyn Fox Keller (1936–2023), physicist turned historian and philosopher of science at MIT’s Program in Science, Technology and Society, MacArthur Fellow. Jacques van Helden, bioinformatician at the Université Libre de Bruxelles and later Aix-Marseille, known for regulatory-sequence analysis tools. Walter Willinger (AT&T Labs–Research, later NIKSUN), David Alderson (Naval Postgraduate School) and John Doyle (Caltech) form the group that replaced statistical-mechanics accounts of network structure with engineering ones; Doyle recurs in T7. Aaron Clauset (Colorado Boulder) and Cosma Shalizi (Carnegie Mellon) wrote the standard power-law fitting paper.

Importance for cybernetics and the VSM. Complexity science is cybernetics’ institutional sibling (shared ancestry in Ashby, von Neumann and McCulloch, per §11.4) and it corrected a structural universality claim about complex systems, at the height of that claim’s popularity, through ordinary adversarial publication. The parallel to the VSM’s five-function recursion is direct: a structural pattern that looks ubiquitous once the lens is adopted is not thereby corroborated. The difference is that “scale-free” was a formal claim with a testable degree distribution, and the venues rewarded attacking it.

Importance for the article. §3.4 and §13.3 use the episode as one of two precedents (with the active-inference community’s absorption of Bruineberg et al.) for corrigibility without infrastructure, and concede C6 §5’s “uncomfortable” implication: “it weakens the claim that infrastructure is what’s missing.” The paper’s answer is that those fields had “formal claims precise enough to be wrong, and venues in which attacking a popular result was a career-advancing move rather than a community betrayal,” that the receipt protocol is a substitute where those conditions are absent, and that Demonstration IV is where the first condition begins to be met. C6 proposed replacing the algedonic framing of §13 with this precedent; v02 kept the metaphor (audited in §13.1) and added the precedent beside it. Reviewer pressure: if precision and adversarial venues are what corrigibility consists in, the register is a stopgap and the paper should be about formalising the VSM rather than governance; §1’s two-programme structure is the answer. The episode also took a decade and needed statisticians from outside the community, which bears on §10.2’s mixed governance.

Sources in the reading list.

the philosopher’s dismantling: how much of a model’s authority came from rhetoric rather than fit
what proper statistical tests did to the biological claims, and what survived
the engineering rebuttal; the clearest case of asking what data would have falsified the claim

Other important sources and authors.

Barabási, A.-L., & Albert, R. (1999). Emergence of scaling in random networks. Science, 286(5439), 509–512. — the original claim; know what it actually asserted before citing its dismantling
Clauset, A., Shalizi, C. R., & Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review, 51(4), 661–703. — the statistical test that made the claim losable; the methodological turning point of the episode
Broido, A. D., & Clauset, A. (2019). Scale-free networks are rare. Nature Communications, 10, 1017. — the census of nearly a thousand networks; the episode’s closing result, and the debate it provoked
Stumpf, M. P. H., & Porter, M. A. (2012). Critical truths about power laws. Science, 335(6069), 665–666. — a short statement of what a power-law claim must show to count
Li, L., Alderson, D., Doyle, J. C., & Willinger, W. (2005). Towards a theory of scale-free graphs: Definition, properties, and implications. Internet Mathematics, 2(4), 431–523. — the Doyle group’s formal treatment; shows that “scale-free” had never been given a definition precise enough to test
Carlson, J. M., & Doyle, J. (1999). Highly optimized tolerance: A mechanism for power laws in designed systems. Physical Review E, 60(2), 1412–1427. — the rival mechanism that produces power laws from design constraints; the bridge to T7’s robust-yet-fragile argument

What you should be able to say after this theme

The unit of appraisal is the programme, not the claim: a receipt fixes one belt modification in advance, and the register’s programme-appraisal field records whether the tradition’s modifications ever predicted anything new and whether it was corroborated.
Schwaninger and Scheef’s reinterpretation of H3 is a protective-belt manoeuvre in Lakatos’s sense, and the paper says so as a critique of inference, not of good faith.
§12.8 applies the Lakatosian criterion to the paper’s own programme: ten receipts with no changed teaching claim, a register with only retentions and narrowings, or no Registered Reports track within a stated period, each names a specific withdrawal.
The many-analysts base rate (odds ratios 0.89–2.93 across 29 teams; a few percent of variance explained across 73 teams) is why Demonstration IIIa preregisters low agreement and moves the discriminator to the conditional question.
Preregistration without enforcement drifts (2 of 27 clean; 9 undisclosed) and Registered Reports change the positive-result rate from 96 to 44 percent, which is why the adoption mechanism is a journal track, not a standalone register.
Cox et al. is the executed precedent for coding a literature against a set of design principles, and its zero negative studies is why the register enforces rather than encourages negative results.
Krippendorff’s alpha with bootstrapped confidence intervals and a preregistered target precision is the named reliability statistic; reliability among analysts does not establish that the levels are there, which is why IIIb and IIIc exist.
Complexity science corrected scale-free network theory without a register because its claims were formal enough to be wrong and its venues rewarded attack; the receipt protocol is a substitute where those conditions are absent, and the paper concedes it is not more than that.