A Conceptual Framework for Computational Reproductions: Formal Definitions and Epistemic Functions

Abstract

Reproductions, which we define as “redoing activities” that work with the same collected data (or simulation source code) and aim to keep the computation as similar as possible, enable the verification of a scientific work. We seek to advance the understanding and conduct of reproductions by providing a conceptual framework and guidance to reproducers. In the first part, the epistemic function of reproductions and replications regarding mistakes—and consequently, faults—is discussed. In the second part, the logic underlying reproductions is formulated, accompanied by formal definitions of central terms. The third and final part provides practical guidance on conducting and reporting reproductions. In short, a reproduction involves performing another computation and comparing the results for consistency, exploring the impact of computational choices on results before suspecting faults as a last resort. Computational choices by reproducers are necessitated by obstacles such as underspecification and external constraints. Regarding faults, four cases are emphasized: (1) incoherent descriptions, (2) different collected data (or simulation source code), (3) different data analysis source code, and (4) incorrect reporting of results. Separate from evaluating the consistency of results, reproductions can investigate their support for a claim. Finally, improper reporting practices can reduce the epistemic value of scientific works and their reproductions. Therefore, reproducers should also report the scope of their reproduction by detailing, among other things, which results were targeted, which computational stages were redone, and which descriptions were consulted.

Keywords: error, reproduction, meta-science, conceptual framework, verification, epistemic function, uncertainty

Introduction and Previous Research

The increasing number of scientific works relying on computations requires conceptual clarity on how their trustworthiness can be improved. In the literature, early recommendations focused on the authors (e.g., Kelly et al., 2009; Nagler, 1995) and still do to a large degree. For example, Sandve et al. (2013) and Ziemann et al. (2023) both present lists of software development practices that support the dissemination of computational results. With a similar goal in mind, the UK Government Analytical Community (2020), Mineault et al. (2021), and Kohrt et al. (2024) created self-paced tutorials that equip researchers with a relevant skillset. Other resources draw explicitly from the authors’ own reproduction (e.g., Krafczyk et al., 2021) or training experience (e.g., Wilson et al., 2017). Further, the idea of reporting standards for communicating computational results has been put forward, for example, in the fields of health sciences (Altman et al., 2008), statistics (Stodden, 2015), machine learning (Albertoni et al., 2023; Nordlie et al., 2009), and agent-based modeling (Grimm et al., 2014, 2020; Monks et al., 2019).

A second, less developed line of research specifically addresses the verification of computational results (e.g., Arguillas et al., 2022; Ivimey-Cook et al., 2023; Pilgrim et al., 2023). At present, researchers face several challenges during redoing activities:1 scientific activities that redo some phases of a previous study. For example, when verifying the data analysis published in a previous study, the study may contain multiple results, and the researchers’ time and resources may not be sufficient to calculate all of them. Then, it is up to the reproducers to select an appropriate and indicative set of results to reproduce, even if the original study does not contain any hints in this regard. Furthermore, the collected data may be unavailable or only available in preprocessed form. Similarly, the data analysis source code may not have been shared (or only partially) and may not run. Moreover, even if results are obtained eventually, they may differ from the original results, and it is unclear whether the verification was successful. Finally, the authors are often no longer available to ask for clarification.

This article sets out to address these challenges. Unlike some of the articles cited above, it does not give recommendations on what tools and software development techniques to use or which technical parameters to report. We rather aim to advance the conceptual understanding and the practical conduct of redoing activities with an emphasis on the computational aspects, particularly the data analysis.

Terminological Issues

In existing conceptual work on redoing activities, many have noted that fields differ in how various redoing activities are named (Barba, 2018), contributing to a “confused terminology” (Plesser, 2018), although—or maybe because—many unifying definitions are proposed (Dreber & Johannesson, 2025; LeBel et al., 2018; National Information Standards Organization, 2021; Nichols et al., 2017; Patil et al., 2016a; Schöch, 2023). When creating typologies, researchers usually make distinctions that are important to their own field and research method, possibly creating jingle-jangle conflicts with other typologies where the same term refers to different concepts (jingle) or different terms refer to the same concept (jangle). For example, many typologies differentiate between redoing a study with the same data vs. using new data, by calling one “reproduction” and the other “replication”—but vary in their nomenclature, which is which (e.g., National Academies of Sciences, Engineering, and Medicine, 2019; Schöch, 2023). Adding to the confusion is the ambiguity of the term “data”, which can refer to the output of a data collection phase (collected data, i.e., basis for scientific inference) but also to the input to a data collection phase (parameter data, i.e., calibration or training data). Consequently, for some typologies using the same data means that no new experiment (nor sample or observation) is involved (e.g., Simkus et al., 2025), while for others both reproduction (same data) and replication (different data) involve conducting a new experiment (e.g., Association for Computing Machinery, 2020). The latter case is already the result of harmonization efforts driven by Heroux et al. (2018) who called for a compatible usage of the terms “reproduction” and “replication” across disciplines, suggesting a focus on internal consistency for the former (“doing things right”) and a focus on external consistency for the latter (“doing the right things”). In the field of computing science, the entire experiment may happen in silico rather than in natura, making another data analysis alone less common than running both experiment and data analysis again. In silico means that collected data are solely obtained by means of computer programs, in an automated fashion, and pertain to computer programs. In other words, the data are about properties of the programs and not their environment. In contrast, in natura means that collected data pertain to the physical or social reality. Non-experimental research is not considered in all of these cases. This article focuses on collected data (the basis for scientific inference) and distinguishes “simulation source code” (any source code that generates collected data, even in the absence of a proper simulation) and “data analysis source code” (for scientific inference such as statistical analysis).

Finally, many definitions do not explicitly discern the redoing activity, the ability to conduct it, and its outcome—let alone agreeing on names for them (Penders et al., 2019)—making undetected conflicts even more likely. An exception to this comes from Goodman et al. (2016), who specifically differentiate the provision of details (methods reproducibility) and obtaining the same results (results reproducibility). They even add another category: drawing similar conclusions (inferential reproducibility). In the context of research assessment, Ulpts and Schneider (2024) suggest differentiating redoing (e.g., reproducing a previous study) from enabling (e.g., being transparent about the data analysis), as enabling can be a meaningful practice even when redoing is not (e.g., with certain qualitative research). They hold that in the case of confusion researchers should state explicitly what is being enabled or redone and for what reason. As an interim conclusion, the existing literature on redoing activities is in no agreement about which redoing activities to discern or what to call them. However, the literature suggests differentiating the act of enabling, the act of redoing, obtaining the same result, and reaching the same conclusion. The present article relies on the following novel, unifying definition of reproduction (or synonymously computational reproduction2): a redoing activity that works with the reported collected data (or the reported simulation source code, in the case of in silico works) and that aims to keep the data analysis as similar as possible. Further, the term replication will be used to mean a redoing activity that works with newly collected data (or new simulation source code). This usage is largely compatible with both computing science and other fields.

Epistemic Functions of Redoing Activities

The reason for conducting a redoing activity—its epistemic function—has also been studied from a meta-scientific perspective. For example, Matarese (2022) details how changing irrelevant or relevant factors of a study allows for evaluating its reliability and validity. Schmidt (2009) specifically discusses five epistemic functions of replications: controlling for sampling variability, internal validity, and fraud, as well as generalizing results and verifying hypotheses. Bayarri and Mayoral (2002) highlight four goals of a replication: reduction of random error, validation (confirmation) of conclusions, extension of conclusions, and bias detection. Following an analysis of the logical structure of experiments in science, Buzbas et al. (2023) study what can be learned from a sequence of replications, given the information available to the replicators. They highlight the importance of reporting “modelling assumptions, model structure and parameters” (Buzbas et al., 2023, p. 19). Gundersen (2021) analyzes the scientific method to argue that repeating an experiment can increase trust in the correctness of its conclusion, but not in the underlying hypothesis or theory, which requires different experiments. Steiner et al. (2019) have developed a framework that allows for a causal interpretation of replication failures. López-Ibáñez et al. (2021) also consider redoing activities that recreate digital objects like source code and mention that they are unlikely to contain the same faults, thus hinting at their function for fault detection. Clemens (2017) is even more explicit in this regard: Redoing activities that use the reported data set or sample and thus expect “materially the same results” (p. 327) can identify issues with measurement, coding, and data set construction, as well as scientific misconduct. Finally, Martel García (2013), LeBel et al. (2018), and Nuijten et al. (2018) describe reproductions as an early, less resource-intensive part of a multistep approach to evaluating studies. To summarize, the existing literature on the epistemic function of redoing activities focuses primarily on replications but mentions the ability of reproductions to detect faults and the possibility of combining them. The present article further elaborates on that, laying bare the logic and diversity of reproductions and what can be learned from the combination of multiple reproductions.

Evaluating the Results of Redoing Activities

Existing research also investigated what constitutes "materially the same results" when new data are collected. For example, Patil et al. (2016b) and Stanley and Spence (2014) describe realistic expectations for replications, and Heyard et al. (2025) compiled 27 replication metrics from the literature. Further, Axtell et al. (1996) differentiate three categories of model equivalence in their work on redoing simulation models: numerical identity (i.e., exactly the same numbers), distributional equivalence (same statistical properties; for which Sego et al. (2024) developed the Empirical Characteristic Function Equality Convergence Test), and relational equivalence (same relationships between variables). Similar categories are described by Ivie and Thain (2019). However, a recent need has been formulated for adding another category in case the original work contains insufficient information (Zhang & Robinson, 2021).

In contrast, reproductions have received considerably less attention in that regard. One approach from the field of psychology suggests a 10% rule for comparing the original with the reproduction result (Hardwicke et al., 2018; LeBel et al., 2018). This rule balances rigor and usefulness, but it falls short of accounting for ambiguity in the description of the data analysis. On this matter, Artner et al. (2021) propose to differentiate correctness from vagueness when evaluating the reproducibility of numerical results. Here, correct results can be reproduced without violating the descriptions in the study. Further, the reproducibility is vague if the descriptions allow for multiple correct results. This distinction allows putting results into the context of the underlying description. Finally, Dreber and Johannesson (2025) propose two indicators for systematic bias of reproductions: the statistical significance indicator and the relative effect size indicator. The present article establishes a new standard for the consistency of results, which accounts for uncertainty in the model description.

Conceptual work on reproductions has also been paired with practical recommendations, for example, by Ankel-Peters et al. (2024) and Brodeur et al. (2024). The related Guide for Accelerating Computational Reproducibility in the Social Sciences (Berkeley Initiative for Transparency in the Social Sciences, 2020) provides detailed instructions for conducting and reporting reproductions, also covering improvements, tests of a work’s robustness, and communication with the original authors. However, it is less formal regarding the comparison of results. The present article fills this gap by systematically discussing reasons for differences in results and deriving criteria for their consistency. Further, the importance of checking the coherence of relevant instructions is highlighted.

About this Article

Building on the existing research on the epistemic functions of reproductions, the second section of this article ("The Verifying Function of Reproductions") introduces their verifying function and explains what can be learned from their conduct. The third section ("Defining Reproductions") explains their underlying logic and defines central terms around them. Finally, the fourth and fifth sections ("Performing a Reproduction" and "Discussion and Recommenedations") provide practical recommendations on their conduct and reporting and discusses their possible outcomes.

Although this article focuses on the concepts rather than the appropriateness of their labels, we aim to use consistent language. For example, a scientific work that is the target of a reproduction attempt is referred to as the original work, its authors as original authors, and its results as original results. The counterparts to these terms are a reproduction, reproducers, and reproduction result. We suggest avoiding the term "reproduced result" to avoid ambiguity. Also note that the term "reported data" always refers to the data provided by the original authors. Further, we use the terms "mistake" and "fault" according to international standards (International Organization for Standardization, 2017) by distinguishing a human action (a mistake) and its manifestation (a hardware or software fault).

The challenges described initially relate to different issues when verifying existing research:

• Target selection refers to which specific results (out of multiple original results) a redoing activity should focus on.

• Issues with data and code relate to their lack of availability, provenance, correctness, and executability.

• Evaluation of results concerns how original and reproduction results are compared and what the conclusion is.

• Synthesis of results involves integrating and reporting multiple results, both on the level of individual studies and across them.

This article will focus on issues with data and code and the evaluation of results, though it will also touch on the other issues.

The Verifying Function of Reproductions

In the following section, we describe where mistakes in the research process can occur, which prototypical redoing activities can detect the resulting faults, and how their combination can be insightful.

Research producers (i.e., the authors of an original work) commonly recognize and communicate their state of uncertainty about obtained results and try to quantify parts of it by making stochastic assumptions, for example, when inferring from a sample to a population and by providing confidence intervals. Similarly, most fields consider the effects of measurement mistakes, for example, using structural equation models in the social sciences (Hoffmann et al., 2021) or via propagation of uncertainty in physics. In the computing sciences, interval analysis can be used to investigate the limits of floating-point operations (Diethelm, 2012).

However, research consumers (i.e., an original work’s readers) are confronted with more potential sources of uncertainty that the research producers have not considered or not reported, contributing to an information asymmetry between them (Young, 2009). A prominent example is researcher degrees of freedom, that is, the multitude of data collection and analysis options, which may call the robustness of a given result into question if only one path is reported (e.g., Botvinik-Nezer et al., 2020; Breznau et al., 2022; Huntington-Klein et al., 2021; Menkveld et al., 2024; Silberzahn et al., 2018). Equally, publication bias and questionable research practices might distort the results of a meta-analysis (Carter et al., 2019). As is the case with the uncertainty of research producers, research consumers can attempt to assess their uncertainty, as is exemplified by robustness checks and studies that analyze publication bias (Franco et al., 2014). Note that robustness checks have a multitude of names across disciplines, for example multiverse analysis (Steegen et al., 2016), sensitivity analysis (Saltelli, 2008), specification search (Leamer, 1978), vibration of effects (Patel et al., 2015), multimodel analysis (Young & Holsteen, 2017), thick modeling (Granger & Jeon, 2004), or Bayesian model averaging (Chatfield, 1995).

This article considers another instance of uncertainty among research consumers that concerns the existence of faults3 within the original work. While we will not address all possible sources of uncertainty, we will additionally address underspecification in "Defining Reproductions". Internal consistency checks referred to as “verification” assess uncertainty due to faults. They deliberately redo some phases of a previous work while inheriting all remaining phases. An inherited phase is only performed once across original work and reproduction and its product feeds as-is into the following phases of the redoing activity, where it is reused. Verification concerns whether a phase has been conducted as intended and is often contrasted with validation, which checks whether the intent itself is appropriate. However, there are many more possible epistemic functions of reproductions—for example, in their review of the literature, Ulpts and Schneider (2024) also found mentions of the functions of learning, training, and understanding. In the following, however, only the verifying function of reproductions will be detailed. In the following paragraphs, however, we focus on three reproduction prototypes as discerned in the literature that all work with the reported collected data (or the reported simulation source code that generates them) and discuss a replication in comparison:

• Re-executing the original data analysis source code on the original data (“re-execution reproduction”). In the following, it is assumed that the source code is not checked for coherence with the methods section.

• Re-implementing the data analysis source code and running it on the original data (“re-implementation reproduction”). Usually, it is only based on an article’s methods section, not considering the original data analysis source code.

• Re-executing the reported simulation source code (“re-collection reproduction”), in the case of in silico works. The data analysis source code may be re-implemented, which we will assume in the following.

• Collecting new data or, in the case of in silico works, re-implementing the simulation source code (“replication”, also called direct replication in psychology). In the following, we only discuss the case where the data analysis source code is also re-implemented.

All four activities represent particular sets of phases that are redone out of all possible phases of a simplified research process, as detailed in Figure 1: data collection, data analysis, and reporting. Here, every phase covered by the colored area of a redoing activity is done again according to the description in the original work. In contrast, all remaining phases are directly inherited (i.e., not redone) from the original work. For example, for a re-implementation reproduction, one inherits the data collection of an original work, attempts to re-analyze the data, and reports the results again. The data collection phase is further differentiated depending on the type of work. For in natura works, this phase can be divided into sampling, experimental manipulation, and measurement. For in silico works, it can alternatively be separated into implementation and execution, where execution involves running an implementation within a computational environment.4 Finally, the data analysis phase also consists of implementation and execution.5 Of course, not all phases are part of every research process—for example, no experimental manipulation occurs when collecting historical data. Also, these four redoing activities represent prototypical cases that may blend in reality. For example, a re-implementation reproduction might incorporate portions of the original data analysis source code rather than writing everything anew. However, for this section, we only consider the prototypical case. Note that we do not recommend relying on our prototypes alone for describing a particular reproduction—instead, see "Discussion and Recommendations" (specifically Box 5 in this article) for recommendations.

While inherited phases should be identical between the original work and the redoing activity, redone phases may deviate despite aiming for equality. For every phase, Figure 1 indicates associated reasons for differences in results as circled letters (a) through (k). These should be considered if the respective phase is redone and the obtained result of a redoing activity is not the same, meaning exact numerical agreement at the maximum precision that the representation of a result allows. Both identical and non-identical results influence the reproducers’ state of knowledge about associated reasons by providing evidence for or against them. This simplified research process assumes that solely the choices by the reproducers and the mistakes by original authors or reproducers are responsible for different results obtained by reproductions. It also assumes that no choices are possible while reporting results, and no mistakes can happen during the execution of a data analysis or an in silico data collection. Further, we differentiate implementation mistakes from reporting mistakes. Reproductions can detect both,6 as will be detailed next. Mistakes happening during the data collection of in natura works and the resulting deviations are not the primary focus of a reproduction. We use “deviation” as a generic term for the difference between an original and a redone phase, which can have various reasons, including statistical variability, biases, mistakes, and varying choices. Mistakes that happen later, when interpreting the results, are also mostly ignored by reproductions, though we touch on them in the "Discussion and Recommendations" section of this article when discussing inappropriate inferences.

Figure 1 details the verifying function of certain prototypical redoing activities and helps to attribute differences in results to their possible reasons. For example, consider the case where the data analysis described in a work has been re-implemented, resulting in a different effect size than originally reported. The re-implementation reproduction in Figure 1 is represented with a blue area covering three phases (data analysis implementation and execution, as well as reporting) that are redone, and consequently, four potential reasons for differences in results: choices by the reproducers affecting the environment (j) or the implementation (h) of the data analysis, mistakes during its—original or redone—implementation (i), or reporting mistakes during manuscript preparation by the original authors or the reproducers (k). The collected data are inherited from the original work; thus, nothing during data collection can account for differences in results. Note that this allows no statement of whether the inherited phases were performed sensibly and appropriately. If the original authors made nonstandard or suboptimal decisions during data collection, an otherwise reasonable data analysis does not fit, and one should possibly criticize the original work rather than the reproduction for this.

The same information is presented in condensed form in Table 1. It displays the state of knowledge about reasons for differences in results for the prototypical redoing activities mentioned above, depending on the method of comparison. For now, only the comparison for identity is of interest. In the example, all columns with a large disk in the row "Re-implementation reproduction Identity Non-identical" are possible reasons for different results: mistakes during reporting or the data analysis implementation (either by the original authors or by the reproducers) and choices regarding the data analysis environment or implementation (by the reproducer, e.g., due to an insufficient description of the computation). If, in contrast, the re-implementation reproduction had resulted in identical results, all potential reasons indicated by a small disk can be considered unlikely. The equal signs of that row indicate that the re-implementation reproduction shares the sampling variability and any other deviations with the original work.

Returning to the scenario where the re-implementation reproduction resulted in non-identical results, one cannot conclude which of the potential reasons with a large disk in Table 1 are responsible for the difference. To narrow it down further, a re-execution reproduction can be performed. If this leads to identical results as originally reported, reporting mistakes (k) and data analysis environment choices (j) common to the re-implementation and the re-execution reproduction can be considered unlikely because they have a small disk in the respective row of Table 1. Hence, implementation choices (h) and implementation mistakes (i) concerning the data analysis (either while conducting the original work or the reproduction) remain as potential reasons. The final section of this article will discuss how these can be discerned by evaluating the consistency of results.

Note. The original and the reproduction result can be compared for identity or consistency (see section "Discussion and Recommendations"). The size of disks indicates the importance of various reasons for differing results. A small disk—with circle () and without ()—indicates that there is no difference or that a reason is less important. A large disk ( ) indicates a potentially important reason for non-identical or inconsistent results (but we do not know which one, if a row contains more than one). A circle around a small disk () indicates that a reason is possible, but the caused difference is probably not larger than for reasons with a large disk. The equal sign (=) indicates that the phase containing this reason is inherited and, therefore, assumed to be identical. Therefore, it cannot be responsible for differences in results by itself, though issues in that phase cannot be detected.

Most informative are cases where a large set of reasons can be ruled out (indicated by small disks) or a small set of reasons is identified (indicated by a large disk). If successful, we learn more from a re-implementation than from a re-execution reproduction, while it is the other way around if they produce non-identical results. The previous example also illustrates how different redoing activities can be combined to learn more about reasons for differences in results. Notably, a re-execution reproduction can produce no evidence against data analysis implementation mistakes (i) unless the data analysis source code is additionally checked for coherence with the methods section. Similarly, one cannot rule out data collection implementation mistakes (f) with a re-collection reproduction, where the data collection is merely re-executed but not compared with the corresponding descriptions. Because the term "reproduction" can refer to any of the prototypes, which differ in their verifying function, we recommend always indicating which exactly is conducted. However, when it comes to atypical cases, even this distinction is not sufficient; hence, the section "Defining Reproductions" introduces the concept of a reproduction’s scope, which we also recommend reporting.

Of course, by using Figure 1 and Table 1 to identify possible reasons for differences in results, one makes certain assumptions, which, if violated, invalidate the conclusions. First, one assumes linear progress through the various phases and the points where mistakes and choices by the reproducers can influence results. Second, one assumes that inherited phases are indeed not redone and consequently that their product is identical. For example, if reproducers obtain the data analysis source code from the original authors of a study, they operate on the belief that it is indeed the exact version that produced the study’s result. Instead, the authors could have (unintentionally) sent an old version of the code. Only if data, computation, and reporting are kept together via executable research compendia (Gentleman & Temple Lang, 2007; Nüst et al., 2017) can one be reasonably confident that they stayed constant. As another example, the collected data might have been preprocessed, either for usability reasons or, by necessity, due to resource limits. This leaves some mistakes undetectable during re-implementation reproductions, possibly without the knowledge of reproducers. For this reason, Table 1 should only be treated as a heuristic guide, providing evidence for or against types of mistakes but not making definitive statements. These and other possibilities are considered in the next section of this article, which introduces a unifying definition of reproductions instead of relying on the distinction between the prototypes.

Defining Reproductions

The Logic of Reproductions

Next, we describe the logic of a reproduction: one possible goal, the procedure to meet that goal, and the preconditions. Afterwards, we explain how the presence of faults is determined. Finally, we describe two obstacles that are commonly faced during reproductions.

One goal of a reproduction can be to verify that a (past) computation on collected data corresponds to the reported artifacts—this is the kind of reproduction the present article focuses on. The artifacts are the descriptions (detailing specified choices), the collected data (or the simulation source code, in the case of in silico works), and the results. The computation itself is uncertain7 but constrained by these three artifacts. Therefore, a reproduction with a goal of verification involves creating another such computation using the descriptions and the collected data (or the simulation source code), then comparing the original results with those of the reproduction effort. The comparison between original and reproduction results is thus the main outcome of the reproduction effort. It follows that the availability of at least one description, one result, and one set of collected data (or the simulation source code) is a precondition to performing a reproduction. Beyond the comparison of the results, optionally, the support for the claim can be evaluated.

In the second section of this article, "The Verifying Function of Reproductions", the comparison for identity has been discussed. If the results are exactly the same, one can conclude that the original computation on the collected data (and any prior repeated computation) has likely been performed as described.8 However, if they are not identical, Table 1 lists both mistakes and computational choices (i.e., environment choices and implementation choices) as possible reasons. Only mistakes cause a verification failure because computational choices are necessitated by insufficient descriptions or a lack of resources (such as special software requirements), neither of which automatically threatens the internal consistency of a work.9 Consequently, such obstacles should be taken into account: Rather than demanding identity, we suggest that results are called “consistent” if they differ no more than the largest expected difference given the obstacles encountered and how these obstacles were resolved. This also provides an alternative to the suggestion by LeBel et al. (2018) to always allow for a difference of 10%.

Therefore, it is necessary to understand the reason(s) for the observed difference between the original and the reproduction result. For example, if it is known that the result of a stochastic algorithm is robust up to the fifth digit across multiple runs, one can demand that results only match up to that precision. Any larger differences can then be confidently attributed to faults.

In the following, actions that cause a computation to be at odds with the reported artifacts in the scope are considered mistakes, manifesting as faults. A common mistake in the context of reproductions is writing incoherent descriptions. Other erroneous actions, such as choosing inappropriate statistical methods, might be considered mistakes in different contexts (e.g., during a validation) but are not the primary focus of a reproduction. Such concerns can be raised if a reproduction also chooses to evaluate the claim(s) of the original work. Once faults are suspected, limited resources may prevent reproducers from investigating further, even if identifying them would be a valuable contribution. The identification typically involves the reproducers obtaining the same faulty result and a proper explanation. Alternatively, such an explanation might be provided by the original authors. In addition to the identification of faults, a reproduction can improve the scientific literature if it also reports the results after correcting all faults.

Obstacles can prevent reproducers from obtaining a result. In the following, two common types of obstacles are differentiated (although there may be others). First, the descriptions of the computational steps may be underspecified, providing too little information for the reproducers to know exactly what to do. To obtain any result, the reproducers need to choose an option they deem appropriate. Second, even with sufficient information, the reproducers might be unable to follow the descriptions due to external constraints, such as a lack of resources. This requires them to deviate deliberately from the descriptions. In general, obstacles require the reproducers to make computational choices that potentially cause deviations from the original computation.

Box 1

Summary: Logic of Reproductions

• Goal: A reproduction can have the goal of verifying a computation on collected data. To this end, it checks whether a selected set of results from such a computation can be obtained by following the descriptions and involving the collected data (or the simulation source code, in the case of in silico works). We suggest comparing the original and reproduction results for consistency rather than identity.

• Procedure: Original and reproduction results are compared. If differences are observed, reproducers attempt to systematically explain them by attributing them to computational choices due to resolved obstacles or due to faults.

• Obstacles: Obstacles can prevent reproducers from obtaining a result. These include underspecification and external constraints.

• Mistakes: Actions that cause a computation to be at odds with the reported artifacts in the scope are mistakes, such as writing incoherent descriptions. They are either identified by closely studying the available materials, possibly with the help of the original authors, or they are suspected by ruling out other possible reasons for differing results. If identified, reproducers may provide corrected results.

• Results: The reproduction results are consistent with the original results if their difference can be expected given the choices necessitated by obstacles.

If the original authors have shared the full source code, the reproducers might be able to directly identify faults as the reason for an inconsistency. However, if the source code has not been shared, underspecified descriptions and external constraints can complicate the identification of faults. This is because both have the potential to completely explain differences between results. For example, if the reproduction uses a different statistical software, at first glance, this might be considered the sole reason for a different result. Only by investing additional effort and attempting to assess how much difference between results the choice of software explains can one reasonably suspect or rule out faults as an additional reason for different results.

Readers seeking an abridged exposition may skip the following formal definitions and immediately jump to their summary in Box 2.

In the rest of this section, we present formal definitions for central terms around reproductions. Starting with computations and descriptions, we delineate what reproductions are about at their core. We continue by defining scientific works and reproductions before exploring three reasons why a reproduction result can differ: underspecification, external constraints, and faults. Finally, we highlight the central importance of source code for reproductions.

Certain statements are accompanied by figures in the supplementary material representing formalized depictions of our definitions using the visual argument structure tool (VAST; Leising et al., 2023). We refer to them using the figure number and the specific number of that statement within that figure. If a statement defines a particular term, the term appears in bold font. Parentheses are used for optional parts of the term, which can be left out for brevity. Any further terms used for the first time are defined in their own indented statement directly following.

Computations

For this article, a reproduction’s primary target is a result produced by a (past) computation on collected data and reported in a scientific work (together with collected data or simulation source code and a derived claim, if available). It is also possible that one reproduction targets multiple results or claims.

Computation (see S8 n₁): A particular calculation of one or multiple results that can be dependent on (1) specific data, (2) a specific computational environment, and (3) specific code.

• Result (see S9 n₂): Data that is the product of a computation. May be used as support for one or multiple claims. For example, a result can be a single number, a table, a figure, a part of a table or a figure (such as a table cell or a plot reading), or a collection of such elements.

  • Claim (see S10/S11 n₂): A conclusion drawn from one or multiple results. For the purpose of a reproduction, the support for a claim can be found to be strengthened, weakened, or undetermined. It is at the reproducers’ discretion to decide which parts of a scientific work comprise a claim, as reproductions of single studies typically allow for a higher granularity than mass reproductions of multiple scientific works. For example, one statement by the original authors could be split into multiple claims by the reproducers.

• Data (see S7 n₆): Digital information

• (Computational) Environment (see S8 n₂): The hardware and software setup in which a computation is performed, for example, versions and (default) parameters of utilized software (dependencies), invisible states (such as those of random number generators), and the architecture of hardware (such as the CPU or the specific pocket calculator).

• (Source) Code (see S5 n₁): A low-level description of a computation that can be executed by a computer.

Put simply, a computation is the act of performing a calculation. Examples include:

• using a pocket calculator and a t-distribution table to perform a t-test;

• using statistics software with a graphical user interface, such as JASP or jamovi, to conduct a Bayesian regression;

• executing an R script to perform a permutation test;

• running an agent-based simulation in C; or

• training a machine learning model in Python.

Note that the first three examples are computations on collected data, while the final two examples generate (“collect”) data in the first place. For this article, during a reproduction, the source code for all of them may get re-executed, but the simulation source code of the final two examples does not get re-implemented, as this would be part of a replication.

Descriptions

Descriptions of computations are central to reproductions, as they enable the verification of results (International Committee of Medical Journal Editors, 2024; Livingston, 2020). In the following, we adopt a very broad notion of what constitutes a description:

Description (of a computation) (see S2 n₁): Instructions on performing a computation detailing computational choices, possibly referring to specific data, results, or other descriptions and mentioning a particular computational environment.

• (Computational) Choices (see S4 n₁): Decisions related to a computation.

An obvious example of a description is the methods section of a journal article detailing the analytical treatment of data. However, descriptions can also have other locations, such as the preregistration, the supplementary material provided with a journal article, or emails from the original authors. Finally, descriptions may not only be written in prose, but also in a particular programming language as source code which can be executed by a computer. Treating source code as a (human-readable) description highlights its usefulness for understanding and reproducing the original work long after it stopped being immediately re-executable.

Scientific Works

(Scientific) Work (see S10/S11 n₁): A compilation of descriptions, results, and, potentially, data, claims, and rules of inference that relate to each other. Other parts, such as theories, may also be part of a scientific work but are less relevant for reproductions. A prominent example is a journal article referencing publicly available open data. However, computational notebooks, blog posts, and other non-traditional research outputs can also qualify as scientific works.

• Rule of inference (see S10/S11 n₅): Reasoning behind making a claim based on a result (or multiple results).

Reproductions

Having established what the target of a reproduction is (i.e., the result of a past computation on collected data) and how a particular computation is communicated (with one or multiple descriptions), it is now considered how a reproduction can reach its goal, that is, verifying that the computation corresponds to the reported artifacts in the scope.

It is by using one or multiple descriptions, all choices left unspecified, and, potentially, data, one can infer a particular computation (see S3 n₁). If this second (redone) computation gives a sufficiently similar result to the one reported in the original work, the verification is successful.

A reproduction (see S12/S13/S14 n₉) is a research activity with the goal of verifying that a computation on collected data corresponds to the reported descriptions, collected data (or simulation source code, in the case of in silico works), and results. At its core, it involves performing a second computation inferred from the descriptions in the scope, from unspecified and conflicting choices, and using collected data in the scope (possibly generated by re-executing simulation source code). Then, the results are compared and (optionally) the support for the claims is evaluated.

• Collected data (see S7 n₁): The product of the data collection phase, potentially preprocessed (see Figure 1).

• Description scope (see S12/S13/S14 n₁₀): The set of descriptions that during a reproduction (a) informs a computation, (b) is checked for coherence, and/or (c) is re-executed (in the case of source code). For example, a journal article might describe a particular computation in its methods section, as well as in the linked preregistration and source code. If only the source code is re-executed, but its coherence with the other descriptions is not checked, the description scope solely consists of the source code being re-executed. For re-collection reproductions, separate description scopes cover the data analysis source code and the simulation source code.

  • Coherent descriptions (see S6 n₁): Descriptions in which none of the instructions conflict with each other.

• Collected data scope (see S12/S13/S14 n₄): Collected data commonly go through multiple computational stages. The collected data scope describes which of these stages are considered (i.e., redone) during the reproduction. If all stages are considered, one works with the primary collected data. Examples for computational stages are preprocessing, statistical testing, and visualization.

  • Primary collected data (see S7 n₅): First digital representation of the raw collected data (Gollwitzer et al., 2021).

    • Raw collected data (see S7 n₄): Original records, that is the first non-volatile form of collected data. In the case of electronic data collection, raw collected data and primary collected data coincide (Gollwitzer et al., 2021).

• Unspecified choices (see S4 n₃): Computational choices not prescribed by the descriptions in the scope, filling gaps due to underspecification. For example, if the original authors do not describe an algorithm’s required parameters, these would be unspecified choices during the implementation concerning the source code. Similarly, if the version of utilized software is not disclosed, that would be an unspecified choice during the execution concerning the computational environment.

• Conflicting choices (see S4 n₄): Computational choices that are at odds with the descriptions in the scope, for example, if reproducers choose to use a different computational environment than the original authors.

Of course, the results may simply differ because the unspecified choices varied between the original work and the reproduction. Also, reproducers may make choices that conflict with the descriptions out of necessity. For this reason, we recommend evaluating the consistency of results—rather than the identity—by exploring alternative unspecified choices and estimating the largest expected difference given any conflicting choices (see also "Discussion and Recommendations").

Verifying whether a computation corresponds to the descriptions in the scope requires the descriptions to be coherent in the first place; they cannot contain contradicting instructions. Thus, broadening the description scope can provide an additional chance to detect faults. Similarly, with a broader collected data scope, fewer computational steps remain unchecked, making the outcome “consistent results” more useful. The "Discussion and Recommendations section" of this article will also consider other senses in which a reproduction is restricted to a certain scope worth reporting.

To summarize, a reproduction verifies whether a computation on collected data corresponds to the reported artifacts in the scope by comparing the reproduction result with the original result. A reproduction may also attempt to verify the results of multiple computations, possibly involving multiple claims. This way, a scientific work in its entirety may become the target of a reproduction. Notably, in our definition, a reproduction is the act of attempting a second computation after meeting the preconditions (having available the necessary descriptions, collected data—or simulation source code—and original results). Performing a reproduction does not imply that any reproduction results are actually obtained (because obstacles may prevent that) or a specific relationship to the original results. Consequently, merely writing that a work has been reproduced is ambiguous regarding the outcome of the reproduction, which should always be mentioned in addition.

Differences Between Results

Suppose that when comparing the result of a re-implementation reproduction with the original result, one observes a difference. According to Figure 1, this means that either the second computation deviates from the original computation (due to the reproducers’ computational choices (h) and (j) or an implementation mistake (i)) or there was a reporting mistake (k).

When using computers, many minor deviations can cause different results. Simply changing the underlying hardware might already be enough to obtain a different result (e.g., see PyTorch Contributors, 2023; Stan Development Team, 2024). In the case of hardware acceleration (GPU programming) and parallelization, even repeatedly using the same hardware may lead to a different result (Diethelm, 2012; Pham et al., 2020). Similarly, a difference may occur due to added debugging statements (Monniaux, 2008), a different operating system (e.g., Bhandari Neupane et al., 2019; Glatard et al., 2015; Scaria, 2018), different compilers (Hong et al., 2013), different random seeds (Pham et al., 2020), updated dependencies with fixed bugs, new features or different defaults (e.g., Van Den Bergh et al., 2023), different algorithms (e.g., Crook et al., 2013; McCullough & Vinod, 1999) or a different statistical software with new defaults (e.g., Herberich et al., 2010; Hodges et al., 2022; W.D., 2019).

Reasons for Differences

If the reproducers followed the descriptions of the original work, a different result raises the question of why the computations changed. In principle, we differentiate between two broad reasons: Computations may have changed (1) due to the reproducers’ computational choices necessitated by obstacles or (2) because of mistakes, resulting in faults.

By obstacle, we mean everything that can prevent reproducers from obtaining a result (and that is not a fault). We discuss two common ones: underspecified descriptions (where the reproducers do not know what to do) and external constraints (where the reproducers know what to do but cannot do it). These require that reproducers make additional choices, which may deviate from those by the original authors, leading to changing computations. Computational choices can manifest while implementing the source code and during execution in a particular environment, see (h) and (j) in Figure 1.

The uncertainty arising from the possible existence of underspecification and faults has also been called “hidden uncertainty” (Auspurg & Brüderl, 2024, p. 3). In the following, we first focus on underspecified descriptions and external constraints before turning to faults.

Underspecified Descriptions

A first common obstacle and hence a possible reason for varying computations is underspecification, that is, the insufficient description of a computation. Whenever a part of the description seems underspecified to the reproducer in the sense that they can imagine multiple appropriate ways forward, they cannot know which one was taken by the original authors. For example, articles published in journals that impose a word limit will not mention every aspect of the computation. Procedures that are common in a field are not described in detail but are either referred to by a literature reference or are only mentioned by their name (e.g., “one-sided t-test” or “structural equation modeling”). If the reference or name is ambiguous (e.g., Weissgerber et al., 2018) or requires additional choices, the reproducers might make other unspecified choices than the original authors. Underspecification might be particularly high for fields with a pronounced technical task uncertainty, where “the use of technical procedures [is] highly tacit, personal, and fluid” (Whitley, 2000, p. 121).

The methods section in journal articles is frequently found to contain less information than deemed necessary by the reproducers (e.g., Crüwell et al., 2023; Hardwicke et al., 2021; Seibold et al., 2021). Providing the source code of a computation can alleviate this problem (Ince et al., 2012), in the best case accompanied by information about the computational environment. However, studies frequently notice missing source code (e.g., Crüwell et al., 2023; Laurinavichyute et al., 2022; Sharma et al., 2024) or missing information about the computational environment (e.g., Herbert et al., 2021). Most requests directed at original authors to provide the source code remain unsuccessful (Krähmer et al., 2023). Of course, source code cannot be made available when using software that is exclusively controlled by a graphical user interface.

Underspecification might also be a deliberate, though misguided, decision by the original authors. We see two reasons for deliberate underspecification: First, leaving out details may be used as an abstraction device (Gervasi & Zowghi, 2010), signaling what is important about a computation.10 However, as this complicates further investigations, a recommended alternative is explicitly marking what is important rather than omitting what is not (Benureau & Rougier, 2018). Second, underspecification might be employed as strategic ambiguity (Frankenhuis et al., 2023), making it difficult to ascertain faults. Such ambiguity decreases the worth of the scientific work to its readers, both in terms of falsifiability and as something to build on (Ince et al., 2012). For example, if the source code is withheld, it cannot be found to conflict with the methods section in an article. Then, different results might be (falsely) ascribed to the method’s stochasticity rather than faults. Also, a vague description generally allows for more conformant computations. Furthermore, such ambiguity raises the effort of reproductions, maybe preventing them in the first place.

External Constraints

Sometimes, it is impossible to follow the description of a computation up to the last detail due to external constraints. This is the case, for example, when reproducers do not have access to paid or discontinued software, lack sufficient computational resources, or do not have the necessary domain-specific knowledge (e.g., how to use a particular programming language or where to find the relevant result among many lines of output). In these cases, either the reproducers do not proceed at all (e.g., Crüwell et al., 2023) or they make choices conflicting with the description, possibly causing a different result (e.g., Seibold et al., 2021).

Faults

Another possible reason for a differing result is a fault due to a mistake—either by the original authors or the reproducers themselves. In the second section of this article, "The Verifying Function of Reproductions", we differentiated mistakes by the phase of the research process they happen in, particularly implementation vs. reporting mistakes, either during the data analysis or during an in silico data collection. What the resulting faults have in common is that they both cause the computation to be at odds with the reported artifacts in the scope. This is the type of fault we consider in the following.

In particular, we define faults as instances where the computation does not correspond to the reported descriptions, collected data (or simulation source code, in the case of in silico works), or results (see S15 n₂₄). For example, during the data analysis, the two experimental conditions might have gotten confused, effectively leading to the opposite claim.

Note that this explicitly does not cover faults that only manifest when applying the same code to newly collected data, as the definition is dependent on the specific results and collected data reported by the original authors. Also, from all the faults covered by this definition, there are four cases that are specifically mentioned in the following, as they are probably not frequently thought of—for example, cases 2 through 4 are not covered by the heuristic guide introduced earlier in this paper. Reproducers might benefit from considering them as well if they try to identify the precise reason for different results.

Fault Case 1 (see S15 n₂₃): The descriptions may not be coherent, for example, if the source code contradicts the methods section of a journal article or the preregistration gives different instructions for the computation than the methods section does.

Fault Case 2a (in natura works, see S15 n₂₂): The collected data reported by authors may differ from the collected data utilized to obtain the result, for example, because the data in a repository were changed by their provider (see Goes, 2023).

• Reported data (see S7 n₃): Data published alongside or referenced from a scientific work.

• Utilized data (see S7 n₂): Data used during a computation.

Fault Case 2b (in silico works, see S16 n₁₈): The simulation source code reported by authors may differ from the one executed to obtain the primary collected data.

• Reported code (see S5 n₄): Code made available by its authors.

• Executed code (see S5 n₂): Code run during a computation.

Fault Case 3 (see S15 n₂₁): The data analysis source code reported by authors may differ from the one executed to obtain the result. Even employing version control does not protect against accidentally using a different version.

Fault Case 4 (see S15 n₂₀): The result reported by authors may not correspond to the obtained result. For example, for elaborate computations, the actual result must be identified among many lines of output and subsequently transferred to the publication, two actions during which mistakes can happen.

• Reported result (see S9 n₃): The result given in a scientific work.

• Obtained result (see S9 n₁): The result calculated during a computation.

Of course, researchers can make more kinds of mistakes than are covered here.

Notably, mistakes, according to our definition, do not extend to conceptual mistakes. Imagine that a researcher intends to analyze data from two independent groups with a dependent t-test. This is conceptually wrong and will lead to results lacking validity. However, if the code correctly implements the (wrong) intention, according to our definition, it is not a mistake in the context of reproductions, which aim to verify computations.

Nonetheless, reproducers are encouraged to incorporate their concerns about substantive flaws in the computation or aspects of the study design and data collection into their appraisal of whether the claim is still supported. Finally, they can also evaluate the rule of inference employed in the original work.

Box 2

Summary of Formal Definitions

In the preceding definitions, we introduced the results of a computation on collected data as the target of a reproduction. By descriptions, we refer to anything that characterizes the computation, such as parts of a methods section in a journal article or source code. Descriptions, together with results and collected data (or simulation source code, in the case of in silico works) and claims, are part of scientific works. During a reproduction, another computation is performed based on the descriptions in the scope, and the results are compared for consistency. Underspecification, external constraints, and faults are all reasons for a reproduction to produce different results than the original work. Human actions that cause a computation to be at odds with the reported artifacts in the scope, either during implementation or during reporting, are mistakes manifesting as faults. Four cases of faults are emphasized: (1) incoherent descriptions, (2) difference between the utilized and the reported collected data (or simulation source code), (3) difference between the executed and the reported data analysis source code, and (4) mismatch between the results one obtains and the results one reports.

The Central Importance of Source Code for Reproductions

From the discussion so far, it becomes clear that there is a subjective side to reproductions: Underspecification is a property ascribed by the reproducers to a description and consequently depends on their knowledge and the methods standards they apply. Similarly, it is conceivable that original authors and reproducers differ in their interpretation of a particular description. The implications of this subjectivity are twofold: First, communication between the original authors and reproducers can be key to resolving uncertainty. Second, scripted computations are in a unique position to facilitate a fruitful reproduction due to their dual nature: They are human-readable and they are used to control computers, thus providing an accurate (though not complete11) record of a computation at a certain point in time. This deterministic automation sets them apart, for example, from statistical software with only graphical user interfaces, where a description of the performed manual steps is at a higher risk of being insufficient or inaccurate.

Checking the coherence of source code with the other descriptions in the scope is, therefore, a key activity that can be performed during a reproduction. One can discern three cases:

• The source code may clear up otherwise underspecified aspects of the descriptions.

• The source code may be partially or completely missing. In this case, it needs to be re-implemented.

• The source code may agree with or disagree with what is stated in the other descriptions. The latter represents a fault that needs to be resolved to obtain any result during the reproduction. Usually, statements from the preregistration take precedence over subsequent implementations.

Crucially, this requires the availability of source code as well as prose. It is not preferable for source code to be the only means of describing a computation, because it might not contain the reason for that computation (Benureau & Rougier, 2018) nor their authors’ expectations about the result (Peng, 2022). Further, a precise description of a computation does not obviate the need for robustness tests (Rahal, 2024; Varoquaux, 2016). Finally, while source code improves the precision, reproducers still need certain domain-specific knowledge to make use of it.

With the definitions introduced in this section, the difference between the three reproduction prototypes can be formulated precisely. Starting with the re-execution and the re-implementation reproduction, first, they differ in their description scope—the former would consider the data analysis source code by inheriting (i.e., re-executing) it but not check this source code for coherence with the methods section. In contrast, the latter would not consider the source code at all. Second, and consequently, the number of unspecified choices is expected to be more pronounced for a re-implementation reproduction. A re-collection reproduction, in contrast, is only possible for in silico works. It usually has no data, but includes the simulation source code. In addition, this source code will be executed as part of the re-collection reproduction. Therefore, it also comes with a whole new opportunity for unspecified choices. However, because there are also atypical cases, merely stating which prototype has been performed is generally not sufficient. For example, a re-implementation reproduction may base its data analysis re-implementation on the original source code, or a re-execution reproduction might also check the original data analysis source code for coherence with the methods section. Therefore, in the final section of this article, we give specific recommendations on how to report a reproduction. Also note that Figure 1 and Table 1 do not cover such atypical cases.

Performing a Reproduction

In the following, building on the definitions, we sketch the reproduction procedure to verify that a computation on collected data corresponds to the reported artifacts (i.e., the descriptions, collected data—or simulation source code in the case of in silico works—and results) in the scope. It consists of (a) defining the scope of the reproduction in terms of the involved works, claims, results, collected data, and descriptions, (b) ensuring the availability of descriptions, results, and collected data (or simulation source code), (c) performing the computation to obtain results, potentially by resolving obstacles, (d) evaluating the results’ consistency, potentially by assessing the largest expected differences in results, and, optionally, (e) evaluating whether the claims are supported given the information acquired during the reproduction. We conclude with remarks on reporting the outcome of a reproduction.

(a) Defining the Scope

First, reproducers must choose which scientific works, claims, results, descriptions, and collected data (or simulation source code) are part of the reproduction’s scope:

• Works: Which scientific works—in natura or in silico—are attempted to be reproduced? Reproductions can focus on individual or multiple works.

• Claims: Which claims within a work are being evaluated? Reproducers can focus only on a subset of all claims presented in a scientific work, for example, those that appear in an abstract. Or disregard all claims and only focus on results if no claims have been made.

• Results: Which results are being evaluated as the reproduction’s target? For example, this can be a particular figure, a table, or a set of numbers. There are many degrees of freedom for reproductions when selecting the evaluated results. One possibility is to focus on all results that (supposedly) support the claims chosen before.

• Obtained collected data or simulation source code:

  • In the case of in natura works: Which computational stages have already been performed for the collected data obtained for this reproduction and which are performed again? In other words, how do the obtained data differ from the raw measurements? In the best case, reproductions obtain primary collected data, but this is not always possible. Then, faults in the processing up to the point where the collected data are obtained are more difficult to identify.

  • In the case of in silico works: Which computations in the research process are performed again? In other words, to what extent is the simulation source code being re-executed?

• Involved descriptions: Reproducers may only focus on a subset of the available descriptions to inform their computation, be checked for coherence, and be re-executed. Is the re-executed code used as is, or is it being modified or extended? The smaller the description scope, the fewer faults can be detected.

• Applied methods standards: Which methods standards are followed when making unspecified choices, checking the coherence, estimating the largest expected differences, and evaluating the appropriateness of specified choices? This will likely get more detailed and can be elaborated on throughout the reproduction.

For reproducers performing mass reproductions of multiple scientific works, rather than evaluating all results, they can evaluate those that support claims that are mentioned in the title, the abstract, or the first table or figure, following Hardwicke et al. (2018) and Alipourfard et al. (2021).

(b) Checking the Preconditions

If the descriptions or the collected data (or simulation source code) in the scope are not available or if the original results cannot be identified, the reproduction cannot be conducted. The original authors or any other relevant stakeholders should be contacted at this point to provide what is missing, or the scope needs to be adjusted to what is available. Otherwise, the reproduction outcome for the affected results and claims is undetermined due to failing a precondition (e.g., lack of collected data).

From this step on, readers can compare the procedure with Figure 2. It depicts a flowchart for performing a reproduction in the case of one result and one claim. A more detailed version is available in the Appendix as Figure A1.

(c) Obtaining Results

After ensuring the preconditions are met, reproducers need to be able to compute results. This requires them to understand the descriptions in the scope and the reported collected data (or simulation source code). The descriptions need to be coherent, that is, not contradict themselves or each other. Also, the reproducers must have the necessary resources to perform the computation, for example, in terms of knowledge, time, software, and hardware. Finally, they need to be able to identify the results in the output. All these requirements can be obstructed by obstacles, which require reproducers to make computational choices not evident from or contradicting the descriptions. Reproducers frequently encounter obstacles but may not acknowledge them as such, for example, when they use a different operating system (e.g., McCullough & Vinod, 2003) during execution than the original authors.

If reproducers identify faults, such as incoherent descriptions, they need to correct them. When reproducers encounter obstacles, in many instances a helpful strategy is to contact the original authors and ask for clarification. However, this may be impossible due to limited reachability and time constraints, or it might not help resolve the obstacle. In that case, another strategy is to start by making the most likely choice according to chosen standards to obtain any results (Martel García, 2013). For example, if it is not clear whether a psychological study has performed a two-sample t-test for equal or unequal variances, psychologists might want to start with the latter (Delacre et al., 2017). However, choosing among multiple options might not be possible if the reproducers are limited by their resources. Hence, as a third strategy, reproducers might need to make a choice conflicting with the descriptions, for example, by simplifying computations or using different software. Finally, if the obstacles cannot be resolved, the reproduction’s outcome concerning the affected results and claims must be regarded as undetermined by stating the respective obstacle.

(d) Evaluating the Consistency of Results

Once results have been computed, reproducers can evaluate their consistency with the originally reported results. For this, each original result is compared with the respective reproduction result. While LeBel et al. (2018), following Hardwicke et al. (2018), generally consider a result consistent if it matches within a 10% margin of error, we propose an approach that is aware of the obstacles faced during the reproduction and allows attributing differences to faults with more confidence, relative to a set of methods standards.

In the easiest case, the original and the reproduction result can be deemed consistent because they are identical (i.e., they match at the maximum precision their representation allows). If a study reports a numerical result with four digits, for example, all four digits need to match after the reproduction result has been rounded to the same precision. In some instances, such identical results can be obtained by using information provided by the original authors. For example, if asked about a non-identical result, the original authors might argue that the reproducers misunderstood a description and provide them with a way of obtaining exactly the same result, in which case the results can be regarded as consistent. On the other hand, the original authors might confirm the existence of a fault. If the fault is found to be responsible for the non-identical result, the result can be regarded as inconsistent. Otherwise, other possible reasons for the difference need to be considered, as detailed below.

If the results are not identical, it is still reasonable to consider them consistent if their difference can occur due to how obstacles were resolved by the reproducers. For example, if the difference between results can be explained solely by the fact that a different operating system has been used (and it is not too large), most people would still consider them consistent. Then, it is up to the readers to decide whether the change between the two computations (here, a different operating system) is critical to them. Therefore, reproducers should investigate whether the results differ no more than the largest expected difference given how any obstacles were resolved. In the following, we describe two strategies for deciding on the consistency of results. It is recommended to try the first strategy if the number of possible computations is finite and the second strategy if it is infinite.

The first strategy is that any choice to resolve underspecification can be varied to another appropriate option until identical results are obtained. Remaining faults as defined in this article can now be considered less likely, and the results are deemed consistent. For example, imagine that the description of the original authors only mentioned a “t-test”. If the reproducers initially calculated a two-sample t-test for unequal variances and obtained a different p-value, a test assuming equal variances can be tried next—both analytical pathways are consistent with the underspecified description. If the p-value from the reproduction is now identical to the original one, they can be regarded as consistent, and faults can be considered less likely. If no identical result is obtained despite trying all appropriate choices, the original and the reproduction result can be regarded as inconsistent. Of course, this evaluation is highly dependent on the chosen methods standards.

However, there may be instances where it is impossible to try all possible (appropriate) combinations of choices. Also, sometimes the instructions cannot be followed, leading to conflicting choices. Therefore, a second strategy is to estimate the impact of deviations in terms of differences between results—similar to an estimation of the model variance during a robustness check (Young, 2009)—and determine whether the original and the reproduction result differ more than expected. For example, if no random seeds are reported, one can perform multiple computations with multiple random seeds and check whether the original result stands out from the reproduction results (e.g., Sego et al., 2024). Or, if the operating system is different, reproducers can make an educated guess concerning the tolerable differences due to floating point arithmetic. Empirical norm values for such standard cases are rarely available, for example, concerning the largest expected difference when changing a software package (for exceptions, see Jen et al., 2023; Letter, 2021; see also Hodges et al., 2022; Keeling & Pavur, 2007; McCoach et al., 2018). Again, the reproduction results can be regarded as inconsistent or consistent with the original result, depending on whether they differ more than the largest expected difference or not. As before, this evaluation crucially depends on the chosen methods standards.

If no strategy is feasible to decide on the consistency of results, it remains unclear whether the observed difference between original and reproduction result can be attributed to a fault or simply to a choice in the step of resolving an obstacle. In this case, despite having obtained a result, the consistency must be declared undetermined, stating all relevant obstacles.

Box 3

Possible Outcomes for the Consistency of Results

• Consistent. For example:

  • identical results by only following the descriptions

  • identical results after using information provided by the authors

  • identical results after trying other appropriate options

  • actual difference of results no larger than the largest expected difference

• Inconsistent. For example:

  • out of options after getting non-identical results from all appropriate computations

  • the smallest obtained difference of results is still larger than the largest expected difference

  • a fault is found to be responsible for the difference

• Undetermined. For example:

  • failed precondition (i.e., missing collected data / simulation source code, description, or result)

  • unresolved obstacles, and hence no result obtained

  • estimation of the largest expected difference is infeasible

Until this point, the reproducers made a separate consistency evaluation for every original result selected for the reproduction. Per original result, the reproducers either did not obtain a reproduction result (due to failed preconditions or unresolved obstacles), they obtained an identical result, or they obtained no identical result and (if possible) assessed the largest expected difference between the results given how any obstacles were resolved. If the results are inconsistent or the consistency is undetermined, a reproduction can also explore why. This is an iterative process where the computation is modified until the same (faulty) result is obtained. It is most likely to succeed if the original data analysis source code is provided.

One can now revisit Table 1, which also displays the state of knowledge implied by evaluating the consistency of results. The example discussed in "The Verifying Function of Reproductions", earlier in this paper, concluded with implementation choices and implementation mistakes concerning the data analysis as potential reasons for a different result. By evaluating whether the result of the re-implementation reproduction is consistent (rather than identical), one answers the question of whether the difference is possible given how obstacles were resolved by the reproducers and depending on the chosen methods standards. Suppose the results are found to be inconsistent. Then, one can say that the most important reasons for differing results is a data analysis implementation mistake (i; indicated by a large disk)12 and any potential deviations due to choices concerning the data analysis implementation (or the environment) are negligible in comparison (indicated by a circle around a small disk). If, in contrast, the results are deemed consistent, one can conclude that data analysis implementation choices (and possibly environment choices) are among the most important reason for differences, and any potential implementation mistakes (and reporting mistakes) regarding the data analysis are probably not larger in comparison—again, contingent upon the applied methods standards.

Table 1 also includes an example of evaluating the consistency of a replication’s results. In addition to computational choices, the possible effects of sampling variability need to be considered here, and the results are inconsistent only if both cannot plausibly explain their difference. Evidently, replications do have a verifying function of their own, as consistent results change the belief about reporting and implementation faults (which are the two main threats to internal consistency discussed in this article): While their absence is not guaranteed, their consequences are small compared to computational choices and sampling variability. Inconsistent results, however, can have many possible reasons, and a lack of internal consistency is only one of multiple explanations.

(e) Evaluating the Support for the Claim

If the reproducers have selected claims for evaluation, it is also of interest whether these are still supported, given the information acquired during the reproduction. Consistent results are neither necessary nor sufficient for a claim to be still supported. Evaluating the support is optional, although it is recommended. We will start by discussing the simple case where a claim is backed by one result in the original work. Then, it is sufficient to evaluate the rule of inference for this one result, along with the collected data and specified choices that led to it.

First, one considers the collected data, which might be deemed inappropriate to support a claim. This consideration includes the simulation source code, in the case of in silico works. For example, the sampled population might be considered unfit to make a particular claim, the operationalization of a construct could be regarded as unsuccessful, or the reproducers might argue that the study design does not allow for causal claims. Also, the reproducers might not know whether the collected data are appropriate for the result to support the claim. In all these cases, the reproducers can conclude that the support for the claim is undetermined, given the collected data. Otherwise, if the reproducers have no concerns about the collected data, the evaluation of the claim continues.

Second, the reproducers might have issues with the original authors’ specified choices regarding a claim. For example, the reproducers might find that the utilized statistical methods deviate from best practices in the respective field or that inadequate preprocessing calls the claim into question. Also, the reproducers might not know whether the specified choices are appropriate for the result to support the claim. Consequently, in these cases, the support for the claim might be regarded as undetermined due to an inadequate computation. If the reproducers have followed up with an appropriate computation, they can continue evaluating the claim. Similarly, if the reproducers do not find the computation to invalidate the claim, they can continue with its evaluation.

Finally, it is necessary to check whether a claim is still supported, given the rule of inference used in the original work and the results obtained by the reproducers. If this is undecidable because the rule of inference is unknown or unfit, reproducers can choose an alternative rule of inference or regard the claim’s support as undetermined. The reproducers might have performed multiple computations per original (focal) result due to obstacles and hence obtained multiple results. In that case, all reproduction results for that particular original result need to be evaluated. If all of them support a claim or do not support it, its support can directly be regarded as strengthened or weakened, respectively. For example, if multiple results have been calculated (because of underspecification) until one was identical and all of them support the claim, the support for the claim can be considered strengthened. In another example, suppose the reproducers have estimated a largest expected difference. If all (hypothetical) results within this difference support the claim, it can directly be considered strengthened. If only some reproduction results support the claim, no general rule exists, and one must decide on the strength of the support on a case-by-case basis. Of course, one may also conclude that the support for the claim is undetermined.

As mentioned earlier, a claim supposedly supported by a result that the reproducers could not compute at all (due to failed preconditions or unresolved obstacles) must be regarded as having undetermined support by stating the reason why no result was obtained. Moreover, even identical results do not guarantee support for the claim: The original authors might have employed an inappropriate rule of inference. For example, the reproducers might write the following: “Although we could exactly reproduce the originally reported numerical result, the support for the claim is weakened because the employed rule of inference is inappropriate: The confirmatory nature of the study warrants correction for multiple testing, which is missing. When the α level is correctly reduced, the result is not significant anymore.”

However, if a claim is backed by multiple results in the original work, all of them need to be considered. Because the rule of inference can be an arbitrary function that maps the results onto a claim, it might not be possible to consider the results individually in their support for the claim. Of course, the original authors may have performed such a “holistic” assessment of the results implicitly, making its criteria unknown to the reproducers. The case of multiple results is not covered by Figures 2 and A1.

As an example of multiple results, significant improvements in speed and accuracy might support the claim that an experimental condition improves performance. Consequently, the reproducers need to check that both speed and accuracy indeed improve significantly. In this example, if only one of the reproduction results shows a significant improvement, the support for the claim would be considered undetermined, and if neither shows significant improvements, the support for the claim would be considered weakened. In general, this depends on the logical function that maps the results onto claims.

Box 4

Possible Outcomes for the Support for a Claim by One Result

• Strengthened. For example:

  • all computations for the same (focal) result support the claim

  • some computations for the same (focal) result do not support the claim, but the strength of the support is still sufficient

• Weakened. For example:

  • no computation for the same (focal) result supports the claim

  • some computations for the same (focal) result do support the claim, but the strength of the support is insufficient

• Undetermined. For example:

  • inappropriate collected data

  • inappropriate specified choices, and no alternative computation was conducted

  • unresolved obstacles, and hence no result obtained

  • some computations for the same (focal) result do support the claim, but the strength of the support cannot be determined

  • rule of inference unknown or unfit, and no alternative employed

(f) Reporting the Outcome

Reporting a reproduction should be more nuanced than just announcing its success or failure for at least two reasons. First, although reproductions should give very similar results, reproducers may underestimate the largest expected difference in results, leading to premature statements about their failure. Second, there is considerable variability in the activities that all count as reproductions and similarly in the conclusions that can be drawn from them. Therefore, summaries of reproductions should (1) use careful language, (2) also evaluate the substantive claim, and (3) be transparent about their limitations.

We recommend that a reproduction separately reports the consistency of the results and the support for the claims. Consistent results mean that the original computation has largely been performed as described, and among the most important reasons for differing results is the way obstacles were resolved. Faults may exist, but their impact on the results is probably not larger in comparison.13 Inconsistent results, on the other hand, might have multiple possible reasons:

• The methods standards chosen by the reproducers did not cover the computational choice by the original authors.

• Aspects of the descriptions were meant differently by the original authors than interpreted by the reproducers.

• Faults in the original work or the reproduction.

Note that for the consistency evaluation, the original result is compared with the reproduction result of a computation that only includes corrections due to faults. When evaluating the support for the claim, however, one uses the reproduction result of a computation that also includes other corrections that reproducers deem necessary to decide on the support (such as changing the computation or the rule of inference because the one originally proposed did not follow current best practice).

The outcome of a reproduction (i.e., the evaluations of consistency of results and support for the claims) critically depends on the reproduction’s scope. For example:

• Interpreting the outcomes from a mass reproduction of multiple works depends on the article inclusion criteria.

• Interpreting the outcome of a reproduction of a single work depends on which results and claims have been selected as targets.

• Interpreting the consistency evaluation of a result depends on the collected data scope, which descriptions were involved, the way any obstacles were resolved, and how the largest expected difference was estimated. In turn, this depends on the methods standards applied by the reproducers.

• Interpreting the support for the claim depends on the results that were included in the evaluation and on the methods standards that were used to judge the appropriateness of specified choices.

Therefore, we provide a set of recommended pieces of information that every reproduction should report, summarized in Box 5 and elaborated in the supplementary material S1. Besides what has already been mentioned as part of the scope, this also includes the reproducers’ expertise.

Box 5

What to Report in a Reproduction

• Reproducers’ expertise

  • Subject-matter (theory and claims)

  • Methods (modeling and computation)

  • Did the reproducers contribute to the original work?

• Scope

  • Works: Report any permanent identifiers of the investigated works and whether they are in silico works.

  • Results: Report which results are being investigated (mention page or table where the results can be found).

  • Claims: Report which claims, if any, are being investigated.

  • Collected data / simulation source code: To what extent have the obtained collected data been preprocessed? To what extent is the simulation source code being re-executed?

  • Descriptions: Which descriptions inform the computation, are checked for coherence, and/or are re-executed?

• • Methods standards: Which methods standards are followed when evaluating computational choices and deviations? Which unspecified choices are explored, and how is the expected difference estimated?

• Preconditions (select all that apply)

  • ☐ At least one description was available

  • ☐ Collected data (or simulation source code) was available

  • ☐ The original results were available for comparison

• Obstacles: Were any obstacles encountered, like underspecification or external constraints, and (how) were they resolved?

• Faults: Were any faults identified?

• Outcome for every evaluated result

  • Reproduction result (with potential faults corrected)

  • ❍ Consistent

  • ❍ Inconsistent

  • ❍ Undetermined (give reason)

• Outcome for every evaluated claim

  • Reproduction result (with all potential corrections)

  • ❍ Support strengthened

  • ❍ Support weakened

  • ❍ Undetermined (give reason)

• About the reproduction process

  • Were the original authors contacted (e.g., for clarification), and did they respond?

  • Are there any suggested improvements (e.g., better documentation, initial sharing of collected data)?

Discussion and Recommendations

This article contributes to the literature on redoing activities—scientific activities that redo some phases of a previous study—in particular on (computational) reproductions. We define reproductions as redoing activities that work with the originally collected data (or the reported simulation source code, in the case of in silico works) and aim to keep the data analysis as similar as possible. This article explicates their logic and provides a conceptual framework for describing their different flavors and understanding their epistemic function, that is, what one learns from their conduct. Further, it establishes a new standard for the consistency of results, which accounts for uncertainty in the model description and provides recommendations on reporting the outcome of a reproduction.

In the section "The Verifying Function of Reproductions", we presented a heuristic guide detailing the epistemic function of reproductions in comparison to replications. By comparing how these redoing activities provide evidence for or against which types of mistakes (see Figure 1 and Table 1), we clarify their respective roles in verification. We discerned three reproduction prototypes, depending on whether the original data analysis source code is re-executed, re-implemented, or—in the case of in silico works—data are re-collected by re-executing the reported simulation source code. Further, reproductions are discerned from replications, which involve collecting new data (or re-implementing the simulation source code). We introduced data analysis implementation and reporting mistakes and explained that a re-execution reproduction alone cannot detect the former. As a remedy, one can combine re-execution with a re-implementation reproduction or additionally check the source code for coherence with the other descriptions. The verifying function of redoing activities is their ability to detect mistakes that threaten the internal consistency of the computational aspects of a scientific work. One way of detection is by comparing the results for identity or consistency. As replications cover more types of mistakes, attributing differences in results to any one of them is more difficult.

Differentiating three prototypes is not sufficient, however, to describe the actual variety and nuances of reproductions. Therefore, we introduced formal definitions for reproductions and surrounding concepts, from which recommendations were derived. In addition, the research process depicted in Figure 1 is simplified, and the derived heuristic guide is dependent on certain assumptions also modeled formally in the "Defining Reproductions" section: First, one assumes linear progress through the various phases and the points where mistakes and choices by the reproducers can influence results. Second, the conclusions drawn from Figure 1 and Table 1 require that code and collected data (or simulation source code) were indeed used by the original authors as reported (i.e., fault cases 2 and 3 are not present). Finally, one could question the asymmetry with which the heuristic guide treats inherited phases as set. Consequently, choices made by the original authors during these phases are not considered as reasons for different results. For example, if missing values during a data collection in natura were recorded with “$-$999” and the original data analysis source code correctly removes these values, whereas the re-implemented code does not (expecting “NA”), according to this guide a different result would be attributed to a mistake by the reproducers whereas common sense would rather attribute this to the original authors’ unconventional data collection choice.

In "Defining Reproductions", we sketched the logic reproducers can follow to answer whether a computation corresponds to the reported descriptions (e.g., methods section or source code in an online repository), collected data, and results, or has a fault. A reproduction involves performing another computation and comparing the results. Naturally, it requires the availability of some descriptions, results, and collected data (or simulation source code, in the case of in silico works). Further, it involves checking the coherence of the descriptions in the scope. At any point, reproducers may see no, exactly one, or multiple paths forward, depending on their interpretation of the descriptions—guided by the chosen methods standards—and any obstacles they encounter, such as underspecification and external constraints.

If the reproducers obtain a result, it may differ from the original result due to varying computational choices or because of faults. However, because a simple comparison for identity cannot discern them, we recommend evaluating the consistency by exploring the impact of computational choices on results before suspecting faults as a last resort. Results are deemed consistent if they differ no more than the largest expected difference given how any obstacles were resolved, and inconsistent otherwise. For example, results can be demanded to only match up to a certain precision, given that a different numerical optimizer has been used. Importantly, both consistent and inconsistent results are contingent on the methods standards the reproducers chose. The consistency can also be undetermined if no result was obtained or if it is infeasible to estimate the largest expected difference.

While the consistency is threatened only by faults that cause the computation to be at odds with the reported descriptions, collected data (or simulation source code), or results in the scope, a reproduction can separately evaluate the support for any claims connected to the results, thus also considering the appropriateness of the collected data, the specified choices, and the rule of inference. The support for claims can be considered strengthened, it can be undetermined, or, in some instances, it may even be deemed weakened. Finally, reproducers should report the scope of their reproduction by detailing which descriptions were involved and to what extent any collected data were already preprocessed.

We demonstrated how underspecification can reduce the potential epistemic value of scientific works and their reproductions. For example, if the data analysis in a study is underspecified, and a reproduction obtains results that are inconsistent with the original article, more uncertainty remains about the reason for the inconsistency: Either the reproducers made different implementation choices, or a mistake has happened. For researchers aiming to facilitate verification and allow informative reproductions, this might serve as an additional incentive to be as transparent as possible when communicating the results of one’s research.

We discerned four special cases of faults, which can guide reproducers on where to look for the source of differences: (1) incoherent descriptions, (2) different collected data (or simulation source code), (3) different data analysis source code, and (4) incorrect reporting of results. However, even for consistent results, faults may be left if the difference they introduce is not larger than the difference caused by obstacles such as underspecification.

The approach described here is subject to some limitations. First, the consistency evaluation heavily depends on the chosen methods standards and how the expected difference is estimated. Second, a consistent result from a re-implementation reproduction does not guarantee that the implementations are free of any type of fault. This is because there can be faults that just do not manifest for the collected data or cancel each other out. Third, even identical results suggest but do not guarantee that the original computation has been performed as described. For example, if the domain of possible results is constrained, the original computation might contain a fault, but the reproducers might have gotten the same result through a different approach. The reproducers might even have made a mistake on their own, possibly the same as the original authors. Fourth, we treat the consistency of results as a binary variable, which only approximates a continuous reality. Fifth, in many instances, we currently lack empirical norm values for standard cases of deviations and the resulting differences. Sixth, we do not discern between mistakes and fraud, that is, instances where the descriptions were written in a way to deliberately deviate from the actual computation conducted.

Taken together, we provide a conceptual framework to describe computational reproductions and their epistemic function of verification in detail. Revisiting the challenges described in the opening section of this article, the procedure we recommend addresses them in several ways:

First, by providing standards, common situations during reproductions can be dealt with consistently. For example, if the data analysis source code or the collected data have not been shared, there is a defined way of dealing with it (i.e., changing the scope or reporting an undetermined outcome due to failed preconditions). If the reproduction results differ from the original results, the determination of the outcome is not arbitrary. Moreover, assistance by the authors is welcome, but the proposed definitions do not depend on their availability. In addition, we sketch how results can be synthesized on the level of individual papers by evaluating the support for corresponding claims.

Second, we believe checking the coherence of descriptions is integral to reproductions. While we acknowledge that source code is often not executable by reproducers, by treating source code as a regular description reproducers can still obtain valuable insight into the original computational choices.

Third, by scoping the outcome of a reproduction, reproducers are transparent about its limitations. The described reproduction prototypes serve different purposes, and the description scope makes that known. Similarly, a reproduction working with preprocessed collected data has its worth if one keeps in mind that only faults after the preprocessing can be identified by it. Lastly, by being explicit about which results are part of the scope, it becomes obvious that a scientific work is typically not reproduced as a whole, but that a reproduction always makes a statement about a particular result, selected with a particular strategy.

Finally, with the conceptual framework in mind, we would like to provide a few recommended actions. On the one hand, we suggest that original authors include a computation summary in their appendix, detailing (1) the claims they make, (2) which results support these claims, (3) the rules of inference connecting claims and results (i.e., when a claim is considered strengthened or weakened), and (4) which are the most important claims and results. On the other hand, reproducers should consider the precise question they would like to answer through strategic choice of the reproduction’s scope and combination with other redoing activities. Further, if possible, reproducers should use the primary collected data, that is, the first digital representation of the original records collected for analysis, so fewer faults can go unnoticed. In addition, ideally all descriptions available to the reproducers should become part of the scope, thus checked for coherence with each other and informing the computation. Every deviation from this default should be specifically indicated when stating the scope. As one possible implementation, we provide a template in the supplementary material S1 which is ongoing work and will be updated based on practical experiences. We invite all readers to apply the conceptual framework presented here to the reproductions they encounter in their own field and treat it as a starting point, ready to be improved and expanded on.

Author Note

FK was funded by the German Research Foundation (DFG SCHO 1334/6-1, awarded to Felix Schönbrodt) in the META-REP Priority Program (SPP 2317). FM was supported by the Austrian Science Fund (FWF, grant DOI: 10.55776/ESP133).

The authors would like to thank Daniel Krähmer, Laura Schächtele, and Julian Möbus for their helpful comments on earlier drafts of this paper.

Supplementary Material

The supplementary material is available at Zenodo: https://doi.org/10.5281/zenodo.20202051

References

Albertoni, R., Colantonio, S., Skrzypczyński, P., & Stefanowski, J. (2023). Reproducibility of machine learning: Terminology, recommendations and open issues. arXiv:2302.12691v1. https://doi.org/10.48550/ARXIV.2302.12691

Alipourfard, N., Arendt, B., Benjamin, D. M., Benkler, N., Bishop, M. M., Burstein, M., Bush, M., Caverlee, J., Chen, Y., Clark, C., Dreber, A., Errington, T. M., Fidler, F., Field, S., Fox, N. W., Frank, A., Fraser, H., Friedman, S., Gelman, B., … & Wu, J. (2021). Systematizing confidence in open research and evidence (SCORE). SocArXiv, 46mnb_v1. https://doi.org/10.31235/osf.io/46mnb

Altman, D. G., Simera, I., Hoey, J., Moher, D., & Schulz, K. (2008). EQUATOR: Reporting guidelines for health research. The Lancet, 371(9619), 1149–1150. https://doi.org/10.1016/S0140-6736(08)60505-X

Ankel-Peters, J., Brodeur, A., Dreber, A., Johannesson, M., Neubauer, F., & Rose, J. (2025). A protocol for structured robustness reproductions and replicability assessments. Q Open, 5(3), qoaf004. https://doi.org/10.1093/qopen/qoaf004

Arguillas, F., Christian, T.-M., Gooch, M., Honeyman, T., Peer, L., & CURE-FAIR WG. (2022, July). 10 things for curating reproducible and FAIR research (Version 1.1). Zenodo. https://curating4reproducibility.org/10things/. https://doi.org/10.15497/RDA00074

Artner, R., Verliefde, T., Steegen, S., Gomes, S., Traets, F., Tuerlinckx, F., & Vanpaemel, W. (2021). The reproducibility of statistical results in psychological research: An investigation using unpublished raw data. Psychological Methods, 26(5), 527–546. https://doi.org/10.1037/met0000365

Association for Computing Machinery. (2020, August 24). Artifact review and badging – Version 1.1. https://www.acm.org/publications/policies/artifact-review-and-badging-current

Auspurg, K., & Brüderl, J. (2024). Toward a more credible assessment of the credibility of science by many-analyst studies. Proceedings of the National Academy of Sciences, 121(38), Article e2404035121. https://doi.org/10.1073/pnas.2404035121

Axtell, R., Axelrod, R., Epstein, J. M., & Cohen, M. D. (1996). Aligning simulation models: A case study and results. Computational and Mathematical Organization Theory, 1, 123–141. https://doi.org/10.1007/BF01299065

Barba, L. A. (2018). Terminologies for reproducible research. arXiv:1802.03311v1. https://doi.org/10.48550/ARXIV.1802.03311

Bayarri, M. J., & Mayoral, A. M. (2002). Bayesian design of “successful” replications. The American Statistician, 56(3), 207–214. https://doi.org/10.1198/000313002155

Benureau, F. C. Y., & Rougier, N. P. (2018). Re-run, repeat, reproduce, reuse, replicate: Transforming code into scientific contributions. Frontiers in Neuroinformatics, 11, Article 69. https://doi.org/10.3389/fninf.2017.00069

Berkeley Initiative for Transparency in the Social Sciences. (2020). Guide for advancing computational reproducibility in the social sciences. https://bitss.github.io/ACRE/

Bhandari Neupane, J., Neupane, R. P., Luo, Y., Yoshida, W. Y., Sun, R., & Williams, P. G. (2019). Characterization of leptazolines A–D, polar oxazolines from the cyanobacterium Leptolyngbya sp., reveals a glitch with the “Willoughby–Hoye” scripts for calculating NMR chemical shifts. Organic Letters, 21(20), 8449–8453. https://doi.org/10.1021/acs.orglett.9b03216

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, N., Beaton, D., Beitner, J., … & Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582, 84–88. https://doi.org/10.1038/s41586-020-2314-9

Breznau, N., Rinke, E. M., Wuttke, A., Nguyen, H. H. V., Adem, M., Adriaans, J., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Balzer, D., Bauer, G., Bauer, P. C., Baumann, M., Baute, S., Benoit, V., Bernauer, J., Berning, C., … & Żółtak, T. (2022). Observing many researchers using the same data and hypothesis reveals a hidden universe of uncertainty. Proceedings of the National Academy of Sciences, 119(44), Article e2203150119. https://doi.org/10.1073/pnas.2203150119

Brodeur, A., de la Guardia, F. H., Miguel, E., Seung, Y. S., & Vilhuber, L. (2024). Standardizing and crowd-sourcing analysis to assess reproducibility in economics [Unpublished manuscript].

Buzbas, E. O., Devezer, B., & Baumgaertner, B. (2023). The logical structure of experiments lays the foundation for a theory of reproducibility. Royal Society Open Science, 10(3), Article 221042. https://doi.org/10.1098/rsos.221042

Carter, E. C., Schönbrodt, F. D., Gervais, W. M., & Hilgard, J. (2019). Correcting for bias in psychology: A comparison of meta-analytic methods. Advances in Methods and Practices in Psychological Science, 2(2), 115–144. https://doi.org/10.1177/2515245919847196

Chatfield, C. (1995). Model uncertainty, data mining and statistical inference. Journal of the Royal Statistical Society. Series A (Statistics in Society), 158(3), 419–466. https://doi.org/10.2307/2983440

Clemens, M. A. (2017). The meaning of failed replications: A review and proposal. Journal of Economic Surveys, 31(1), 326–342. https://doi.org/10.1111/joes.12139

Crook, S. M., Davison, A. P., & Plesser, H. E. (2013). Learning from the past: Approaches for reproducibility in computational neuroscience. In J. M. Bower (Ed.), 20 Years of Computational Neuroscience (pp. 73–102). Springer New York. https://doi.org/10.1007/978-1-4614-1424-7_4

Crüwell, S., Apthorp, D., Baker, B. J., Colling, L., Elson, M., Geiger, S. J., Lobentanzer, S., Monéger, J., Patterson, A., Schwarzkopf, D. S., Zaneva, M., & Brown, N. J. L. (2023). What’s in a badge? A computational reproducibility investigation of the open data badge policy in one issue of psychological science. Psychological Science, 34(4), 512–522. https://doi.org/10.1177/09567976221140828

Delacre, M., Lakens, D., & Leys, C. (2017). Why psychologists should by default use Welch’s t-test instead of Student’s t-test. International Review of Social Psychology, 30(1), 92–101. https://doi.org/10.5334/irsp.82

Diethelm, K. (2012). The limits of reproducibility in numerical simulation. Computing in Science & Engineering, 14(1), 64–72. https://doi.org/10.1109/MCSE.2011.21

Dreber, A., & Johannesson, M. (2025). A framework for evaluating reproducibility and replicability in economics. Economic Inquiry, 63(2), 338–356. https://doi.org/10.1111/ecin.13244

Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

Frankenhuis, W. E., Panchanathan, K., & Smaldino, P. E. (2023). Strategic ambiguity in the social sciences. Social Psychological Bulletin, 18, Article e9923. https://doi.org/10.32872/spb.9923

Gentleman, R., & Temple Lang, D. (2007). Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics, 16(1), 1–23. https://doi.org/10.1198/106186007X178663

Gervasi, V., & Zowghi, D. (2010). On the role of ambiguity in RE. In R. Wieringa & A. Persson (Eds.), Requirements engineering: Foundation for software quality (Vol. 6182, pp. 248–254). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-14192-8_22

Glatard, T., Lewis, L. B., Ferreira Da Silva, R., Adalat, R., Beck, N., Lepage, C., Rioux, P., Rousseau, M.-E., Sherif, T., Deelman, E., Khalili-Mahani, N., & Evans, A. C. (2015). Reproducibility of neuroimaging analyses across operating systems. Frontiers in Neuroinformatics, 9, Article 12. https://doi.org/10.3389/fninf.2015.00012

Goes, I. (2023). New data, new results? How data sources and vintages affect the replicability of research. Research & Politics, 10(2). https://doi.org/10.1177/20531680231170502

Gollwitzer, M., Abele-Brehm, A., Fiebach, C. J., Ramthun, R., Scheel, A., Schönbrodt, F., & Steinberg, U. (2021). Management und Bereitstellung von Forschungsdaten in der Psychologie: Überarbeitung der DGPs-Empfehlungen. Psychologische Rundschau, 72(2), 132–146. https://doi.org/10.1026/0033-3042/a000514

Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016). What does research reproducibility mean? Science Translational Medicine, 8(341), Article 341ps12. https://doi.org/10.1126/scitranslmed.aaf5027

Granger, C. W. J., & Jeon, Y. (2004). Thick modeling. Economic Modelling, 21(2), 323–343. https://doi.org/10.1016/S0264-9993(03)00017-8

Grimm, V., Augusiak, J., Focks, A., Frank, B. M., Gabsi, F., Johnston, A. S. A., Liu, C., Martin, B. T., Meli, M., Radchuk, V., Thorbek, P., & Railsback, S. F. (2014). Towards better modelling and decision support: Documenting model development, testing, and analysis using TRACE. Ecological Modelling, 280, 129–139. https://doi.org/10.1016/j.ecolmodel.2014.01.018

Grimm, V., Railsback, S. F., Vincenot, C. E., Berger, U., Gallagher, C., DeAngelis, D. L., Edmonds, B., Ge, J., Giske, J., Groeneveld, J., Johnston, A. S. A., Milles, A., Nabe-Nielsen, J., Polhill, J. G., Radchuk, V., Rohwäder, M.-S., Stillman, R. A., Thiele, J. C., & Ayllón, D. (2020). The ODD protocol for describing agent-based and other simulation models: A second update to improve clarity, replication, and structural realism. Journal of Artificial Societies and Social Simulation, 23(2), Article 7. https://doi.org/10.18564/jasss.4259

Gundersen, O. E. (2021). The fundamental principles of reproducibility. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2197), Article 20200210. https://doi.org/10.1098/rsta.2020.0210

Hardwicke, T. E., Bohn, M., MacDonald, K., Hembacher, E., Nuijten, M. B., Peloquin, B. N., deMayo, B. E., Long, B., Yoon, E. J., & Frank, M. C. (2021). Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study. Royal Society Open Science, 8(1), Article 201494. https://doi.org/10.1098/rsos.201494

Hardwicke, T. E., Mathur, M. B., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., Hofelich Mohr, A., Clayton, E., Yoon, E. J., Henry Tessler, M., Lenne, R. L., Altman, S., Long, B., & Frank, M. C. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science, 5(8), Article 180448. https://doi.org/10.1098/rsos.180448

Herberich, E., Hothorn, T., Nettle, D., & Pollet, T. V. (2010). A re-evaluation of the statistical model in Pollet and Nettle 2009. Evolution and Human Behavior, 31(2), 150–151. https://doi.org/10.1016/j.evolhumbehav.2009.12.003

Herbert, S., Kingi, H., Stanchi, F., & Vilhuber, L. (2024). Reproduce to validate: A comprehensive study on the reproducibility of economics research. Canadian Journal of Economics/Revue canadienne d’économique, 57(3), 961–988. https://doi.org/10.1111/caje.12728

Heroux, M. A., Barba, L. A., Parashar, M., Stodden, V., & Taufer, M. (2018). Toward a compatible reproducibility taxonomy for computational and computing sciences. Sandia Report, SAND2018-11186. Sandia National Laboratories. https://doi.org/10.2172/1481626

Heyard, R., Pawel, S., Frese, J., Voelkl, B., Würbel, H., McCann, S., Held, L., Wever, K. E., Hartmann, H., Townsin, L., & Zellers, S. (2025). A scoping review on metrics to quantify reproducibility: A multitude of questions leads to a multitude of metrics. Royal Society Open Science, 12(7), Article 242076. https://doi.org/10.1098/rsos.242076

Hodges, C. B., Stone, B. M., Johnson, P. K., Carter, J. H., Sawyers, C. K., Roby, P. R., & Lindsey, H. M. (2022). Researcher degrees of freedom in statistical software contribute to unreliable results: A comparison of nonparametric analyses conducted in SPSS, SAS, Stata, and R. Behavior Research Methods, 55, 2813–2837. https://doi.org/10.3758/s13428-022-01932-2

Hoffmann, S., Schönbrodt, F., Elsas, R., Wilson, R., Strasser, U., & Boulesteix, A.-L. (2021). The multiplicity of analysis strategies jeopardizes replicability: Lessons learned across disciplines. Royal Society Open Science, 8(4), Article 201925. https://doi.org/10.1098/rsos.201925

Hong, S.-Y., Koo, M.-S., Jang, J., Esther Kim, J.-E., Park, H., Joh, M.-S., Kang, J.-H., & Oh, T.-J. (2013). An evaluation of the software system dependency of a global atmospheric model. Monthly Weather Review, 141(11), 4165–4172. https://doi.org/10.1175/MWR-D-12-00352.1

Huntington-Klein, N., Arenas, A., Beam, E., Bertoni, M., Bloem, J. R., Burli, P., Chen, N., Grieco, P., Ekpe, G., Pugatch, T., Saavedra, M., & Stopnitzky, Y. (2021). The influence of hidden researcher decisions in applied microeconomics. Economic Inquiry, 59(3), 944–960. https://doi.org/10.1111/ecin.12992

Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012). The case for open computer programs. Nature, 482, 485–488. https://doi.org/10.1038/nature10836

International Committee of Medical Journal Editors. (2024, January). Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly work in Medical Journals. https://www.icmje.org/news-and-editorials/icmje-recommendations_annotated_jan24.pdf

International Organization for Standardization. (2017). Systems and software engineering—Vocabulary (ISO Standard 24765:2017). https://www.iso.org/standard/71952.html

Ivie, P., & Thain, D. (2019). Reproducibility in scientific computing. ACM Computing Surveys, 51(3), 1–36. https://doi.org/10.1145/3186266

Ivimey-Cook, E. R., Pick, J. L., Bairos-Novak, K. R., Culina, A., Gould, E., Grainger, M., Marshall, B. M., Moreau, D., Paquet, M., Royauté, R., Sánchez-Tójar, A., Silva, I., & Windecker, S. M. (2023). Implementing code review in the scientific workflow: Insights from ecology and evolutionary biology. Journal of Evolutionary Biology, 36(10), 1347–1356. https://doi.org/10.1111/jeb.14230

Jen, M.-H., Varney, B., Lee, K., Arancibia, B., Qi, M., Taylor, L., Fillmore, C., Rickert, J., Stackhouse, M., & Rimler, M. (2023). Key considerations when understanding differences in statistical methodology implementations across programming languages – An introduction to the CAMIS project (WP-077). PHUSE. https://psiaims.github.io/CAMIS/publication/white_paper.html

Kambouris, S., Wilkinson, D. P., Smith, E. T., & Fidler, F. (2024). Computationally reproducing results from meta-analyses in ecology and evolutionary biology using shared code and data. PLOS ONE, 19(3), Article e0300333. https://doi.org/10.1371/journal.pone.0300333

Keeling, K. B., & Pavur, R. J. (2007). A comparative study of the reliability of nine statistical software packages. Computational Statistics & Data Analysis, 51(8), 3811–3831. https://doi.org/10.1016/j.csda.2006.02.013

Kelly, D., Hook, D., & Sanders, R. (2009). Five recommended practices for computational scientists who write software. Computing in Science & Engineering, 11(5), 48–53. https://doi.org/10.1109/MCSE.2009.139

Kohrt, F., Zerna, J., & Scheffel, C. (2024). Code publishing tutorial. Zenodo. https://lmu-osc.github.io/code-publishing/. https://doi.org/10.5281/zenodo.16631977

Krafczyk, M. S., Shi, A., Bhaskar, A., Marinov, D., & Stodden, V. (2021). Learning from reproducing computational results: Introducing three principles and the Reproduction Package. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2197), Article 20200069. https://doi.org/10.1098/rsta.2020.0069

Krähmer, D., Schächtele, L., & Schneck, A. (2023). Care to share? Experimental evidence on code sharing behavior in the social sciences. PLOS ONE, 18(8), Article e0289380. https://doi.org/10.1371/journal.pone.0289380

Laurinavichyute, A., Yadav, H., & Vasishth, S. (2022). Share the code, not just the data: A case study of the reproducibility of articles published in the Journal of Memory and Language under the open data policy. Journal of Memory and Language, 125, Article 104332. https://doi.org/10.1016/j.jml.2022.104332

Leamer, E. E. (1978). Specification searches: Ad hoc inference with nonexperimental data. Wiley.

LeBel, E. P., McCarthy, R. J., Earp, B. D., Elson, M., & Vanpaemel, W. (2018). A unified framework to quantify the credibility of scientific findings. Advances in Methods and Practices in Psychological Science, 1(3), 389–402. https://doi.org/10.1177/2515245918787489

Leising, D., Grenke, O., & Cramer, M. (2023). Visual Argument Structure Tool (VAST) Version 1.0. Meta-Psychology, 7. https://doi.org/10.15626/MP.2021.2911

Letter, J. (2021, February 3). JASP Output Verified With Other Statistical Packages. JASP – Free and User-Friendly Statistical Software. https://jasp-stats.org/2021/02/03/the-jasp-verification-project/

Livingston, E. H. (2020). Study design and statistics. In AMA Manual of Style Committee (Ed.), AMA manual of (11th ed., pp. 977–1096). Oxford University Press. https://doi.org/10.1093/jama/9780190246556.003.0019

López-Ibáñez, M., Branke, J., & Paquete, L. (2021). Reproducibility in evolutionary computation. ACM Transactions on Evolutionary Learning and Optimization, 1(4), 1–21. https://doi.org/10.1145/3466624

Martel García, F. (2013). Scientific progress in the absence of new data: A procedural replication of Ross (2006). SSRN Electronic Journal. https://doi.org/10.2139/ssrn.2256286

Matarese, V. (2022). Kinds of replicability: Different terms and different functions. Axiomathes, 32, 647–670. https://doi.org/10.1007/s10516-021-09610-2

McCoach, D. B., Rifenbark, G. G., Newton, S. D., Li, X., Kooken, J., Yomtov, D., Gambino, A. J., & Bellara, A. (2018). Does the package matter? A comparison of five common multilevel modeling software packages. Journal of Educational and Behavioral Statistics, 43(5), 594–627. https://doi.org/10.3102/1076998618776348

McCullough, B. D., & Vinod, H. D. (1999). The numerical reliability of econometric software. Journal of Economic Literature, 37(2), 633–665. https://doi.org/10.1257/jel.37.2.633

McCullough, B. D., & Vinod, H. D. (2003). Verifying the solution from a nonlinear solver: A case study. American Economic Review, 93(3), 873–892. https://doi.org/10.1257/000282803322157133

Menkveld, A. J., Dreber, A., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Neusüß, S., Razen, M., Weitzel, U., Abad-Díaz, D., Abudy, M. (Meni), Adrian, T., Ait-Sahalia, Y., Akmansoy, O., Alcock, J. T., Alexeev, V., Aloosh, A., Amato, L., Amaya, D., … & Zwinkels, R. (2024). Nonstandard errors. The Journal of Finance, 79(3), 2339–2390. https://doi.org/10.1111/jofi.13337

Mineault, P., Nozawa, K., & The Good Research Code Handbook Community. (2021). The good research code handbook. Zenodo. https://goodresearch.dev/. https://doi.org/10.5281/ZENODO.5796873

Monks, T., Currie, C. S. M., Onggo, B. S., Robinson, S., Kunc, M., & Taylor, S. J. E. (2019). Strengthening the reporting of empirical simulation studies: Introducing the STRESS guidelines. Journal of Simulation, 13(1), 55–67. https://doi.org/10.1080/17477778.2018.1442155

Monniaux, D. (2008). The pitfalls of verifying floating-point computations. ACM Transactions on Programming Languages and Systems, 30(3), 1–41. https://doi.org/10.1145/1353445.1353446

Nagler, J. (1995). Coding and good computing practices. Political Science & Politics, 28(3), 488–492. https://doi.org/10.2307/420315

National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and replicability in science. National Academies Press. https://doi.org/10.17226/25303

National Information Standards Organization. (2021). Reproducibility badging and definitions (NISO Recommended Practice RP-31-2021). NISO. https://doi.org/10.3789/niso-rp-31-2021

Nichols, T. E., Das, S., Eickhoff, S. B., Evans, A. C., Glatard, T., Hanke, M., Kriegeskorte, N., Milham, M. P., Poldrack, R. A., Poline, J.-B., Proal, E., Thirion, B., Van Essen, D. C., White, T., & Yeo, B. T. T. (2017). Best practices in data analysis and sharing in neuroimaging using MRI. Nature Neuroscience, 20(3), 299–303. https://doi.org/10.1038/nn.4500

Nordlie, E., Gewaltig, M.-O., & Plesser, H. E. (2009). Towards reproducible descriptions of neuronal network models. PLOS Computational Biology, 5(8), Article e1000456. https://doi.org/10.1371/journal.pcbi.1000456

Nuijten, M. B., Bakker, M., Maassen, E., & Wicherts, J. M. (2018). Verify original results through reanalysis before replicating. Behavioral and Brain Sciences, 41, Article e143. https://doi.org/10.1017/S0140525X18000791

Nüst, D., Konkol, M., Pebesma, E., Kray, C., Schutzeichel, M., Przibytzin, H., & Lorenz, J. (2017). Opening the publication process with executable research compendia. D-Lib Magazine, 23(1/2). https://doi.org/10.1045/january2017-nuest

Patel, C. J., Burford, B., & Ioannidis, J. P. A. (2015). Assessment of vibration of effects due to model specification can demonstrate the instability of observational associations. Journal of Clinical Epidemiology, 68(9), 1046–1058. https://doi.org/10.1016/j.jclinepi.2015.05.029

Patil, P., Peng, R. D., & Leek, J. T. (2016a). A statistical definition for reproducibility and replicability. bioRxiv. https://doi.org/10.1101/066803

Patil, P., Peng, R. D., & Leek, J. T. (2016b). What should researchers expect when they replicate studies? A statistical view of replicability in psychological science. Perspectives on Psychological Science, 11(4), 539–544. https://doi.org/10.1177/1745691616646366

Penders, B., Holbrook, J. B., & De Rijcke, S. (2019). Rinse and repeat: Understanding the value of replication across different ways of knowing. Publications, 7(3), 52. https://doi.org/10.3390/publications7030052

Peng, R. (2022, July 29). Is code the best way to represent a data analysis? Simply Statistics. https://simplystatistics.org/posts/2022-07-29-is-code-the-best-way-to-represent-a-data-analysis/

Pham, H. V., Qian, S., Wang, J., Lutellier, T., Rosenthal, J., Tan, L., Yu, Y., & Nagappan, N. (2020). Problems and opportunities in training deep learning software systems: An analysis of variance. Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 771–783. https://doi.org/10.1145/3324884.3416545

Pilgrim, C., Kent, P., Hosseini, K., & Chalstrey, E. (2023). Ten simple rules for working with other people’s code. PLOS Computational Biology, 19(4), Article e1011031. https://doi.org/10.1371/journal.pcbi.1011031

Plesser, H. E. (2018). Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics, 11, 76. https://doi.org/10.3389/fninf.2017.00076

PyTorch Contributors. (2023). Reproducibility. In PyTorch 2.5 Documentation. https://pytorch.org/docs/stable/notes/randomness.html#reproducibility

Rahal, C. (2024). On the responsible use of pseudo-random number generators in scientific research [Computer software]. https://github.com/crahal/seeds

Saltelli, A. (Ed.). (2008). Sensitivity analysis (Paperback ed). Wiley.

Sandve, G. K., Nekrutenko, A., Taylor, J., & Hovig, E. (2013). Ten simple rules for reproducible computational research. PLOS Computational Biology, 9(10), Article e1003285. https://doi.org/10.1371/journal.pcbi.1003285

Scaria, L. (2018). A framework to evaluate pipeline reproducibility across operating systems [Master’s Thesis, Concordia University]. https://spectrum.library.concordia.ca/id/eprint/984061/

Schmidt, S. (2009). Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2), 90–100. https://doi.org/10.1037/a0015108

Schöch, C. (2023). Repetitive research: A conceptual space and terminology of replication, reproduction, revision, reanalysis, reinvestigation and reuse in digital humanities. International Journal of Digital Humanities, 5, 373–403. https://doi.org/10.1007/s42803-023-00073-y

Sego, T. J., König, M., Fonseca, L. L., Fain, B., Knapp, A. C., Tiwari, K., Hermjakob, H., Sauro, H. M., Glazier, J. A., Laubenbacher, R. C., & Malik-Sheriff, R. S. (2024). EFECT – A method and metric to assess the reproducibility of stochastic simulation studies (Version 1). arXiv:2406.16820v2. https://doi.org/10.48550/ARXIV.2406.16820

Seibold, H., Czerny, S., Decke, S., Dieterle, R., Eder, T., Fohr, S., Hahn, N., Hartmann, R., Heindl, C., Kopper, P., Lepke, D., Loidl, V., Mandl, M., Musiol, S., Peter, J., Piehler, A., Rojas, E., Schmid, S., Schmidt, H., … & Nalenz, M. (2021). A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses. PLOS ONE, 16(6), Article e0251194. https://doi.org/10.1371/journal.pone.0251194

Sharma, N. K., Ayyala, R., Deshpande, D., Patel, Y., Munteanu, V., Ciorba, D., Bostan, V., Fiscutean, A., Vahed, M., Sarkar, A., Guo, R., Moore, A., Darci-Maher, N., Nogoy, N., Abedalthagafi, M., & Mangul, S. (2024). Analytical code sharing practices in biomedical research. PeerJ Computer Science, 10, Article e2066. https://doi.org/10.7717/peerj-cs.2066

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahník, Š., Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., … & Nosek, B. A. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646

Simkus, A., Coolen-Maturi, T., Coolen, F. P. A., & Bendtsen, C. (2025). Statistical perspectives on reproducibility: Definitions and challenges. Journal of Statistical Theory and Practice, 19, Article 40. https://doi.org/10.1007/s42519-025-00459-x

Stan Development Team. (2024). Reproducibility. In Stan Reference Manual. https://mc-stan.org/docs/reference-manual/reproducibility.html

Stanley, D. J., & Spence, J. R. (2014). Expectations for replications: Are yours realistic? Perspectives on Psychological Science, 9(3), 305–318. https://doi.org/10.1177/1745691614528518

Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637

Steiner, P. M., Wong, V. C., & Anglin, K. (2019). A causal replication framework for designing and assessing replication efforts. Zeitschrift Für Psychologie, 227(4), 280–292. https://doi.org/10.1027/2151-2604/a000385

Stodden, V. (2015). Reproducing statistical results. Annual Review of Statistics and Its Application, 2, 1–19. https://doi.org/10.1146/annurev-statistics-010814-020127

UK Government Analytical Community. (2020). Quality assurance of code for analysis and research. https://best-practice-and-impact.github.io/qa-of-code-guidance/

Ulpts, S., & Schneider, J. W. (2024). A conceptual review of uses and meanings of reproducibility and replication. MetaArXiv, entu4_v1. https://doi.org/10.31222/osf.io/entu4

Van Den Bergh, D., Wagenmakers, E.-J., & Aust, F. (2023). Bayesian repeated-measures analysis of variance: An updated methodology implemented in JASP. Advances in Methods and Practices in Psychological Science, 6(2), Article 25152459231168024. https://doi.org/10.1177/25152459231168024

Varoquaux, G. (2016). Beyond computational reproducibility, let us aim for reusability. IEEE CIS Newsletter on Cognitive and Developmental Systems, 13(2), 7. https://hdl.handle.net/21.11116/0000-0008-0384-F

W.D. (2019, August 31). Scikit-learn’s defaults are wrong. https://ryxcommar.com/2019/08/30/scikit-learns-defaults-are-wrong/

Weissgerber, T. L., Garcia-Valencia, O., Garovic, V. D., Milic, N. M., & Winham, S. J. (2018). Meta-Research: Why we need to report more than “Data were analyzed by t-tests or ANOVA.” eLife, 7, Article e36163. https://doi.org/10.7554/eLife.36163

Welch, I. (2019). Reproducing, extending, updating, replicating, reexamining, and reconciling. Critical Finance Review, 8(1–2), 301–304. https://doi.org/10.1561/104.00000082

Whitley, R. (2000). The degree of task uncertainty and the organization of scientific fields. In R. Whitley, The Intellectual and Social Organization of the Sciences (pp. 119–152). Oxford University Press. https://doi.org/10.1093/oso/9780199240531.003.0004

Wilson, G., Bryan, J., Cranston, K., Kitzes, J., Nederbragt, L., & Teal, T. K. (2017). Good enough practices in scientific computing. PLOS Computational Biology, 13(6), Article e1005510. https://doi.org/10.1371/journal.pcbi.1005510

Young, C. (2009). Model uncertainty in sociological research: An application to religion and economic growth. American Sociological Review, 74(3), 380–397. https://doi.org/10.1177/000312240907400303

Young, C., & Holsteen, K. (2017). Model uncertainty and robustness: A computational framework for multimodel analysis. Sociological Methods & Research, 46(1), 3–40. https://doi.org/10.1177/0049124115610347

Zhang, J., & Robinson, D. T. (2021). Replication of an agent-based model using the Replication Standard. Environmental Modelling & Software, 139, Article 105016. https://doi.org/10.1016/j.envsoft.2021.105016

Ziemann, M., Poulain, P., & Bora, A. (2023). The five pillars of computational reproducibility: Bioinformatics and beyond. Briefings in Bioinformatics, 24(6), Article bbad375. https://doi.org/10.1093/bib/bbad375

Appendix

Figure A1: Detailed Flowchart for Performing a Reproduction of One Result