A Blog of Trial and Error

GRAIL EOSC Gravity Update Month 3 and 4

Maura Burke — Wed, 23 Sep 2026 13:24:12 GMT

GRAIL EOSC Gravity Project Month 3 and 4 Update

The Grant Reuse and Abandoned Ideas Library is an EOSC Gravity funded Preparatory Project exploring what it would look like to participate in the Federations’ Tripartite Node Enrolment. In our first blogpost, we detailed our orientation process and reflected on the obstacles we had experienced in getting used to working within the EOSC framework. In this post, we will focus on the work we have been doing behind the scenes to put together our recently submitted Project Charter, a document whose function is to describe what GRAIL would be if it were enrolled as a Node, a function it took us quite a while to fully understand.

For most of month 3, we were focused on doing the actual work we promised we would deliver in our project plan – finalizing and analyzing the qualitative interviews we conducted, building out spec pages and digital architecture for the repository itself, working on the legal framework for the functioning of a repository for rejected grant proposals and abandoned ideas, determining the governance and organizational structure of GRAIL, and establishing the network in which we would be embedded as a digital service provider. Our technical team was able to get a demo of the repository live, and we were feeling quite happy with our progress.

In month 4, we began translating this work into the project charter, each working group taking responsibility for a section that corresponded to their area of expertise. We consulted publicly available examples of past Project Charters for inspiration and insights and had filled out most of the document by the time the EOSC Academy session on Project Charters occurred, a few days ahead of the due date. It was there that we learned that the Charter should not present any of our preparatory grant work – the interviews, the legal review, the metadata scoping, not even the demo. Instead, we were to fill out the Project Charter as if we were applying to the Tripartite Node Enrolment without including any of the project milestones described in the project plan of the Preparatory Project. To be fair, there were two footnotes on sections related to specific timelines and milestones that we should make a new timeline unrelated to the Preparatory Grant. However, it was not clear to us that this meant the entire Project Charter should be executed as a disjoint deliverable.

The EOSC Academy session was, therefore, immensely useful for us as a project team. It does make me think that perhaps the offering dates could be more strategically aligned with project deliverables, as this academy session was only a few days ahead of the final due date. However, our experience nonetheless highlights the value of the Academy for participants of the EOSC Federation funding schemas.

Ultimately, the reframing did help us clarify some of the details related to GRAIL. The Charter’s technology-readiness (TRL) self-assessment, for instance helped us evaluate how deployable the elements of our demo were. Currently, our demo runs a working OIDC/PKCE authentication flow against ORCID, registers test DOI4s under our Crossref prefix and carries a submission through a full metadata pipeline end to end. There were also aspects that we recognize are far from the required technology readiness level 7. The Project Charter covers resource-catalogue registration, helpdesk federation, service monitoring, and a FitSM-aligned Service Management System that we just simply cannot develop with the resources currently available to us. This raised a question for us as a team – should we envision the future of GRAIL as dependent upon continued funding from EOSC, or should we develop the infrastructure as possible to host independently?

Relatedly, we spent some time thinking about how GRAIL could best fit into the existing EOSC Node landscape. There already exists a Node in the Netherlands – the SURF node. We also are aware that there are private discussions with other major service providers in the Netherlands to develop either additional Nodes or one large national Node under which more specific Nodes are deployed. As I mentioned in previous posts, GRAIL is unique within the Gravity Preparatory Proposal system as we are neither a pre-existing infrastructure nor a national thematic Node. Maybe it would be best for us to be a part of a larger, national network of Nodes?

In the coming months, we will think about these questions within the GRAIL team while we finalize the last deliverables for this grant, our proposed contribution to the EOSC Academy Matrix and our final presentation of our efforts undertaken in this period. After our very positive experience with the Academy in this period, we are looking forward to developing our contribution. Until then…

Maura Cassidy Burke on behalf of the GRAIL team

GRAIL EOSC Gravity Project Month 1 and 2 Update

Maura Burke — Fri, 10 Jul 2026 08:30:00 GMT

GRAIL – the Grant Reuse and Abandoned Ideas Library – is an EOSC Gravity funded Preparatory Project. We imagine GRAIL as a repository for research products that otherwise never get shared or credited. The preparation of grant applications requires substantial fiscal and human resources, yet the vast majority never result in funding. A rejection does not necessarily signal poor quality. Given the volume of applications submitted by researchers of comparable quality, funding bodies inevitably reject proposals that are scientifically sound. This challenge is only likely to increase as generative AI tools lower the barrier to drafting competitive grant proposals.

At the Center of Trial and Error, we have long been guided by a commitment to sharing scientific products on the basis of their intrinsic merit, rather than whether they conform to prevailing evaluative metrics or academic conventions. We do so because we see that existing incentive structures within academia can encourage questionable research practices and reduce the transparency, efficiency, and reliability of scientific research. GRAIL emerged from conversations within the Center dating back to 2019 and reflects our desire to address some of these misaligned incentives by making overlooked scholarly work visible and reusable.

Going into this preparatory phase, we assumed that the central idea behind GRAIL was relatively straightforward. People submit materials, we make them findable, searchable, and citable – et voilà, a repository. Our IT team has very kindly walked us through why that is not, in fact, the case.

Developing GRAIL as a potential EOSC Node has required us to think far beyond the repository itself. The ethos of the EOSC Federation is to make open science services interoperable, sustainable, and accessible across Europe. That means governance, metadata, security, authentication, and long-term sustainability are fundamental design questions for us.

One realization during M1 and M2 has been that GRAIL occupies a somewhat unusual position within the EOSC ecosystem. We are not an existing service seeking federation, we are not a research community or competence center. We are designing an entirely new infrastructure project. As a consequence, we have occasionally found it challenging to determine how existing EOSC guidance applies to our project.

Our IT team, for example, is still determining whether GRAIL should be developed to meet the requirements of a Federated EOSC Node, or what the practical implications of the SSO requirements will be. Our project manager continues to explore how the sustainability of GRAIL is related to its status as an EOSC Node, and to what extent GRAIL should make design decisions that require access to greater fiscal funding. Meanwhile, our qualitative researcher has been considering how to frame the community insights garnered from participant interviews with the objectives and deliverables of the EOSC Gravity program.

However, I would like to give credit to EOSC in how they are engaging with these sorts of challenges. EOSC is a growing initiative, and throughout our interactions is has become clear that the Federation’s governance documents and overall visions are still evolving. Rather than presenting these documents as fixed, EOSC has been inviting the community to contribute to their development and operations. I was pleased to participate in the 15 June workshop on applying the EOSC Federation Handbook, in which organizers and participants worked together to better understand both the content of the Handbook and its practical implications for developing future EOSC Nodes.

Several discussions struck me as particularly relevant to GRAIL. For example, should EOSC Nodes primarily be organized geographically or thematically? At present there is no EOSC Node-NL, although there is the Netherlands-based EOSC Node SURF. Should the Federation eventually pursue a model in which each Member State hosts a single national Node supporting multiple services? Or, should thematic Nodes exist alongside geographically organized ones, allowing infrastructures like SURF and a future GRAIL Node to coexist?

A related question concerns coordination between Nodes. Currently, GRAIL is independently establishing relationships with existing infrastructure providers, including SURF. Could there be a larger role for the EOSC Federation itself in facilitating these connections, identifying existing expertise, and helping emerging Nodes navigate the broader EOSC ecosystem? As this blog series continues, we hope to share more reflections that may contribute to these ongoing conversations within EOSC.

Perhaps the most unexpected experience during these first months has been encountering what might be called the paradox of plenty. Historically, when developing new initiatives at the Center of Trial and Error, the limits of scope have largely been imposed by available time, funding, and personnel. We have rarely had to ask what should or should not be included because those decisions were largely made for us by practical constraints. The EOSC Gravity funding has created a rather different situation in which we have the opportunity to think intentionally about what GRAIL should become before committing ourselves to a particular implementation.

One of the key outcomes of our most recent internal GRAIL Charter Workshop was therefore the realization that we need greater clarity regarding the scope of the repository itself. To develop the technical architecture for GRAIL, we still need to provide our IT team with more concrete answers about how users will engage with GRAIL as a repository – how will they log in, will they be able to save entries, what sort of documents will be uploaded, how will they navigate the repository? The answers to these questions directly bear on what steps are next for our IT team, and scoping will therefore be a priority for us in the upcoming weeks.

In answering these questions, we will be drawing on the very valuable insights our qualitative research team has been collecting. We have been conducting interviews to better understand how GRAIL could become a genuinely useful research tool. One early finding recently shared with the team concerns the importance of changing the culture surrounding these kinds of research products. If we hope researchers will share rejected grant applications or other unrealized research materials, then greater recognition of their scholarly value will also be necessary.

The interviews suggest that researchers are enthusiastic about the prospect of sharing and discovering rejected grant applications, but the idea of abandoned ideas has prompted more questions. That response is understandable – unlike grant applications, abandoned ideas are not a clearly established scholarly object. When we use the term, we are thinking of materials such as unpublished position papers, unrealized symposium proposals, or research plans that never progressed further. These documents may nevertheless contain valuable ideas that could inspire future work, and they may also provide useful starting points for students, early-career researchers, or collaborators seeking new directions. Making these materials discoverable could contribute to more decentralized forms of collaboration and mentorship that extend beyond institutional or geographical boundaries.

At the same time, these conversations and the conversations with our IT team have prompted us to reconsider whether abandoned ideas should be included in GRAIL’s first iteration. We may instead choose to begin with a narrower focus on rejected grant applications, allowing our IT team to develop simpler and more robust metadata structures and technical workflows before expanding to additional research products. We are still firmly in the scoping phase of the project, and conversations with researchers, infrastructure providers, legal experts, and colleagues within EOSC remain essential to answering many of these questions.

Alongside these discussions, we have made encouraging progress with our legal consultant and with representatives from existing research infrastructures – and of course we are happy that our first deliverable has been approved by EOSC! Ensuring that GRAIL can integrate smoothly into the wider open science ecosystem remains one of our central priorities, and in the coming months we will continue to refine our governance, technical architecture, and understanding of researcher needs while working towards creating an infrastructure that is genuinely useful to the communities it hopes to serve.

On Behalf of the GRAIL Team,

Maura Burke

Cultures of Trial and Error: When the Spell Breaks: Magic, Money, and the EPR Effect

Declan Kuch — Sun, 17 May 2026 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

How nanomedicine conjured billions from thin air and what happened when the enchantment failed

“Nano scale [is] magic scale.” This was not a late-night laboratory joke. It was official U.S. government policy. The phrase appeared verbatim in early National Science Foundation solicitations for the National Nanotechnology Initiative (NNI). Nano-evangelist Mihail Roco coined it, and the very first NNI programme description placed it alongside technical aims (Mody 2004; Roco 2023). When President Clinton announced the program in 2000, he promised breakthroughs that were “only imagination away.” Magic sat next to milestones in official funding calls. The government had converted a charismatic promise into a bureaucratically sustained future.

What follows offers a case study in how weak scientific claims gain extraordinary performative power. It also shows how the entanglement of technoscience with financial capital can disable science’s self-correcting mechanisms. The protagonist is the Enhanced Permeability and Retention (EPR) Effect: the hypothesis that tumours’ leaky blood vessels would passively accumulate drug-loaded nanoparticles. For over two decades, EPR organised billions in research funding, launched dozens of biotech ventures, and became nanomedicine’s constitutive fiction. Its empirical basis was tenuous from the start. It worked reliably only in certain tumours transplanted into certain mice.

Drawing on our article in Social Studies of Science (Rasmussen & Kuch, 2026), this post uses the concept of enchantment to illuminate how EPR’s spell was cast, why it held for so long, and what its collapse reveals about contemporary technoscience. Max Weber (1991 [1919]: 139) called enchantments “mysterious incalculable forces” and denied their presence in modern science. Science and Technology Studies (STS) scholars have tracked them nonetheless, through concepts like promissory economies (Brown 2003), fictional expectations (Beckert 2013), and sociotechnical imaginaries (Jasanoff & Kim 2015).

The Claim That Launched a Thousand Startups

In 1986, Japanese pharmacologists Matsumura and Maeda observed that a polymer-modified cancer drug accumulated preferentially in mouse tumours. They proposed a mechanism. Tumours have chaotic, leaky blood vessels (enhanced permeability) and poor lymphatic drainage (enhanced retention). Macromolecules and particles of appropriate size should therefore concentrate in tumours passively, without molecular targeting (Matsumura & Maeda 1986). If true, this “EPR Effect” would turn cancer’s chaos into therapeutic advantage. Leaky vessels would become gateways. The drug would find the tumour on its own.

The empirical basis was tenuous from the outset. EPR worked reliably only in specific tumours transplanted into specific mice. Not all tumours showed enhanced permeability. Results varied considerably. As early as 1992, pharmacologist Len Seymour noted that discouraging results from liposome researchers had “suppressed passive drug targeting research in general” (Seymour 1992: 151). This early warning would be buried under subsequent enthusiasm.

Yet EPR did not fade. Through the 1990s, it circulated within a small drug delivery community. Researchers like Ruth Duncan and Karel Ulbrich developed polymer-drug conjugates designed to exploit the Effect. Duncan’s landmark 1999 review positioned EPR as the linchpin for a new generation of “polymer therapeutics” (Duncan 1999). At that point, none of these drug classes carried the “nano” prefix. That rebranding was about to arrive.

From Hypothesis to Holy Grail

The NNI’s launch in 2000 changed everything. Billions in federal funding and explicit invocations of “magic” transformed EPR’s prospects. Coordinated across agencies including NSF, NIH, DOE, and NASA, NNI spending climbed from roughly $500 million in 2001 to nearly $2 billion annually by 2010. By 2023, cumulative spending passed $40 billion (Roco 2023). Similar nano-roadmaps in France, the UK, Japan, China, and elsewhere reinforced a transnational funding ecology. Funders treated EPR as a ready-made portal to futuristic cancer cures, years before robust human data existed (Kearnes & Rip 2009).

Under these conditions, EPR became what sociologists call a “boundary object” (Star & Griesemer 1989). It was flexible enough to mean different things to different communities, yet coherent enough to coordinate collective action. Polymer chemists saw a design principle. Venture capitalists saw a platform technology. Federal agencies saw a tangible health benefit to justify massive investment. Universities saw patents and spin-offs. Each community found what it needed in the same three-letter acronym.

EPR-based cancer therapeutics were not a minor corner of nanomedicine. They were its centre of gravity. By the mid-2010s, cancer therapeutics accounted for roughly 40% of nanopharmaceutical value. Nanopharma itself represented about 90% of projected nanomedicine market value. EPR underpinned more than a third of nanomedicine’s overall worth (Loubaton 2012).

The Wizard and the Magic Bullet

Every enchantment needs a magician and a talisman. The EPR story found its charismatic figure in Robert Langer, the MIT scientist whose thousand-plus patents made him legendary in academic and investment circles alike. Media profiles called him a “wizard” of biotech and compared him to Thomas Edison (Prokesch 2017). His reputation operated as transferable charisma: symbolic capital that conveyed credibility through association. To have Langer’s name on your paper, his technology in your startup, his blessing on your grant: these conferred value.

Through strategically placed review articles in journals like Nature Nanotechnology, Langer positioned EPR-based nanocarriers as the natural heirs to Paul Ehrlich’s century-old “magic bullet” dream. Ehrlich, the early twentieth-century pharmacologist, had envisioned drugs that seek and destroy pathogens while sparing healthy tissue (Lenoir 1988; Bosch & Rosich 2008). In a prominent 2007 review, Langer and colleagues described the nanocarrier-based approach as “an important modality within therapeutic and diagnostic oncology,” with EPR as its core mechanism (Peer et al. 2007: 758).

Langer was candid about the logic. Publication in “highly selective and competitive journals validates that the idea may be a significant breakthrough,” he wrote (Langer 2013: 487). Peer review became a charisma-generating machine.

When Stories Become Securities

Here the political economy of promising becomes explicit. EPR did not merely organise research. It created financial value. Patents incorporated EPR assumptions as foundational claims. Investment prospectuses treated it as validated science. When Langer and colleague Omid Farokhzad founded BIND Therapeutics in 2006, they secured nearly $70 million in private capital by the end of 2007, plus billion-dollar milestone payments from Big Pharma partnerships (Maine & Thomas 2017). Many of BIND’s foundational patents depended explicitly on EPR. The Effect underwrote the value of the firm.

By the time BIND floated its IPO in 2013, its drug delivery platform was protected by 16 issued U.S. patents and 50 patent applications (Neuman & Chandhok 2016). The prospectus invoked EPR as the mechanism by which its Accurin® particles would reach tumours, asserting that “Accurins represent the next stage in the evolution of targeted therapies and nanomedicine” (BIND Therapeutics 2013). To many investors, BIND’s EPR-based “story” made for a compelling “value proposition,” according to one investment newsletter’s post-mortem (Nanalyze 2017).

Economic sociologist Jens Beckert (2013) calls such narratives “fictional expectations”: stories whose empirical truth matters less than their capacity to coordinate action under radical uncertainty. But fictionality alone cannot explain why some stories hold collective imagination for decades while others evaporate quickly. EPR’s empirical weakness was evident early on. Yet belief persisted. What Beckert’s framework captures in coordination, it underspecifies in effect.

This is where enchantment adds analytical depth. Drawing on Jane Bennett’s (2001) concept of secular enchantment and Josephson-Storm’s (2017) work on modernity’s persistent magical thinking, we use “enchantment” to describe the sticky, affective commitment that binds heterogeneous actors into shared belief. Scientists, investors, regulators, patients: EPR bound them through something that preceded and exceeded rational calculation. Mouse experiments became talismans. Langer’s reputation became portable charisma. The “magic bullet” metaphor promised transformation, not mere treatment. It promised to convert “leaky” from pathological liability into investment opportunity.

Why Self-Correction Failed

Here we confront a troubling question: why didn’t science’s self-correcting mechanisms identify and reject EPR’s weak empirical foundations?

Part of the answer lies in what we call “distributed complicity.” No single actor committed fraud or conscious deception. Yet the system collectively sustained a fiction. The concept draws on Mirowski’s (2011) analysis of the “commercialisation of epistemology” but extends it by emphasising the distributed, emergent quality of the problem. No conspiracy was needed. Each node in the network optimised rationally within local constraints. The aggregate effect was that a weak claim became structurally impossible to dislodge.

Scientists needed NNI funding and could access it by framing existing research as “nanomedicine.” Peer reviewers faced reputational costs in challenging papers from prestigious laboratories. Funding officers needed success narratives to justify continued investment to Congress. Universities needed patent royalties and spin-off companies. Venture capitalists needed compelling stories to deploy capital. Regulatory agencies lacked resources for aggressive pre-clinical scrutiny. Each actor’s behaviour was locally rational. The collective outcome was epistemic capture.

The classic Mertonian norms of organised scepticism presume disinterestedness. Financialisation erodes that disinterestedness. When everyone from graduate students to university presidents has material stakes in a narrative, correction requires more than better evidence. It requires collective willingness to write off massive sunk costs in careers, patents, institutional identities, and investment portfolios. Enchantment makes those costs prohibitive by distributing complicity so widely that no individual actor can afford to break ranks.

Even ambiguous evidence was systematically reinterpreted to sustain the spell. When BIND’s Phase I trials showed modest results (two of three patients exhibiting some response based on surrogate measures), this became “promising” data justifying three Phase II trials and a jump in company valuation (Fonseca et al. 2014). The field had developed sophisticated interpretive practices for extracting hope from equivocal findings.

What Finally Broke the Spell

The spell did not break through peer review, replication failures, or internal scientific debate. It broke through a market correction.

BIND’s Phase II trials showed limited benefit. In January 2016, the company dropped its prostate cancer indication. In April, results for lung cancer and a multi-tumour trial showed marginal or no efficacy. Layoffs came the same day. Within a month, BIND had filed for bankruptcy. Two months later, Pfizer acquired the remaining assets for $40 million, down from a peak valuation exceeding $230 million.

This was not unusual for cancer drugs. Phase II trials are not designed to prove efficacy, and many promising candidates fail at this stage. But BIND had come to symbolise EPR’s promise so conspicuously that its collapse carried disproportionate weight.

Drug delivery scientist Fabienne Danhier (2016: 109) concluded: “The verdict has been handed down: the EPR effect works in rodents but not in humans. What is the future of nanomedicine?” The question confirmed the point. EPR-based cancer drugs had been nanomedicine’s most valuable fruit. Some senior scientists went further, arguing that the EPR fiasco reflected a political problem with the field’s distribution of resources (Park 2017).

The timing reveals something important. The spell broke from outside the scientific community, through bankruptcy, rather than from within. Financial failure is not epistemic correction. But it can trigger epistemic correction by making continued belief too costly to sustain. In this sense, the market compensated for science’s structural failure. Scientists still revised their views. The impetus came from outside.

The COVID Sequel: Re-Enchantment

Enchantments rarely stay dispelled. By late 2019, nanomedicine had reached the “trough of disillusionment.” The U.S. National Cancer Institute had discontinued funding for its flagship Centres of Cancer Nanotechnology Excellence. In October 2019, French nanomedicine researcher Patrick Couvreur blamed overinvestment in EPR for the field’s parlous state (Couvreur 2019: 319–320).

Barely a month later, COVID-19 changed everything. Within months, Moderna’s and Pfizer-BioNTech’s lipid-nanoparticle mRNA vaccines proved spectacularly effective. Nanomedicine’s leaders seized the opportunity. A March 2020 editorial called for deploying “nanotechnology advances as frontline tools” against the coronavirus (Chan 2020). A later Nature Nanotechnology editorial framed the impending vaccine approvals as “a milestone for nanomedicine,” capable of restoring faith lost in the EPR debacle (Anon. 2020).

Robert Langer illustrates how charisma migrates under financialised technoscience. He had co-founded Moderna in 2010, six years before BIND’s collapse. The lipid-nanoparticle technology was technically distinct from EPR-based polymer platforms. The vaccines worked by a different mechanism and targeted a different disease. What transferred was not science but reputational capital: the aura of the “wizard” whose track record made investors confident. At its 2018 IPO, Moderna was valued at nearly $8 billion before it had a single approved product.

This rapid re-enchantment demonstrates what Fortun (2008) calls “speculative realism”: promissory narratives can survive empirical failure by migrating to new domains. Under financial capitalism, charisma and narrative transfer across applications faster than evidence accumulates. The COVID vaccines were genuine achievements. But the speed with which they rehabilitated nanomedicine as a whole owed much to enchantment’s fungibility.

Beyond Nanomedicine: Recognising the Pattern

The EPR story is not unique. Similar dynamics appear wherever promissory narratives, charismatic figures, and venture capital intertwine (Borup et al. 2006; Brown & Michael 2003).

Consider the current AI landscape. “Scaling laws” and “emergent capabilities” function much as EPR once did: they organise research programs, justify massive capital allocation, and resist falsification because too many actors have too much invested (Galanos 2023; Kotliar 2025). The charismatic figures are familiar types. The gap between laboratory benchmarks and real-world utility echoes the gap between mouse tumours and human patients. The interpretive practices for extracting hope from ambiguous evidence mirror the endpoint drift that sustained EPR (Schaefer et al. 2024).

Quantum computing tells a similar story. Repeated promises of imminent “quantum advantage” have organised funding for two decades, yet practical applications remain elusive (Verhoeven 2021). Foundational assumptions are rarely questioned in funding contexts. Longevity biotech follows the same grammar: massive capital, promissory narratives, wizard founders, and a gap between cellular demonstrations and clinical reality.

This is not to say these fields are fraudulent. EPR was not fraudulent either. The point is structural. When technoscience and finance entangle under these conditions, certain patterns recur. Recognising them is the first step toward better epistemic hygiene.

Navigating Enchantment: Notes for Early-Career Researchers

For PhD students and postdocs, the EPR story may feel uncomfortably familiar. You may be building your research program on foundational assumptions you have inherited rather than tested. You may suspect a paradigm is shaky but lack the career security to say so.

A few practical observations.

Learn to recognise enchantment from the inside. Ask yourself: What would falsify my core assumption? Has anyone looked? If the assumption were wrong, who would lose, and how much? If the answers are “nothing would,” “no,” and “everyone, enormously,” you may be inside an enchantment.

Understand the costs of scepticism. Questioning foundational assumptions is professionally dangerous. The costs fall hardest on those with least security: graduate students, postdocs, researchers without tenure. This asymmetry is how enchantments sustain themselves.

Diversify your epistemic portfolio. If your entire research program depends on a single foundational assumption, you are vulnerable. Where possible, develop expertise that would retain value even if the paradigm shifted.

Find your sceptics. Every field has people who harbour doubts but stay quiet for career reasons. Find them. Build relationships. Create spaces where concerns can be voiced without professional penalty.

Remember that enchantments end. EPR organised billions and careers for two decades, then collapsed in months. The researchers who adapted fastest had privately maintained critical distance. The researchers who suffered most had fully identified with the paradigm.

What Can We Learn?

Spectacular failures create privileged observation windows. When spells break, previously invisible infrastructure surfaces. The vacuum left by a collapsed enchantment reveals the coordination work it had been doing: review articles positioning EPR as inevitable, patents assuming it into existence, investment narratives making it bankable.

Metaphors perform work. When technologies promise to turn chaos into control, ask what the metaphor accomplishes. “Magic bullet” is not innocent description. It is strategic enchantment that canalises capital, labour, and imagination.

Scepticism has material costs. Questioning enchantments requires material security, not just intellectual courage. This is why EPR was broken by bankers rather than bench scientists. The costs of questioning fell hardest on those inside the scientific community.

Correction comes from unexpected places. For highly financialised technoscience, we may need to rethink where epistemic accountability operates. Bankruptcy triggered what peer review could not.

Which Enchantments, For Whom?

Max Weber argued that modern rationality means “there are no mysterious incalculable forces” and that “the world is disenchanted.” He was wrong. The EPR saga shows why. The boundary between rational calculation and magical thinking is more porous than we admit, especially when technoscience and finance intertwine.

The question is not whether science should dispel enchantment. Enchantment may be constitutive of the collective action that ambitious research requires. The question is which enchantments serve which communities and how we can make their operation visible before they collapse under their own weight.

A closing invitation. What enchantments structure your own field? What foundational assumptions have you inherited but never tested? What would it cost you to voice scepticism? And if those costs feel prohibitive, what does that tell you about the epistemic health of your discipline?

These are not comfortable questions. But they are the ones most worth asking.

This post draws on Nicolas Rasmussen and Declan Kuch’s article “‘Nano Scale [is] Magic Scale’: On EPR, Unicorns, and Enchantment in Nanomedicine,” published in Social Studies of Science. DK gratefully acknowledges support for some of this research by the ARC Centre of Excellence in Convergent Bio-Nano Science and Technology (CE140100036).

References

Anon. (2020). Nanomedicine and the COVID-19 vaccines. Nature Nanotechnology, 15, 963.

Beckert, J. (2013). Imagined futures: Fictional expectations in the economy. Theory and Society, 42(3), 219–240.

Bennett, J. (2001). The Enchantment of Modern Life. Princeton University Press.

BIND Therapeutics. (2013). Form S-1, United States Securities and Exchange Commission, filed 12 August 2013.

Borup, M., Brown, N., Konrad, K. & van Lente, H. (2006). The sociology of expectations in science and technology. Technology Analysis & Strategic Management, 18(3–4), 285–298.

Bosch, F. & Rosich, L. (2008). The contributions of Paul Ehrlich to pharmacology. Pharmacology, 82(3), 171–179.

Brown, N. (2003). Hope against hype. Science & Technology Studies, 16(2), 3–21.

Brown, N. & Michael, M. (2003). A sociology of expectations: Retrospecting prospects and prospecting retrospects. Technology Analysis and Strategic Management, 15(1), 3–18.

Chan, W.C.W. (2020). Nano research for COVID-19. ACS Nano, 14(4), 3719–3720.

Couvreur, P. (2019). Nanomedicine: From where are we coming and where are we going? Journal of Controlled Release, 311, 319–321.

Danhier, F. (2016). To exploit the tumor microenvironment. Journal of Controlled Release, 244, 108–121.

Duncan, R. (1999). Polymer conjugates for tumour targeting. Pharmaceutical Science and Technology Today, 2(11), 441–449.

Fonseca, N.A. et al. (2014). Bridging cancer biology and the patients’ needs. Cancer Treatment Reviews, 40(5), 626–635.

Fortun, M. (2008). Promising Genomics. University of California Press.

Galanos, V. (2023). A Historical Sociology of Expectations and Expertise in AI. PhD thesis, University of Edinburgh.

Jasanoff, S. & Kim, S.H. (2015). Dreamscapes of Modernity. University of Chicago Press.

Josephson-Storm, J.A. (2017). The Myth of Disenchantment. University of Chicago Press.

Kearnes, M.B. & Rip, A. (2009). The emerging governance landscape of nanotechnology. In S. Gammel et al. (Eds.), Jenseits von Regulierung. AKA.

Kotliar, D.M. (2025). Can’t stop the hype: Scrutinizing AI’s realities. Information, Communication & Society. https://doi.org/10.1080/1369118X.2025.2531165

Langer, R. (2013). A personal account of translating discoveries. Nature Biotechnology, 31(6), 487–489.

Lenoir, T. (1988). A magic bullet. Minerva, 26(1), 66–88.

Loubaton, B. (2012). Exploring the quantitative dimensions of the economic impact of nanomedicine. OECD & US NNI International Symposium, Washington DC.

Maine, E. & Thomas, V.J. (2017). Raising financing through strategic timing. Nature Nanotechnology, 12(2), 93–98.

Matsumura, Y. & Maeda, H. (1986). A new concept for macromolecular therapeutics. Cancer Research, 46(12.1), 6387–6392.

Mirowski, P. (2011). Science-Mart. Harvard University Press.

Mody, C.M. (2004). Small, but determined. Hyle, 10, 99–128.

Nanalyze. (2017). WTF Happened to BIND Therapeutics? Nanalyze, 1 August 2017.

Neuman, D. & Chandhok, J.N. (2016). Nanomedicine patents highlight importance of production methods. Nature Reviews Drug Discovery, 15(7), 448–450.

Park, K. (2017). The drug delivery field at the inflection point. Journal of Controlled Release, 267, 2–14.

Peer, D. et al. (2007). Nanocarriers as an emerging platform for cancer therapy. Nature Nanotechnology, 2, 751–760.

Prokesch, S. (2017). The Edison of medicine. Harvard Business Review, March–April.

Rasmussen, N. & Kuch, D. (2026). ‘Nano Scale [is] Magic Scale’: On EPR, Unicorns, and Enchantment in Nanomedicine. Social Studies of Science. https://doi.org/10.1177/03063127261424493

Roco, M.C. (2023). National Nanotechnology Initiative at 20 years. Journal of Nanoparticle Research, 25, 197–224.

Schaefer, M. et al. (2024). Misrepresented technological solutions in imagined futures: The origins and dangers of AI hype in the research community. arXiv:2408.15244.

Seymour, L.W. (1992). Passive tumor targeting. Critical Reviews in Therapeutic Drug Carrier Systems, 9(2), 135–187.

Star, S.L. & Griesemer, J.R. (1989). Institutional ecology, translations, and boundary objects. Social Studies of Science, 19(3), 387–420.

Verhoeven, P. (2021). On the social shaping of quantum technologies: An analysis of emerging expectations through grant proposals from 2002–2020. Minerva, 59, 421–440.

Weber, M. (1991 [1919]). Science as a vocation. In From Max Weber: Essays in Sociology (pp. 129–156). Routledge.

Behind the Scenes: How to Set Up a Diamond Open Access Journal?

Muriel Strange — Wed, 11 Mar 2026 23:00:00 GMT

In this post, we explore the practical steps and considerations involved in setting up a Diamond Open Access (OA) journal. From securing funding to obtaining ISSNs and DOIs, and building the necessary infrastructure: we provide a comprehensive guide for researchers and institutions interested in scholarly-led, freely accessible publishing. Whether you’re starting a new journal or looking to improve an existing one, this FAQ offers insights drawn from our real-world experience with JOTE.

Q1 - What is Open Access, and what does it mean?

Open Access (OA) is a movement that provides immediate, free, online access to scholarly research. It removes financial barriers (paywalls) or legal barriers and allows anyone to read, download, reuse, and share academic publications and data. This significantly increases knowledge dissemination and societal impact of scholarly research. In short, Open Access aims to transform scholarly communication from a restricted, pay-per-view system to a free, open, and collaborative model for the benefit of science and society.

However, this may not be the case for some types of Open Access. Open Access exists in several models. Green Open Access allows authors to deposit a version of their work in institutional or subject repositories, such as university libraries. Gold Open Access ensures articles are immediately available on the publisher’s website, but usually requires authors to pay article processing charges. Hybrid Open Access offers optional open access within subscription-based journals for a fee, while Diamond Open Access provides full open access without any costs for authors or readers, relying instead on institutional and community support.

Q2 - How can you fund Diamond Open Access?

There is no one-size-fits-all approach to making academic content freely available. If there were, we recognize many more people would do it. The truth of the matter is, funding Diamond OA journals requires ingenuity, persistence, and the willingness to sacrifice one’s own time in service of something you believe in. Funding can come from a variety of sources; if we were to name a few, the most notable would be local or international grants and subsidies, strategic partnerships with other entities, crowdfunding, and the reallocation of funds from non-publishing activities. Note that currently almost no structural grants or subsidies exist for Diamond OA journals, so another source of income is crucial. If you run your journal as a non-profit organization, you might be eligible for free services from big tech companies.

We recommend applying for some one-time grants and contacting universities and university libraries to get the initial funding needed to set up a Diamond OA journal, as well as making use of free services available to non-profits.

We are seeing an increase in ‘researcher-driven’ journals managed entirely by researchers as part of their university commitments. This could be a feasible path towards greater publishing sovereignty for universities. In addition, a lot of Diamond OA journals are run by scientific societies, which is another way to ensure scholarly-led publishing as well as procure funding through membership fees.

It is important to note that operating a Diamond OA journal requires significant personal time sacrifices. Many unpaid hours are required to manage and arrange open-access content. However, as briefly mentioned earlier, Open Access is a movement that is just beginning to gain recognition at the institutional level. It is our hope that in the future this work will bring more and more monetary and non-monetary value to academia and society.

It is our mission to ensure access for researchers, students, and the public, regardless of their institution’s budget.

Q3 - What is a DOI? How do I get one, and why might a DOI be useful?

A DOI (Digital Object Identifier) is a permanent, unique identifier assigned to scholarly works, for example, journal articles, datasets, books, and even blogs like this one. It ensures that research outputs can always be found and cited, even if their web locations (URL) change. Unlike a regular URL, DOIs remain accessible when a website changes its structure or moves content to a new server. DOIs are essential for proper scholarly attributions, long-term accessibility, and integration into academic indexing and citation systems.

To obtain a DOI, you need to join a DOI registration agency! The most popular agency is Crossref, but there are competitors such as DataCite. Once registering with them, you’ll then receive a unique DOI prefix that can be connected to the articles you publish through your publishing platform. You’ll also need to obtain an International Standard Serial Number (ISSN) to assign DOIs to your articles. For more information on this aspect, please see the FAQ on ISSNs. Whichever DOI registration agency you use will require that you submit metadata for each journal so that the article is reliably identified.

Q4 - What is an ISSN and how can you obtain an ISSN (in the Netherlands)?

An ISSN consists of eight digits: seven digits plus a check digit. The number sequence makes serial publications, such as newspapers, magazines, and series, available for sale and easy to find for retailers and consumers. Good news - obtaining an ISSN is a straightforward process in the Netherlands. You can request an ISSN from the ISSN Center Netherlands at https://www.kb.nl/uitgevers/issn-aanvragen. The ISSN Centre Netherlands is housed within the Royal Library of the Netherlands. You can apply for a free International Standard Serial Number (ISSN) with them. They recommend including the ISSN in the colophon of each issue of the publication.

This brings us to some of the benefits of an ISSN, such as the ISSN remaining the same for all issues within the series, as long as the title remains the same. It also helps you gain international visibility for your journal (other examples include a newspaper, magazine, or series). ISSNs also live and are collected in an international database, ISSN.org. This simplifies the processes for searching digital files and processing and exchanging data.

Q5 - How do you register a DOI?

The Journal has created a step-by-step guide for how to register a DOI. Please note that this guide is specific to our journal, and that we use PubPub as our publishing platform. The guide is as follows:

Navigate to https://manage.crossref.org/records and log in to your account
Navigate to ‘New record’, then select ‘Journal Article’ and continue
Fill in the first set of fields according to the following example. Pay close attention to these points:
1. The DOI follows your proper format, and you are absolutely sure it is correct.
2. You have named your PubPub page already, so the link is definite.
3. Remove/draft from the end of the PubPub link.
Next, add contributors.
1. These are only the listed authors.
2. Make sure the ORCiD IDs are correct, and affiliations are complete.
For other fields:
1. The published online date is the date on which you make the article available online
2. Page and article number are optional, add funding only if applicable
3. We do not use a similarity check URL, as license we use: [https://creativecommons.org/licenses/by/4.0/]
If applicable, we enter the relationship between the article and other articles. These relationships are documented one-way on the supplement. For example, when we have a main article and a reflection, we add the relationship ‘IsSupplementTo’ the reflection article, and relate it to the DOI of the main article. When it is a commentary, we add the relationship to the commentary, relating to the main article. We do not add these relationships to the DOI of the main article.
Simply paste in the reference section from the article. The cleanest way to do this is by copying it from the finalized PubPub article.
You can now move to the next page, adding the journal data (ISSN, title, and abbreviation). We do not add a crossmark update policy.
On the next page, you can add the issue data if this is already known. If the article isn’t part of an issue yet or soon, this can be left blank.
And you’re done! Review your data, and submit when ready.

Q6 - What infrastructure is needed to create an OA journal?

To build your infrastructure, you will need a publishing platform in which the publication process unfolds. This is where authors submit papers, editors record decisions, peer reviewers submit reviews, and copyediting is posted. There are great open-source solutions available, for example Open Journal Systems. It is important not to rely on emails and Excel files, as it is incredibly difficult to scale without some sort of dedicated platform.

You should also use an archiving service like LOCKSS to ensure that your published material is available in the future. An ISSN and Crossref membership are essential to this as discussed in previous questions.

You will then need a website where your journal lives. On this website, you will publish all of your policies which also need to be designed in order to achieve academic or industry standards. You must also write up policies on your licenses, copyrights, OA approach, editorial guidelines, peer review standards, etc. While this can feel overwhelming, rest assured that in recent years, some excellent services have emerged to make the process of establishing an OA journal much easier. We recommend interested people to check out the OA Journals Toolkit.

Q7 - What is the Directory of Open Access Journals (DOAJ) and how do you get into it?

The Directory of Open Access Journals (DOAJ) is an independent, curated database of high-quality, peer-reviewed open-access journals. Being listed in DOAJ increases a journal’s visibility, credibility, and trustworthiness within scholarly communities. The DOAJ sets strict and diverse standards for transparency, licensing, and editorial quality, and only indexes open-access journals that meet these criteria. For simplicity, we have provided a list including all of the criteria at the end of this page.

In addition, we wish to highlight some elements that are required of such an OA journal: it must be open access, have an internationally accessible website, and transparently list a host of policies related to licensing, copyrighting, and quality controls. Journals must also have a proven history of publication, either by publishing at least 10 OA articles, or by publishing at least 1 edition a year for 1 year minimally. The standards demanded by DOAJ ensure that the indexed journals are of a high quality, lending prestige and reliability to smaller and newer publishers. Lastly, we recommend you check out the guidance page on their website here for more information.

Cultures of Trial and Error: The Narrative Side of (Open) Science

Sheena F. Bartscherer — Mon, 02 Feb 2026 23:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

Heterogenous communities with explicit commitments to science corrections or what this blog series summarises under the descriptor ‘cultures of trial and error’ have existed for the longest time in science (Knorr, 1979; Shapin & Schaffer, 1985; Derksen, 2019; Hesselmann & Reinhart 2024; Antonakaki et al., 2025). Some might even argue that the supervision and monitoring of scientific practices is a key concern of science, equally as important as its primary fact-finding mission (Derksen, 2019). One reason being that with the flawed human nature, mistakes, – or even worse – intentional fraud, as well as other kinds of issues and redundancies will be an inevitability. And if left unchecked, ‘perverse incentive structures’ will grow rampant, jeopardising public trust in science, undermining its credibility, and thereby directly threatening the scientific endeavour itself (see e.g., Hubbard 2016; Lilienfeld, 2017; Crane, 2018; Klein et al. 2018; Dirnagl, 2019; Wingen et al., 2019; Penders, 2022; Methner et al. 2023; Brodeur et al., 2024; Balafoutas et al., 2025). Naturally, these issues need to be addressed. This is especially true, when circumstantial evidence of the non-replicability of central research findings is pointing towards not just individual problems occurring within a single discipline, but rather systemic issues spanning across multiple research disciplines. As was the case in social psychology and biomedicine, when large-scale replication studies failed to replicate key findings of their respective fields (see Crane, 2018; Zwaan et al., 2018; Malich & Munafò, 2022). The uncovering of these cross-disciplinary issues has led many to the declaration of an urgent crisis of confidence or a replication crisis in the mid-2010s (see Pashler & Harris, 2012; Earp & Trafimow, 2015; Ioannidis, 2015; Nosek et al., 2022). Science seemed ‘broken’ and inspired researchers and other non-academic stakeholders to come together in a grassroots-like manner to find practical solutions to these issues. For example, by creating international replication initiatives (Bartscherer & Reinhart, 2025), new publishing formats such as pre-prints and registered reports or the publishing of null results (Nosek et al., 2015; Baxter & Burwell, 2017; Penders, 2022; Balafoutas et al., 2025). Other solutions included the creation of new reporting and teaching guidelines (McLoughlin & Drummond, 2017; Perrault, 2023), automated labs (Adam, 2024; Ulpts et al., 2025-2), as well as continued campaigning about these issues and through releasing manifestos (e.g., Munafò et al., 2017).

However, what this recounting or story of science tends to overlook, is that it is simply that: a story. A very convincing and highly successful one, given how many actors in academia and its periphery have since adopted it and joined the Open Science (OS) movement as active reformers (Nelson et al., 2021; Penders, 2022; Peterson & Panofsky, 2023; Bartscherer & Reinhart, 2025; Ulpts et al., 2025-2). Many of them are highly influential stakeholders from academia and beyond (Bartscherer & Reinhart, 2025). All having seemingly embraced the OS mission, all agreeing that science is deeply broken and needs to fundamentally change its ways. Primarily, by becoming more open, replicable, reproducible, reliable, efficient, transparent, robust, rigorous, inclusive, diverse, etc. pp. (Ulpts et al., 2025-2).

Building on Bruner’s (1991) idea that narratives are a cornerstone of how we, as humans, construct our social realities and Polkinghorne’s (1995) extension of it into the method of narrative inquiry, in this blogpost, I will attempt to provide a first schematic overview of the narrative mosaic that pieced together, help create this highly successful story the OS movement tells about science. I will highlight some of its more prominent narrative elements (subplots) and contextualise the demands that are formulated in them.¹ The overall story they help to form, is intended to convince others of the need for what it is promoting: fundamental (and lasting) changes to the way that science is being done. The OS story is thus purposeful, influential, and by no means neutral. Hence why we shall now take a closer look at the ideas that this story of science promotes and the perspectives that underpin it (Bruner, 1991; Crespo López et al., 2025).

To do so, I will be drawing from my work in the DFG-funded research project Replication as a Social Movement, in which we mapped a total of 175 international replication initiatives (RIs) via virtual snowball sampling of this core community in the OS movement as well as through conducting research interviews (n = 28) with people active in these RIs to learn about their work and what motivated them to engage with it in the first place (Bartscherer & Reinhart, 2025). This material, an extensive literature review, and participant observations of key conferences and events of the OS movement (2023 – 2025) are the basis for the now following analysis.

Six Prominent Subplots in the Open Science Movement’s Story of (a broken) Science

Open Science as ‘good science’ that is fighting a corrupt system:

As part of their story about science, reformers commonly describe academia as a deeply ‘broken system’ that is dominated by biased researchers (see ‘publication bias’; Flis, 2019), who get awarded despite their corrupt behaviours and ‘questionable research practices’ (QRPs; also see ‘perverse incentives’, Stephan, 2012; Chambers et al., 2014; Crane, 2018; Penders, 2024; Balafoutas et al., 2025). This subplot usually promotes a rather Manichean characterisation of the social dynamics at play, commonly speaking of a fight between ‘good’ versus ‘bad science’. Despite such over-simplifications, the wider community seems to agree that there is in fact a ‘better science’ that they are helping to produce (Nosek et al., 2012; Farnham et al., 2017; Goldensher, 2023; Crespo López et al., 2025). As part of this subplot, reformers tend to depict themselves as (anti)heroes who are trying to counteract the ‘broken system’ by bringing about the urgently needed (positive) changes and reforms. And they do all this, despite not being acknowledged or rewarded for their efforts by those in power (e.g., not receiving awards or funding for their OS work; see Buck, 2023). This first subplot provides a core explanation as to why reformers assume that the replication crisis was able to occur in the first place (see corruption and biases) and why their OS work matters as much: because they are fighting the (flawed) human nature and the self-serving interests of researchers and corrupt academic institutions.

Reformers are outsiders & Open Science is driven by grassroots activism:

When discussing their positions in the current science system, reformers often stylise themselves as outsiders (see e.g., Buck, 2023), who belong to an active grassroots movement that is predominantly driven by early career researchers, seeking to create change bottom-up (Nicholas et al., 2019, 2020; Toribio-Flórez et al., 2021). Consequentially, reformers tend to portray themselves as those who ‘shake things up’, speaking inconvenient truths to power (see publication of null-results; Govaart et al., 2022), and pushing for urgently needed (credibility) revolutions (O’Donohue et al., 2022). This ‘underdog’ or ‘outsider’ subplot of OS as a ‘revolutionary’ (scientific) counter culture movement (Bartling & Friesike, 2014), is also illustrated well by the names that reformers choose for their projects, such as for example, the ‘RIOT Science Club’, which was founded in 2018 by researchers at King’s College, London. Their inconvenience and rioting are, however, always seen as serving a greater good (science). This subplot seems most relevant for the reformers’ communal identity-building and is usually brought up to publicly justify their (often rather harsh) critiques of existing protocols and (academic) institutions. However, given the financial ties of central OS institutions (e.g., Center for Open Science, US) to private funders (e.g., Arnold Ventures) and the direct involvement of highly influential actors from politics and private business (e.g., ex-Alphabet CEOs in the newly founded Metascience Alliance), the claims in this subplot about OS being ‘counter-culture’ or ‘grassroots’ appear to stand in stark contrast to the current organisational practices of the movement (Bartscherer & Reinhart, 2025).

Science is in crisis & Open Science will save it:

Another important subplot in their story on science promotes the idea that science is not only corrupt (see first subplot) but that it is (or has been) in an urgent crisis. Namely a ‘replication crisis’ or a ‘crisis of confidence’ (Hesselmann & Reinhart, 2024; Crespo López et al., 2025). A handful of highly publicised cases of academic fraud and QRPs of renowned researchers in the mid-2010s were commonly interpreted not as outliers, but as symptoms of a sick and corrupt system (see first subplot) (Penders, 2024; Bartscherer & Reinhart, 2025). The failed large-scale replication attempts that followed them (see e.g., Klein et al. 2018) were thus seen as undisputable proof that issues of irreproducibility were indeed systemic and wide-spread, and that a crisis was indeed unfolding (Morawski, 2022; Ulpts et al., 2025-2). As a consequence, reformers commonly warn about a loss of the public’s trust (in the institution of science) if these issues and ‘low standards’ are left unaddressed (see e.g., Camerer et al., 2016; Penders et al., 2019; Roettger et al., 2019; Wingen et al., 2020; Edlund et al., 2022; Methner et al., 2023; Penders, 2024). The loss of the public’s trust in science thus serves as justification for the urgency with which reformers move to implement their drastic and universal reforms. This urgency can be seen as a narrative motor behind the reformers’ push to act immediately and implement OS reforms as quickly as possible (Penders, 2024). An additional reason being that OS and its reforms, in this subplot, are usually depicted as the (research) programme that will save science as a whole (Mirowski, 2018; Crespo López et al., 2025). Allegedly by making the system more resilient. Thereby preventing similar crises from happening in the future (Stengers, 2016).

Open Science as the correct way of ‘doing science’:

Building on the crisis subplot and the claim that OS is able to save science, is the subplot that OS is bringing science back to its roots, its original ideals, and (with its proposed reforms) is going to undo all the ‘bad tendencies’ that have been allowed to grow rampant for so long. As a part of this subplot, OS is usually hailed as a panacea that is going to help fulfil science’s ‘original’ promises to society (Mirowski, 2018; Bartscherer, 2025; Crespo López et al., 2025; Ulpts et al., 2025-1). With a somewhat melancholic undertone, OS herein is typically stylised as the actual embodiment of an imagined past, a utopia of ‘good science’ that will finally serve the common good (again) (Bartscherer & Ulpts-1, 2025-1; Penders, 2025). Slogans such as ‘Open Science: Just science done right!’ embody this idea (Imming & Tennant, 2018). Closely related to the second subplot of OS as a counter culture movement, herein OS is commonly characterised as actively trying to resist, what some have called the ‘neoliberalisation of science’ (Nosek et al., 2012; Mirowski, 2018; Hostler, 2024). Discourse on ‘publication biases’ (Marks-Anglin & Chen, 2020), the ‘gaming of metrics’ (Biagioli & Lippmann, 2020) or the ‘publish or perish’ mentality (Elbanna & Child, 2023; Knöchelmann, 2024) are often pointing to this problematised aspect of the commodification of science to explain why things need to fundamentally change and why OS reforms should be implemented (Hostler, 2024).

Open Science will foster a ‘slow science’ & increase its overall quality, democratisation, equity, inclusion, and sustainability:

Closely related to the fourth sublot are claims in which OS reforms are depicted as inevitably resulting in a positive ‘slowing down of science’. Reformers argue that through the active shift of incentive structures towards OS practices, the pace of science will become ‘humane’ again. Primarily through the anticipated rewards of getting more ‘correct’ claims and replications of central studies published (which takes time), instead of the quick and constant publication of loads of papers annually (with low quality claims), e.g. in the form of ‘salami slicing’ (Nosek et al., 2012; Stengers, 2016; Frith, 2020; Antonakis, 2023; Hostler, 2024; Köstenbach & Oransky, 2024). It is then usually assumed that through following OS ideas, pressure on researchers to follow academia’s ‘perverse incentives’, is going to be effectively alleviated and research as well as career decisions are going to be made more consciously (Berkowitz & Delacour, 2020; Balafoutas et al., 2025). This ‘anti-capitalist’ promise of OS (as a better future) also goes hand in hand with the idea that OS is helping science become more sustainable through its practices and new formats – e.g., Open Access publishing or preprinting. The idea is that they are effectively ‘saving human, monetary, and natural resources in all research fields’, ‘stimulating international collaborations’, and making science overall more efficient (Fraser et al., 2021; Govaart et al., 2022).

Furthermore, OS is argued to positively influence scientific progress and the scientific exchange of knowledge. This is supposed to be achieved through, e.g., eliminating gatekeeping functions of journals or peer review (e.g. through pre-printing), making access to knowledge (primarily online) more democratic, and increase its inclusion (Desmond, 2024). Equity and inclusion are thus often brought up as core values of OS, making their (structural) increase within the science system a primary goal. To realise this goal, in this subplot to their larger story on science, reformers usually advocate for the need to make research (findings) more (financially) accessible to marginalised researchers, particularly to those in the global south, e.g., through open data practices and the elimination of paywalls (Serwadda et al., 2018; Desmond, 2024; Govaart et al., 2022). If these measures have actually been effective and how much they have contributed to the de-marginalisation of global south researchers (if at all), however, has been put into serious question (Mirowski, 2018; Bartscherer & Ulpts, 2025-2).

Humans are the problem; standardisation & automation are the (mechanical) solutions:

To offer explanations as to why so many replication attempts have failed (and continue to fail) and why cases of fraud and QPRs were able to occur in the first place, reformers often point to the flawed human nature as a primary reason for concern: in true rationalist fashion, researchers are seen as ‘only human’ and thus as deeply flawed beings, who – given the opportunity – will always lie, cheat, steal. Humans thus introduce uncertainties to any scientific endeavour, no matter how noble the cause; some intentionally, but most unintentionally (Ulpts et al., 2025-2). To eradicate these (human) uncertainties from research processes, reformers commonly present automation and standardisation (through strict protocols) as viable solutions to reduce said (inherently human) biases and uncertainties (see Flis, 2019; Penders, 2022), increase (scientific) efficiency (Peterson & Panofsky, 2021), and help produce a ‘better science’ overall: A ‘mechanically objective’ science – that will make research decisions and practices more transparent, traceable, and thus reproducible. Research that has been produced in this manner, is depicted as serving public as well as private interests more effectively, introducing a cost-calculation-logic to the discourse (Daston & Galison, 2010). The use of computerised tools is often brought up in this context as well, usually of artificial intelligence (AI), to help estimate replicability of studies in a faster, more reliable, and efficient way (see e.g., Yang et al., 2020). Another related example would be ‘automated labs’ that, with the help of AI, promise to reduce human interventions to an absolute minimum (see Mirowski, 2018; Ulpts et al., 2025-2). Reflections about concerns regarding the irreproducibility of AI decision-making are seemingly absent from these suggestions. At its core this subplot seems to follow the positivist conviction that there is an ‘objective’ truth that science ought to uncover and (human) obstacles need to be removed from this process as much as possible (see Derksen, 2019; Graves et al., 2025).

The Story Reformers Tell About (Open) Science

The six subplots that I outlined in this blogpost are to be understood as condensed versions of some of the more prominent narrative elements one encounters when talking to OS reformers about their work, when reading their papers, or listening to their talks and conference presentations as well as their public online discussions on everything (open) science. They are not to be understood as having hard and clear boundaries. Rather, as one might have already noticed when reading this blogpost, they tend to blend into each other as they build and expand on one another, typically occurring together in actual discourse. Their distinction into six separate subplots on my end is therefore more so a theoretical aid to help render certain notions and ideas within the OS story on science distinguishable and therefore discussable. Woven together, these and other elements spell out the (sometimes contradictory) ‘communal story telling’ of the OS movement: about science in general, their own work, their goals, but also their perceived ‘outside’. A story I attempted to retell in the first paragraph of this blogpost. This story can be seen as a partial account of reality, and according to Bruner (1991), is by no means a ‘neutral’ form of communicating information. Instead “There seems indeed to be some sense in which narrative, rather than referring to ‘reality’, may in fact create or constitute it, as when ‘fiction’ creates a ‘world’ of its own – […].” (Bruner, 1991, p.13). Using the lens of narratives or stories when looking at the discourse around OS allows us, the (scientific) audience, to take a step back and critically reflect on how the OS movement makes sense of the world. Instead of taking their story of science as fact, we now can take it for what it is: a summary of personal accounts and attempts at making sense of the (scientific) world and what reformers believe to have occurred in it. A prime example being the so-called replication crisis, which reformers are vehemently insisting has in fact occurred. Whereas their critics call it a moral panic and claim that the cases of fraud and problematic research practices were simply overblown. Critics further counter that the failed replication attempts that followed this panic were more so an issue of skill, incomplete protocols or tacit knowledge that those attempting to replicate the works seemed to be lacking (see, e.g., Monin & Oppenheimer et al., 2014).

As demonstrated with the chosen examples, the story that OS tends to tell about the (more recent) history of science consists of many universal claims and recurring themes, usually with a strong ‘positivist’ undertone (Derksen, 2019; Graves, 2025). Most often this story and its subplots convey reformers’ deep convictions that there are indeed universal and objective truths that they have helped to uncover: about science being in crisis, corrupt, and vulnerable due to human errors and biases. All while promoting ideas of a ‘good’ vs ‘bad science’ (or right vs wrong science), and thus rather simplistic Manichean characterisations of the world of science. It is this story that reformers tell about science, that has given them the confidence to urgently push their reforms onto the larger science system. Reforms that they claim will fix a (currently) broken science system. OS for them – and increasingly more actors beyond academia – seems to have become the answer to all questions (Mirowski, 2018; Bartscherer & Reinhart, 2025; Crespo López et al., 2025). Yet, their story of science promotes understandings of our (scientific) realities that have apparent (positivistic) biases themselves: ideals of hard facts and undeniable truths that can be uncovered and corrected with the right (=OS) methods; all intended to lead us towards an idealistically proclaimed ‘scientific utopia’ (see, e.g., Nosek, et al., 2012; Bartscherer & Ulpts, 2025-1; Penders, 2025). Noticeably, the solutions that they produce, show a strong practice orientation, materialising in a multitude of mechanical and standardised OS guidelines, protocols, and recommendations that are to be implemented immediately and with urgency (Penders, 2024; Ulpts et al., 2025-2). The OS story has, however, not only produced practical solutions, it has also introduced a strong sense of uncertainty into the overall discourse about science (in crisis). Beyond cohorts of anxious students and researchers (‘what/who can we still trust?’), some negative consequences have since manifested in the form of unwanted fraternisations from autocratic figures in the political domain. The Trump admin notoriously picked up elements of the OS story and language in a White House memorandum (‘restoring gold standard science’) to justify their own political agenda and exert control over the scientific domain (Hesselmann & Reinhart, 2024; Penders, 2024; O’Grady, 2025; Ulpts et al., 2025-2).

The OS story is undeniably a demonstratively successful one, given the large uptake by (non)academic actors across many different social domains (Bartscherer & Reinhart, 2025). Its impact can be felt across academia and beyond, despite theirs being only one story amongst many of the (recent) history of science (Daston, 2017). Its cursory adoption as fact, however, jeopardises our collective memory of the scholarly community as well as the imagination of its possible future(s) (Devezer & Penders, 2023; Crespo López et al., 2025). When engaging with the OS story and the proposed solutions by reformers to (systemic) issues, issues they claim to have identified across all of science, caution or at the very least, a heavy dose of scepticism seems more than warranted.

Footnotes:

[1]: Also see Nelson et al., (2021) for a more detailed analysis of the discursive dimensions of the replication crisis and more specifically of reproducibility in the literature.
[2]: Replication initiatives were defined by us as “(temporary) projects and institutions that have or had the explicitly stated mission2 to, either, replicate or reproduce findings of existing (central) studies and experiments themselves (i.e., replication studies), or, support the increase of replication numbers and the reproducibility of research findings (e.g., via providing digital or institutional infrastructures, funding, awards, university curricula, publishing guidelines, workshops, teaching material, etc.).” (see Bartscherer & Reinhart, 2025).

References:

Adam, D. (2024). The automated lab of tomorrow. Proceedings of the National Academy of Sciences,121(17), e2406320121. https://doi.org/10.1073/pnas.2406320121

Antonakaki, M., Sanchez, C. F., Barbeitas, M. (2025). Cultures of Trial and Error. Identifying and Overcoming Barriers in Science Correction. Blogpost. https://doi.org/10.36850/dc83cd30-11ad

Antonakis, J. (2023). In support of slow science: Robust, open, and multidisciplinary. The Leadership Quarterly, 34(1), 101676. https://doi.org/10.1016/j.leaqua.2023.101676

Balafoutas, L., Celse, J., Karakostas, A., & Umashev, N. (2025). Incentives and the replication crisis in social sciences: A critical review of open science practices. Journal of Behavioral and Experimental Economics, 114, 102327. https://doi.org/10.1016/j.socec.2024.102327

Bartling, S., & Friesike, S. (Eds.). (2014). Opening Science. Springer International Publishing. https://doi.org/10.1007/978-3-319-00026-8

Bartscherer, S. F. (2025). 'People tend to be kind by default'. Presentation. Let’s Talk about Science Reform! A Workshop on Theoretical and Methodological Approaches to Investigating the Open Science Movement, Bartscherer, Sheena F. et al., 2025. https://doi.org/10.5446/70580

Bartscherer, S. F., & Reinhart, M. (2025, August 9). The (Non)Academic Community Forming around Replications: Mapping the International Open Science space via its Replication Initiatives. Preprint. https://doi.org/10.31235/osf.io/rbyt6_v2

Bartscherer, S. F., & Ulpts, S. (2025, July 26). Stories from the Imagined Utopia of an Open Science. Contextualizing Reformers' Conceptions of "Good Science". Presentation. Evidence & Uncertainty in Science: Methodological, Philosophical and Meta-Scientific Issues, Tübinger Forum für Wissenschaftskulturen, Universität Tübingen. Zenodo. https://doi.org/10.5281/zenodo.16453233

Bartscherer, S. F., & Ulpts, S. (2025, July 28). The Broken Promises of Open Science. How Open is Science Today Truly and for Whom?. Inclusive Open Science. From Global Asymmetries to Pluriversal Design, Berlin, Germany. Zenodo. https://doi.org/10.5281/zenodo.16531429

Baxter, M. G., & Burwell, R. D. (2017). Promoting transparency and reproducibility in Behavioral Neuroscience: Publishing replications, registered reports, and null results. Behavioral Neuroscience, 131(4), 275–276. https://doi.org/10.1037/bne0000207

Berkowitz H., & Delacour H. (2020). Sustainable Academia: Open, Engaged, and Slow Science. M@n@gement, 23(1), 1-3. https://doi.org/10.37725/mgmt.v23.4474

Biagioli, M., & Lippman, A. (Eds.). (2020). Gaming the Metrics: Misconduct and Manipulation in Academic Research. The MIT Press.

Brodeur, A., Esterling, K., (…), Young, L. (2024). Promoting Reproducibility and Replicability in Political Science. Research & Politics, 11(1), 20531680241233439. https://doi.org/10.1177/20531680241233439

Bruner, J. (1991). The Narrative Construction of Reality. Critical Inquiry, 18(1), 1–21.

Buck, S. (2023, August 13). Metascience Since 2012: A Personal History [Substack newsletter]. The Good Science Project. https://goodscience.substack.com/p/metascience-since-2012-a-personal

Camerer, C. F., Dreber, A., (…), Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science. 351(6280), 1433-1436. https://doi/10.1126/science.aaf0918

Chambers, C. D., Feredoes, E., D. Muthukumaraswamy, S., & J. Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. https://doi.org/10.3934/Neuroscience.2014.1.4

Crane, H. (2018). The Fundamental Principle of Probability: Resolving the Replication Crisis with Skin in the Game. Unpublished Manuscript. https://harrycrane.com/FPP-final.pdf

Crespo López, M., Pallise Perello, C., de Ridder, J., & Labib, K. (2025). Open Science as Confused: Contradictory and Conflicting Discourses in Open Science Guidance to Researchers. Preprint. https://doi.org/10.31222/osf.io/zr35u_v3

Daston, L. (ed.). (2017). Science in the Archives: Pasts, Presents, Futures. University of Chicago Press, Chicago Scholarship Online, 2017. https://doi.org/10.7208/chicago/9780226432533.001.0001.

Daston, L., & Galison, P. (2010). Objectivity (First paperback edition). Zone Books.

Derksen, M. (2019). Putting Popper to work. Theory & Psychology, 29(4), 449–465. https://doi.org/10.1177/0959354319838343

Derksen, M. (2021). A Menagerie of Imposters and Truth-Tellers: Diederik Stapel and the Crisis in Psychology. In S. Woolgar, E. Vogel, D. Moats, & C.-F. Helgesson (Eds.), The Imposter as Social Theory: Thinking with Gatecrashers, Cheats and Charlatans (1st ed., pp. 53–76). Bristol University Press. https://doi.org/10.2307/j.ctv1p6hphs.8

Desmond, H. (2024). Gatekeeping should be conserved in the open science era. Synthese, 203(5), 160. https://doi.org/10.1007/s11229-024-04559-2

Devezer, B. & Penders, B. (2023). Scientific Reform, Citation Politics and the Bureaucracy of Oblivion. Quantitative Science Studies 4 (4):857–859. doi:10.1162/qss_c_00274.

Dirnagl, U. (2019). Rethinking research reproducibility. The EMBO Journal, 38(2), e101117. https://doi.org/10.15252/embj.2018101117

Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6. https://doi.org/10.3389/fpsyg.2015.00621

Edlund, J. E., Cuccolo, K., Irgens, M. S., Wagge, J. R., & Zlokovich, M. S. (2022). Saving Science Through Replication Studies. Perspectives on Psychological Science, Vol. 17(1), 216-225. https://doi.org/10.1177/1745691620984385

Elbanna, S., & Child, J. (2023). From ‘publish or perish’ to ‘publish for purpose’. European Management Review, 20(4), 614–618. https://doi.org/10.1111/emre.12618

Farnham, A., Kurz, C., (…), Hettne, K. (2017). Early career researchers want Open Science. Genome Biology, 18(1), 221. https://doi.org/10.1186/s13059-017-1351-7

Flis, I. (2019). Psychologists psychologizing scientific psychology: An epistemological reading of the replication crisis. Theory & Psychology, 29(2), 158–181. https://doi.org/10.1177/0959354319835322

Fraser, N., Brierley, L., (…), Coates, J. A. (2021). The evolving role of preprints in the dissemination of COVID-19 research and their impact on the science communication landscape. PLOS Biology, 19(4), e3000959. https://doi.org/10.1371/journal.pbio.3000959

Frith, U. (2020). Fast Lane to Slow Science. Trends in Cognitive Sciences, 24(1), 1–2. https://doi.org/10.1016/j.tics.2019.10.007

Goldensher, L. O. (2023). Problems of knowledge, problems of order: The open science field site. Frontiers in Sociology, 8, 1149073. https://doi.org/10.3389/fsoc.2023.1149073

Govaart, G. H., Hofmann, S. M., & Medawar, E. (2022). The Sustainability Argument for Open Science. Collabra: Psychology, 8(1), 35903. https://doi.org/10.1525/collabra.35903

Graves, T. A., Pownall, M., Prosser, A. M. B. (2025). Getting Creeped Out? Open Science, Qualitative Methods, and the Dangers of Positivism Creep. Preprint. https://doi.org/10.31235/osf.io/nphjc_v1

Hesselmann, F., & Reinhart, M. (2024). From scandal to reform: Approaches to research integrity at a turning point. Journal of Responsible Innovation, 11(1), 2414491. https://doi.org/10.1080/23299460.2024.2414491

Hostler, T. J. (2024). Open Research Reforms and the Capitalist University: Areas of Opposition and Alignment. Collabra: Psychology, 10(1), 121383. https://doi.org/10.1525/collabra.121383

Hubbard, R. (2016). Corrupt research. SAGE Publications, Inc., https://doi.org/10.4135/9781506305332

Imming, M. & Tennant, J. (2018). Sticker open science: just science done right (ENG). Zenodo. https://doi.org/10.5281/zenodo.1285575

Ioannidis, J. P. A. (2015). Failure to Replicate: Sound the Alarm. Cerebrum, Nov. 2015

Klein, R. A., Vianello, M., (…), Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225

Knorr K. D. (1979). Tinkering toward success: Prelude to a theory of scientific practice. Theory and Society, 8(3), 347‐376. https:// doi.org/10.1007/BF00167894

Knöchelmann, M. (2024). Formal authorship in the wake of uncertain futures: The narrative of publish or perish in the humanities. Research Evaluation, 33, rvae044. https://doi.org/10.1093/reseval/rvae044

Köstenbach, T., & Oransky, I. (2024). Salami slicing and other kinds of scientific misconduct: A faux pas for the author, a disaster for science: An interview by Tamara Köstenbach with Ivan Oransky in October 2022 for the research project “Summa cum fraude – Wissenschaftliches Fehlverhalten und der Versuch einer Gegenoffensive”. Information – Wissenschaft & Praxis, 75(1), 1–6. https://doi.org/10.1515/iwp-2023-2041

Lilienfeld, S. O. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4), 660–664. https://doi.org/10.1177/1745691616687745

Malich, L., & Munafò, M. R. (2022). Introduction: Replication of Crises - Interdisciplinary Reflections on the Phenomenon of the Replication Crisis in Psychology. Review of General Psychology, 26(2), 127–130. https://doi.org/10.1177/10892680221077997

Marks‐Anglin, A., & Chen, Y. (2020). A historical review of publication bias. Research Synthesis Methods, 11(6), 725–742. https://doi.org/10.1002/jrsm.1452

McLoughlin, P., & Drummond, G. (2017). Publishing replication studies to support excellence in physiological research. Experimental Physiology, 102(9), 1041–1043. https://doi.org/10.1113/EP086344

Methner, N., Dahme, B., & Menzel, C. (2023). The “replication crisis” and trust in psychological science: How reforms shape public trust in psychology. Social Psychological Bulletin, 18, e9665. https://doi.org/10.32872/spb.9665

Mirowski, P. (2018). The future(s) of open science. Social Studies of Science, 48(2), 171–203. https://doi.org/10.1177/0306312718772086

Morawski, J. (2022). How to True Psychology’s Objects. Review of General Psychology, 26(2): 157–171.

Monin, B., Oppenheimer, D.M., Ferguson, M.J., Carter, T.J., Hassin, R.R., Crisp, R.J., … Kahneman, D. (2014). Commentaries and Rejoinder on. Social Psychology. 45(4): 299–311.

Munafò, M. R., Nosek, B. A., (…), Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021. https://doi.org/10.1038/s41562-016-0021

Nelson, N. C., Ichikawa, K., Chung, J., & Malik, M. M. (2021). Mapping the discursive dimensions of the reproducibility crisis: A mixed methods analysis. PLOS ONE, 16(7), e0254090. https://doi.org/10.1371/journal.pone.0254090

Nicholas, D., Watkinson, A., (…), Herman, E. (2019). So, are early career researchers the harbingers of change? Learned Publishing, 32(3), 237–247. https://doi.org/10.1002/leap.1232

Nicholas, D., Jamali, (…), Polezhaeva, T. (2020). A global questionnaire survey of the scholarly communication attitudes and behaviours of early career researchers. Learned Publishing, 33(3), 198–211. https://doi.org/10.1002/leap.1286

Nosek, B. A., Hardwicke, T. E., (…), Vazire, S. (2022). Replicability, Robustness, and Reproducibility in Psychological Science. Annual Reviews Psychology, 2022. 73:719-48. https://doi.org/10.1146/annurev-psych-020821114157

Nosek, B. A., Alter, G., (…), Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374

Nosek, B.A., Spies, J.R. and Motyl, M. (2012). ‘Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability’, Perspectives on Psychological Science, 7(6), pp. 615–631. Available at: https://doi.org/10.1177/1745691612459058

O’Donohue, W., Masuda, A., & Lilienfeld, S. (Eds.). (2022). Avoiding Questionable Research Practices in Applied Psychology. Springer International Publishing. https://doi.org/10.1007/978-3-031-04968-2

O’Grady, C. (2025). Science’s reform movement should have seen Trump’s call for ‘gold standard science’ coming, critics say. Science. https://doi: 10.1126/science.zlwoaxz

Roettger, T. B., Schmalz, X., Chambers, C. D., Fraser, H., Mellor, D. T., Ostojic, L., & Schoof, T. (2019). We still need more trust in science: The need for broader adoption of Registered Reports. https://doi.org/10.31222/osf.io/4sb7c

Pashler, H., & Harris, C. R. (2012). Is the Replicability Crisis Overblown? Three Arguments Examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401

Penders, B. (2022). Process and Bureaucracy: Scientific Reform as Civilisation. Bulletin of Science, Technology & Society 42 (4):107–116. doi:10.1177/02704676221126388.

Penders, B. (2024). Scandal in scientific reform: The breaking and remaking of science. Journal of Responsible Innovation, 11(1), 2371172. https://doi.org/10.1080/23299460.2024.2371172

Penders, B. (2025). Cultures of Trial and Error: Wissenschaftsschmerz. Blogpost. https://doi.org/10.36850/fd2ec6bc-702c

Penders, B., Holbrook, J. B., & De Rijcke, S. (2019). Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing. Publications, 7(3), 52. https://doi.org/10.3390/publications7030052

Perrault, E. K. (2023). Teaching replication through replication to solve the replication “crisis”. Communication Teacher, 37(3), 220–226. https://doi.org/10.1080/17404622.2022.2123110

Peterson, D., & Panofsky, A. (2021). Arguments against efficiency in science. Social Science Information, 60(3), 350–355. https://doi.org/10.1177/05390184211021383

Peterson, D., & Panofsky, A. (2023). Metascience as a Scientific Social Movement. Minerva, 61(2), 147–174. https://doi.org/10.1007/s11024-023-09490-3

Polkinghorne, D. E. (1995). Narrative configuration in qualitative analysis. International Journal of Qualitative Studies in Education, 8(1), 5–23. https://doi.org/10.1080/0951839950080103

Serwadda, D., Ndebele, P., (…), Wanyenze, R. K. (2018). Open data sharing and the Global South—Who benefits? Science, 359(6376), 642–643. https://doi.org/10.1126/science.aap8395

Shapin, S., & Schaffer, S. (1985). Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton University Press.

Stengers, I. (2016). ‘Another Science is Possible!’. A Plea for Slow Science. In H. Letiche, G. Lightfoot, & J.-L. Moriceau, Demo(s). BRILL. https://brill.com/view/title/37881

Stephan, P. (2012). Perverse Incentives. Nature, 484, 29–31. https://doi.org/10.1038/484029a

Toribio-Flórez, D., Anneser, L., (…), & on behalf of Max Planck PhDnet Open Science Group. (2021). Where Do Early Career Researchers Stand on Open Science Practices? A Survey Within the Max Planck Society. Frontiers in Research Metrics and Analytics, 5, 586992. https://doi.org/10.3389/frma.2020.586992

Ulpts, S., Bartscherer, S. F., Field, S. M., & Penders, B. (2025-1). The social replication of replication: Moving replication through epistemic communities. https://doi.org/10.31235/osf.io/pqc4v_v1

Ulpts, S., Bartscherer, S. F., Penders, B., & Nelson, N. (2025-2). Epistemic oligarchies: capture and concentration through scientific reform. Preprint. https://doi.org/10.5281/zenodo.17136864

Wingen, T., Berkessel, J. B., & Englich, B. (2020). No Replication, No Trust? How Low Replicability Influences Trust in Psychology. Social Psychological and Personality Science, 11(4), 454–463. https://doi.org/10.1177/1948550619877412

Yang, Y., Youyou, W., & Uzzi, B. (2020). Estimating the deep replicability of scientific findings using human and artificial intelligence. Proceedings of the National Academy of Sciences, 117(20), 10762–10768. https://doi.org/10.1073/pnas.1909046117

Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972

Stop Formatting Before the Desk Reject: A Proposal for Staged Submissions

Bert Bakker — Mon, 26 Jan 2026 23:00:00 GMT

Many flagship journals reject most manuscripts at the desk-review stage, with rates of 40–80% being common (e.g., Dolan et al., 2024; Jiang et al., 2019; Tripp & Dion, 2024). As submission volume increases (Thelwall & Sud, 2022), including an apparent rise in low-quality contributions facilitated by large language models (Corker, 2025), early rejections are likely to become even more frequent. This is understandable: editors process many submissions, reviewers are scarce, and early decisions keep the system functioning. Yet, high desk-rejection rates create a rarely acknowledged structural problem: authors must comply with extensive submission requirements - cover letters, collaborator lists, funding statements, questionnaires, formatting templates, and citation rules - before editors decide whether a manuscript is worth sending out for review. Preparing these materials can take hours or even days (Jiang et al., 2019). The work has no value for authors who receive a desk-rejection, and it is unclear how many of these requirements inform editorial decisions. Moreover, they are often idiosyncratic, making them largely useless for future submissions. Early-career researchers who often submit collaborative work and scholars with heavier teaching loads are especially affected, as fine-tuning submissions comes at the expense of already scarce time for research or teaching. The result is an inefficient and inequitable system. A more efficient system in which authors provide only information needed at each stage of evaluation is possible. That is what we propose in this blog.

While others have discussed the burden of formatting requirements and the need to streamline submission processes (e.g., Budd, 2017; Jiang et al., 2019; Zon et al., 2021), we believe that rising submission volumes, partly driven by the use of large language models, underline the need to reform publishing systems. Our point also concerns a different part of the process: journals often request information at first submission that is unnecessary for the initial decision. A more effective approach would be to sequence these requirements, requesting additional details only as the manuscript progresses. This aligns with fundamental principles of good scientific practice: reducing unnecessary burdens, improving transparency, and ensuring that administrative requirements serve a clear purpose.

The first stage should be a minimal submission. At this point, editors need to determine whether a manuscript should be sent out for review. They typically need author information, the title and abstract, the manuscript as a PDF file in any reasonable typesetting format meeting basic requirements such as word counts, and a concise statement or checklist covering ethics, conflicts of interest in research (Ahn et al., 2017; Ebrahim et al., 2016), and supporting information related to open-science practices (Aczel et al., 2020). Editors do not - and should not - base decisions on recent collaborators, funding sources (aside from clear conflicts of interest), conference presentations, or formatting details. Cover letters should not be required either, as they commonly add no information beyond the manuscript, and their role in editorial decisions is unclear. With minimal information, editors can focus on the scientific contribution before making the key decision: desk reject or peer review.

If editors decide that a manuscript merits external review, the process should move to a second stage. Authors may then be asked to provide information needed for reviewer selection: suggested or opposed reviewers and individuals who should be excluded due to close collaboration or involvement with the work. Authors can supply this within a reasonable time window. The manuscript then returns to the editor, who can use this information to assemble reviewers. Crucially, this information becomes relevant only after the decision to send the manuscript out for review. This prevents authors from spending time preparing materials for manuscripts that will never reach reviewers and avoids requiring editors to evaluate information that is irrelevant at first submission.

A third stage occurs only after a manuscript receives conditional acceptance. At this point, authors can be asked to format their manuscripts according to journal style and citation requirements, provide figures in specific formats, and include more details on funding sources and acknowledgments. These tasks are appropriate once publication is within reach, and only a small fraction of authors should be asked to complete them. While not all complex formatting requirements should be outsourced to authors, asking the small number who progress through peer review to meet formatting standards is reasonable and aligns administrative efforts with expected benefits.

We believe that the advantages of this staged approach outweigh potential disadvantages. As more journals have increasingly high desk-rejection rates, authors will no longer need to spend hours preparing submissions to meet requirements irrelevant to the first editorial decision. Editors, in turn, avoid reviewing unnecessary information that does not help them assess scientific merit. Some may worry that simplifying initial requirements could invite more low-quality or spam submissions; yet with LLMs lowering the effort required to generate and submit such manuscripts, existing hurdles are unlikely to deter these attempts, and new filtering processes would be required in any case. It would also be valuable to consult handling editors, such as the first author of this commentary, when designing concrete changes. This blog post is not intended as a definitive solution to rising submission and desk rejection rates, but as a call to begin a serious discussion about structural changes that the publication system can no longer postpone.

Our proposed reform requires only modest adjustments. It does not introduce new obligations; it simply moves existing ones to the stages where they matter. A staged submission model would make the publication process more efficient and focus editorial attention on what matters most: the science itself.

This project has received funding from the European Union, under the Horizon Europe MSCA Doctoral Networks programme (Call HORIZONMSCA-2021-DN-01, Grant Agreement No. 101072992) with participation from the UKRI Horizon Europe guarantee’ scheme.

References

Aczel, B., Szaszi, B., Sarafoglou, A., Kekecs, Z., Kucharsky,` S., Benjamin, D., Chambers, C. D., Fisher, ˇ A., Gelman, A., Gernsbacher, M. A., et al. (2020). A consensus-based transparency checklist. Nature human behaviour, 4(1), 4–6. https://doi.org/10.1038/s41562-019-0772-6

Ahn, R., Woodbridge, A., Abraham, A., Saba, S., Korenstein, D., Madden, E., Boscardin, W. J., & Keyhani, S. (2017). Financial ties of principal investigators and randomized controlled trial outcomes: Cross sectional study. BMJ, 356. https://doi.org/10.1136/bmj.i6770

Budd, J. (2017). Reformatting wastes public funds. Nature, 543(7643), 40–40. https://doi.org/10.1038/ 543040e

Corker, K. (2025, August). Changes to moderation at psyarxiv preprints [Blog post]. PsyArXiv. Retrieved January 15, 2026, from https://blog.psyarxiv.com/2025/08/20/changes- to- moderation- atpsyarxiv-preprints/

Dolan, K., Lawless, J. L., Boehmke, F. J., Cohen, E., Reiter, D., Robertson, G., Siegel, D., & Salvatore, J. (2024). American journal of political science: Annual report to the executive council of the midwest political science association (tech. rep.) (Co-Editors in Chief: Kathleen Dolan and Jennifer L. Lawless; Associate Editors: Frederick J. Boehmke, Elizabeth Cohen, Dan Reiter, Graeme Robertson, David Siegel; Managing Editor: Julia Salvatore). American Journal of Political Science. https://ajps.org/wp-content/uploads/2024/04/FINAL-COPY-AJPS-editorial-reportMarch-2024.pdf

Ebrahim, S., Bance, S., Athale, A., Malachowski, C., & Ioannidis, J. P. (2016). Meta-analyses with industry involvement are massively published and report no caveats for antidepressants. Journal of clinical epidemiology, 70, 155–163. https://doi.org/10.1016/j.jclinepi.2015.08.021

Jiang, Y., Lerrigo, R., Ullah, A., Alagappan, M., Asch, S. M., Goodman, S. N., & Sinha, S. R. (2019). The high resource impact of reformatting requirements for scientific papers. PLoS One, 14(10), e0223976. https://doi.org/10.1371/journal.pone.0223976

Thelwall, M., & Sud, P. (2022). Scopus 1900–2020: Growth in articles, abstracts, countries, fields, and journals. Quantitative Science Studies, 3(1), 37–50. https://doi.org/10.1162/qss a 00177

Tripp, A., & Dion, M. (2024). American political science review: Annual editorial report (tech. rep.) (Prepared on behalf of the Editors). American Political Science Review. https://apsanet.org/wpcontent/uploads/2024/10/APSR-2023-24-Annual-Report-.pdf

Zon, L. I., Boisvert, J. D., Moreau, H., Chan, I., Weiss, J., Barbano, J., Smith, M., Weber, M., Prasad, M., Stanhope, M., et al. (2021). A uniform format for manuscript submission. Cell, 184(7), 1654– 1656. https://doi.org/10.1016/j.cell.2021.01.030

Cultures of Trial and Error: Field-specific Barriers to Performing Replications

Candida F. Sánchez Burmester — Sun, 07 Dec 2025 23:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

“Reproducibility is a defining feature of science,” opens one article describing a large-scale replication project (Open Science Collaboration, 2012, p. 657). [1] Many advocates for replication begin similarly, framing replicability as a universal, incontrovertible quality of good science. Discussions about what hinders replications are similarly general, focusing on obstacles common to most fields: researchers are not incentivized to conduct replications and editors are not incentivized to publish them (Open Science Collaboration, 2014; Romero, 2018; Vachon et al., 2021), key parameters are not reported in methods sections (Clayson et al., 2019), infrastructures for sharing/reusing data are insufficient (Horbach et al., 2025; Vachon et al., 2021), and little funding is available specifically for replications (Lilienfeld, 2017).

In this piece, we call for more comparative research on field-specific replication dynamics. Replicability is not a goal that makes sense for all research fields, nor should it be taken uncritically as an indicator of research quality (Leonelli, 2018). Even in fields where replication is desirable, the obstacles to replication may be different in kind or scale. As work in science and technology studies (STS) has shown, scientific communities vary widely in their social structures, material practices, and the epistemic strategies they pursue (Knorr Cetina, 1999). Some disciplines invest heavily in increasing their control over the material world, bringing objects of interest into the laboratory where they can be purified, modified, and recombined (Latour, 1983). In other fields, in contrast, scientists constrain the messiness of the world through statistical manipulation rather than investing in 'bench-building' (Peterson, 2015, p. 1215). In this post, we explore how such differences impact the local texture of the challenges faced by those attempting replications, drawing on our interdisciplinary expertise in STS, history of science and technology, metascience, astronomy, and cell biology.

The Reproducibility Projects

The Center for Open Science’s large-scale replication projects demonstrate how access to specialized skills sets and resources constrain what is possible, and how the magnitude of these constraints differs among fields. The Reproducibility Project: Psychology (RP:P), launched in 2011, aimed to replicate studies from prominent psychology journals. Of the 158 studies selected, only 111 studies 'matched' with collaborators willing to replicate them, with the unmatched studies typically requiring specialized samples (such as people with autism or nonhuman primates) or specialized resources (such as functional magnetic resonance imaging or eye-tracking equipment) (Open Science Collaboration, 2015, aac4716-2). Of the studies selected, however, the group’s completion rate was high: 100 out of 111 attempted replications were completed on time and the meta-analysis of these results was published less than four years after the project began.

While the RP:P certainly faced problems with access to skills and materials, its sister project, the Reproducibility Project: Cancer Biology (RP:CB), faced even greater resource constraints. In contrast to the RP:P, which conducted their experiments with a grassroots network of volunteers, the RP:CB initially budgeted $25k USD for replicating 50 studies in commercial fee-for-service labs (funded by a private foundation grant and subsidized by donations from over a dozen biomedical reagent suppliers). Even these resources were not enough, however: project leaders reduced the number of studies to be replicated to 37 in 2015, then to 29 in 2017, and finally to just 18 in 2018 (Kaiser, 2018). Cost was a major barrier (studies cost $60k USD on average), but not the only one. In some cases reagents were no longer available from commercial providers; in other cases negotiating material transfer agreements took so long that the team opted to rebuild the needed materials from scratch (Errington et al., 2021a). In all, it took eight years from announcement of the RP:CB to publication, with each individual project taking an average of nearly four years from design to completion (Errington et al., 2021a; 2021b). Other projects have similarly been able to replicate a greater volume of social science studies. The Dutch NWO replication project, for example, completed 18 social science replications but only three medical replications (Derksen et al., 2024).

Comparing these two cases also reveals domain specific differences in seemingly similar problems. Both psychology and cancer biology face constraints on access to proprietary materials needed to conduct replications (Hays et al., 2017; Murray, 2010), but many of psychology’s core tools are protected by copyright while biology’s are protected by patents. This distinction is important because copyright accrues automatically to researchers, and US American universities have historically treated copyright as belonging to the author (AAUP, 1999). This creates the possibility for actions at the individual level which preserve access to scientific tools, such as using open source/open access licenses. In contrast, researchers must apply for patent protection, and the fees associated with this process limit who can engage with it and incentivize those who do to monetize their scientific tools. By law US American universities hold the rights to inventions made using federal funds (Patent and Trademark Law Amendments Act, 1980), and so would-be replicators must negotiate access to patented tools with institutions rather than individuals, as in the case of the RP:CB.

Replications in the nanosciences

The nanosciences span multiple disciplines, so attempts at replication in the nanosciences require a broad range of skills for working with different scientific objects and instruments. Thus, the first formalized replication project in the nanosciences - in which some of us are involved as replicator, advisor, or ethnographer - includes researchers from physics, biochemistry, and cell biology. The NanoBubbles replication project examines whether and to what degree (percentage-wise) nanoparticles escape intracellular vesicles called endosomes and access the cytosol where they can be used for sensing (and perhaps thus for diagnostics). Learning to produce (i.e., synthesize) nanoparticles and investigate their properties (i.e., characterize) requires skills that can take years to acquire. One advisory board member has “come up with 3 year[s] as estimate to really reproduce one paper with things new to the group” since “for all new materials one has to do all characterization from the beginning to really understand what happens” (Email to Research Participants 3, 2, and 1, 11.04.2024). This scientist thus plans for the replication of one paper to span an entire PhD project.

Acquiring an interdisciplinary skill set requires time but also money and social connections. Similar to RP:CB, costs represented obstacles for the NanoBubbles replication project and prompted the team to produce one organic molecule from scratch. As it was too expensive to buy, the biochemist-postdoc suggested synthesizing it. Since it was outside of the expertise of the team members, they collaborated with an organic chemist, who supervised the postdoc in synthesizing the organic molecule, which lasted four-months and was not even considered part of the replication as it was a preparatory step. For the replications, the biochemist synthesizes and characterizes the nanoparticles, while the biologist is responsible for studying them inside cells. In both parts, an in-depth understanding of different microscopy techniques is crucial. Lacking the time, money or personnel for establishing such internal division of labor and collaborations with external colleagues, which enabled the replication team to cover a broad, interdisciplinary set of materials, instruments and skills associated with them, can impede replications.

Working across different disciplines to conduct replications in the nanosciences also presents spatial challenges (and potential opportunities) (Choi & Shields, 2015; Yaneva, 2022). Rather than working in a “hybrid lab” (Yaneva, 2022, p. 738), the NanoBubbles replication project established a chain of labs, spatially illustrating how the nanosciences range from basic research with materials to applied science in biological environments. This chain of labs also challenges the homogeneity of the epistemic culture, as knowledge production is not based on norms, practices, and materials coming from a single field but of several that are moving across fields. A repercussion that stems from a heterogeneous epistemic culture as such is that scientists from different fields may have a different ways of reasoning, and different standards. For example what may serve as a necessary control for a biologist may seem needless for a chemist. The NanoBubbles team is based in an inorganic chemistry lab, while also working in the aforementioned organic chemistry lab. The replicators also use the chemistry repository Chemotion to re-characterize, store and share the data and samples of the organic molecule with other researchers. Additionally, the nanoparticles inside cells were planned to be investigated in a biology lab linked to a hospital. The replicators furthermore collaborated with two microscopy facilities, one in their own country and one in another country. Thus, the team created a network of labs, microscopy facilities, and repositories which enabled them to perform and verify different steps of their replication. Establishing such a network requires extensive preparation and effort, again hindering replications in this field.

Barriers between fields have been present since the beginnings of the nanosciences, and throughout that time disciplinary barriers have - occasionally - facilitated as well as hindered replications. One of the instruments credited with enabling nanoscience research was the scanning tunneling microscope (STM), invented in the early 1980s. However, the STM’s inventors were (mostly) unaware that they were replicating an earlier, similar instrument called the Topografiner. Other researchers, particularly surface scientists, believed that the Topografiner was a failed experiment and thus had not attempted to replicate it. In fact, surface scientists labored under an incorrect understanding of why the Topografiner failed (Mody, 2011); unencumbered by that disciplinary knowledge, the STM’s inventors (who were not surface scientists) merrily forged ahead. In turn, when the first STM results were announced, surface scientists flocked to replicate it but could not obtain the needed tacit knowledge from its inventors. Getting past that barrier required all the early replicators to meet together, thus seeding formation of an STM community that lowered barriers to future replications.

Indeed, they were perhaps lowered too far. In the early 1990s, multiple groups reported atomic resolution of organic molecules deposited on graphite in air - a result surface scientists tried, unsuccessfully, to convince other fields was improbable. Certain features of graphite and air STM were, however, later found to make ‘replication’ of spurious artifacts laughably easy. Air STM of biomolecules was thus largely abandoned, though no results were retracted - and barriers between fields (especially between experimental physicists on the one hand and materials scientists, chemists, and biologists on the other) have hindered flows of knowledge about common microscopy artifacts and thus fostered similar controversies over improbable ‘replications’ (Cesbron et al., 2012).

Observational astronomy

Astronomy instructively illustrates what replications entail and how equipment shapes the ability to undertake replications. Large telescope projects - both ground-based ones like the Square Kilometre Array (SKA) and space-based ones like the James Webb Space Telescope (JWST) - take several decades to plan, design and build, involving many thousands of individuals with a range of skills. The imperative to coordinate around large publicly-funded infrastructure significantly molds research cultures of observational astronomy. By the time they come on-line, explicit trust in large telescopes is high (the aberrations in the Hubble Space Telescope’s primary mirror being a rare example of astronomers’ doubts being made public). Once operational, the time available on these devices is limited and strictly rationed between elite professional astronomers (McCray, 2000). The volume of data collected on even a single night’s observation is vastly greater than what could be collected in a lifetime a century ago, presenting problems for data retention; for some datasets “it is not possible to keep all the raw data after analysis is complete” (Pepe et al., 2014, p. 1). Astronomy also shows how the boundary between replicability and reproducibility can be blurry. As an observational science, data collected is understood to be in principle unrepeatable, although given the timescale over which most astronomical systems evolve this is only a theoretical concern. Nonetheless, replication of measurements is quite rare. On the other hand, with strong confidence in underlying theoretical mechanisms, observations of other similar target systems are considered as probative as direct replications and are treated equivalently. As a result, for astronomers, replication primarily entails discovering, locating, retrieving and storing pre-existing data (Sands et al., 2013). Most working astronomers no longer control primary data collection and disputes are addressed through the assemblage of resources and producing interpretations. These re-use practices are no more likely to refer to primary data; as one participant in the study by Sands et al. (2013) noted “there are probably ten people on the face of earth that ever re-reduced Sloan images.” Data collected in vast amounts that are almost unmanageable, through a small number of observational facilities, create research cultures of data reuse that shape the possibilities of replication studies.

Towards field comparative research

These cases from psychology and cancer biology, the nanosciences, and astronomy illustrate that limited access to specialized and proprietary research objects and equipment, the skills associated with them, their timescales, and the infrastructures in which they are embedded can constrain replications in those fields. The understanding of what a replication entails can also differ greatly across research cultures, illustrated for example by how astronomers consider observations of other similar target systems as evidential as direct replications. While some barriers might be similar across fields, their magnitude can differ among disciplines, as illustrated by the comparison between the Reproducibility Project: Psychology and the Reproducibility Project: Cancer Biology. This can slow replication projects in certain disciplines and reduce their sample of replicated articles. In nanotechnology, replications are made more difficult by the variety of scientific objects, instruments, and standards coming together in this highly interdisciplinary field. Building a team with skills in materials science, cellular biology, and different microscopy techniques and establishing a network of labs ranging from basic to applied science can be time-consuming and resource-intensive. The history of the STM community, which formed partly because of a meeting aimed at understanding failed replications, furthermore illustrates how barriers between fields have hindered replications of earlier scientific instruments and have sparked controversies over easily replicable microscopy artifacts. The material infrastructure in astronomy complicates further the universal calls for replications since scientific equipment may need decades to be built, access is limited, and the equipment generates a large amount of raw data, which is collected by only a few observational facilities and is hard to preserve. We encourage more elaborate comparative research on field-specific barriers to replications that may help predict how likely some disciplines are to engage in replications and that can also provide insights needed to facilitate replications in these fields.

Acknowledgments

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393. CSFB, MMS and CCMM have individually received funding from the ERC-NanoBubbles project.

Footnotes

[1]: Early works use reproducibility and replication interchangeably; in this post we use the more currently accepted definition of replication as collecting new data by closely following the approach of a previously conducted study.

References

American Association of University Professors. (1999). Statement on Copyright. https://www.aaup.org/reports-publications/aaup-policies-reports/topical-reports/statement-copyright

Cesbron, Y., Shaw, C. P., Birchall, J. P., Free, P., & Lévy, R. (2012). Stripy nanoparticles revisited. Small, 8(24), 3714-3719. https://doi.org/10.1002/smll.201001465

Choi, H., & Shields, B. (2015). A place for materials science: Laboratory buildings and interdisciplinary research at the University of Pennsylvania. Minerva, 53(1), 21-42. https://doi.org/10.1007/s11024-015-9265-6

Clayson, P. E., Carbine, K. A., Baldwin, S. A., & Larson, M. J. (2019). Methodological reporting behavior, sample sizes, and statistical power in studies of event‐related potentials: Barriers to reproducibility and replicability. Psychophysiology, 56(11), e13437. https://doi.org/10.1111/psyp.13437

Derksen, M., Meirmans, S., Brenninkmeijer, J., Pols, J., de Boer, A., van Eyghen, H., Gayet, S., Groenwold, R., Hernaus, D., Huijnen, P., Jonker, N., de Kleijn, R., Kroll, C. F., Krypotos, A.-M., van der Laan, N., Luijken, K., Meijer, E., Pear, R. S. A., Peels, R., … de Winter, J. (2024). Replication studies in the Netherlands: Lessons learned and recommendations for funders, publishers and editors, and universities. Accountability in Research-Policies and Quality Assurance, 32(7), 1285-1303. https://doi.org/10.1080/08989621.2024.2383349

Errington, T. M., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021a). Challenges for assessing replicability in preclinical cancer biology. elife, 10, e67995. https://doi.org/10.7554/eLife.67995

Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021b). Investigating the replicability of preclinical cancer biology. elife, 10, e71601. https://doi.org/10.7554/eLife.71601

Hays, R. D., Weech-Maldonado, R., Teresi, J. A., Wallace, S. P., & Stewart, A. L. (2018). Commentary: Copyright restrictions versus open access to survey instruments. Medical Care, 56(2), 107-110. https://doi.org/10.1097/MLR.0000000000000857

Horbach, S. P., Cole, N. L., Kopeinik, S., Leitner, B., Ross-Hellauer, T., & Tijdink, J. (2025). How to get there from here? Barriers and enablers on the road towards reproducibility in research. https://osf.io/gx9jq/download

Kaiser, J. (2018). Plan to replicate 50 high-impact cancer papers shrinks to just 18. Science, 243(10.1126). https://doi.org/10.1126/science.aau9619

Knorr Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University press.

Latour, B. (1983). Give me a laboratory and I will raise the world. In K. Knorr Cetina & M. Mulkay (Eds.), Science observed: Perspectives on the social study of science (pp. 141-170). Sage.

Leonelli, S. (2018). Rethinking reproducibility as a criterion for research quality. In L. Fiorito, S. Scheall, & C. E. Suprinyak (Eds.), Including a symposium on Mary Morgan: curiosity, imagination, and surprise (pp. 129-146). Emerald Publishing Limited. https://doi.org/10.1108/S0743-41542018000036B009

Lilienfeld, S. O. (2017). Psychology’s replication crisis and the grant culture: Righting the ship. Perspectives on psychological science, 12(4), 660-664. https://doi.org/10.1177/1745691616687745

McCray, W. P. (2000). Large telescopes and the moral economy of recent astronomy. Social Studies of Science, 30(5), 685–711. https://doi.org/10.1177/030631200030005002

Mody, C. C. M. (2011). Instrumental Community: Probe Microscopy and the Path to Nanotechnology. MIT Press.

Murray, F. (2010). The oncomouse that roared: Hybrid exchange strategies as a source of distinction at the boundary of overlapping institutions. American Journal of sociology, 116(2), 341-388. https://doi.org/10.1086/653599

Open Science Collaboration. (2012). An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspectives on Psychological Science, 7(6), 657-660. https://doi.org/10.1177/1745691612462588

Open Science Collaboration (2014). The reproducibility project: A model of large-scale collaboration for empirical research on reproducibility. In: V. Stodden, F. Leisch, & R. D. Peng (Eds.) Implementing Reproducible Research (pp. 299-324). CRC Press, Taylor & Francis Group.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Patent and Trademark Law Amendments Act, 35 U.S.C. §§ 200–212 (2018).

Pepe, A., Goodman, A., Muench, A., Crosas, M., & Erdmann, C. (2014). How do astronomers share data? Reliability and persistence of datasets linked in AAS publications and a qualitative study of data practices among US Astronomers. PLoS ONE 9(8), e104798. https://doi.org/10.1371/journal.pone.0104798

Peterson, D. (2015). All that is solid: Bench-building at the frontiers of two experimental sciences. American Sociological Review, 80(6), 1201-1225. https://doi.org/10.1177/0003122415607230

Romero, F. (2018). Who should do replication labor?. Advances in Methods and Practices in Psychological Science, 1(4), 516-537. https://doi.org/10.1177/2515245918803619

Sands, A., Borgman, C.L., Wynholds, L. & Traweek, S. (2013), Follow the data: How astronomers use and reuse data. Proceedings of the Association for Information Science and Technology, 49(1), 1-3. https://doi.org/10.1002/meet.14504901341

Vachon, B., Curran, J. A., Karunananthan, S., Brehaut, J., Graham, I. D., Moher, D., ... & Grimshaw, J. M. (2021). Changing research culture toward more use of replication research: a narrative review of barriers and strategies. Journal of clinical epidemiology, 129, 21-30. https://doi.org/10.1016/j.jclinepi.2020.09.027

Yaneva, A. (2022). The architecture of the hybrid lab: Spacing graphene research. Social Studies of Science, 52(5), 733–757. https://doi.org/10.1177/03063127221114721

Trial & Error Symposium 2025: Honesty, Openness, and Cultural Change

JOTE — Wed, 05 Nov 2025 23:00:00 GMT

On October 7th, 2025, the University Museum Utrecht hosted the first-ever Trial & Error Symposium - a day dedicated to rethinking how science values failure, uncertainty, and openness. The event brought together researchers, editors, students, and communicators to confront a familiar problem: our cultural reification of positive and novel insights over and above negative, neutral, and replicative studies.

At the Center of Trial & Error, we believe in transforming this culture so that failure, negative and null results, and rejections are finally understood as a natural and vital part of the scientific process, rather than an obstacle to be hidden or overcome. The symposium explored how to make that belief a reality in research culture, through talks, workshops and even art. Topics ranged from publication bias and communicating uncertainty to the ethics of retractions and the value of sharing failure as collective progress.

Cultural change and uncertainty
We began the day with a keynote talk by Dr. Noémie Aubert Bonn, whose talk ‘The Problem of Positive Publication Bias and How to Bring About Cultural Change in Science,’ set the tone for the event. Noémie explored how systemic incentives shape what kinds of results are seen as valuable, and challenged us to consider how to mitigate bias while redefining what success could look like in scientific research.

Following Noémie’s keynote, participants took part in an interactive icebreaker activity organized by our Chair Stefan Gaillard. The participants split into groups and were given sets of scientific findings, some real, some misleading and some entirely false. Through debate and discussion, the participants had to decide what they thought of these findings - some groups focused on statistical abnormalities while others relied on their own expertise to judge the findings. When we reconvened, one theme emerged: we were, by and large, highly skeptical of all the findings.

Of course, there’s a selection effect at play, but nonetheless it was interesting to see how distrustful we were of the findings. This points towards something that we’ve believed at the Center of Trial & Error: increasing the visibility of scientific failures rather than only publishing positive results will help make scientific research more trustworthy. By taking away the incentive to fraudulently present failed research as successful, positive findings become more credible and scientifically meaningful.

This exercise ended up flowing perfectly into the next talk after the coffee break, where Dr. Lucie White gave a talk on ‘Communicating Uncertainty.’ Lucie’s presentation addressed how scientists can engage with the public honestly about the limits of knowledge, without diminishing trust. She argued that uncertainty, when framed correctly, can actually increase credibility by showing that science is self-correcting and evidence-based rather than dogmatic.

Throughout the day we exhibited art made through a process of trial-and-error by students of SintLucas, as well as an exhibition about historical cases of trial-and-error in science, developed together with the Museum of Failure. The digital variant of this exhibition can be found here.

AI and transformational leadership
In the afternoon, Dr. Maximilian Roßmann, presenting research conducted with Auste Valinciute, delved into the often misunderstood world of retractions in academic publishing, looking into the causes, consequences, and potential for reform. Max’s talk emphasized that retractions should not be understood as shameful, but as a form of scientific accountability. Retractions are evidence that the scientific system can correct itself, a shining example of the way in which scientific failure plays an important and positive role in scientific research.

Our final guest speaker of the day, Dr. Veronika Cheplygina led a workshop entitled, ‘Showcasing Your Failures,’ in which participants were guided through CVs of failures and explored all the ways in which failure presents itself as a normal part of our daily experience. The workshop was a reminder that vulnerability can be a source of strength in academia.

The day culminated in the Trial & Error Award Ceremony, presented by Stefan Gaillard. The Trial and Error Award celebrates the courage to share scientific failures publicly, recognizing that such transparency strengthens science as a whole. This year, we opened nominations to anyone who has published scientific failures in any venue. Our team narrowed down the nominations to four groups of contenders that we felt best exemplified the spirit of the award: the authors of ‘AI-based virtual assistant and transformational leadership in social cognitive theory perspective: a study of team innovation in construction industry’, the authors of ‘No support for a causal role of primary motor cortex in construing meaning from language: An rTMS study’, the authors of ‘Proactive and retroactive effects of novelty and rest on memory’, and the authors of ‘Limited data prevent assessment of role of climate change in deadly floods affecting highly vulnerable communities around Lake Kivu’.

We decided to choose the final winner through a lottery because we have not yet set up objective criteria to determine the winner of the award - something we aim to establish for next year. This year’s winners were the authors of ‘AI-based virtual assistant and transformational leadership in social cognitive theory perspective: a study of team innovation in the construction industry’. They investigated how AI-powered virtual assistants and transformational leadership interact to influence team innovation, focusing on the construction industry through the lens of social cognitive theory (SCT). Their initial hypothesis predicted a positive correlation between the use of AI tools, leadership style, and innovation outcomes. However, pilot tests revealed several methodological challenges, failed attempts at causal verification, and noticeable participant resistance to AI adoption. As a result, the team revised their theoretical framework multiple times before finally settling on a more complex interpretation grounded in SCT.

Normalizing failure
We are deeply grateful to everyone who made this day possible, especially our speakers, volunteers, collaborators, audience, and funders. The symposium was a shared achievement. Meaningful cultural change requires the collaboration of countless stakeholders such as scientists, funders, institutions, journals, and the public. We are proud that the Center of Trial & Error is emerging as one of the loci where that transformation begins.

When we began as a small group of students determined to talk about the role of negative and null results in science, we could not have imagined standing here today hosting a full symposium on our core ideals. Over the years, the Center of Trial & Error has grown into a platform for open conversation, scholarly experimentation, and cultural critique. Our mission remains clear: to normalize failure, negative and null results, and to question incentives that distort knowledge production.

A special thank you to our collaborators, the Museum of Failure, Utrecht Young Academy, Young Academy Twente, and The New Utrecht School, to our funders: The Descartes Centre, the GKG Impact Grant, the NWO OS Fund, Utrecht Young Academy, to The New Utrecht School, and to the students of SintLucas for their art works.

We look forward to the next chapter of the Center of Trial & Error, and to a scientific culture that is braver, more honest, and more just.

Please see below for the PowerPoint slides of the talks, the workshop, the award ceremony and the icebreaker activity.

‘The Problem of Positive Publication Bias and How to Bring About Cultural Change in Science’, by Dr. Noémie Aubert Bonn

The Icebreaker Activity, by Stefan Gaillard

Understanding Uncertainty, by Dr. Lucie White

Framing Retractions in Scientific Media, by Dr. Maximilian Roßmann

‘Showcasing Your Failures’, by Dr. Veronika Cheplygina

Presentation of Trial and Error Award Nominations, by Stefan Gaillard

Cultures of Trial and Error: Self, Correction, and Science in Conflict-Laden Science

Sergio Sismondo — Sun, 05 Oct 2025 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

Once upon a time, when it was said that science self-corrects, the image was of a relatively unified culture, a body of practices guided by shared norms of disinterested inquiry. Thus, for example, in the middle of the twentieth century the philosopher Herbert Feigl wrote, “The self-corrective aspect of science has rightly been stressed as its most important characteristic …” (1949, p. 142). The posts in this series, and the wide-ranging articles in The Journal of Trial and Error, challenge both that image of unity and the efficacy of one-size-fits-all interventions to promote self-correction.

The medical science I study is often the product of diverse, overlapping, and sometimes competing institutions, actors, and interests, especially those closely entangled with the pharmaceutical industry.

Pharmaceutical companies don’t merely provide funding that creates conflicts of interest among scientists. Rather, they actively graft their research and publication efforts onto the body of medical science, resulting in an integrated whole. As I and many others have shown, clinical trials are designed, run, and publicized under company control (e.g. Bernisson & Sismondo, 2024; Cosgrove et al., 2023; Fugh-Berman, 2010; Gagnon & Dong, 2023; Ross et al., 2008; Sismondo, 2018). Academic medical researchers and other apparently independent physicians serve as authors on ghostwritten publications, lending legitimacy and a gloss of independence to corporate research. This isn’t science gone astray due to isolated moral failings; it’s science shaped by commercial objectives, objectives that even determine what counts as exemplary research and trustworthy knowledge. Think less ‘rogue scientists break norms’ and more ‘teams of professionals in blazers execute publication strategies’. Then pharmaceutical companies hire (or anoint) ‘key opinion leaders’, who will go on speaking tours to repeat and amplify the messages of corporate research. Their messages go on to circulate in continuing medical education courses, advertisements, etc. The resulting medical science has become permeated with commercial interests, to the point that those interests are largely taken for granted.

If we hope to see self-correction in this arena, what is the scientific ‘self’ that will do the correcting? Can we find it in some original body of medical science – some idealized core that might challenge the grafted parts? Yes, many people are trying to act as part of that idealized core, but they are mostly gleaners of data, marginal to the industrial harvesting of research results for broad medical interventions. They include medical science sleuths who file freedom of information requests to access protocols and clinical study reports submitted for regulatory approval. Sometimes, they dig up fascinating data and subject it to revealing scrutiny and analysis. But they rarely alter the deeply furrowed lanes that pharmaceutical companies have carved.

In my email feed, as I write this, is a post to the blog Mad in America. Pharmaceutical industry critic Peter Gøtzsche (2025) reports on his failed request that the UK’s Medicines & Healthcare products Regulatory Agency (MHRA) change the package inserts for antidepressants. Gøtzsche insists that the MHRA should remove all claims that depression is caused by chemical imbalances in the brain. As has been well known for decades, there has never been any evidence for the imbalance thesis, or for the more specific claim that depression is caused by a shortage of serotonin. Yet the imbalance claim continues to be recited by physicians, embedding patients’ depressions in possibly long-term brain chemistry – the recitations include some by physicians who also say that the chemical imbalance claim is just an ‘urban legend’ (Ang et al., 2022). The MHRA is not willing to participate in any correction of this widely circulating lore. Nor is the industry interested in correction, given that the imbalance thesis grounds immensely profitable drugs – though occasionally companies challenge the lore to replace old drugs with new ones, as when Eli Lilly successfully converted physicians’ diagnoses of depression with ones of bipolar disorder, converting prescriptions of its off-patent Prozac with its in-patent Zyprexa (Applbaum, 2009).

These issues become even more concerning in the light of a new study by Lenzer and Brownlee (2025), also in my email feed as I write. Through painstaking work, these researchers showed that 73% of drugs approved by the US Food and Drug Administration (FDA) had not been supported by studies that met the FDA’s basic criteria for “substantial evidence” of effectiveness. Cancer drugs were especially likely to have been approved without evidence. And a number of the approved drugs have been shown to be dangerous. I very much doubt that anybody expects to see many, or perhaps even any, of these drugs withdrawn in the wake of Lenzer and Brownlee’s study. Instead, some of them are widely prescribed, with the FDA’s approval taken as reflecting evidence of safety and efficacy. The ‘self’ of self-correcting science here includes institutions and practices deeply structured by the interests of powerful actors, and so often resistant to correction – though see below on the “Make America Healthy Again” revolution at the FDA!

This brings us to ‘correction’. In the storybook version, correction happens when flawed findings are identified and replaced through internal processes of critique, replication, or the production of new evidence. But what happens when the conditions of possibility for correction are shaped by the same forces that produced the flawed knowledge? If pharmaceutical companies ghost-manage research (Sismondo, 2018), they can also ghost-manage the avoidance of corrections. Negative trials may go unpublished; unfavorable data can be spun or buried; publication strategies may flood the literature with positive results, shaping what later reviews and meta-analyses can find. Correction may still occur, but it is slower, harder, and often partial, because the conditions for it are compromised from the outset.

Let me switch to a very different example of conflicted science, where the dominant forces are overtly political and even geopolitical: the origins of the virus SARS-CoV-2 and the disease Covid-19. Early in the pandemic, the ‘natural origins’ hypothesis (in terms of ‘zoonotic spillover’) was dominant. For example, a substantial epidemiological analysis arguing that a seafood market in Wuhan was the epicentre of early Covid cases (Worobey et al., 2022) received wide coverage. Alternative hypotheses, including that the virus originated in a lab, were treated as fringe or conspiratorial (Lewandowsky et al., 2023). In 2020, the ‘lab leak’ idea’s (evidence-free) promotion by President Donald Trump, alongside racist labels for the virus, only helped to discredit the hypothesis (Singh et al., 2020).

Only gradually, through trickles of evidence and shifting political contexts, did the lab leak hypothesis gain respectability and attract serious scientific and journalistic attention. Today, the WHO’s committee on the virus’s origins recognizes that the evidence is inconclusive, though it still considers zoonotic spillover the best-supported hypothesis (SAGO, 2025). Nonetheless, numerous scientific papers and reports have argued that the lab leak hypothesis needs to be taken seriously. US agencies, from the FBI (under a Democratic administration) to the CIA (in a report prepared for a Democratic and then released under a Republican administration), have announced that the lab leak hypothesis is the more likely of the two, though they acknowledge that the evidence is weak (CBS News, 2023; Honderich, 2025), especially given China’s refusal to allow a full external investigation.

This shift might be seen as an instance of self-correction: new evidence and arguments challenging an initial consensus. But here too, the boundaries of the ‘self’ and ‘science,’ and the dynamics of ‘correction’, are complex.

Covid has contributed to sweeping political and cultural polarizations (e.g. Bharti & Sismondo, 2024), and the main competing views on the virus’s origins have fallen on one side or the other of a key dividing line (e.g. Horton, 2025). The zoonotic spillover hypothesis has become, roughly, the establishment stance, endorsed by mainstream institutions, major public health agencies, and much of the scientific publishing ecosystem. Here the ‘establishment’ is aligned with recognized expertise, professional authority, and institutional legitimacy.

The lab leak hypothesis has become, also roughly, a populist stance, reflecting skepticism toward elites and elite institutions. Here ‘populism’ is a set of attitudes, identities, and movements propelled by rhetorics of anti-elitism, suspicion of authority, and appeal to ‘the people’ and their common sense.

The establishment/populist divide is not about power. For example, the current Trump administration is staying close to its populist roots, even though it controls almost all of the important levers of power in the US. Especially relevant here, Robert F. Kennedy Jr., the current US Secretary of Health and Human Services, has been using his position to push forward his anti-establishment “Make America Healthy Again” campaign. This has involved weakening the government’s support for vaccines, challenging entrenched relations between large corporations and the FDA – which, in the view of this author, may lead to some needed and healthy corrections – and overturning the leadership of the Centers for Disease Control (CDC). On the last of these, Kennedy recently accused the CDC of lying to Americans about masks, vaccines, and social distancing: “I need to fire some of those people and make sure this doesn’t happen again” (Thomson Reuters, 2025).

In the contest over the origins of SARS-CoV-2, what counted as admissible evidence, whose voices were heard, and what kinds of inquiry were funded and published. All were shaped by broader institutional, political, and geopolitical contexts. The boundaries of science have never been clean, of course, but on this issue the FBI and CIA (!) became relevant voices in a supposedly scientific debate, given their roles in interpreting Chinese actions and statements. One could be forgiven for wondering whether peer review needed a counterintelligence wing. Geopolitical tensions, national security concerns, institutional reputations, and media narratives all helped determine which hypotheses gained traction in different moments, audiences, and contexts. Correction, if that’s the right word, is not an internal matter of better evidence displacing worse. It is the result of shifting configurations of authority, interest, and possibility.

The principle of symmetry in STS urges us to explain beliefs we take to be true and those we take to be false in the same terms. Successes and failures, knowledge and error, are to be analyzed using the same kinds of causes. Most readers of this journal and this series of blog posts, this author included, are part of the establishment. Nonetheless, as analysts we should resist the temptation to treat errors as the intrusion of external forces and corrections as the return of autonomous scientific reason. Both are products of broadly the same kinds of conditions. The shifts in SARS-CoV-2 origin theories reflect changing configurations of knowledge, power, and interest.

Where does this leave the idea of self-correction in conflict-laden science? It’s not that science cannot or does not correct itself. But in cases like those above, the metaphor conceals as much as it reveals. The ‘self’ is heterogeneous, structured by relations of power, interest, and legitimacy that exceed any simple internal process. The ‘correction’ is not a clean triumph of better over worse, but the outcome of struggles over what counts as knowledge, what counts as error, and who gets to decide.

And so the storybook version, with brave truth-seekers armed with data and rationality and slaying the dragons of error, turns out to be a fable, albeit one with staying power. In its place, we find a messier but more instructive tale, in which science sometimes corrects itself, sometimes evades correction, and always does so amid tangled alliances and competing narrators. Recognizing this complexity can help us develop more robust practices of critique and reform – ones that attend not just to errors in reasoning or evidence, but to the conditions that make some errors easy to perpetuate and some corrections hard to achieve. If there is to be a moral to this story, it is that self-correction may be real, but it is not simple.

References

Ang, B., Horowitz, M., and Moncrieff, J. (2022). Is the chemical imbalance an ‘urban legend’? An exploration of the status of the serotonin theory of depression in the scientific literature. SSM – Mental Health 2 (December): 100133.

Applbaum K. (2009) Getting to yes: Corporate power and the creation of a psychopharmaceutical blockbuster. Culture, Medicine and Psychiatry. (2009) 33:185–215. doi: 10.1007/s11013-009-9129-3

Bernisson, M., & Sismondo, S. (2024). Promoting opioids, a story about how to influence medical science and opinions. Frontiers in Medicine, 11, 1327939. doi: 10.3389/fmed.2024.1327939

Bharti, N., & Sismondo, S. (2024). Political prescriptions: three pandemic stories. Science, Technology, & Human Values, 49(2), 371-402. https://doi.org/10.1177/01622439221123831

CBS News (2023, 1 March). FBI chief says agency feels COVID pandemic likely started with Chinese lab leak. CBS News (1 March). https://www.cbsnews.com/news/chris-wray-covid-origin-likely-wuhan-labl-leak/#:~:text=FBI%20chief%20says%20agency%20feels,World

Cosgrove, L., Patterson, E. H., & Bursztajn, H. J. (2023). Industry influence on mental health research: Depression as a case example. Frontiers in Medicine, 10. doi: 10.3389/fmed.2023.1320304

Fugh-Berman A, J. (2010). The haunting of medical journals: how ghostwriting sold ‘HRT’. PLoS Med 7:e1000335. doi: 10.1371/journal.pmed.1000335

Gagnon, M. A., & Dong, M. (2023). What did the scientific literature learn from internal company documents in the pharmaceutical industry? A scoping review. Cochrane Evidence Synthesis and Methods (2023) 1:e12011. doi: 10.1002/cesm.12011

Gøtzsche, P. C. (2025). Protecting the false narrative about antidepressants. Mad in America (7 July). https://www.madinamerica.com/2025/07/protecting-the-false-narrative-about-antidepressants/

Honderich, H. (2025, 25 Jan.). CIA says lab leak most likely source of Covid outbreak. BBC (25 January). https://www.bbc.com/news/articles/cd9qjjj4zy5o

Horton, L. (2025). ‘Zoonati’ vs. ‘epistemic trespassers’: Scientific identity in contentious online advocacy campaigns on the origins of SARS-CoV-2. Social Studies of Science 55(3),

Lenzer, J., & Brownlee S. (2025). FDA approved – and ineffective. The Lever (June 5). https://www.levernews.com/fda-approved-and-ineffective/

Lewandowsky, S., Jacobs, P. H., & Neil, S. (2023) Leak or leap? Evidence and cognition surrounding the origins of the SARS CoV-2 virus. In Butter, M., & Knight, P. (Eds.) Covid conspiracy theories in Global perspective. Taylor & Francis, ch. 2. https://doi.org/10.4324/9781003330769

Ross, J. S., Hill, K. P., Egilman, D. S., & Krumholz, H. M. (2008). Guest authorship and ghostwriting in publications related to rofecoxib: a case study of industry documents from rofecoxib litigation. JAMA, 299(15), 1800-1812. doi: 10.1001/jama.299.15.1800

SAGO [Scientific Advisory Group for the Origins of Novel Pathogens]. (2025). Independent assessment of the origins of SARS-CoV-2. https://www.who.int/groups/scientific-advisory-group-on-the-origins-of-novel-pathogens-(sago)

Singh, M., Davison, H., & Borger, J. (2020). Trump claims to have evidence coronavirus started in Chinese lab but offers no details. The Guardian (1 May). https://www.theguardian.com/us-news/2020/apr/30/donald-trump-coronavirus-chinese-lab-claim

Sismondo, S. (2018). Ghost-managed medicine: Big pharma's invisible hands. Manchester: Mattering Press (2018). http://doi.org/10.28938/9780995527775

Thomson Reuters. (2025, 4 Sept.). RFK grilled by both Republicans and Democrats during raucous hearing. https://www.cbc.ca/news/world/rfk-vaccines-raucous-senate-hearing-1.7625329

Cultures of Trial and Error: Engines of Science Correction in the Biosciences - And How They Sometimes Run Out of Steam

Melpomeni (Melina) Antonakaki — Mon, 01 Sep 2025 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

Scientific correction is often imagined as a clear-cut, procedural, and largely self-contained response to error: identify the problem, notify the editor, issue a retraction, update the record, move on. For small, innocent mistakes, contradictory findings, or methodological flaws, this process might as well work as intended. But when correction involves facing allegations laden with social, economic and emotional weight, contested responsibility, institutional silence, or reputational risks, the situation becomes far murkier.

Recent developments - what this blog post series has aptly framed as cultures of trial and error - have challenged longstanding assumptions about how scientific errors are recognized and addressed. From the so-called replication crisis in psychology and the biosciences, to the rise of watchdog initiatives such as Retraction Watch, the sufficiency of traditional quality controls has come under sustained scrutiny (Antonakaki et al., 2025). In the currently emerging meta-research framing, retractions, corrigenda, and institutional investigations are just the visible parts of a largely hidden ecosystem of responses to perceived errors and misconduct. These responses are not merely technical or ethical; they are deeply social and material: they need both humans and non-humans (lab material and records, research data & infrastructures etc.) to come together in the controlled space of a trial specifically designed to produce corrective insight.

This blog post examines one site where that heterogeneous ecosystem was tested and reshaped: the Molecular Biology Society of Japan (hereafter, MBSJ or Society), one of the country’s largest representative bodies of bioscientists. Between 2006 and 2014, the Society was repeatedly confronted with cases of suspected misconduct—each involving distinct pressures, publics, and possibilities for action [1]. I ask: What drove the MBSJ to take on corrective work in their field of research? What kind of service did they see themselves as offering their community and others beyond?

More importantly, each case the MBSJ engaged was blurring the line between error and outright misconduct, making them especially valuable case-studies for Science and Technology Studies (STS) research, which analytically engage with, even invest in, the murky (For a recent review of the STS controversy studies tradition, see Elam, 2024). During this period, the MBSJ experimented with a model of scientific self-regulation that sometimes aligned with institutional norms, and at other times ran counter to them. They weren’t just navigating murky waters—they were actively trying to chart them, drawing lessons from experience and testing ways to secure they would always find their way, or a good way, through them. And, as I will show, they paid a hefty price for it.

In an effort to understand the conditions that make charting a path for scientific correction practically im/possible, while doing justice to the unresolved complexities of the empirical story, I introduce the idea of engines of science correction. This conceptual framework is inspired by Ian Hacking’s engines of discovery (2007), a framework he developed to describe the historically situated mechanisms that drive knowledge production in (what he calls) the sciences of the Human—a category broad enough, in his usage and mine, to include the biosciences, psychology, psychiatry, economics, political science, anthropology, and more. In Hacking, I found not just a language for naming and ordering mechanisms along a sequence, but a generative way to surface the tacit labor, interpretive judgment, and institutional friction involved in how scientific communities attempt to correct themselves.

But naming and ordering, here, are not only critical moves; they are also reparative. As Eve Sedgwick has argued, reparative reading resists the urge to preemptively expose or condemn (2003). Instead, it seeks to assemble meaning and possibility under conditions of partial knowledge, institutional constraint, or even despair, as “the reader has room to realize that the future may be different from the present, [as] it is also possible for her to entertain such profoundly painful, profoundly relieving, ethically crucial possibilities as that the past, in turn, could have happened differently from the way it actually did.” (Sedgwick, 2003, p. 146). It is in this reparative spirit that I approach the MBSJ's story - as an attempt to engage bioscientists not just as subjects of critique, but as colleagues facing somewhat familiar struggles in the science systems we inhabit. Moreover, the concept allows me, in the concluding remarks, to shift into a more autoethnographic register - one shaped by my own participation in projects of critique, repair and service to the STS community.

The MBSJ as a culture of trial and error: initiatives, mistakes, consequences

To explore how engines of science correction operate in practice, I narrate four key episodes in the recent history of the MBSJ. I focus on the period between 2006 and 2014, when the MBSJ faced a series of high-profile cases involving allegations of research misconduct among its members. Each case triggered different responses—ranging from independent investigation to frustrated, chronic anticipation for relevant disclosures, to persistent public interventions and national media engagement - and together, they offer a rare window into how corrective activity emerges, evolves, or falters under pressure. These are not stories of linear reform. Rather, they reveal how access to material and testimony, the authority to assess those implicated, and the legitimacy to draw and communicate a composite perspective, are continually negotiated as corrective moves and how they set precedents, both good and bad, that guide communities from one episode to the next.

From tragedy to agency: the possibility of independent investigation

The episode now known within the MBSJ as the Sugino problem (Mondai, in Japanese) began in late summer 2006, when it became public knowledge that Professor Sugino of Osaka University, who had previously chaired the MBSJ Annual Meeting, was implicated in allegations of data fabrication. The paper at the center of the controversy had been published just a month earlier in the Journal of Biological Chemistry and was co-authored by members of Sugino’s lab at the Graduate School of Frontier Biosciences. It was the co-authors themselves who alerted the university, alleging that certain experimental data had been included in the paper without their consent. Although technical in nature, these accusations quickly escalated due to their broader implications for authorship, hierarchy, and lab management. (For a brief overview in English, see the translated excerpts of the final report by the Scientific Ethics and Research Conduct Committee at the Graduate School of Frontier Biosciences, Osaka University, 2007; and also, for a raw reaction to the case, see Yanagida, 2006b).

The situation deepened into tragedy soon after, when one of Sugino’s assistant professors committed suicide inside the space of the laboratory (Fuyuno & Cyranoski, 2006). This loss cast a long shadow over the local research community, with conversation points raised about how his death exposed the extreme pressure placed on young researchers and the isolation that often surrounds allegations of misconduct (see, for example, this Request by the Graduate Students Community of Kyoto University at an online forum, Onegai: YK Senpai No Fuhō Yori - Kyōtodaigaku Daigakuin, 2006; or, this moving blog post in English by Professor Yanagida, 2006a). Osaka University first attempted to separate its inquiry from media speculation about the conditions of death (Chair of the investigation was quoted at Asahi Shinbun, 2006), and later guaranteed future protections for whistleblowers and complainants (Osaka School of Frontier Biomedicine, 2006).

Not fully convinced by the university’s internal report - particularly its reluctance to grant Sugino the right to appeal - the MBSJ established its own Working Group (WG) in early 2007 to conduct an independent investigation (see Sec1:3–Sec1:4 at the Bulletin No.86 of the Molecular Biology Society Japan, 2007a, pp. 3–4). This move challenged the prevailing assumption that universities alone should oversee error correction. It would later be reported that the WG took just over a year to gain access to materials and testimonies, and to reach a position from which it could present its findings with both authority and legitimacy (Tsurimoto et al., 2009). By publishing its full report in Genes to Cells (the MBSJ Journal), alongside a formal rebuttal from Sugino, the Society reimagined scientific correction as a practice rooted in transparency, dialogue, and fairness, such that extends to cover those who have erred (Sugino, 2009; Yanagida, 2009).

Building reform: The emergence of the Wakate (Young Researcher) Symposia

In the aftermath of Sugino’s dishonorable discharge from Osaka University, the MBSJ sought not only to reckon with what had gone wrong, but to shift the cultural conditions that might allow such problems to repeat. One of the key vehicles for this effort became the Wakate Symposia - a new initiative aimed at cultivating ethical reflection and open dialogue among young researchers at the MBSJ. Launched in 2007 under the stewardship of Professor Kato of Tokyo University, the symposia functioned as both pedagogical experiment and cultural intervention (see the Wakate WG's report at the MBSJ News outlet, “Kenkyū Rinri Iinkai Wakate Kyōiku Wākingugurūpu Kara No Go Hōkoku,” 2007, pp. 9–10). The slogan for the first two iterations of the Wakate was Now Is the Time to Show the Conscience of Scientists - A Collective Reflection on Scientific Misconduct (Ima koso shimesou kagakusha no ryoushin - Minna de kangaeru kagakuteki fusei mondai) (The MBSJ has released full transcripts of the discussions, see Molecular Biology Society Japan, 2007b, 2008).

At the inaugural session, Professor Kato offered a metaphor that would come to shape the spirit of the Wakate series: each lab, he suggested, is like a takotsubo, a clay pot traditionally used as an octopus trap (Molecular Biology Society Japan, 2007b, pp. 27–28). A takotsubo lures the animal in with a false sense of safety and comfort. The implication was subtle but powerful: while the enclosed, familiar environment of a lab can foster productivity, it can also conceal problems - ethical, procedural, or interpersonal. The Wakate Symposia encouraged young researchers to reflect critically on these closed dynamics and to look beyond their immediate supervisors for support and guidance.

Limits of independence in doing corrective work: The Kato Problem

The MBSJ’s effort to institutionalize scientific correction through the Wakate Symposia was soon tested by a deeply uncomfortable turn of events. In January 2012, Professor Kato - who had not only helped frame the Sugino case but had also become the public face of the Wakate initiative - was himself accused of serious misconduct (katolab-imagefraud, a blog by JuuichiJigen, 2012; Normile, 2012). The allegations spanned multiple publications and several years, prompting the University of Tokyo to launch an internal investigation. That investigation, which lasted nearly three years, ultimately painted a bleak picture of systemic abuses of power, questionable research practices, and image or data reuse across numerous publications from the Kato Laboratory.

In contrast to the Sugino problem, the circumstances surrounding Kato unfolded in a markedly different way. While the earlier case had drawn visibility through tragedy and mobilized sympathy within the community, the Kato case was met with silence and opacity. The University of Tokyo conducted its investigation largely behind closed doors, and information remained scarce (The first public acknowledgement came almost two years after the launch of investigation, see President's statement, the University of Tokyo, 2013). The only partial exception to this silence came from watchdog platforms like Retraction Watch, which documented retractions and updates in real time (14 posts between April 2012 and December 2014. I cite the earliest as an example, Retraction Watch, 2012). Still, for the MBSJ, the lack of access to materials and personnel - a key factor that had enabled their independent inquiry into the Sugino problem - made any parallel investigation effectively impossible. As a result, the Society was left on the sidelines, unable to assert the community-based corrective role it had begun to develop in earlier years. Minutes from the December 2012 Board of Directors meeting reveal intense internal debate: members lamented the slow response, questioned whether the Society should issue a public statement, and expressed concern that inaction might erode members’ and public trust. Some acknowledged the limits of the Society’s authority, recognizing that the University controlled all information and insisting that the MBSJ could not proceed without verified evidence (Molecular Biology Society Japan, 2012, online, agenda item 5).

As the Society’s leadership saw no other option but to push for formal updates from Tokyo University (President's statement, the Molecular Biology Society Japan, personal communication, November 8, 2012; President's statement, the Molecular Biology Society Japan, personal communication, August 7, 2013), growing frustration within the MBSJ membership became apparent. Doubts surfaced over the leadership’s apparent inaction and questions were raised about the viability of continuing the Wakate symposia, given that their public face was now embroiled in scandal (Science in Japan, a discussion site for Japanese science [Nihon no kagaku o kangaeru], 2013a, 2013b). The resulting inner debate is a fascinating tale, which I do not have the adequate space to develop as part of this blog post, but I can, at least, direct the curious reader to the full transcript of one MBSJ forum session in 2013 that was held in English (Therein, president Osumi Noriko summarizes the Kato problem for the MBSJ, Molecular Biology Society Japan, 2013b, pp. 5–9).

Past and Parallel Experiences Shaped the MBSJ’s Response to the now infamous STAP Cell case

By early 2014, the MBSJ was again confronted with a major case of alleged scientific misconduct. Allegations had emerged against two publications on the so-called STAP (stimulus-triggered acquisition of pluripotency) cells, a controversial and ultimately discredited stem cell culture technique (For detailed analysis, see Antonakaki, 2019; Meskus et al., 2018). The Society now faced the complex task of selecting its mode of response to a scandal under intense national and international scrutiny, while still managing the unresolved fallout from the Kato problem within its own membership. This placed the Society in a moment of acute institutional tension, where its authority and legitimacy as a corrective agent came under unprecedented pressure (The President's Report for 2013 nicely summarizes the tensions and priorities, in English, see Molecular Biology Society Japan, 2013a).

As RIKEN—the prominent national research organization that oversaw the laboratories where the STAP cell research was conducted - began its investigation at the end of February 2014, the MBSJ took a notably proactive public stance. This time, rather than waiting for updates from the sidelines, the Society issued a series of public presidential statements (Molecular Biology Society Japan, 2014a, 2014b, 2014c). Each urged a thorough and transparent investigation into the STAP cell papers, with particular emphasis on creating the conditions necessary for independent verification of investigatory findings. The implication was that the MBSJ would take on the challenge of independent verification, as they had done in the past, provided that they were given the material opportunity: The initial presidential request to RIKEN (March 3, 2014) can be interpreted as a tactical effort, primarily aimed at indirectly criticizing and pressuring Tokyo University for its opacity and lack of responsiveness. The subsequent requests (March 11 and July 4, 2014) focused on calls for genetic analysis of the STAP cell samples (calls that were emerging inside and outside RIKEN, but were not reflected in the scope and priorities of the first investigation) - reflecting the MBSJ’s belief that, as practicing bioscientists, they could meaningfully evaluate the integrity of an investigative report, provided the raw data and analytical methods were transparently disclosed. This stance is consistent with how the Society had constructed their independent inquiries in the Sugino case, and how they were experiencing the limits of what they could offer in the Kato one.

The flurry of public-facing activity revealed both how much had changed since 2006 - and how much had not: The experience of the Sugino case had taught the MBSJ the value of independent, self-directed inquiry. Yet the limits of that independence had already been made painfully clear during the Kato case, and those constraints remained - especially in the Society’s ability to navigate high-stakes controversy without exposing itself to harmful levels of scrutiny, speculation, or backlash. This tension came into sharp focus following the death of Sasai Yoshiki in August 2014. Sasai, one of the co-authors of the STAP cell papers, died by suicide at his workplace - another tragic loss in the history of science correction in Japan and one that sent shockwaves through the community. Again, as in the 2006 assistant professor’s passing, grief and shock quickly gave way to a wave of online speculation. This time, however, the MBSJ came under fire for having possibly amplified Sasai’s despair with their public critique. Just days before his death, members of its research ethics WG and the Wakate initiative had appeared as expert commentators on a nationally televised investigative program by NHK, Japan’s public broadcaster, offering technical rebuttals to the STAP cell claims.

In response, President Osumi issued a public letter to the MBSJ community, firmly rejecting claims that the Society had singled out Sasai or applied undue pressure (Molecular Biology Society of Japan, 2014, online, §3). She emphasized their requests had been procedural and systemic in nature, not personal. Still, the damage had been done: the Society’s earlier emphasis on transparency and accountability had become uncomfortably entangled with tragedy. When President Osumi later convened the forum Maintaining Fairness in Life Science Research at the 2014 Annual Meeting, the tone was noticeably subdued. In her remarks, she invoked Heinrich’s Law - the principle that visible failures are often preceded by numerous hidden risks - not to assign blame, but to highlight the need for collective vigilance in addressing deeply rooted problems (Molecular Biology Society Japan, 2014d, p. 2). Another senior member of the community pleaded in response that,

"Although the (Kato) problem caused a setback and left the Society with a sort of trauma, I believe that the Society - as represented by the Board of Directors - should properly come to terms with it. After that, it would be better to have a bottom-up discussion about what kind of programs to create and what the community can do going forward." (Molecular Biology Society Japan, 2014d, p. 12)

Yet by that point, the Society’s appetite for such public visibility and intervention had begun to wane. Interventions vis-á-vis the STAP cell case effectively stopped. Moreover, that forum marked a quiet farewell to the Wakate era altogether. The established mode of MBSJ critique and redress had run out of steam (Cf. Latour, 2004), something that reflects the exhaustion of the previous script for corrective action, without the introduction of a novel one. It begs the question: was it all for nothing?

MBSJ’s Engines of Correction in the Wakate Era

Between 2006 and 2014, at least four distinct engines of science correction operated within the MBSJ. Together, these form the space of the corrective trial - a domain where evidence, interpretation, and legitimacy are actively negotiated amid social, institutional, and communal pressures. Drawing inspiration from Hacking’s practice of naming his engines with imperative verbs and exclamation marks, I offer the following short elaborations in the same spirit:

Access material sources and personnel at stake!

Participants in a culture of trial and error require access to the raw materials of laboratory life - data, notes, protocols, and people. This access need not be complete; partial proxies can suffice to construct a meaningful and nuanced understanding. In the Sugino case, the Working Group leveraged their familiarity with the Osaka/Kyoto research community to broker access to both material evidence and personal testimony. They approached the university’s published report as if they were its critical replicators and peer reviewers, systematically comparing its findings with insights they had developed independently.

By contrast, in the Kato case, the University of Tokyo’s informational opacity effectively foreclosed any parallel or independent investigation. Despite high public interest and institutional commitment, the MBSJ’s corrective efforts could not gain traction between 2012-2013. Confronted with this impasse, the MBSJ leadership redirected its energies in 2014 toward RIKEN, hoping that a public, good-faith critique and collaborative engagement in the STAP cell case might serve as a constructive example - one that could encourage the University of Tokyo to adopt a more transparent and cooperative stance as its own investigation reached its final stages. However, the tragic events that unfolded during this period - most notably the suicide of Sasai Yoshiki - and the subsequent handling of blame and responsibility disrupted these efforts. In the aftermath, public interventions aimed at securing access and independently verifying the investigative reports in either case were effectively abandoned midstream.

But, returning to the Sugino case where ‘access’ was achievable - via public disclosures, open data, leaks, or institutional cooperation - it activated a second engine:

Assess the measurement at stake!

Correction depends not only on access to materials or testimony but on whether established standards were embedded, assumed, or violated in publications. In the Sugino inquiry, testimony from twenty employees and former students, coupled with lab workflow mapping, exposed mismatches between published findings and (missing) primary data. When this engine fires, it invites the activation of a third:

Assess the measurer at stake!

Who conducted the measurement, under what authority, and within what environment? The Working Group engaged directly with Sugino, tracing his lab’s evolution from the mid-1990s into the 2000s. Their report painted a picture of a laboratory increasingly populated by early-career researchers and a steady influx of students, with an often-absent leader due to travel or multiple affiliations. Concerns about managerial style and authorship responsibility influenced both blame distribution and reform imaginations. With engines two and three in motion, a fourth one becomes available:

Assess the propriety of actions versus the contents of allegations!

Correction involves judging the seriousness of deviations from norms - were they violations, and was harm done? This engine combines scientific and moral reasoning and underpins decisions about public action. Osaka University might have ultimately dismissed Sugino dishonorably, but the MBSJ saw this as reflecting the gravity of the assistant’s suicide more than a fair assessment of the publication error itself. The Society’s decision to allow Sugino a public response can be seen as partly an attempt to remediate harm in the community, and partly an effort to close the episode with some dignity for the living.

Conclusion

Although the period of the Wakate Symposia has largely faded from institutional memory - and I want to extend a degree of sympathy toward that purposeful erasure - I must admit, as an early-career researcher and a feminist in STS, that the story of the MBSJ’s corrective work has stayed with me. I have tried to learn from it, to “think with it” (as STSers like to say), and to share it outside Japan and beyond the biosciences. After all, placing trust in the wrong person (as the MBSJ did when appointing Kato as a champion of young researchers, only for subsequent revelations to show that he had abused his position and power) is hardly unique. Is it too far-fetched to suggest that this could happen to the best of us - in all domains of life, science included?

In recent years, and while still a doctoral candidate, I have taken on roles such as serving the stsing e.V. community as ombudsperson, and the EASST community as co-chair of its Ethics Committee. These roles emerged at a time when STS itself was being asked - by insiders and interlocutors alike - to turn its critical gaze inward (For a short review see, Antonakaki, 2023). I do not miss the irony - or the precarity - of being appointed myself to positions of ethical stewardship as an early-career scholar in a field newly attuned to its own hierarchies and exclusions. But perhaps that, too, is part of corrective work: learning to hold responsibility while still navigating one’s own vulnerability.

That said, what continues to resonate with me in the MBSJ story is the moral weight of it all and how boldly, even if not perfectly, they attempted to pick it up: acknowledging harm, seeking accountability and thinking which forms of redress and repair could become available after the fact. Those who shepherd early-career paths, ethics training, or integrity initiatives in the sciences carry a quiet, often invisible responsibility: to help shape how communities acknowledge harm and seek to remediate it - not flawlessly, but meaningfully. Science correction can have (as the sociotechnical narratives of many meta-researchers and open access advocates suggest) transformative, even “revolutionary” effects. But - and here is the crux for feminists like myself - if we cannot do reparative work along the way, then it is not our science correction.

Acknowledgements

This blog post benefitted from two thoughtful peer reviews received in July 2025. My co-editors, Candida F. Sánchez Burmester and Mady Barbeitas, also offered gracious, insightful, and encouraging comments on the submitted draft. I am grateful to all four colleagues for their close and generous reading of my work.

I am also deeply grateful to my PhD writing-accountability partner, Adina Dymczyk, for two formative conversations on Ian Hacking’s work (Oct & Dec 2024). These discussions were invaluable in helping me build the confidence to extend Hacking’s concept beyond discovery and into the realm of science correction. I also thank her for thoughtful feedback on an early version of the blog post.

The effort to trace common threads across the three cases of misconduct investigation - and to weave them into a shared story centered on the experience of the MBSJ - would not have been possible without the ongoing support of the post/doctoral colloquium led by Prof. Stathis Arapostathis at the National Kapodistrian University of Athens. Over the past few years, the opportunity to present (as a guest member) each case in detail and receive generous, sustained feedback from the colloquium members has shaped this work in critical ways. I am sincerely grateful to all participants, and to Stathis in particular.

Footnotes

[1]: Clarification: The MBSJ continued to face cases of misconduct after 2014; the date range here reflects the period covered by my dissertation research.

References

Antonakaki, M. (2019). Mapping laboratories and pinpointing intentions: The entanglement of audit and reproducibility in the STAP case. In C. Hauskeller, A. Manzeschke, & A. Pichl (Eds.), Genetics and Society Ser. The Matrix of Stem Cell Research (pp. 149–169). Routledge. https://doi.org/10.4324/9781315104386-10

Antonakaki, M. (2023). Scientific integrity in-action: How to follow STS reflexivities after MeToo_Academia, and especially #MeTooSTS. EASST Review, 42(1), online. https://easst.net/easst-review/easst-review-volume-421-july-2023/scientific-integrity-in-action-how-to-follow-sts-reflexivities-after-metoo_academia-and-especially-metoosts/

Antonakaki, M., Sánchez Burmester, C. F., & Barbeitas, M. (2025). Cultures of Trial and Error: Identifying and Overcoming Barriers in Science Correction [A JOTE x NanoBubbles, peer reviewed, blog post series]. Journal of Trial and Error. https://blog.trialanderror.org/cultures-of-trial-and-error-identifying-and-overcoming-barriers-in-science-correction

Asahi Shinbun (2006, September 6). Handai kyōju-ra no ronbun ni `gimon' shiteki no joshu ga jisatsu: (EN translation: Assistant that pointed out "doubts" over professor's papers at Osaka University commits suicide). https://web.archive.org/web/20061106202708/http://www.asahi.com/kansai/news/OSK200609060061.html

Elam, M. (2024). Controversy studies and analysis. In U. Felt & A. Irwin (Eds.), Elgar Encyclopedia of Science and Technology Studies (pp. 244–253). Edward Elgar Publishing.

Fuyuno, I., & Cyranoski, D. (2006, September). Mystery surrounds lab death. Nature News, 443. https://doi.org/10.1038/443253a

Hacking, I. (2007). Kinds of People: Moving Targets. Proceedings of the British Academy, 151, 285–318.

katolab-imagefraud, a blog by JuuichiJigen. (2012, January 19). SUMMARY of alleged image manipulation. http://katolab-imagefraud.blogspot.com/

Kenkyū rinri iinkai wakate kyōiku wākingugurūpu kara no go hōkoku: (EN translation: Report from the Research Ethics Committee / Young Researcher Education Working Group) (11.2007). MBSJ News, No.88. https://www.mbsj.jp/archive/bulletin/kaihou-88.pdf

Latour, B. (2004). Why Has Critique Run out of Steam? From Matters of Fact to Matters of Concern. Critical Inquiry, 30(2), 225–248. https://doi.org/10.1086/421123

Meskus, M., Marelli, L., & D'Agostino, G. (2018). Research misconduct in the age of Open Science: The case of STAP stem cells. Science as Culture, 27(1).

Molecular Biology Society Japan. (February 2007a). Bulletin No.86. https://www.mbsj.jp/archive/bulletin/kaihou-86.pdf

Molecular Biology Society Japan. (2007b, December 13). Ima koso shimesou kagakusha no ryoushin — Minna de kangaeru kagakuteki fusei mondai: (EN translation: Now is the time to show the conscience of scientists - Collective consideration of the scientific fraud problem). https://web.archive.org/web/20160311105400/http://mbsj.jp/admins/ethics_and_edu/investigation/20071213_sympo_kagakusha_ryoshin.pdf

Molecular Biology Society Japan. (2008, December 9). Ima koso shimesou kagakusha no ryoushin 2008 — Minna de kangaeru kagakuteki fusei mondai: (EN translation: Let's Show the Conscience of Scientists 2008 - Scientific Misconduct Issues for Everyone to Consider). https://web.archive.org/web/20130925092036/http://www.mbsj.jp/admins/ethics_and_edu/doc/081209_wakate_sympo_all_final.pdf

Molecular Biology Society Japan. (2012, December 10). Tokuteihieirikatsudōhōjin nihonbunshiseibutsugakkai dai 17-ki dai 3-kai riji-kai kiroku: (EN translation: Record of 3rd meeting of the 17th term Board of Directors). https://www.mbsj.jp/admins/report/board/report1212-17th.html

Molecular Biology Society Japan (2013a, June). President's Report. https://www.mbsj.jp/en/admins/statement/20130612.pdf

Molecular Biology Society Japan (2013b, December). Session 3. Academic Journal Systems for Preventing Scientific Misconduct. https://www.mbsj.jp/admins/committee/ethics/doc/2013forum/session3_full_e.pdf

Molecular Biology Society Japan (2014a, March). President’s Statement: Response to STAP Cell Papers. https://www.mbsj.jp/en/admins/statement/20140303.pdf

Molecular Biology Society Japan (2014b, March). President’s Statement: Further Request regarding Response to STAP Cell Papers. https://www.mbsj.jp/en/admins/statement/20140311.pdf

Molecular Biology Society Japan (2014c, July). Statement by the President of MBSJ Regarding Response to STAP Cell Research Papers (Third Statement). https://www.mbsj.jp/en/admins/statement/20140704.pdf

Molecular Biology Society Japan. (2014d, November 25). Seimei kagaku kenkyū no kōsei-sei o tamotsu tame ni: (EN translation: Maintaining the Fairness in life science research). https://www.mbsj.jp/admins/committee/ethics/doc/2014forum.pdf

Molecular Biology Society of Japan. (8.2014). Message from the President (Late Summer 2014). https://web.archive.org/web/20150403131135/http:/www.mbsj.jp/en/admins/messages/18th_osumi_201408.html

Normile, D. (2012, January 25). Whistleblower Uses YouTube to Assert Claims of Scientific Misconduct. ScienceInsider. http://web.archive.org/web/20120129170318/http://news.sciencemag.org/scienceinsider/2012/01/whistleblower-uses-youtube-to-assert.html

Onegai: YK senpai no fuhō yori - Kyōtodaigaku daigakuin: (EN translation: Request by the Graduate Students Community of Kyoto University: News of the passing of YK-senpai). (2006, September 29). https://mixi.jp/view_bbs.pl?comm_id=45001&id=10857428

Osaka School of Frontier Biomedicine. (2006, September 21). Shinri o tankyū suru kagaku kenkyū no kōseina hatten no tame ni: (EN translation: For the fair development of scientific research that seeks truth). https://web.archive.org/web/20070206191350/http://www.fbs.osaka-u.ac.jp/reports/index.html

President's statement, the University of Tokyo (2013, October). A High Standard of Research Ethics as the Ethos of the University of Tokyo: (translated from JP by Euan, McKay). https://www.u-tokyo.ac.jp/content/400008640.pdf

Retraction Watch. (2012, April 9). Endocrinologist Shigeaki Kato resigns amidst University of Tokyo misconduct investigation. https://retractionwatch.com/2012/04/09/endocrinologist-shigeaki-kato-resigns-amidst-university-of-tokyo-misconduct-investigation/

Science in Japan, a discussion site for Japanese science. (2013a, April 6). Hon saito ni tsuite: (EN translation: About this site). http://web.archive.org/web/20130406043013/http://scienceinjapan.org/about

Science in Japan, a discussion site for Japanese science. (2013b, June 26). Netsuzō mondai ni motto ikari o: (EN translation: Get more angry at the fabrication problem (by Anonymous)). http://web.archive.org/web/20130712040338/http://scienceinjapan.org/topics/20130625.html

Scientific Ethics and Research Conduct Committee at the Graduate School of Frontier Biosciences, Osaka University. (2007, January 19). Extracts from the "Report on the Two Papers Suspected of Involving Scientific Misconduct”. Osaka School of Frontier Biomedicine. http://www.fbs.osaka-u.ac.jp/eng/reports/investigationreportsummary.pdf

Sedgwick, E. K. (2003). Paranoid and Reparative Reading. In E. K. Sedgwick (Ed.), Touching Feeling: Affect, Pedagogy, Performativity. Duke University Press. https://www.moodle.tum.de/pluginfile.php/2883768/mod_resource/content/2/Paranoid%20and%20Reparative%20Reading.pdf

Sugino, A. (2009). My response to the Report from the Working Group of the Molecular Biology Society of Japan. Genes to Cells : Devoted to Molecular & Cellular Mechanisms, 14(8), 909–910. https://doi.org/10.1111/j.1365-2443.2009.01330.x (Letter to the Editor).

Tsurimoto, T. (. o. t. W., Itoh, T., Katayama, tsutomu, Kimura, H., & Shinohara, A. (2009, April). Report from the Working Group of the Molecular Biology Society of Japan for the investigation of fraud in research papers. Genes to Cells, 14, 903–908. https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1365-2443.2009.01329.x (Letter to the Editor).

Yanagida, M. (2006a, September 14). Blogpost: The death of Dr Yasuo Kawasaki. Blog 'My Dear Friends 2008 by Yanagida Mitsuhiro'. https://mthiei.exblog.jp/4553710/

Yanagida, M. (2006b, September 23). Blogpost: Professor Akio Sugino's case. Blog 'My Dear Friends 2008 by Yanagida Mitsuhiro'. https://mthiei.exblog.jp/4608423/

Yanagida, M. (2009). Editor’s Note. Genes to Cells, 14(8), 901. https://doi.org/10.1111/j.1365-2443.2009.01331.x

Cultures of Trial and Error: The Promise of Open Notebook Science

Anne Clinio — Wed, 30 Jul 2025 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

A lesser known yet powerful aspect of the Open Science movement, open notebook science challenges the traditional ways we produce and share knowledge, creating a more transparent and collaborative approach to research that could foster science correction.

Imagine a world where scientific research isn’t hidden behind paywalls or only shared after months - or even years - of polishing and peer review. Instead, every step of the journey, from brainstorming to experimentation (even the ones that fail), is open to everyone, everywhere, in real time. That’s the essence of open notebook science, a groundbreaking approach to how science could be done and shared, that was first implemented in chemistry education programs for training future professionals, and in research areas with limited commercial viability, such as antimalarial compounds.

In this post, we introduce this innovation through findings from comprehensive documentary research focused mainly on Jean-Claude Bradley's records - who we identify as a recruiter of followers for a new epistemic culture and its knowledge machinery (Knorr-Cetina, 1999). Vividly articulated by Bradley, who is a chemistry professor and researcher at Drexel University (USA), open notebook science challenges the traditional practice of keeping scientific findings confidential until they are formally published in a scientific journal, inviting anyone interested to dive into the details - from raw data to unresolved challenges - as they unfold in open notebooks. Ultimately, it aims to promote “faster science and better science” (Bradley, 2006a) through transparency, data provenance and open collaboration enabled by digital platforms and open licenses.

According to Bradley, open notebook science means “making all your research freely available to the public and in real time” (Bradley, 2010b). He defined open notebook science in the following way:

“[…] I will use the term Open Notebook Science, which has not yet suffered meme mutation. By this I mean that there is a URL to a laboratory notebook (like this) that is freely available and indexed on common search engines. It does not necessarily have to look like a paper notebook but it is essential that all of the information available to the researchers to make their conclusions is equally available to the rest of the world. Basically, no insider information.” (Bradley, 2006a)

Open notebook science, as a concept, was described in numerous blog posts by Jean-Claude Bradley and stemmed from both his advocacy for open knowledge and his personal ambition to expand the impact of science - from educational and research practices to innovative approaches in drug discovery. However, the practical implementation of this innovation was far from a solo endeavor. It relied on a broad and collaborative network of educators, students, researchers, synthetic chemists, computational chemists, biochemists, chemical companies, programmers, mathematicians, journalists, librarians, open knowledge advocates, and a variety of digital tools and platforms. [1] Together, they supported the development of open notebook prototypes and their surrounding research ecosystem. [2]

A new form of scientific communication: from lab secrets to open laboratories notebooks

Science, as often idealized, holds a unique position in shaping our understanding of the world. For centuries, its foundation has relied on the narrative of a specific methodology - the production of facts through careful experimentation, carried out by a select few within the controlled environment of laboratories. These individuals, often perceived as geniuses and extraordinary builders of facts, have been entrusted with the discovery of hidden truths about nature. Their findings, meticulously crafted into matters of fact, became the cornerstone of what we now consider Modern Science (Shapin & Shaffer, 1985).

The scientific article, as we know it today, is the primary instrument (or literary technology) for sharing these discoveries (Shapin & Shaffer, 1985). Published in peer-reviewed journals, it presents research as ready-made science (Latour, 1987) - polished and persuasive narratives that establish the validity of a new scientific fact. This perspective, known as the diffusion model (Latour, 1987), focuses on delivering coherent stories designed to convince the audience of the discovery's truthfulness.

According to Shapin and Shaffer (1985), the origins of this literary technology can be traced back to the 17th century, with figures like Robert Boyle and other natural philosophers. In their experimental essays, they aimed to communicate phenomena observed in private laboratory spaces to those who could not witness the experiments firsthand. Their work pioneered the concept of virtual witnesses (Shapin, 1984) - readers who could imagine the experiment's execution through detailed descriptions and visual representations, bridging the gap between observation and understanding.

To achieve this goal, early scientists developed a distinct style of reporting, one that brought experiments to life for readers. These reports were complemented by visual aids - drawings and diagrams - to fill gaps in imagination and ensure clarity. Over time, this literary method became intertwined with material technologies, like scientific instruments, and social conventions, such as peer validation, to legitimize findings and navigate controversies. Until today, scientific articles tend to limit scientific communication by showcasing only 'successful' experiments, which are filtered by editors from an avalanche of submitted manuscripts.

On the other hand, open notebook science advocates for a translation model (Latour, 1987) that illuminates in real time the complex network of contributors and interactions involved in research activity. This innovation aims to reveal the full research process in detail, including hypotheses, experimental procedures, setbacks, incompleteness, and so-called 'failure' experiments.

The origins of open notebook science

As a science worker, both researcher and university professor, Bradley identified that the peer review process (closed and double-blind) has several structural constraints and inherent biases that harm teaching and research activities, limit innovation, and restrict access to valuable information. His primary critique highlighted that scientific journals, including the most renowned ones, had limited capacity of evaluation since peer review is restricted to anonymous evaluators who deal with important gaps in information - this critique is particularly relevant if we consider that, at the time (early 2000s), scientific journals did not ask authors to submit raw data as supplemental material. Besides that, Bradley considered the voluntary work of reviewers precarious given the shortage of time to issue an opinion and the impossibility of replicating experiments.

In that sense, the author believed that a major and structural flaw of the evaluation process - established to ensure that a scientific paper meets established standards of quality, validity, and relevance - is its heavy reliance on trust. The way he saw it: readers trust editors to choose qualified anonymous reviewers. In turn, reviewers trust authors to summarize their findings accurately. And main authors trust their collaborators, students, and postdocs to produce accurate data (Bradley, 2007b). [3]

Bradley believed that this perspective dialogued with a core principle of the Scientific Revolution - the shift from blindly trusting authorities to systematically questioning everything. He considered it deeply troubling that even in the 21st century, researchers could still make claims based primarily on their position of authority rather than open, transparent, and reproducible evidence. As he puts it: "In chemistry, facts need references back to experimental data. Only opinions can stand without proof - and then the author's reputation becomes the only reference" (Bradley, 2005).

Secondly, the author criticized that scientific articles format often present experimental procedures in a highly condensed and imprecise manner - frequently lacking the level of detail necessary for even a competent peer to replicate the work (Bradley, 2012a). In some cases, the omissions were so significant that he, a senior professional, could not determine whether the error lay in his own replication attempt or in the original publication. For this reason, he envisioned unrestricted access to laboratory notebooks and raw data as the most effective way to share scientific knowledge. Such openness would enable others to trace every step of an experiment without having to rely on assumptions or incomplete information.

Finally, another criticism focused on what he viewed as a needless and systematic waste of valuable research being lost due to restrictive publication practices of only publishing what is considered a successful result. Bradley estimated that 87% of his own scientific production would never go beyond the walls of his lab because it consisted of experiments that had failed to achieve the expected results, being therefore considered as failures by gatekeepers who define what is to be published in scientific journals (Bradley, 2013). To him, science is made up not only of success histories, but also of failures.

For him, this distortion in scientific communication creates a pervasive reluctance among researchers to share ongoing work, a procrastination driven by two interconnected disfunctions: a culture of secrecy rooted in fears of being scooped, and an obsessive need to present only 'completed' scientific facts. The consequence is a systemic bottleneck in knowledge dissemination, where findings remain concealed until they can be polished into seemingly flawless, incontrovertible claims.

In contrast to the prevailing peer review process, structurally built on an ‘excess of trust’, Bradley advocated for open notebooks to minimize assumptions and enhance transparency in scientific communication. His primary concern was not malicious fraud but rather what he termed the ‘trusted source cascade’: errors introduced into literature under the pressure to publish that are subsequently propagated through citations in future articles. In his words, “once these mistakes are published, they are difficult to identify and even harder to correct - if they are discovered at all. Their spread is further amplified by the ease with which content is copied and disseminated via online databases” (Bradley, 2007b). [4]

In this context, making research data and detailed documentation openly available through open laboratory notebooks would greatly enhance the potential for scrutiny, correction, refutation, enrichment, collaboration, validation, and learning across a broader scientific community.

Training skeptical scientists: from information consumers to curators

As a professor of ‘Chemical Information Retrieval’, Bradley encouraged his students to question trusted sources and emphasized that peer review should not be seen as an unquestionable validation. To challenge the students' mindset, he asked them to gather information about a common experiment (synthesizing dibenzalacetone) from five different sources and compare them. The objective was to analyze protocols and specific requirements described in each article to make students identify and understand inconsistencies in literature. To his surprise, several learners used ethyl acetate in the recrystallization step, which didn’t make sense to him since ethyl acetate does not mix with water. Intrigued by how this mistake became widespread in the classroom, Bradley investigated major chemistry references and found that the mistake has been perpetuated since a paper was published back in 1903 (Bradley, 2012b; 2013).

Through this kind of exercise, he aimed to change the students' belief system and attitude toward scientific literature, transitioning their role from passive information consumers into active information curators. Or even act as peers that correct (and open up) science. He believed that future professionals should be capable of collecting, selecting, and evaluating relevant content in Organic Chemistry. Above all, they should know how to verify the reliability, integrity, and reproducibility of experimental data. In his words: “This exercise should teach them not to trust a source implicitly. [...] There are no trusted sources” (Bradley, 2010a).

Asking someone to be skeptical of others’ work may sound disagreeable, as sincere mistakes can happen to anyone. It’s important to make clear that this proposal did not aim to create a surveillance-like environment of constant oversight over others’ work. On the contrary, the genuine purpose of open notebook science was to highlight and value something fundamental to good science, often limited to a small audience: high quality research documentation regardless of the experiment's outcome. In his own words:

“Any chemistry grad student can tell you that there is tremendous value in discussing failed experiments with others who are equally or more knowledgeable. However, this discussion is usually limited to lab co-workers. By recording ongoing experiments in blogs, I can help you just by knowing what you are trying to do, even if you have not yet succeeded.” (Bradley, 2006c)

Open notebook science is part of a strategy to encourage more consistent scientific discussions and promote open collaboration. In that sense, data provenance must be preserved since it is essential to access the history of the shared dataset, trace its origin, understand the transformations it underwent, and review the analyses and interpretations made by scientists. Due to the importance placed on documentation, we refer to its epistemic culture as a matter of proof (Clinio, 2016; 2017), contrasting with the matter of fact of Modern Science.

What does open notebook science look like?

When Jean-Claude Bradley began exploring open notebook science in 2006, he used the Blogger platform to document and share research activities. Blogger’s simple features, like posting updates, enabling comments, and providing RSS feeds, were enough to get started. However, as Bradley's vision of open collaboration grew, he realized that true collaboration required more than commenting or critiquing. It demanded tools for editing, rewriting, and reshaping content.

To meet these needs, Bradley's UsefulChem Project adopted a bliki format - a blend of a blog and a wiki. [5, 6] The blog component was used to update a general audience on the project’s progress, while the wiki recorded detailed experimental work. This structure allowed researchers to collaborate actively, making changes and contributions directly without needing administrator permissions. For the wiki, Bradley introduced a standardized format for experiment pages to ensure consistency and encourage reuse of data. The experiment page consisted of nine sections: 1) the number of the experiment; 2) the graphic representation of the experiment, 3) the name of the researcher; 4) the objective; 5) procedures; 6) results; 7) discussion; 8) conclusion and 9) log.

Figure 1. Experiment n. 269, done by the student Matthew McBrid

Available at: http://open notebook sciencechallenge.wikispaces.com/EXP269
Accessed on: 10 Aug. 2015

Bradley (2006c) emphasized that the log section is the most critical part of an experiment record. Ideally, it captures every step of the experiment as it unfolds, ensuring accuracy and a rich level of detail. The log is written in a clear and objective style, avoiding unnecessary embellishments. It answers the essential questions: who did what, where, how, when, and what partial results were achieved.

This approach removes the need for a polished narrative, which is often required in traditional scientific articles. Instead, it creates what Bradley (2007a) calls a ‘storyless experiment’ - a straightforward, result-focused record that places equal importance on all outcomes, whether or not they align with the experiment's original goals.

Figure 2 : Experiment n. 269 page detail – the log section

Available at: http://open notebook sciencechallenge.wikispaces.com/EXP269
Accessed on: 10 Aug. 2015

Building an ecosystem for open collaboration

Bradley conceived open notebooks as hubs for collaboration. Its ‘architecture of attention’ aims to connect the right people to the right problems at the right time. In that sense, its ecosystem would function as a social networking platform that connects people that share interests and challenges. On this attempt of ‘reinventing discovery’ (Nielsen, 2011), openly sharing ‘negative results’, incomplete, or ambiguous experimental data should be interpreted as an invitation to collaborate with others. The idea is to take advantage of latent micro-expertise to provide just-in-time solutions for specific research bottlenecks, amplifying collective intelligence (Nielsen, 2011). Nonprofit initiatives such as the OpenSource Malaria project, with which Bradley collaborated, and more recently, the Montreal Neurological Institute-Hospital (The Neuro) are initiatives that have embraced open notebooks as a tool to boost research for neglected diseases.

Open notebook science has great potential to redefine how researchers solve problems. However, in order to achieve its objective to promote ‘faster science, better science’ (Bradley, 2006b), its open collaboration ecosystem requires robust infrastructures and a new culture between science workers. In brief, these shifts could promote that scientific knowledge would be communicated to others without prior control by an editorial committee (de-editorialized), through an open peer review process that is inclusive (accessible to everyone), transparent (visible to all), continuous (with no expiration date), interactive (allowing for responses and counter-responses), and dynamic (welcoming new comments and corrections over time).

For Bradley, ’faster and better science’ could be achieved through open notebook science, which fosters rigorous debate among all scientific stakeholders, such as professional researchers, students, amateurs, and even bots. It could also reduce redundant or misguided research and minimize wasted time and resources. By advocating those well-documented experiments be recognized as valid scientific contributions - regardless of their outcomes - open notebook science promises to foster meticulous record-keeping and a culture of continuous improvement. It could transparently establish priority among scientists through three key mechanisms: time-stamped contributions (automatically recorded), version-control systems (tracking all edits and additions), and third-party administration (ensuring impartial record-keeping).

Collectively these features create an auditable trail of intellectual contributions that could help resolve authorship disputes while maintaining collaborative integrity. Moreover, open notebook science could significantly enhance replicability and encourage open collaborations, potentially proving more effective than ready-made (and closed) science frameworks.

This text is based on the thesis “New laboratory notebooks and new epistemic cultures: between the politics of experiment and the experiment of politics” (2016), written by Anne Clinio. For further information about open notebook science in English, please consult the scientific article “Open notebook science as an emerging epistemic culture within the Open Science movement” published on the Revue Française des Sciences de l'information et de la communication.

Footnotes

[1]: The idea that open notebooks can accelerate progress and enhance the quality of science was a common argument among advocates like Matthew Todd, Anthony Salvagno, and Steve Koch, as well as open science evangelists such as Cameron Neylon and Peter Murray-Rust. Matthew Todd, for instance, a professor at the University of Sydney who collaborated closely with Bradley on an open drug discovery project, highlights a key advantage: “Selfishly, from a scientist’s point of view, it helps to be open because other people can correct your mistakes before you make them.” This openness, he explains, prevents wasted time on unproductive research paths (Todd, 2014).

[2]: During the prototyping phase, early enthusiasts strategically customized well established information tools to minimize learning curves and adoption barriers, incorporating corporate solutions when practical. They established an integrated digital ecosystem combining wiki-based experimental pages, Blogger for research communication, and Google Spreadsheets for collaborative data management. This documentation framework was enhanced by multimedia platforms - Flickr for experimental imagery and YouTube for procedural demonstrations. The ecosystem also implemented advanced chemical informatics through semantic web standards, InChI/InChIKey identifiers, and dual indexing via both Google and ChemSpider for optimal data discovery. Early enthusiasts developed specialized search through Google Custom Search and leveraged the Collaborative Drug Discovery (CDD) platform for targeted data access. The innovation adopted Creative Commons licenses, which was supplemented by a visual four-icon system denoting documentation granularity (full/partial) and dissemination timing (real-time/delayed). Their sharing practices also included material technologies, such as reagents and instruments, when a new lab partner accepted to adopt open research practices. By that time, task automation had already emerged as a key discussion point.

[3]: Professionals who often lack access to raw data, as well as the time and resources necessary to replicate experiments.

[4]: Preceding quotes in this paragraph come from the same source: Bradley, 2007b.

[5]: Originally named for its research directive ("chemists focusing their attention on chemical solutions to real immediate problems" (Bradley, 2006a)), UsefulChem became the prototype open notebook ecosystem, evolving from simple blogging to integrated, structured documentation platforms.

[6]: See: http://usefulchem.blogspot.com, and http://usefulchem.wikispaces.com

Bibliography

Bradley, J.-C. (2005, July 24). Peer review and supported documents [Blog post]. Drexel Coas e-Learning. http://drexel-coas-elearning.blogspot.com.es/2005/07/peer-review-and-supported-documents.html

Bradley, J.-C. (2006a, May 7). Bird flu chemistry progress [Blog post]. UsefulChem. http://usefulchem.blogspot.com.br/2006/05/bird-flu-chemistry-progress.html

Bradley, J.-C. (2006b, September 26). Open Notebook Science [Blog post]. Drexel COAS E-learning. http://drexel-coas-elearning.blogspot.com.br/2006/09/open-notebook-science.html

Bradley, J.-C. (2006c, February 13). Peer review in Google age [Blog post]. Drexel Coas e-Learning. http://drexel-coas-elearning.blogspot.com.es/2006/02/peerreview-in-google-age.html

Bradley, J.-C. (2007a, August 13). C&E News article on writing [Blog post]. UsefulChem. http://usefulchem.blogspot.com.br/2007/08/c-news-article-on-writing.html

Bradley, J.-C. (2007b, October 4). Science is about mistrust [Blog post]. UsefulChem. http://usefulchem.blogspot.com.es/2007/10/science-is-about-mistrust.html

Bradley, J.-C. (2010a, January 2). Dangerous data: Lessons from my Cheminfo Retrieval class [Blog post]. UsefulChem. http://usefulchem.blogspot.com.es/2010/01/dangerous-data-lessons-from-mycheminfo.html

Bradley, J.-C (2010b, September). The impact of Open Notebook Science: Interview with Richard Poynder. Information Today. https://www.infotoday.com/it/sep10/Poynder.shtml

Bradley, J.-C. (2012a, August 20). Shining a light on chemical properties with Open Notebook Science and open strategies [Video]. YouTube. https://www.youtube.com/watch?v=7mK6SY-jXqk

Bradley, J.-C. (2012b, September 27). A recrystallization app based on open services [Video]. Drexel Chemistry Mini-Symposium. https://www.youtube.com/watch?v=vMndyT-DSXM

Bradley, J.-C. (2013, February 12). Open education [Video]. Tuesday Tech Talks. https://www.youtube.com/watch?v=BN8UjULNG9A

Clinio, A. (2016). New laboratory notebooks and new epistemic cultures: Between the politics of experiment and the experiment of politics [Doctoral thesis, IBICT/UFRJ-ECO]. RIDI IBICT Repository. http://ridi.ibict.br/handle/123456789/943

Clinio, A., & Albagli, S. (2017). Open notebook science as an emerging epistemic culture within the Open Science movement. Revue française des sciences de l’information et de la communication, 11. https://doi.org/10.4000/rfsic.3186

Knorr-Cetina, K. (1999). Epistemic cultures: How the sciences make knowledge. Harvard University Press.

Latour, B. (1987). Science in action: How to follow scientists and engineers through society. Harvard University Press.

Nielsen, M. (2011). Reinventing discovery: The new era of networked science. Princeton University Press.

Shapin, S. (1984). Pump and circumstance: Robert Boyle's literary technology. Social Studies of Science, 14(4), 481–520.

Shapin, S., & Schaffer, S. (1985). Leviathan and the air-pump: Hobbes, Boyle, and the experimental life. Princeton University Press. https://doi.org/10.1515/9781400838493

Cultures of Trial and Error: Hoaxes, Thought Experiments & Science Sleuths: Sniffing Out Ways to Correct the Scientific Record

Mady Barbeitas — Sun, 29 Jun 2025 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

In August 2024, a peculiar letter to the editor titled “Practice of Neurosurgery on Saturn” was published in the International Journal of Surgery Case Reports. The title suggests that the article is either a spoof or a hoax. However, continued reading reveals a scientific and formal tone adopted by the authors as they describe three clinical cases of chirurgical interventions on SATURN, in a country called ILLUSIONLAND, where 60 million Homo sapiens sapiens have lived since migrating from Earth (Mostofi & Peyravi, 2024). In fact, after reading the entire article, it becomes clear that the authors are conducting a thought experiment, using absurdity and humor as rhetorical devices to highlight issues that happens in real settings.

This piece might have gone unnoticed by the broader neurosurgical community, given that the Elsevier journal in which it was published has an impact factor of 0.6. This indicates that, on average, articles published in the journal were cited 0.6 times in recent years. Consequently, at least 40% of the journal’s publications receive no citations, while the remainder are cited only rarely, highlighting the journal’s relatively marginal status within the field of surgery. Nevertheless, this particular piece managed to catch the attention of “science sleuths”.

“Sleuths” is the most common name used by journalists to refer to a group of individuals “who are willing to read texts, evaluate images, design screening tools, run through statistical analyses of a publication’s data, and share their findings and views on websites, blogs, wikis and social media” (Biagioli & Lippman, 2020, p.17). The findings that science sleuths share or flag often involve inconsistencies, errors, or even fraud cases within the scientific and academic literature. They believe that the academic corpus should be clean, correct and pure. [1] From their perspective, an article like “Practice of Neurosurgery on Saturn” may represent a blemish – a spot of contamination that undermines the credibility of the scientific record. This raises important questions: Is there truly no place for humor in scientific literature? What can be learned from such unorthodox contributions? And how do science sleuths, peer reviewers, and the broader academic community respond to thought experiments, hoaxes, or spoofs?

There are, in fact, different ways in which hoaxers and sleuths aim to address and correct the scientific record. Hoaxers typically expose weaknesses in the editorial or peer review process by submitting intentionally flawed or absurd work, while sleuths focus on identifying and flagging specific cases of problematic research. They often use PubPeer, a post-publication peer review platform, to flag issues, leave comments, and publicly express their concerns and dissatisfaction about scientific articles. On this independent platform, anyone can comment on any published paper. This system has sparked criticism, particularly around the question of who is qualified to assess the quality of scientific work (Horbach & Halffman, 2018). In addition, some argue that PubPeer has become more of a space for denunciation, or an incubator for sleuths, rather than a forum for constructive scientific dialogue (Blatt, 2015). Nevertheless, several significant fraud cases – and the subsequent corrections and retractions – were addressed following comments on PubPeer (Dubois & Guaspare, 2019). PubPeer is likely to play an increasingly important role in reshaping how scientific articles are collectively evaluated. Articles once certified as valid through peer review can become contested objects through discussions on PubPeer, prompting greater awareness and initiating a process of reassessment by some editors.

Unsurprisingly, Practice of Neurosurgery on Saturn sparked some reactions on PubPeer, where commentators demanded answers: “Could the authors and the journal explain the rationale behind the publication of this ‘case study’?” (Anonymous, 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #1) In other words, how did a piece like this make it into the scientific literature? Some comments were more provocative: “At least informed consent was obtained from all participants. No violations of research ethics on Saturn.” (Anonymous, 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #3) One commenter addressed the journal editor directly: “My guess is that this spoof was primarily designed to expose poor peer review and editorial standards. It took the journal eight days to accept! Publishing charge $1,945!” (Thornton, Jim G 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #5)

The authors quickly disclosed on Pubpeer that the article was a parody. Their intention was to highlight the challenges neurosurgeons may face in certain professional contexts – challenges that could not be illustrated using real clinical cases due to ethical constraints. As a result, the scenarios had to be presented hypothetically, which is why the authors chose to set the case on another planet.

The use of hoaxes in the scientific literature

Some hoaxes have gained popularity and become highly cited, but what has the academic community learned from them? Are they an effective way of correcting the scientific record? I am particularly interested in examining the reaction from the academic community, including the sleuths, towards articles later recognized as hoaxes, as well as the shared expectations that were disrupted or disappointed upon their exposure.

One of the most prominent academic hoaxes is Alan Sokal’s article, “Transgressing the Boundaries: toward a Transformative Hermeneutics of Quantum Gravity”, published in the journal Social Text. (Sokal, 1996b) The article was filled with nonsensical and illogical claims in physics, but it was so densely packed with jargon, buzzwords and references that it was unlikely to be recognized as nonsense, especially because those claims were beclouded by the main message, which was ideologically and politically aligned with the field’s purpose (Cultural Studies) and the editors’ views. Immediately after publication, Sokal revealed the hoax in another journal, Lingua Franca. The story was quickly picked up by major newspapers and received extensive media coverage (Lynch, 1997).

In the Sokal hoax, the primary aim was not to challenge the peer review process, but rather to critique the field of Cultural Studies. The hoax was used to question its epistemic standards and its complicated language. In contrast, other hoaxes have primarily targeted editorial practices, particularly in response to the rise of fee-charging Open Access journals with poor editorial standards, commonly known as predatory journals.

The Bohannon’s “experiment” serves as a clear example of this type of hoax. The science journalist John Bohannon conducted an “experiment” in which he submitted a fabricated and deliberately flawed pharmaceutical study to 304 journals with a biological, chemical, or medical title (Lagerspetz, 2020). Of the 255 journals that rendered a final decision within the project’s timeframe, 157 accepted the paper, while only 98 rejected it. In each case, Bohannon withdrew the submission prior to publication and disclosed the hoax in an article published in Science (Bohannon, 2013).

The article Practice of Neurosurgery on Saturn isn’t exactly a hoax; it’s rather a thought experiment, an imaginary scenario designed to make a point or test a theory, much like Schrödinger’s cat. [2] While hoaxes typically aim to deceive or mislead, the authors of Practice of Neurosurgery on Saturn employ humor to provoke reflection, without ever claiming to present genuine surgical cases. Nevertheless, some sleuths have misread and responded to the piece as though it were a deliberate hoax.

The blurred boundary between fraud and hoax

What is the difference between fraud and hoax, particularly in the cases cited above? According to The Proceedings of the National Academy of Science, one of the first U.S. federal frameworks and educational guides developed around the term: “Fraud encompasses a wide spectrum of behaviors. It can range from selecting only those data that support a hypothesis and concealing the rest (“cooking” data) to changing the readings to meet expectations (“trimming” data) to outright fabrication of results.” In other words, “fraud arises when scientists intentionally publish inaccurate results.” (National Academy of Science, 1989) [3]

In Sokal and Bohannon cases, the authors violated editorial standards by submitting papers containing results they knew to be inaccurate. However, an important question arises: do hoaxes constitute ethical transgressions, or are they a legitimate means of exposing deficiencies in the editorial and publishing process? Can it be argued that the end justifies the means? To address this, we must first clarify the desirable end of both fraud and hoax.

Fraudulent publications are typically motivated by personal gain. The incentives for committing fraud may include obtaining or maintaining an academic position, gaining prestige or influence, attracting research funding, or confirming a belief in a particular hypothesis by “cutting corners” to reach a desired conclusion. In all cases, the fraudulent author aims to benefit by adding a new publication to the scientific record.

In contrast, hoaxers often have different motivations. Their primary aim is usually to expose institutional practices or deficiencies in the editorial and peer review processes of scientific journals. By successfully publishing demonstrably nonsensical or deliberately flawed work, they seek to reveal that editorial standards are insufficient and that manuscripts are sometimes accepted without appropriate scrutiny by technically competent reviewers. Thus, while a hoax is technically similar to fraud in definition, its underlying motivations differ significantly.

Reactions to hoaxes

In order to achieve their purpose – namely, to test how institutional practices function – hoaxes need to dismiss any accusation of ethical violation. As Lagerspetz observes, the term hoax is often used in a more conciliatory manner. “The perpetrators themselves tend to favor the term “experiment”, implying both knowledge creation and a context that can be argued to justify manipulative practices.” (Lagerspetz, 2020, p. 406)

The Sokal hoax received extensive academic and media attention, winning a place in the public spotlight, due in part, though not exclusively, to Sokal’s dismissal of any accusation of fraud, as noted by some authors (Hilgartner, 1997 & Lynch, 1997). Sokal claimed that all the works cited were real, and none were invented. According to him, the only deliberate inaccuracy concerned the physics content, which, as he stated, “any competent physicist or mathematician would realize… is a spoof” (Sokal, 1996a). This claim creates a paradox for those who accuse Sokal of scientific fraud. Such an accusation would hold weight only if his article had been submitted to and accepted by a physics journal – which it was not. Social Text is a journal in humanities; it does not publish empirical studies or experiments subject to verification for scientific accuracy (Lynch, 1997).

On the other hand, the Bohannon hoax reaped both praise and criticism, particularly regarding publishing ethics and research design (Buckland et al., 2013). Surprisingly, some sleuths expressed strong disapproval, raising several concerns on PubPeer (PubPeer, 2013). These reactions trigger an important question: why do hoaxes tend to frustrate and disappoint certain sleuths, whose primary goal is to correct and clean the scientific literature? Are hoaxes, in fact, an ineffective or inappropriate strategy to correct the scientific record? In conversations with some of these sleuths, a common sentiment emerged – readers do not want to feel deceived by scientific publications. From their perspective, authors are expected to maintain a tone of seriousness and transparency, leaving no room for irony or satire. In their view, authors need to stand firmly behind their work and ideas.

As one sleuth clearly stated on Pubpeer in reference to the case of Practice of Neurosurgery on Saturn: “Satire is rare in scientific publishing, and readers are generally not expecting to have to spot such pieces; also, sadly, ostensibly serious articles that express ideas almost as bonkers that of neurosurgery taking place on another planet do appear in the literature with depressing regularity” (Brown, Nicholas 2024, PubPeer thread on ‘Practice of neurosurgery on Saturn’, pp. online, #7)

Conclusion: What we could learn from hoaxes?

In terms of concrete outcomes for the academic publishing system, Bohannon’s hoax prompted the Directory of Open Access Journals (DOAJ) to implement stricter quality control measures, resulting in the delisting of 114 journal titles. (Teixeira da Silva & Al-Khatib, 2016) The case of Practice of Neurosurgery on Saturn, triggered a public letter from the editor of the International Journal of Surgery Case Reports, in which he endorsed the explanation provided by the authors on PubPeer. In his words: “Letters are not reports that churn out rare or strange cases, and I believe we can, at times, publish letters that are of interest. It is of no surprise to me that people have posted this letter on social media, as they feel it is different, light-hearted but with a serious message.” (Rosin, 2025)

From a sociological perspective, hoaxes can offer valuable insights. They can bring a productive friction that stimulates critical reflection within scholarly communication. Placing hoaxes at the center of the analysis means we can learn from the responses they provoke to the “appearance-reality puzzle” they embody as well as the countermeasures they elicit and anticipate (Coopmans, 2022). One particularly important line of critical thinking raised by hoaxes concerns the limitations of peer review and editorial practices. Similarly, the thought experiment, Practice of Neurosurgery on Saturn, has raised questions about what constitutes an acceptable style in scholarly publication.

In the context of science sleuths, hoaxes can serve as instructive provocations. By mimicking the structure of fraud, while remaining distinct, they compel sleuths to clarify, refine, and sometimes revise the criteria they use to flag, critique, and protect the scientific record. In this way, hoaxes illuminate the values that sleuths hold, particularly their commitment to ‘clean’ the scientific literature. What is this notion of purity or cleanness in the literature they covet so much? Is there truly no room for mockery, irony, or sarcasm in the academic record, even though many comments on PubPeer are laced with precisely those tones?

Footnotes:

[1]: I can affirm this based on my fieldwork research on and with science sleuths, through which I’ve developed a keen interest in the underlying ideal of 'purity' that drives many of their interventions.

[2]: Schrödinger's cat is a thought experiment in the field of quantum mechanics. A hypothetical cat in a sealed box is considered both alive and dead due to its fate depending on a random subatomic event.

[3]: Later frameworks employed alternative terms to distance themselves from the implication of intentional wrongdoing associated with the term fraud. For example, the 1992 report Responsible Science introduced a framework centered on the terms misconduct in science, questionable research practices, and other misconduct. (National Academies of Sciences, Engineering, and Medicine, 2017)

Bibliography

Biagioli, M., & Lippman, A. (Eds.). (2020). Introduction: Metrics and the New Ecologies of Academic Misconduct. In GAMING METRICS: Misconduct and Manipulation in Academic Research (pp. 2–24). The MIT Press.

Blatt, M. R. (2015). Vigilante Science. Plant Physiology, 169(2), 907–909. https://doi.org/10.1104/pp.15.01443

Bohannon, J. (2013, October 4). Who’s Afraid of Peer Review? Science, 342(6154), 60–65. https://doi.org/10.1126/science.2013.342.6154.342_60

Buckland, A., Eve, M. P., Steel, G., Gardy, J., & Salo, D. (2013). On the Mark? Responses to a Sting. Journal of Librarianship and Scholarly Communication, 2(1). https://doi.org/10.7710/2162-3309.1116

Coopmans, C. (2022). Learning from Fakes: a Relational Approach. In S. Woolgar, E. Vogel, D. Moats, & C.-F. Helgesson (Eds.), The imposter as social theory: thinking with gatecrashers, cheats and charlatans. Bristol University Press.

Dubois, M., & Guaspare, C. (2019). “Is someone out to get me?” : la biologie moléculaire à l’épreuve du Post-Publication Peer Review. Zilsel, N° 6(2), 164–192. https://doi.org/10.3917/zil.006.0164

Hilgartner, S. (1997). The Sokal Affair in Context. Science, Technology & Human Values, 22(4), 506–522. https://www.jstor.org/stable/689833?seq=1

Horbach, S. P. J. M., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3(1). https://doi.org/10.1186/s41073-018-0051-5

Lagerspetz, M. (2020). “The Grievance Studies Affair” Project: Reconstructing and Assessing the Experimental Design. Science, Technology, & Human Values, 46(2), 402–424. https://doi.org/10.1177/0162243920923087

Lynch, M. (1997). A so-called “fraud”: moral modulations in a literary scandal. History of the Human Sciences, 10(3), 9–21. https://doi.org/10.1177/095269519701000302

National Academies of Sciences, Engineering, and Medicine. (2017). Context and definitions. In Fostering Integrity in Research. National Academies Press. https://doi.org/10.17226/21896

Mostofi, K., & Peyravi, M. (2024). Practice of neurosurgery on saturn. International Journal of Surgery Case Reports, 122, 110139. https://doi.org/10.1016/j.ijscr.2024.110665

National Academy of Science. (1989). On being a scientist (pp. 9068–9069). National Academy Press. https://www.pnas.org/content/86/23/9053.full.pdf

PubPeer. (2013, October 4). Who’s afraid of peer review? https://pubpeer.com/publications/E902004AA49A4636D480087B2A6BE8#0

PubPeer. (2024, September). Practice of neurosurgery on Saturn. https://pubpeer.com/publications/306A99F8F7084E6F58CBBFEB84FA99

Rosin, D. (2025). Regarding the letter to the editor: “Practice of neurosurgery on saturn.” International Journal of Surgery Case Reports, 126, 110665. https://doi.org/10.1016/j.ijscr.2024.110665

Sokal, A. (1996a). A Physicist Experiments With Cultural Studies. Lingua Franca, 62–64. https://physics.nyu.edu/sokal/lingua_franca_v4/lingua_franca_v4.html

Sokal, A. (1996b). Transgressing the Boundaries: Toward a Transformative Hermeneutics of Quantum Gravity. Social Text, 46/47. https://physics.nyu.edu/sokal/transgress_v2/transgress_v2_singlefile.html

Teixeira da Silva, J. A., & Al-Khatib, A. (2016). Questioning the ethics of John Bohannon’s hoaxes and stings in the context of science publishing. KOME, 4(1). https://doi.org/10.17646/kome.2016.16

Cultures of Trial and Error: Wissenschaftschmerz

Bart Penders — Mon, 02 Jun 2025 22:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

Something is rotten in the state of science. In its most generic form, this is the conclusion attached to the polycrisis in psychology and the life sciences by those who actively seek to reform how science is performed, published and evaluated. Self-tasked with the diagnosis of the ‘rot’, its mapping and its repair, is the community of meta-science (Field, 2022). Peterson and Panovsky (2023) have conceptualized metascience as a ‘scientific social movement’, drawing together scientific and activist dimensions under a single banner. This does not necessarily mean that there is only uniformity in the metascience community (Knibbe et al., 2025), but some shared characteristics and goals do keep the community together (Field, 2022). One of them is the profound dissatisfaction with the current state of science. The framing of that state is in terms of ‘rot’, or brokenness and connected to notions of fraud, sloppy science or irreplicability. It is accompanied by the perceived responsibility to “fix” science through, usually, strict policies, enhanced notions of rigor, commitment to transparency and a series of procedural innovations and technological platforms to support all of them. How do we understand metascience’s dissatisfaction and what does it fuel? To answer this question, I will ultimately draw from Fredric Jameson’s notion of “Nostalgia for the Present”.

The struggles of psychology and the life sciences are occasionally proclaimed as a polycrisis of science as a whole, even though Rubin (2023) qualifies this as a questionable metascience practice. Nevertheless, science exists at the intersection of publish-or-perish culture, the replication crisis (often recast into a credibility crisis, a crisis of confidence, or a theory crisis), the commercialization of research and even the threats to scientific freedom across the globe as universities are under siege by governments in, for instance, the US. Each and every one of these crises has placed increasing pressure on the modernist ideal. Modernism, in the intellectual and philosophical sense, is tied to belief in progress through universalist and dispassionate reason and empirical inquiry, to faith in the human capacity to shape and improve societies, and carries an emphasis on autonomy, often cast into terms of scientific freedom and independence.

In many ways, universities are microcosms of this modernist ideal: the pursuit of truth and progress through free and disciplined inquiry. They are both a product of and a vehicle for modernism (and a lot more). However, the modernist ideal is under pressure from all sides and it produces, especially in and among scientists a deep melancholia; a sense of loss, or the fear of losing something many scientists struggle to put into words: the fear of losing modernist ideals, beliefs and aspirations. Metascience can be read as the operationalization of that melancholia, providing both substance to the loss, and as a form of resistance to the crumbling of the modern ideal.

Metascience and the modernist ideal

The modernist ideal for science is contested though. If anything, it is both aspirational and nostalgic. Merton’s codification of the ethos of science under the banner of communalism, universalism, disinterestedness and organized skepticism (CUDOS) knows, for instance, at least three sets of competitors which each carry within them, a diagnosis of the state of science. Where the Mertonian norms reflect modernist ideals –objectivity, rational critique, and (near-)universal values, Mitroff’s work on counter-norms in the Apollo project show how the local social, political and economic realities find their way into the conduct of scientists and their motivations (Mitroff, 1974). The normative structure of science was a lot like the normative order ‘outside’ of science. The production of certified knowledge was, it turns out, social and political through and through. Ziman (1996) followed Mitroff in the development of the post-academic PLACE norms (Proprietary, Local, Authoritarian, Commissioned, Expert), further embodying the postmodern skepticism toward the grand narratives and ideals of modernity.

Whereas Merton's norms approach an idealized, modernist vision of science, Mitroff’s and Ziman’s norms describe a postmodern vision of science as embedded in institutions, societies, markets, and power structures. From the perspective of most scholars of science beyond metascience, this does not necessarily translate into some form of decline for science, but merely a form of demystification of science. However, this is not a shared diagnosis. In fact, after CUDOS and PLACE, MacFarlane (2024) coined the DECAY norms for science (Differentialism, Egoism, Capitalism, Advocacy). In many ways, the DECAY norms seem modelled after Mitroff’s counter-norms. Where Mitroff offered an expansion of the normative structures of science, offering suggestions that science has either shifted away from a state of modernity, or perhaps that such a state never existed (Latour, 1993), MacFarlane (who does not cite Mitroff at all), frames this very actively as decline and decay of the liberal virtues that Merton’s norms embodied.

While metascience is a diverse community, this framing of decay resonates with a large part of it. In fact, in the opening chapter of the Handbook of Meta-Research, Derrick et al. (2024) write that the label of metascience “was adopted, in particular, by a collection of defining pieces of work referring to a crisis of reproducibility in science and the rise of the recognition of academic misconduct” (p. 3), suggesting that to some in the metascience community, the decline of science is part of its foundational identity. This decline allows us to connect the metascience’s active scandalization that there is a social and political dimension to scientific systems that influences how they operate and what they produce (Hesselmann & Reinhart, 2024; Penders, 2024b) to the realization that the modernist ideal for science is slipping from our hands, at risk of being lost.

However, metascience offers a dual affective state. Next to its melancholia, there is sanguinity, optimism and hope – the hope for a return to modernity. Importantly, though, that return is not a return to the past. Instead, it is deeply future-oriented and rooted in innovations meant to strengthen the scientific process that were technologically impossible in the past, improving upon it along the way. The flexibility of digital publication trajectories allows procedural innovations such as registered report-based publication (Nosek & Lakens, 2014) in which research design and methodology are temporally disconnected from experimentation or data collection. The digital infrastructures available to scientists nowadays allow for digital lab notebooks, open methods, open data, and the sharing of materials, procedures and data to an extent that was impossible only a few decades ago. Innovations also extend into new organizational forms for designing research and research consortia (Doell, 2023; Penders, 2024a). Entire new communities, expertises and job descriptions have been built along the way (Nelson, 2020). Metascience and scientific reform seek to innovate their way back into modernity, one fix at a time.

Nostalgia for the present

How can we understand the coexistence of metascience’s foundational melancholia and its sociotechnical sanguinity? When Nosek et al write that with the aid of metascience, they seek to return science to ‘the simplified model of research taught in elementary school’ (2018, p.2602), they invoke images of an idealized past in which science was simpler, purer, more objective. The paper in which Nosek et al. invoke this image is not nostalgic in the traditional sense though: it seeks to promote action in the present. The paradox metascience brings forward is best caught in Fredric Jameson’s notion of a “Nostalgia for the Present” (1989), the paradoxical situation that even though we are in the present, we experience a sense that the present itself has been lost or fragmented and seek to return to a stabilized present which we can grasp and understand. Or, in more romantic terms, we are painfully experiencing a mismatch between science as it is, fragmented, imperfect, error-prone and in need of repair, and science as it could be in the here and now: objective, transparent, rigorous, self-correcting and trustworthy: Wissenschaftschmerz.

Under the banner of a “Nostalgia for the Present”, we can see how the melancholia that comes with Wissenschaftschmerz is fuel for the innovations that metascience offers and that the sanguinity of metascience is part of its modernist project. Metascience is all about the present and the future, about repairing the loss of a science that does not exist in the here and now, and about reaching for a science that can fulfil the expectations of the mind. As such, metascience is both a progressive, forward-looking endeavor: seeking to build new structures for scientific practice that address contemporary challenges; and an endeavor that wields reactionary nostalgia for the present: longing for a scientific present that is no longer alienated from its own ideals, and in need of repair.

Jameson (2016) developed his thinking on nostalgia through his critical work on postmodernism in literary critique. To him, it is tied to the rise of consumerism in late capitalism, where perceptions of the past and present can be sold as commodities in the present and future. Metascience has excelled at this, at selling its modernist project and a social scientific movement of hope, bundling imaginaries of decay and repair, as a cure to our Wissenschaftschmerz.

Bibliography

Derrick, G. E., Nuseibeh, N., Oancea, A., & Xu, X. (2024). Meta-research as discipline, field, or spectrum. In Handbook of Meta-Research (pp. 1–11). Edward Elgar Publishing. https://www.elgaronline.com/edcollchap/book/9781839105722/book-part-9781839105722-6.xml

Doell, K. C. (2023). Megastudies to test the efficacy of behavioural interventions. Nature Reviews Psychology, 2(5), 263–263.

Field, S. M. (2022). Charting the Constellation of Science Reform [Thesis fully internal (DIV), University of Groningen]. https://doi.org/10.33612/diss.229114775

Jameson, F. (1989). Nostalgia for the Present. South Atlantic Quarterly, 88(2), 517–537.

Jameson, F. (2016). Postmodernism, or the cultural logic of late capitalism. In Postmodernism (pp. 62–92). Routledge.

Knibbe, M., de Rijcke, S., & Penders, B. (2025). Care for the soul of science: Equity and virtue in reform and reformation. Cultures of Science, 8(1), 12–23. https://doi.org/10.1177/20966083251329632

Latour, B. (1993). We have never been modern. Original title "Nous n’avond jamais ete modernes: Essays d’anthropologie symmetrique (1991) (C. Porter, Trans.). Harvard University Press.

Macfarlane, B. (2024). The DECAY of Merton’s scientific norms and the new academic ethos. Oxford Review of Education, 50(4), 468–483. https://doi.org/10.1080/03054985.2023.2243814

Mitroff, I. I. (1974). Norms and Counter-Norms in a Select Group of the Apollo Moon Scientists: A Case Study of the Ambivalence of Scientists. American Sociological Review, 39(4), 579–595.

Nelson, N. C. (2020). The Methodologists: A Unique Category of Scientific Actors. Engaging Science, Technology, and Society, 6, 20–33. https://doi.org/10.17351/ests2020.345

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600–2606. https://doi.org/10.1073/pnas.1708274114

Nosek, B. A., & Lakens, D. (2014). Registered Reports: A Method to Increase the Credibility of Published Results. Social Psychology, 45(3), 137–141.

Penders, B. (2024a). Renovating the Theatre of Persuasion. ManyLabs as Collaborative Prototypes for the Production of Credible Knowledge. OSF. https://doi.org/10.31222/osf.io/vhmk2

Penders, B. (2024b). Scandal in scientific reform: The breaking and remaking of science. Journal of Responsible Innovation, 11(1), 2371172. https://doi.org/10.1080/23299460.2024.2371172

Peterson, D., & Panofsky, A. (2023). Metascience as a scientific social movement. Minerva, 61, 147–174.

Rubin, M. (2023). Questionable Metascience Practices. Journal of Trial and Error. https://doi.org/10.36850/mr4

Ziman, J. (1996). “ Post-academic science”: Constructing knowledge with networks and norms. Science & Technology Studies, 9(1), 67–80.

Breaking Bubbles: Art and Science in a Fractured World

Peter Bergeijk — Sun, 30 Mar 2025 22:00:00 GMT

In an age where trust in science, facts, and authority is rapidly eroding, art may offer a surprising way forward. While science seeks clarity and truth, art embraces ambiguity, interpretation, and emotion—qualities that can foster deeper engagement with the world. Historically, art and science were seen as opposites, yet both rely on observation and curiosity. Can art help restore trust by offering new ways of seeing and understanding? This piece explores the intersection of art and science, questioning whether creative expression can bridge the growing divide in a world overwhelmed by misinformation and ideological bubbles.

Trust worldwide is on the verge of breakdown. Can we trust authorities? Can we trust science? Can we trust facts? Lies, conspiracy theories and fabrications reach all corners of the world and spread quicker than the COVID-19 virus did. This undercurrent of falsehoods threatens to erode communities, societies and even the international order. It feels like a unique phenomenon, but our parents and grandparents have been here before (of course!). People have aways tended to live in bubbles, long before we started to use that word as a metaphor for an isolated sphere of ideas and like-mindedness. Religion, ideology and conspiracy theories are of all times and have always become attractive during periods of significant life-threatening stress. This time science - again - has not been able to restore the balance. But could art be of any help?

During the late 19th and a large part of the 20th century, the worlds of art and science were disconnected. The emotional world of the heart and the rational world of the mind were seen as opposites and disjuncts. Indeed, for academics, policy makers and the public at large, science was held to be exact, rule-bound, descriptive and objective. The arts on the other hand, were seen as emotional, creative, subjective and fuzzy. Of course, there are important distinctions: works of art are often intended to be ambiguous – there is no greater joy than asking spectators what they think that the meaning of an artwork is and then to discover the variety of viewpoints that also include perspectives that never were in the conscious mind of the creator. In contrast, science is expected to be unambiguous and rigorously ‘true’ to the data.

Recently we seem to be returning to an approach more like that of the 18th century and earlier periods, when great scientists might also be gifted painters or composers, such as Samuel Morse, a gifted portrait painter and the inventor of the telegraph. Artists deployed research and new discoveries in their work such as Rembrandt's ‘Anatomy Lesson of Dr. Nicolaes', or the arrival of steam in Monet’s ‘Gare Saint-Lazare’. The renaissance man par excellence is of course Leonardo da Vinci. Initiatives such as ‘I art my science’ or the exhibition Observations provide examples of how art and science relate and can reinforce one another.

What both worlds have in common is ‘observation’: a fervent commitment to observing, interpreting and representing the world. I believe that the most important contribution from the arts is that it allows for a different way of experiencing how we observe, that different – and often unforeseen - perspective exists and also that the purposeful intention of an artwork can be overtaken by new realities. One of my lithographs, originally meant to illustrate the disconnect imposed by economic sanctions (and later used as a book cover), seamlessly fit into the Broken links exhibition during the COVID-19 pandemic.

(HOW) COULD ART BE PART OF A SOLUTION?

Art itself has been often under attack for being seen as part of an ideology. It is telling that in Orwell’s 1984 the ‘Ministry of Truth’ has a special attention for the fine arts. Art has indeed been a tool for the powerful from the Egyptian pharaohs to Chinese Maoism and this recognition is one of the driving forces of the movement towards decolonization of the arts. So, art is certainly not a panacea, but art certainly has something to offer to science.

For one thing, art has the ability to help people remember facts and inconvenient truths. Memorials for the COVID-19 pandemic are one example. Although it can be challenging to create a memorial when the ‘enemy’ is invisible, and the victims dispersed and anonymous, they can still become an important part of collective memory. Spontaneous, informal monuments have popped up, such as the wall of hearts along the Thames, overlooking Parliament, where for every COVID-19 victim family members and volunteers could paint a heart with their name on it.

There are more examples of such monuments in our culture. For instance, various altarpieces and depictions of saints were created in response to the bubonic plague, a disabled water pump on Broad Street in London was erected to commemorate the cholera outbreak caused by contaminated water, and in Amsterdam a structure incorporating an abacus - the HIV/AIDS monument by Jean-Michel Othoniel - has become a spot for regular commemoration meetings. They show that art can carry a message over many generations.

I see two contributions that would seem to fit into the curriculum of most scientific curricula. First is the art of interpretation. Works of art do not only require a creator but also an audience and while the creator may have a clear message in mind with her artwork, the audience may see other things. It is a very useful, sometimes sobering but always joyful experience to learn that others do not see what one intended to show. An art class may provide that lesson in a less stressful and more open way to students.

Second is the art of observation. Many sciences train observation specific to the field of enquiry and often relying on machines or instruments. This makes sense when the observations cannot be done with human senses, but typically even those working with nano or cosmos level observations can learn from observation through human science. Art offers a multidimensional and truly multidisciplinary experience.

ARTFUL OBSERVATION AND SCIENCE

Observation1, in the words of the Vietnamese activist artist Tu Anh Hoang is a fundamental aspect of human experience. It involves more than just seeing; it is an active process of engagement of the self with the world, allowing for deeper understanding and insight. The British geographer, painter and community artist Imogen Bellwood-Howard sees art and science as forms of analysis. Analysis of what takes place in Euclidean space and of social processes and phenomena. The precursor to this analysis is observation, and interpretation of these observations.

The Dutch social scientist and novelist Ruben Gowricharn points out that differences in perception and interpretations have consequences, not only for the political debate but also for science. In the exact sciences and life sciences, you can report an observation of, for example, the behaviour of molecules, without being disputed by the object. In the humanities, however, the observer risks that his report may not go down well with the group being studied.

Political scientist Helen Hintjens writes that observation grounds you in reality. Attentiveness is rooted in observation. Observation in art is not limited to perceiving your subject, but also the observation of your own soul and mind. And beyond that, the soul and mind of others. For example, if you are a writer, you are concerned with the soul and mind of the future viewer of your work (unless it never leaves your studio).

Observations are not only rational; emotions are an important part of them. As Maartje Raijmakers explains: ‘An observation grabs my attention and makes me search for what I am actually looking at, and what experiences it evokes in me'. Sometimes observations can be imagined: Vincent Icke, an astrophysicist and visual artist investigates how aliens would see space. In his artworks he investigates how the human eye has difficulty with different shades of blue, and he developed installations that allow visitors to experience fundamental physics principles.

My own work as a professor and lithographer allows an important role for uncertainty, chance, errors, mistakes and serendipity to play in both formal research strategies and in the strictly prescribed chemical procedures of lithography. Observation is key in this process, as accidental discovery brings gifts as well as risks that emerge both at the macrolevel (the composition or the image as a total), its components (the colour separations and structures), and the details. Since variation occurs at all these levels, and because lithography is a natural process, you need to be constantly alert, not only rely on your eyes. In a lithographic atelier you also need to smell and feel and be aware of temperature and humidity. Even air pressure adds to the physical challenges of operating heavy stones and a press that was constructed around the time Edison invented the light bulb. If you miss the signals on either the stones or on paper (both can behave quite differently) you can end up with distortions and messy, unsharp and weak prints. Not useless, because you will have learned a lot, but certainly not stuff that you can show or sell. Creating a role for chance implies that you will not know what the exact result will be (this is in my experience ‘luck’) but your approach is professional as it creates the right conditions for new and interesting findings. Many presented results are part of the art of economic science but more importantly is that we all know about the graveyard where we burry our non-publishable results. The same is true in a lithographic atelier, when the printer’s secret reigns. Seeing in these contexts requires trained observation and a mind open to finding the opposite of what you expect.

Ultimately, the true power of art lies not in providing answers, but in challenging us to look closer, think deeper, and engage with the world—and each other—with renewed curiosity and openness. By embracing the art of observation, we may find unexpected connections between disciplines, allowing both science and art to contribute to a richer, more nuanced perception of truth. This is an asset in a world where trust in science and facts are increasingly under siege: art offers a unique way of seeing, feeling, and questioning—one that can complement scientific inquiry and open new avenues for understanding.

Cultures of Trial and Error: Replication in Social Psychology

Maarten Derksen — Tue, 25 Mar 2025 23:00:00 GMT

JOTE x NanoBubbles present Cultures of Trial and Error: a peer reviewed blog series on error correction in science.

For the last 13 years psychology has been in the grip of a replication crisis. The crisis was declared because time and again the results of earlier, often influential research could not be reproduced in replication studies. Social psychology in particular was affected, and much of the discussion has focussed on this field. Roughly speaking three different lessons have been drawn from these failed replications.

The first is that there must be structural problems in social psychology. Apparently social psychologists produce a lot of spurious results, which then get published and cited, and may even give rise to entire research programs, despite being false. A research culture has developed in which ‘getting it published’ is more important than ‘getting it right’, and it is easier to publish flashy studies than robust work (Nosek et al., 2012). This diagnosis was shared by a growing number of disaffected psychologists who banded together to reform the discipline. They proposed and implemented a number of solutions, many of them aligned with the principles of the broader Open Science movement.

A different kind of conclusion was already drawn in the 1970s, when failed replications had also attracted attention in social psychology. Ken Gergen, an American social psychologist, caused an uproar when he proposed that social psychology should abandon its search for general laws of behavior (K. Gergen, 1973). There are no such laws, he claimed, the phenomena that social psychologists study fluctuate over time and are culture-specific. Moreover, psychology itself causes its object of study to change: when people learn about the results of psychological research, they may resist and choose to behave differently. As evidence for the instability of behavioral phenomena Gergen pointed among other things to failed replications of famous studies in social psychology (K. J. Gergen, 1976). Clearly, prediction and control are not a viable aim for this field. Instead, Gergen argued, social psychology should be seen as a form of history, cataloguing how people behave, so that they may learn to behave differently if they wish. Social psychology should have an explicitly political, liberatory mission. Gergen’s ideas were indignantly rejected by the majority of his colleagues, but they did inspire a relatively marginal social constructionist school that survives to this day. The replication crisis is grist to their mill.

The third response to the failed replications in social psychology takes over Gergen’s ideas about the changeability of social behavior, but not his radical conclusions. These social psychologists argue that, indeed, social behavior is very sensitive to context, and behavioral phenomena vary over time and place. “In matters of social psychology, one can never step in the same river twice” (Crandall & Sherman, 2016). That is why replications so often fail to reproduce the original results. Because the phenomena it deals with are so fluid, ‘direct’ replications that follow the protocol of the original study are not useful. The theories of social psychology describe the stable mechanisms and processes underlying the variable phenomena. These theories should be developed with the use of ‘conceptual’ replications, different experiments that explore aspects of the same theory. According to these social psychologists, the failed replications that the reformers are up in arms about are not a sign of structural problems but of the special nature of social psychology’s object of study. It requires a slightly amended methodology, but there is no need to go crazy and abandon science the way Gergen had done.

On the basis of the failed replications alone it is not possible to choose between these three responses to the replication crisis. First of all, when a replication fails to reproduce the results of the original study, there are broadly speaking two possible conclusions: there is something wrong with the original study, or there is a problem with the replication. As sociologist of science Harry Collins pointed out long ago, it is impossible to decide on purely empirical grounds where the problem lies (Collins, 1985). As long as there is no agreement about the proper outcome it is always possible to put the results in doubt. Collins called this the experimenters’ regress: a debate about which of the two experiments (original or replication) was competently performed is at the same time a debate about what the right outcome of the experiment is, and the right outcome is that of a competently performed experiment.

Collins focused on competence, in particular on tacit skills. But discrepancies between original and replication may also lead to theoretical discussion. The original researchers may defend their theory by arguing that it is basically correct, but that the problem lies with its auxiliary hypotheses (Lakatos, 1970). Any anomalies can be explained away by modifying its protective belt of hypotheses about boundary conditions, proper instrumentation, and so on. Perhaps the replication failed because instrument XYZ is not such a good way to measure the dependent variable after all. Again, this kind of debate cannot be ended on purely empirical grounds, because any further experiment will be vulnerable to the same kind of argument.

Thus, no matter how many failed replications the reformers in psychology point to, each one can be contested, and indeed the crisis has seen plenty of controversies about replications, in which both arguments about competence and the “lakatosian defense” (Meehl, 1990) have figured prominently. Against the social constructionists, moreover, it can always be argued that further research may, one day, unearth the regularities that they believe do not exist. That argument was brought against Gergen in the 1970s (Manis, 1975; Schlenker, 1976) and it is put forward again today with the appeal to the ‘stable underlying mechanisms’. All in all, it seems that replication cannot be the lever for change that reformers and social constructionist hope that it is.

And yet, much has changed in psychology the past 13 years, due to a great extent to the steady stream of (failed) replications. There are of course researchers who continue their work much as they did before the crisis, but the discipline around them has certainly shifted. For one thing, there has been broad support for raising methodological standards, even though specific innovations such as preregistration remain contentious. Moreover, once-thriving research programs such as priming research have fallen in disarray because of the repeated failures to reproduce their results (Leys, 2024). A replication is never decisive, but multiple failures to replicate are an embarrassment and make it increasingly difficult to defend a claim or a theory. The problems in psychology have also given an opportunity to proponents of qualitative research methods to assert themselves and argue, for example, for the integration of qualitative methods into social psychology (Power et al., 2018) and interaction research (Albert & Ruiter, 2018; de Ruiter & Albert, 2017).

Finally, replication studies have encouraged a reflection on the way the field produces knowledge. As we observed in an ethnographic study of replication that I am involved in, trying to replicate someone else’s work confronts researchers with the minutiae of the research process and raises the question which of these details really matter, and which were merely convenient (Derksen et al., 2024). The often messy process behind the neat ‘methods section’ is foregrounded, and the role of tacit knowledge and skills becomes visible. In turn these replicators also consider their own research practice and how they describe it in their papers. These reflections may also include wider epistemological and social epistemological issues: what is a replication really, and who should be charged with conducting them? Thus, replication studies have led to a kind of autoethnography, in which this field of research observes and analyzes its own practices.

Bibliography

Albert, S., & Ruiter, J. P. de. (2018). Improving Human Interaction Research through Ecological Grounding. Collabra: Psychology, 4(1), Article 1. https://doi.org/10.1525/collabra.132

Collins, H. M. (1985). Changing order: Replication and induction in scientific practice. SAGE Publications.

Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93–99. https://doi.org/10.1016/j.jesp.2015.10.002

de Ruiter, J. P., & Albert, S. (2017). An Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology. Research on Language and Social Interaction, 50(1), 90–107. https://doi.org/10.1080/08351813.2017.1262050

Derksen, M., Meirmans, S., Brenninkmeijer, J., Pols, J., De Boer, A., Van Eyghen, H., Gayet, S., Groenwold, R., Hernaus, D., Huijnen, P., Jonker, N., De Kleijn, R., Kroll, C. F., Krypotos, A.-M., Van Der Laan, N., Luijken, K., Meijer, E., Pear, R. S. A., Peels, R., … De Winter, J. (2024). Replication studies in the Netherlands: Lessons learned and recommendations for funders, publishers and editors, and universities. Accountability in Research, 1–19. https://doi.org/10.1080/08989621.2024.2383349

Gergen, K. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309–320. https://doi.org/10.1037/h0034436

Gergen, K. J. (1976). Social Psychology, Science and History. Personality and Social Psychology Bulletin, 2(4), 373–383. https://doi.org/10.1177/014616727600200409

Leys, R. (2024). Anatomy of a Train Wreck: The Rise and Fall of Priming Research. University of Chicago Press.

Manis, M. (1975). Comment on Gergen’s “Social Psychology as History.” Personality and Social Psychology Bulletin, 1(2), 450–455. https://doi.org/10.1177/014616727500100207

Meehl, P. E. (1990). Appraising and Amending Theories: The Strategy of Lakatosian Defense and Two Principles that Warrant It. Psychological Inquiry, 1(2), 108–141. https://doi.org/10.1207/s15327965pli0102_1

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science, 7(6), 615–631.

Power, S. A., Velez, G., Qadafi, A., & Tennant, J. (2018). The SAGE Model of Social Psychological Research. Perspectives on Psychological Science, 13(3), 359–372. https://doi.org/10.1177/1745691617734863

Schlenker, B. R. (1976). Social Psychology and Science: Another Look. Personality and Social Psychology Bulletin, 2(4), 384–390. https://doi.org/10.1177/014616727600200410

Cultures of Trial and Error: Identifying and Overcoming Barriers in Science Correction

Melpomeni (Melina) Antonakaki — Thu, 27 Feb 2025 07:00:00 GMT

A JOTE x NanoBubbles, peer reviewed, blog post series, with guest editors Melpomeni (Melina) Antonakaki, Candida F. Sánchez Burmester, Mady Barbeitas.

The sentiment that "the truth will prevail" is commonly held in the worlds of science, suggesting that scientific knowledge advances through self-correction, via peer review and replication. However, the arguments and evidence supporting this idea have been continuously debated and updated. This discussion spans academic, industry, and science policy communities, focusing on various normative issues, such as whether corrective measures in science are necessary, what form they should take, whether they should operate on an individual or systemic level, and how to demarcate the "self" in self-correction.

The last decade has witnessed a renewed, practical interest in science correction, often alongside concerns about crises of reproducibility in biomedicine (Mcnutt, 2014), nanotechnology and materials science (Leong et al., 2019), and crises of confidence regarding the experimental methods of social psychology (Pashler & Wagenmakers, 2012) and political science (Janz & Freese, 2019). Furthermore, this has occurred within the context of a larger crisis of higher education (Callard, 2022; Loher & Strasser, 2019), as much of university-based research today depends on short-term contracts, while competitive rankings, research output indicators and audit procedures have been introduced (and resisted) as drivers of academic productivity (Halffman & Radder, 2015; Welsh, 2021).

Today, science correction—whether related to one or a combination of the aforementioned crisis narratives and productivity drives—is debated and practiced across a wide range of fields and forums. Several efforts to correct science exist outside or at the margins of academia. At the same time, specific projects and personalities have received widespread recognition, while events such as the World Conferences on Research Integrity have become powerful institutions in themselves. We refer to these heterogeneous communities of varying degrees of institutionalization, but with explicit commitments to science correction, as 'cultures of trial and error': their participants develop shared standards of practice and communication, just so they can do together the corrective work they set out to do (this is what goes into designing any Trial); at the same time, they cannot alone evaluate the wider significance of corrective work and need other cultures of trial and error, also broader constituencies, to see value in it (this is what goes into identifying any Error).

Several practitioners of cultures of trial and error have reported barriers in the execution and communication of their work, while trying to set the record straight via publication outlets (Gelman, 2013; Lévy, 2022) or challenging questionable practices in science management (see discussions on policy ineffectiveness, harassment and retaliation at Franzen, 2021, p. 137, pp. 173-174; Täuber, 2022). These include rejections or delays in publishing in mainstream journals (as corrective output may not fit existing genres or is considered unsuitable for peer review), dismissive behavior (perceived as fostering mistrust or discord), legal threats (accused of defamation), and lacking support from their home institutions (as such work is not typically covered in contracts). Speaking publicly about the challenges of corrective work is often treated as a threat to the authority of scientific expertise at large, especially when systemic deficits are foregrounded in public discourse, such as the rise in the rate of article retractions (Abritis et al., 2021).

As interdisciplinary researchers and collaborative partners in scientific correction projects, we have observed how barriers tend to intersect, producing an amplified effect for those involved. Who benefits and who suffers at this intersectional predicament? Which lessons can be drawn from the experiences of collectively identifying and developing strategies for overcoming barriers? And how do those lessons enrich the scope and shape of science correction?

As access to formal credentials or supporting infrastructure is often limited at the outset of corrective work, it is not surprising that sharing lived experiences of overcoming barriers and establishing credibility have become valuable community resources. These stories contribute to the strong belief, at least among cultures of trial and error, that efforts to correct the errors and right the wrongs are both just and justified. New agents driving research culture reforms have emerged as a result, including communities of ‘meta-scientists,’ ‘sleuths’ (sometimes called ‘epistemic activists’ or ‘misconduct detectives’), as well as open and/or citizen science initiatives, ‘feminist technoscience laboratories’ and academic labor activists [1]. Through their persistent and creative efforts, particularly over the last fifteen years, emerging cultures of trial and error have reinvigorated longstanding questions about the role of proper scientific conduct in the research outcome.

The upcoming series of twelve blog posts, one published every month (and indicated throughout the text in bold), addresses how practitioners in specific cultures of trial and error experience and deal with barriers to science correction in academia and industry, contributing to an increased understanding of how corrections aim not just for error-free, but better science.

Overview of the series

Efforts to correct science may stem from scientist-driven initiatives and be long-term projects, as the four decades of continuous work for the Society for Scientific Values (hereafter, SSV) in India illustrates. Concerned about a case of unfounded claims and dubious promotion practices in the country, a group of Indian scientists came together in 1981 to promote a “healthy scientific environment” (Jayaraman, 1987, p. 535), founding the SSV in 1986. The SSV has examined cases of misconduct in India. By documenting the investigations on its website, and publishing a biannual newsletter discussing cases and articles, the SSV has curated a rich and open repository. In discussions about science correction, India and other countries in the Global South are often criticized for predatory publishing, while less attention is paid on corrective initiatives in these regions. A more balanced understanding of questionable and corrective practices in those countries requires serious consideration of cultures of trial and error and their national and international networks (Shahare & Roberts, 2020). Melina Antonakaki focuses on a Japanese scientist-driven initiative that aimed for self-correction in molecular biology, discussing what drove, for almost a decade (2006-2014), those researchers active in it, and which factors contributed to its untimely stopping in its tracks.

In other situations, science correction is framed as a problem of better integrating science in society, something that involves scientists and non-scientists. This framing opens up space for politicians, lawyers, government officials, and journalists to shape, to some extent, the standards and available resources for corrective activities. The US Congress hearings held in 1981, following several cases of misconduct, is a milestone in this sense. David Guston sees those hearings as “differ[ent] from the previous inquiries into the political loyalty of individual scientists and the fiscal integrity of the scientific agencies, scrutinizing instead the integrity of the performance of research and the mechanisms of alleged self-regulation of misconduct” (2000, p. 88). Today, established priorities, i.e., integrity and productivity, persist and interact, in public discourse and in scientific practice, with other evaluatory ones, i.e., responsibility, reproducibility, openness, diversity, impact, etc. This is not frictionless. Bart Penders examines contemporary languages of research assessment and reform, particularly ones that posit the sciences as in decline and in pressing need of change, and finds that critics use the trope of nostalgia strategically, namely, for overcoming the dichotomy of tradition and innovation.

Evidence targeting systemic problems impeding corrections contributes today to intense debates about the relation between scientific work (for the benefit of societies) and technological innovation (in global capitalist markets). With the rise of metrics, new forms emerge for “gaming” systems of rewards (Biagioli & Lippman, 2020). Mario Biagioli elaborates on why it is so challenging to contest and correct, what he calls, ‘new species’ of misconduct. Another systemic issue is the way in which the pharmaceutical industry has leveraged and biased the corpus of medical science in service of company goals (Sismondo, 2020, 2021). Sergio Sismondo further develops the point, focusing on the difficulties of identifying and dismantling ‘epistemic corruption’. Yet, financial markets and commercial targets may at times indicate directions for science correction. For example, Declan Kuch and Nicolas Rasmussen trace a nano-medical discovery that attracted significant investment but failed in clinical trials, leading to the collapse of a major biotech company and dwindling interest from academic scientists.

In other cases, flawed discoveries remain uncorrected for a long time. Douglas Allchin (2015) narrates how sloppy or erroneous results can persist for decades without being identified, arguing that scientific self-correction is a myth. However, research communities often draw less dramatic conclusions as to why an error has persisted in their field. For instance, in the case of the miscalculated number of human chromosomes, Aryn Martin has shown that cytologists created 'common sense' explanations for why the error went undetected, which were “both technologically determinist... and asymmetrical” (2004, p. 924). Sahana Srinivasan delves further into error non-correction, analyzing how a refuted hypothesis persists in Alzheimer’s research.

Elsewhere, critics use infamous cases of research misconduct (e.g., Jan Hendrik Schön, Hwang Woo-suk, Diederik Stapel, Paolo Macchiarini, and Olivier Voinnet) to showcase the shortcomings of self-correction on a systemic level, since fraudulent activities in all those cases carried on for years, undetected by journals, colleagues and university bureaucracies alike. Practitioners in corrective projects suggest and implement various measures to address systemic flaws in knowledge validation. Anne Clinio is diving deep into one of them, the ‘open science notebook’ of the late Jean-Claude Bradley, which originated from discussions about open knowledge and structural flaws in a system based on trust.

In recent years, practitioners have also tried to improve peer review, especially with the rise of digital platforms. Observing how peer review and the expectations surrounding it have evolved since the 17^th century, Serge Horbach and Willem Halffman (2018) argue that these different expectations create tensions in contemporary debates. New online review practices, especially post-publication peer review, serve as "incubators" for developing cultures of trial and error. Practitioners often use blogs, social media, and platforms such as PubPeer to document their attempts to uncover misconduct, as well as discuss barriers they encounter along the way. While this uncovers the systemic fragilities of peer-review, Mady Barbeitas asks whether errors or hoaxes in the scientific record are only examples of blatant failure, or whether they could potentially serve as means to scout and tinker with the inner workings of peer review, scientific article corrections, and retractions.

One central topic in discussions around science correction is the ambivalent role of faithful replications. There is growing evidence that experimental replication has been a neglected practice: graduate students are often unprepared to cope with experimental irreplication or failure (Lubega et al., 2023) and, in many fields, research teams replicate each other’s results primarily in ‘integrative’ ways, adjusted to suit the interests of the replicating team (Peterson & Panofsky, 2021). For many, replications are deemed ‘unproductive’ for advancing scientific knowledge [2]. This assessment has only recently been challenged (Derksen & Morawski, 2024; Penders & Janssens, 2018), showing the generative potential of replications, for example in driving self-reflection in social psychology and opening discussions on subject matter variability (in this case, human behavior), as Maarten Derksen explains.

At the same time, we notice that reflections stemming from practical experience are rarely taken seriously as credible accounts worth analyzing, systematizing, and integrating in the scientific record as more-than-anecdotes. In fact, prominent voices in metascience promote a universalizing rhetoric that posits the disciplined experimental protocol and effectively managed bureaucracies as the main sources of corrective potential (Penders, 2022). Speaking in a broad and universal idiom often glosses over crucial differences among research fields and disregards infrastructural needs for science correction. Sheena Fee Bartscherer unpacks elements of the discourse of replication as a social movement, and discusses the wide range of stances, motivations and interests of up-and-coming reformers. In a complementary manner, Martin Bush, Cyrus Mody, Nicole Nelson, Maha Said, and Candida Sánchez Burmester discuss the requirements for doing replications across fields of research and note that these are very different.

To conclude, we believe that learning more about diversity in scientific knowledge cultures, here exemplified by the heterogeneous tales of science correction that the series covers, goes hand-in-hand with critical advocacy against an undifferentiated ‘replication drive’ (Holbrook et al., 2019), which would enforce same standards of conduct and evaluation across sciences and the humanities. By foregrounding the experiences and perspectives of cultures of trial and error, we believe this edited collection contributes to an appreciation of diversity in knowledge creation and correction across science and society.

Acknowledgements

The blog post series resulted from a round table at the 2024 EASST/4S conference and has involved most of the roundtable organizers and speakers as series’ contributors. We are very grateful to them for joining this series and for taking the time to write thought-provoking contributions. We would also like to thank Chakalaka Films for recording the round table and for transforming it into a podcast. Early versions of this editorial benefitted greatly from the friendly readings of Anne Clinio, Maarten Derksen, Sheena Fee Bartscherer and Nicole Nelson. Thanks goes out to the two peer reviewers as well, for their encouraging feedback and constructive criticism. Finally, we want to thank the editor of the Blog of Trial and Error, Marcel Hobma, for an excellent collaboration, as well as the whole JOTE team, especially Stefan Gaillard, for organizing peer-review for 12 posts.

This blog post series has been financially supported by 'NanoBubbles: how, when and why does science fail to correct itself', a project that has received Synergy grant funding from the European Research Council (ERC), within the European Union’s Horizon 2020 programme, grant agreement no. 951393. Mady Barbeitas and Candida F. Sánchez Burmester have individually received funding from the ERC-synergy project 'NanoBubbles', with the grant agreement no. 951393.

[1] Prominent examples of feminist technoscience spaces include the Civic Laboratory for Environmental Action Research (https://civiclaboratory.nl/) and the Access in the Making (AIM) Lab (https://accessinthemaking.ca/). Broadly speaking, feminist technoscience considers gender, along with other structures of power and oppression, to play a constitutive role in science and technology. The hybrid term ‘technoscience’ challenges the idea that there is a clear demarcation line between “pure” science and its downstream applications.

[2] This convention can be understood to some extent, if one were to look into the historical roots of the scientific publication regime. A prevalent focus of scientists, publishers and regulators onto ‘originality’ has facilitated the emergence and, ever since, relative stability of the modern function of the scientific author as owner of the reported work and as party responsible for the fruits of their labor (Biagioli, 2003, p. 256).

References

Allchin, D. (2015). Correcting the ‘self-correcting’ mythos of science. Filosofia E História Da Biologia, 10(1), 19-35.

Biagioli, M. (2003). Rights or Rewards? Changing Frameworks of Scientific Authorship. In M. Biagioli & P. Galison (Eds.), Scientific Authorship (pp. 253–280). Routledge.

Biagioli, M., & Lippman, A. (Eds.). (2020). Gaming the Metrics: Misconduct and manipulation in academic research. MIT Press.

Callard, F. (2022). Replication and Reproduction: Crises in Psychology and Academic Labour. Review of General Psychology, 26(2), 199–211.

Derksen, M., & Morawski, J. (2024). Replications are informative, particularly when they fail. Theory & Psychology, 34(5), 597–603.

Franzen, S. (2021). University Responsibility for the Adjudication of Research Misconduct: The Science Bubble. Springer.

Gelman, A. (2013). It’s Too Hard to Publish Criticisms and Obtain Data for Replication [Ethics and Statistics], 26(3), 49–52. http://www.stat.columbia.edu/~gelman/research/published/ChanceEthics8.pdf

Guston, D. H. (2000). Between Politics and Science: Assuring the Productivity and Integrity of Research. Cambridge University Press.

Halffman, W., & Radder, H. (2015). The Academic Manifesto: From an Occupied to a Public University. Minerva, 53(2), 165–187.

Holbrook, B. J., Penders, B., & de Rijcke, S. (2019, January 21). The humanities do not need a replication drive. CWTS. https://www.cwts.nl/blog?article=n-r2v2a4

Horbach, S. P. J. M. S., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3, 8.

Janz, N., & Freese, J. (2019). Good and Bad Replications in Political Science: How Replicators and Original Authors (Should) Talk to Each Other. https://www.mzes.uni-mannheim.de/openscience/wp-content/uploads/2019/01/Janz-Freese_-Good-and-Bad-Replications-1.pdf

Jayaraman, K. S. (1987). Healthy scientific environment promoted by society in India. Nature, 326(6113), 535. https://www.nature.com/articles/326535b0.pdf

Leong, H. S., Butler, K. S., Brinker, C. J., Azzawi, M., Conlan, S., Dufés, C., Owen, A., Rannard, S., Scott, C., Chen, C., Dobrovolskaia, M. A., Kozlov, S. V., Prina-Mello, A., Schmid, R., Wick, P., Caputo, F., Boisseau, P., Crist, R. M., McNeil, S. E., . . . Pastore, C. (2019). On the issue of transparency and reproducibility in nanomedicine. Nature Nanotechnology, 14(7), 629–635.

Lévy, R. (2022). Is it somebody else’s problem to correct the scientific literature? Rapha-Z-Lab. https://raphazlab.wordpress.com/2022/12/15/is-it-somebody-elses-problem-to-correct-the-scientific-literature/

Loher, D., & Strasser, S. (2019). Politics of precarity: neoliberal academia under austerity measures and authoritarian threat. Social Anthropology, 27(S2), 5–14.

Lubega, N., Anderson, A., & Nelson, N. C. (2023). Experience of irreproducibility as a risk factor for poor mental health in biomedical science doctoral students: A survey and interview-based study. PLOS ONE, 18(11), e0293584.

Martin, A. (2004). Can’t Any Body Count? Social Studies of Science, 34(6), 923–948.

Mcnutt, M. (2014, July 11). Journals unite for reproducibility. American Association for the Advancement of Science.

Pashler, H., & Wagenmakers, E.‑J. (2012). Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence? Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 7(6), 528–530.

Penders, B. (2022). Process and Bureaucracy: Scientific Reform as Civilisation. Bulletin of Science, Technology & Society, 42(4), 107–116.

Penders, B., & Janssens, A. C. J. W. (2018). Finding Wealth in Waste: Irreplicability Re-Examined. BioEssays, 40(12), e1800173.

Peterson, D., & Panofsky, A. (2021). Self-correction in science: The diagnostic and integrative motives for replication. Social Studies of Science, 1-23.

Shahare, M., & Roberts, L. L. (2020). Historicizing the crisis of scientific misconduct in Indian science. History of Science, 58(4), 485–506.

Sismondo, S. (2020). Ghost-Managing and Gaming Pharmaceutical Knowledge. In M. Biagioli & A. Lippman (Eds.), Gaming the Metrics: Misconduct and manipulation in academic research (pp. 123–134). MIT Press.

Sismondo, S. (2021). Epistemic Corruption, the Pharmaceutical Industry, and the Body of Medical Science. Frontiers in Research Metrics and Analytics, 6, 614013.

Täuber, S. (2022). Women Academics' Intersectional Experiences of Policy Ineffectiveness in the European Context. Frontiers in Psychology, 13, 810569.

Welsh, J. (2021). Stratifying academia: ranking, oligarchy and the market‐myth in academic audit regimes. Social Anthropology, 29(4), 907–927.

Video: Professor Peter Bergeijk's 'On the inaccuracies of economic observations'

Marcel Hobma — Thu, 30 Jan 2025 23:00:00 GMT

Economics is plagued by large and significant measurement error problems. 75 years ago, the economist Oskar Morgenstern already highlighted this issue with his monograph 'On the accuracy of economic observations'. Unfortunately the book became a neglected classic, and his urgent call to publish measurement errors was not taken to heart.

Emeritus professor Peter van Bergeijk revisits this critical topic in his recent book 'On the inaccuracies of economic observations: Why and how we could do better'. Bergeijk pays tribute to Morgenstern’s seminal work while demonstrating that the discipline has failed to address these longstanding issues. Economic data accuracy on balance has not improved and the need to transparently report measurement error is as big as ever.

The Blog of Trial and Error premieres a thought-provoking video by Bergeijk discussing his book and the way forward for economics. Don’t miss it—check it out here:

Let’s Focus on Learning From Mistakes Not on Avoiding Mistakes

Gerbrand Koren — Tue, 29 Oct 2024 23:00:00 GMT

Mistakes in a learning context

During my commutes I enjoy listening to the radio to stay up to date with politics, sports, tech developments and other news. Often, I encountered a radio commercial for training primary school children to perform ‘error-free’ calculations: ‘Learn how to calculate without mistakes in just a few weeks!’ Of course, I realise that commercials are often a bit overenthusiastic, provocative or in some cases (over)promising results. But still I wondered: Would this really work? What if a child still makes occasional mistakes after completing this training: can parents then demand a refund? Both seemed unrealistic to me…

After my initial surprise, I found myself fundamentally disagreeing with the underlying message of the commercial. Is it really necessary to focus this much on avoiding mistakes? Errors are an inherent part of the learning process [1-4]. When students do not make any mistakes in a course or assignment, it is too easy and the expected learning efficiency is low. On the other hand, when a course or assignment is too difficult, students might feel discouraged, also resulting in sub-optimal learning. Adjusting the level of teaching is an essential step to maximise the potential for learning. Providing the proper support is a balancing act sometimes referred to as ‘scaffolding’ in pedagogy and didactics [5].

Beyond the pragmatic argument that effective learning can take place through mistakes, one could also reflect more fundamentally on the goals that we want to achieve with our teaching. Do we truly want to focus so much on eradicating mistakes, and are we trying to train a generation of reliable ‘robots’? The world around us is changing rapidly: there are groundbreaking technological developments and at the same time we are confronted with a myriad of complex societal and environmental problems [6,7]. I believe that these circumstances demand us to focus not on creating human calculators, but on teaching children creative problem-solving skills and integrating the technological tools that we have at our disposal into our teaching programs [8,9]. Admittedly, the basic skills learned in primary schools are still the foundation for these more complicated competencies.

My teaching experiences

One of the most important things that I try to achieve as a teacher is to create an atmosphere where students are encouraged to try new things and do not feel too discouraged when they make a mistake. Such a ‘safe’ environment can enhance effectiveness of the learning process [10], while also making it more enjoyable. In particular, creative processes benefit from open exchange of ideas [11], and I try to contribute to such an environment in my research activities.

In some of my courses I have introduced computer practicals in Jupyter Notebooks [12], containing bits of easily modifiable Python code. When teaching these practicals, I noticed that some students ‘freeze’ when they get the first error message and immediately ask for help. My typical response is something along the lines of: ‘What is the error about?’. A surprisingly large fraction of these students do not even consider reading the actual error message (which is usually very informative in Python and the first clue for solving it). Now, I often intentionally steer students towards such errors, making the ability to handle them an essential learning outcome.

Encouraging students to undertake ambitious and unusual projects, even at the risk of failure, is one of the things that I enjoy so much about the Honours education that I coordinate for the bachelors programs in my department. The grading in these programs is not based on a typical numerical scale, but we use pass/fail grading with a strong emphasis on the students’ reflection on what went well, what went wrong, and what they have learned. Exploring real problems in an educational context, with the potential of failing, can provide excellent learning opportunities [13].

Success stories

A powerful way to demonstrate the value of mistakes is through sharing some success stories on how initial mistakes have ultimately led to spectacular results. I vividly remember attending inspiring lectures on entrepreneurship by a visiting professor, who shared how some of the companies that he started went bankrupt while others became highly successful. He explained that his failures were indispensable experiences towards his later entrepreneurial success. Also, within science there are several professors that have shared their ‘CV of failures’ [14], to steer away from the typical success stories and provide more visibility to the rejections and failures that pave the way towards it.

My favourite anecdote from mistakes in a research context is related to the development of chaos theory by Edward Lorenz [15]. He ran a computer model to predict the weather and saved intermediate outcomes of the calculations to a disk. It then turned out that the predictions that started from these intermediate outcomes were different from the calculations that were not written to a file in between. Ultimately, Lorenz concluded that small differences in the input can lead to drastically different output. Making a mistake and then pinning down what causes the mistake requires analytical thinking, but seeing that this ‘bug’ is actually an important finding by itself, also relies on the willingness to truly learn from mistakes.

I feel incredibly proud to be part of organisations where we look beyond simple success stories and appreciate efforts that lead to later success. This is apparent in the reward and recognition policy for promotions at Utrecht University with a strong focus on Open Science [16]. I am a strong supporter of Open Science [17-19] and aim to further stimulate its uptake through my roles as Ambassador of the Open Science Community Utrecht and Member of the Utrecht Young Academy. Through these organisations and initiatives like the Journal of Trial & Error, I feel overwhelming support in my ambition to learn more from mistakes!

References

1: Lorenzet, S. J., Salas, E., & Tannenbaum, S. I. (2005). Benefiting from mistakes: The impact of guided errors on learning, performance, and self-efficacy. Human Resource Development Quarterly, 16, 301–322. https://doi.org/10.1002/hrdq.1141

2: Potts, R. & Shanks, D. R. (2014). The Benefit of Generating Errors During Learning. Journal of Experimental Psychology: General, 143(2), 644–667. https://doi.org/10.1037/a0033194

3: Metcalfe, J. (2017). Learning from Errors. Annual Review of Psychology, 68, 465–489. https://doi.org/10.1146/annurev-psych-010416-044022

4: Young, M. (2019). The utility of failure: a taxonomy for research and scholarship. Perspectives on Medical Education, 8(6), 365–371. https://doi.org/10.1007/S40037-019-00551-6

5: Belland, B.R. (2014). Scaffolding: Definition, Current Debates, and Future Directions. In: Spector, J., Merrill, M., Elen, J., Bishop, M. (Eds) Handbook of Research on Educational Communications and Technology. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-3185-5_39

6: DeFries, R., & Nagendra, H. (2017). Ecosystem management as a wicked problem. Science, 356, 6335, 265–270. https://doi.org/10.1126/science.aal1950

7: Lawrence, M., Homer-Dixon, T., Janzwood, S., Rockström, J., Renn, O., & Donges, J. F. (2024). Global polycrisis: The causal mechanisms of crisis entanglement. Global Sustainability, 7, e6. https://doi.org/10.1017/sus.2024.1

8: Chiu, T. K. F. (2024). Future research recommendations for transforming higher education with generative AI. Computers and Education: Artificial Intelligence, 6, 100197. https://doi.org/10.1016/j.caeai.2023.100197

9: Susha, I., Viberg, O., & Koren, G. (2024). Co-writing an essay with ChatGPT: Experiences and perceptions of students in higher education. European Conference on Information Systems Education Research (ECISER 2024), Cyprus. https://aisel.aisnet.org/eciser2024/3

10: Prashanti, E., & Ramnarayan, K. (2020). Ten maxims for creating a safe learning environment. Advances in Physiology Education, 44, 4, 550–553. https://doi.org/10.1152/advan.00085.2020

11: Castro, D. R., Anseel, F., Kluger, A. N., Lloyd, K. J., & Turjeman-Levi, Y. (2018). Mere listening effect on creativity and the mediating role of psychological safety. Psychology of Aesthetics, Creativity, and the Arts, 12(4), 489–502. https://doi.org/10.1037/aca0000177

12: Koren, G., Edelenbosch, O., van der Wijst K.-I., & Zwaan, A. (2023). Interactive learning experiences with open source Jupyter Notebooks. Teaching and Learning Collection, Centre for Academic Teaching and Learning, Utrecht University. https://teaching-and-learning-collection.sites.uu.nl/project/interactive-learning-experiences-with-open-source-jupyter-notebooks/

13: Lanni, J. (2021). Why I teach my students about scientific failure. Science. https://doi.org/10.1126/science.caredit.acz9901

14: Stefan, M. (2010). A CV of failures. Nature, 468, 467. https://doi.org/10.1038/nj7322-467a

15: MIT News (2008). Edward Lorenz, father of chaos theory and butterfly effect, dies at 90. https://news.mit.edu/2008/obit-lorenz-0416

16: Woolston, C. (2021). Impact factor abandoned by Dutch university in hiring and promotion decisions. Nature 595, 462. https://doi.org/10.1038/d41586-021-01759-5

17: Blanken, P. D., Brunet, D., Dominguez, C., Goursaud Oger, S., Hussain, S. et al. (2022). Atmospheric Sciences Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science. Earth and Space Science, 9, e2021EA002204. https://doi.org/10.1029/2021EA002204

18: Dwivedi, D., Santos, A. L. D., Barnard, M. A., Crimmins, T. M., Malhotra, A., et al. (2022). Biogeosciences Perspectives on Integrated, Coordinated, Open, Networked (ICON) Science. Earth and Space Science, 9, e2021EA002119. https://doi.org/10.1029/2021EA002119

19: Koren, G., Ferrara, V., Timmins, M., & Morrison, M. A. (2022). Global environmental change perspectives on Integrated, Coordinated, Open, and Networked (ICON) Science. Earth and Space Science, 9, e2022EA002231. https://doi.org/10.1029/2022EA002231

Founding JOTE: A Conversation with Stefan Gaillard: “I Believe That Positive Publication Bias is Actively Harming Science”

Marcel Hobma — Sun, 04 Aug 2024 22:00:00 GMT

When establishing JOTE, its founders already had different views on the values and direction of the Journal. In this third and last part of the series on JOTE’s founders, we discuss the past and future direction of JOTE with its current chair Stefan Gaillard, who has a distinctively different view on failure and academic publishing than his former colleagues.

He was editor-in-chief until 2022, and took up the chair position from Martijn van der Meer in 2024. As a PhD candidate he studies overpromising in nanobiology at the Radboud University.

How did you get involved with JOTE?

"Together with the other two co-founders as well as Alex Visser, we attended a lecture on Open Access. One of the big focusses of the lecture was how Open Access would enable a general public to get access to the results of science. However, we disagreed with this take, because there is a lot that does not get published, so even if everything that is published is published Open Access, the general public (and us scientists) won’t have access to all the results of science.

"At the time that was a big part of our motivation in founding the journal. Over time I have become more sceptical whether the general public is really interested in having direct access to the results of science, be they positive or negative. I also believe that my viewpoint at the time was altruistic and that scientists should primarily focus on what benefits themselves, which is high-quality knowledge production (and knowledge of what doesn’t work or what is not the case, is definitely part of that).

"In addition, especially for Max and me, there was personal frustration with how scientists dealt with failed research. During my psychology premaster I conducted an experiment on subliminal messaging. My group failed to replicate the findings of our supervisors – the euro coins that we presented subliminally had no effect on the subsequent task that participants had to perform. Rather than being considered an interesting finding in itself, our supervisors almost immediately assumed that we had done something wrong or there was something faulty with the technology. Of course it was reasonable of them to assume so – I think it’s likely that it was either of those - but the fact that only those two options were considered, was what bothered me. There was also never any follow-up with regards to what exactly must have gone wrong with regards to either how we conducted the research or the used technology.

"So taken together, these were two important reasons to found the journal. After the Open Access lecture, the five of us went to a local café and started hammering out some preliminary ideas. After a while and a few more meetings we figured out that Martijn, Max and I would do most of the grunt work in establishing the journal and laying out the vision, and that Alex would stay peripherally attached and help us out where they could."

What was your role in establishing it?
"That’s a surprisingly difficult question. There was so much going on in the beginning, you could almost do historical research on it. A while ago I was going through some old emails and WhatsApp conversations and there really is a lot to reconstruct. I was involved in trying to set up the website and the submission portal with the help of a far more knowledgeable friend of mine. Still, it was only when Thomas Jorna joined our team that this really got up to speed.

"In addition, we had to work out the vision of the journal and our target audience. There were many, many hours long discussions on what exactly failure and trial-and-error mean, if we should use the word failure, what failure and trial-and error look like in the Humanities and much much more. We then also had to put that on paper, for example in our Manifesto.

"At the same time we were promoting the idea of the journal at various conferences which we divided between us. You might think that that would occur after we had fully worked out our vision and audience, but we took a more iterative approach. During these conferences we gathered feedback that we then took with us when working on the vision and strategy. Besides these main tasks we also divided “smaller” tasks between us such as getting an ISSN number and signing up for the archival service CLOCKSS.

Can you elaborate on your personal motivations, ideas and ideals when establishing it?

"Personally I was mostly motivated by my own dismay with how science operates. Science is our best way of producing knowledge and its contribution to human flourishing is without equal. Precisely because science is such a great enterprise I am very motivated to keep improving it and to protect it from harm. I believe that positive publication bias is actively harming science and at the very least holding knowledge production back.

"Although I think traditional for-profit publishers provide a valuable service, I do believe that from their viewpoint, universities and libraries would be better off spending their money on financing Diamond Open Access journals directly rather than paying for traditional or Green/Gold Open Access journals, because it is cheaper and the quality is, in my opinion, the same. However, it is up to us Diamond Open Access journals to convince universities and libraries of that in the marketplace of ideas. Even though there is a lot of talk from universities and libraries about Diamond Open Access being the supposed gold standard, there is currently almost no funding for it. Clearly, universities and libraries are less enthusiastic about Diamond Open Access than they say they are. Being more mindful of how we spend money as academics was thus also a big motivation, but never because of any anti-profit motives.

"During the founding, I was also convinced that we had a responsibility towards the general public to actively change their picture of the scientific enterprise, but I no longer believe this is something that we should be overly concerned about as scientists, let alone that it is our responsibility or even duty."

How did your ideas differ from the other founders?

"My ideas differed fundamentally in almost every way from the other founders. Sometimes I think that really the only thing we had in common was wanting to tackle positive publication bias. Max and Martijn are both very critical of for-profit publishing in general. I personally think that being a non-profit Diamond Open Access journal is the way to go for JOTE: it lowers the barrier for people to publish their failed research, which is precisely what we want. However, I feel that for other journals it makes sense to be for-profit. They provide valuable services that I myself also use in my capacity as a researcher. Despite all the complaining by scientists, evidently they also think the services of these publishers are worth it because they keep using them.

"Furthermore, I take a fundamentally different view towards how we approach failure. Unlike Martijn and Max, I do not believe it is our mission to problematize failure. Humans are goal-oriented creatures. Success means that we reach our goals, which contributes to our flourishing. In that sense success will always be superior to failure. However, at the same time we are not omniscient creatures. That means that inevitably we will fail – failure is part of life. To deny or ignore that would be foolish. Rather, we can learn from our failures to prevent making similar mistakes in the future, and ideally, we can also learn from the mistakes of others so we can prevent making our own. To me, that’s what the journal is about. But failure is always instrumental in that sense. It helps us succeed."

Does JOTE differ from your first conception of the project, and if so, how?

"Partially yes. When we first conceived of the project, we also had this idea of providing a platform to whistleblowers. In many cases, early career researchers do not own the data of their research, so they are dependent on their PIs to be able to publish their failed research. Those PIs often do not give permission. At the same time, for example in medicine, it is often mandated that these types of results are published, although this is almost never checked. In those cases I think a whistleblower platform for failed results would be a great idea and it’s something I’d still like to implement one day. In other cases intellectual property rights should be respected even though it is a shame that this means that failed research does not get published. Although I am sure my co-founders would disagree with my view on intellectual property."

"Another way that JOTE differs from my first conception is that we now also publish meta-research. I am very happy that we do, because I personally find it a very fascinating field (although not as new as it pretends to be), and also because I think it fits the scope of JOTE really well. Science itself is a process of trial-and-error, so reflecting on that process fits us very well in that sense."

What was the most productive or most notable failure in the establishment of JOTE (and what did you learn from it?)

"One of the most notable failures was the day that an article got published about JOTE in the Volkskrant, a national Dutch newspaper. A day where we expected a great increase in traffic to the website. However, just that very day an error occurred which led to the website being flagged as unsafe to visit. This led to a lot of lost traffic. I was very annoyed by this, which led me to become a bit terse with the people who pointed out that the website was failing. One visitor rightly called me out on this. Thus, that day was a good lesson for myself to be mindful that errors can occur and are there to be overcome, rather than let my annoyance get the best of me."

What do you hope for the future of JOTE?

"I hope that we continue to inspire other journals and platforms to address positive publication bias and perhaps the file drawer problem more broadly. Daniel Lakens and Eline Ensinck recently published an op-ed in Nature with regards to the broader file drawer problem, where not only negative and null results end up unpublished, but also sometimes positive results – for example because the lead researcher gains a new position at another university and starts with a new project there. I would be interested in helping address that issue. In addition, I still want to provide the aforementioned whistleblower platform.

"Eventually it would of course be great if we could solve positive publication bias and the file drawer issue. One could argue that at that point in time the journal becomes superfluous. Just like we don’t have the Journal for Positive Results, we would no longer have journals solely tailored to negative and null results. Every journal would just publish a mix of both, and without skewing towards positive results. However, I think even then, JOTE would have a raison d’être as (one of) the first multidisciplinary metascience journal(s).

"We have a Trial and Error Award which I would like to see expand in scope in the coming years. We are also starting a collaboration with the Museum of Failure, which I hope to see flourish in the future. To be honest there are too many things to count that I hope for the future of JOTE. I have more project ideas than I have time, money or manpower to conduct."

Do you have other thoughts or comments that you would like to add?

"I would sincerely like to thank the Descartes Centre, and in particular David Baneke, Hieke Huistra and Bert Theunissen for their support during the founding and their continued support over the years."

Founding JOTE: A Conversation with Martijn van der Meer. “My Interests Became More Political. I Wanted to Contribute to a Mission-Driven Model of Scientific Publishing”

Marcel Hobma — Mon, 29 Jul 2024 22:00:00 GMT

When establishing JOTE, its founders already had different views on the values and direction of the Journal. In this second part of the series on JOTE’s founders, we speak with Martijn van der Meer, long-time chair of JOTE and the driving force of JOTE’s organisation and foundation. He put down his position as chair and project leader in 2024, and is currently conducting research on the history of preventive child health intervention as a PhD candidate.

How did you get involved with JOTE?

"My motivations sprang from existential insecurities. As an aspiring historian of science, I wanted to know how I could contribute to contemporary struggles scientists were having at the time. Therefore I went to an “Open Science” event at the Utrecht Medical Center. After the event, I figured out that Max and Stefan, fellow students from the History and Philosophy of Science research masters were present as well. We started talking about how great it would be to share “failed research”, as this could potentially solve the publication bias and resulting reproducibility crisis.

"We also liked the idea because it could potentially solve the unrealistic self-image of scientists as chasing clouds and landing on ground breaking discoveries. Especially when Max shared the experiences from his master thesis in neuroscience, I personally became convinced that it was an idea worth exploring practically. We decided to think about it and had a meeting in Café Tilt to discuss the idea further. Over drinks, we decided to at least pretend that we founded the “journal of failed experiments” – later we changed the name to a more constructive one."

Can you elaborate on your personal motivations, ideas and ideals when establishing it?
"My motives shifted over the years. Initially, I was attracted to the idea of correcting the self-image scientific researchers seem to have, and securing a space for constructive reflection on the trial-and-error processes that are central to scientific work. Later, I liked the idea of an interdisciplinary dialogue that we were facilitating among empirical researchers and reflective scholars. Taking research with unintended outcomes as a point of departure seemed a great way of stimulating constructive interaction between History, Sociology, and Philosophy of Science, and those who actually do scientific research.

In the final years of my involvement, my interests became more political. I wanted to practically contribute to a mission-driven model of scientific publishing. We explicitly decided to not collaborate with profit-driven publishers – despite tempting offers – and instead tried to maintain a Diamond Open Access journal that could be practically and financially self-sustaining. For me, while steering our activities in the background, the last years have been to come up with a non-profit business model for facilitating scholarly communication. Moreover, I started to perceive the act of communicating about “failure” in a medium that conventionally focused on “success” to be a fantastic example of ironic anarchism. To me, the Journal of Trial and Error resembles an administrative logic alternative to growth-oriented modes of production."

How did your ideas differ from the other founders?
"I would describe my organisational approach as pragmatic in the sense that I have been focused on the consequence of our activities, and less on the intentions behind our efforts or how these intentions were perceived. I know that Stefan, from the progressive right, believes that profit-oriented modes of organisation can be very productive. Max, on the other hand, was motivated by radically revising profit-driven science. I most certainly agreed often with Max, but I did care less about whether anyone agreed with me. My personal values, in the end, have been less important than the efficacy of our efforts. Maximising the opportunities of how JOTE was misunderstood, in my view at least, has been perfectly legitimate from a pragmatic point of view."

What was the most productive or most notable failure in the establishment of JOTE (and what did you learn from it?)
"We should have been less eager to bring in as many people as possible without having clear operational tasks in mind. I have done a terrible job at getting individuals engaged at times I had other priorities (my studies, my actual job, etc.)."

What do you hope for the future of JOTE?
"I hope that it stays non-profit, and that the activities do not scale too fast – both in terms of people, money, and number of publications."

Founding JOTE: A Conversation with Max Bautista Perpinyà. “I Don’t Think Forbes Has Any Legitimacy in Acknowledging Scientific Success.”

Marcel Hobma — Mon, 22 Jul 2024 22:00:00 GMT

In April this year, JOTE co-founders Stefan Gaillard and Martijn van der Meer appeared on national television to talk about their nomination for the Forbes 30 under 30 listing. “It’s a little bit awkward”, said Van der Meer shortly after the introduction of the Journal. Not only could one of the founders (Max Bautista Perpinyà) not be listed because of his age, but also because JOTE generally tends to be critical of success-focused rankings like these, he said.

The talk show performance hints at the fact that the three founders of JOTE had different ideas and motivations for establishing the journal. In this three-part series, we interview the three founders about these differences, and the history of the Journal, starting with the co-founder that didn’t enter (nor endorse) the Forbes listing: Max Bautista Perpinyà.

Max was part of the initial team that created JOTE in 2018. He fulfilled several roles in the journal – such as sketching its basic principles and aims, developing editorial policies, outreach, establishing institutional contacts, and coordinating design– until 2022. He is currently pursuing his PhD at UCLouvain on the history and philosophy of scientific environmentalism in the Spanish political transition from Francoism towards democracy.

How did you get involved with JOTE?
"I had a role in JOTE from its start, when it was still a joke, then through its period when it served as a sort of ‘cultural intervention’. I left shortly after the publication of the first issue and the serious beginning, when the processes of professionalisation and institutionalisation were gaining traction.

"A key moment that can be seen as JOTE’s start was an Open Science talk where Alex Visser, Martijn van der Meer, Stefan Gaillard and I attended independently. We met there by chance and started talking about how this movement for transparency in science related to what we were reading in our masters’ in the History and Philosophy of Science, like Latour’s black box or Collin’s writings about the difficulty of replicating experiments. We wanted to use these kinds of concepts – to open up science, think critically about what it means to replicate – and articulate them through some kind of technology. JOTE was our technology, our way to materialise our intellectual interests. As an ethos, we all had the ambition to use our critical stance on science to contribute to the academic community in general and to the natural sciences specifically. It’s a bit of a naïve idea, I think, but at that time, at least for me, it was useful to transition from my background in the life sciences to the humanities.

What was your role in establishing JOTE?
"I mostly worked on thinking about how to communicate our project to scientists. I thought of myself as a ‘translator’. I talked a lot about JOTE to former colleagues, many of whom were doing their PhD in Neuroscience in different parts of Europe. I also reached out to the coordinators of my former Neuroscience master’s program and gave talks in several places to spread the message, and very importantly, to receive feedback. It was really important for us to know what the scientific community needed, so talks like the one Martijn and I gave at the Annual Graduate Student Symposium at the University of Cambridge Laboratory of Molecular Biology were key for that, as well as getting recognition and putting the JOTE brand out there.

What were your personal motivations, ideas and ideals when establishing the Journal?
"My time in JOTE allowed me to bridge the gap between my background in the natural sciences and my ambition to research and work in the humanities. At that point, I had accumulated some unsatisfactory experiences in the lab (although I do have many fond memories though!), and I was frustrated by the lack of transparency in the research papers. At the coffee machine in the lab, scientists are very upfront and honest about their daily difficulties, their struggles, and their lack of trust in what they are doing. But in the research papers I was reading while I was a Neuroscience graduate, I would find, instead, big promises about the possibility of addressing complex issues such as depression or schizophrenia, through understanding molecular processes, for instance. This gap –the gap between the coffee machine talk and the big breakthrough promises of science– was troubling for me. It generated unrealistic goals for young scholars, and my motivation came from a need to reduce expectations, embrace the daily life of research, and the mediocrity of academia, to collectively humble ourselves with what we are doing. In a time when researchers are demanded to think about their social impact, I think it’s important to think more about what we are actually doing, and not what we could do if all went well.

Does JOTE differ from your first conception of the project, and if so, how?
"It does. JOTE has become successful in securing funding and professionalising the structure. These are objectives that we set early on and I congratulate those who made that possible. However, in my view, there have been changes in how some of the core principles of the organisation are implemented. Two of JOTE’s so-called co-founders appear now in the Forbes list ’30 under 30’, which in my view, sits squarely against two core ideals of JOTE. First, the initial goal of JOTE on questioning success in academia; and second, the idea that science is a social act best understood by the collective work of many, and not by the success of individuals with great ideas.

"Regarding the first, I don’t think Forbes has any legitimacy in acknowledging scientific success. Forbes – known for publishing an annual list of the wealthiest individuals – is not a referent for any academics who believe in social justice and the role of science in contributing to it. Who makes the list? What background do the decision-makers have? What are the selection parameters? This lack of transparency is expected from a for-profit company that celebrates individual achievement in the capitalist ladder. It is not, and should not, in my opinion, be the same measuring stick for a student-led, non-profit academic journal that aims to change “how science is done.”

"Forbes may see JOTE’s interest in ‘failures’ as simply, and yet again, a way to succeed. In this way, failure is just a stepping stone towards success, but success remains the objective. I am not surprised to see Forbes employing this discourse, but it is not something I would be comfortable with sharing if I were still involved in JOTE. It is also important to remember that the Forbes list has been tainted in recent years by several entrepreneurs ending up in jail for financial and crypto scams. Forbes has now expanded into several categories beyond entrepreneurship, like ‘Sports’, ‘Social Impact', and 'Science & Healthcare’ (the category where JOTE’s ‘co-founders’ were selected for). For me, this is an intrusion into other professional sectors and an attempt to whitewash Forbes’ failures and illustrates the meaninglessness of their rankings. I am disappointed that JOTE is now being used to elevate Forbes’s reputation.

"A second point in which I see JOTE departing from its original stance is seeing science as an exercise of entrepreneurship rather than a complex social phenomenon. JOTE was built by many, not only its official three white cis-male ‘co-founders’. Alex Visser was also at the initial OS meeting, and equally important are their contributions as well as those of other non-cofounders. I am also the first one responsible for using the category of ‘co-founder’, building on the idea that the most important aspect of a successful enterprise is having ‘the great idea’. As historians of science, we could do better than that. Typesetting, designing, administration, outreach, communication, and a big etcetera – those were vital tasks that we all participated in, so I am just not as comfortable with the notion of ‘co-founder’ as I used to be. Perhaps it’s not only that JOTE has changed since the initial conception, but that I have also changed and my values may also lay elsewhere.

What was the most productive or most notable failure in the establishment of JOTE, and what did you learn from it?
"I honestly cannot think of specific ‘failures’, but I can say that the most notable and productive achievement came from a precarious situation. We started the project as students, and many of us also worked part-time, so running the project without funding was not sustainable in the long run. I think the most radical achievement of JOTE is not that it publishes null results or failed experiments, it is its financial and legal structure. As a non-profit, it mostly sustains itself through grants, and donations from research institutes (like the Descartes Centre at Utrecht University), and services provided to university libraries. Setting up these structures took time, effort, and expertise; and importantly, allowed for (some) of the members of JOTE to be paid. I was not there when all this work was done, and since I left, I think this is the most important thing JOTE can teach other organisations. I was proud that they managed to create a sustainable alternative to academic publishing. In the face of the parasitic monopoly of big publishers, this is no easy feat. JOTE shows us that as academics we don’t need to rely on commercial publishers like Springer, Elsevier or Wiley. We have the resources and motivations in-house. The makers of JOTE were completely inexperienced, yet they made it possible.

What do you hope for the future of JOTE?
"I only hope that newer people keep joining the organisation and keep the journal and the centre a radical sustainable alternative to publishing research. Both the content and institutional form of JOTE need to be radical, bottom-up, and committed to social justice. For that, I think JOTE needs to be young to be daring. In the few years since its inception, it has been the venue for great articles and special issues, like the one on the Consequences of the Scientific Reform Movement. We need more issues like this, that allow for critical self-reflexivity for those of us who want to reform (or revolutionise?) the scientific sphere.

JOTE in Conversation: Leo Tiokhin - “Criticism in Science Should be Rewarded”

Marcel Hobma — Mon, 01 Apr 2024 22:00:00 GMT

We present to you the fourth instalment of our “JOTE in conversation” series, where we ask researchers how they experience failure and success in research practices and their careers.

This article is based on a conversation had between JOTE co-founder Stefan Gaillard and Dr. Leo Tiokhin, where they delved into Leo’s interesting career path from evolutionary anthropology to metascience, and discussed the scientific evaluation platform Red Team Market, which he led during his postdoc at Eindhoven University of Technology. Leo is currently a senior researcher at the Rathenau Instituut.

Let’s start with your evaluation project. What is Red Team Market and how does it relate to the open science movement?
“Essentially, what we were trying to build was a way for people to receive criticism whenever they need it by connecting them to the right experts. Using the platform, scientists and other researchers could critically evaluate each other’s work and receive financial compensation for doing so. I think it’s very well-aligned with the open science movement because it aims to improve the efficiency and reliability of science. More generally, we also wanted to change the academic incentive system by rewarding people for providing high-quality criticism. Contrast that with the current situation, where good reviews are treated as a free service. Well-motivated criticism is valuable for improving science and should therefore be rewarded.”

How would you institutionalise this kind of platform?
“I think it is important for scientific research to ensure that there is a way to apply criticism independently of publishers or funders. We envisioned our role as an independent middleman. Reviewers often want to assess certain things that they might not know the nitty-gritty details of, for example whether the code or statistical models were correctly implemented. However, it is important that these things indeed get reviewed, so we tasked ourselves to connect these people with critics with the right expertise.

At the same time, this project was not intended as a “free market” without rules. It was regulated. We did quality control, audited critics and were developing ideas for training programmes for critics in order to improve the quality of our offering. We started with the field of psychology, but ultimately we were aiming for a large marketplace with people from different fields, and eventually a search algorithm would allow people to find the appropriate critics.”

How would it compare with the ‘traditional’ form of peer review in academia?
“Red Team Market had the potential to work as a supplement to peer review. At the time, we were having discussions with journals to see if we could do a supplemental review on a subset of papers that were controversial or where the editor simply desired an extra level of review. You could also potentially conduct Red Team “audits” by checking a random subset of papers at a journal. I suspect that this would substantially increase the quality of the papers: you don’t have to audit very often for it to change people’s behaviour.

Currently, when a journal receives a submission, they rely on a limited database of experts. Often editors need to look for the right reviewer themselves, and that’s difficult: reviewers are not publicly listed, and many will simply ignore your review request. Journals can experience substantial delays because they cannot find the right reviewers. But with a larger marketplace, you could have people set their availability, add their expertise and rates, and so streamline the review process.

So, will it be the journals that would pay for the service? Or is this also something that individual researchers could use?
“The journal is just one type of customer. You could see individual researchers use a platform like Red Team Market when they have a project that could benefit from extra feedback, but they need to have funding for it, for example via their research grant.

You could also see governments or companies use this when they are planning to base a costly decision or policy implementation on scientific research. In situations like this, you want to be as certain as possible that you're not making a mistake, which means there is substantial value in taking a small percent of your money and hiring a red team.

Weird question halfway through, but how did the idea arise? And is the name a reference to something or …?
“Yes, it is. We were inspired by red team practices used in tech companies and in governments for military strategies. In these domains, a red team is a group that critically challenges a certain plan or policy by adopting an adversarial stance. Our whole team was interested in looking for better ways to provide and receive criticism, and had all experienced frustration in our careers regarding this.

One of our co-founders even organised a bug bounty for his own work, where he put up a few thousand euros of his own money that he would pay out if people found mistakes in his paper or code. The larger the mistake, the larger the reward. Drawing on ideas like these, we wanted to see whether providing financial incentives for finding scientific errors - could feasibly be professionalised in science.

We ran a trial with a paper from one of our team members and it turned out great! The critics were happy to get paid for something that they would normally do for free, and they were all very critically minded so they enjoyed the process. He [the team member] ended up not even submitting his paper and instead conducting additional experiments, because the red team had brought up issues that he hadn’t considered.”

So how did you end up working on Red Team Market during your postdoc, and how does that relate to your academic background?
“The Red Team project was not something I had planned when starting my postdoc: it came up organically throughout the process. My background is in evolutionary anthropology, where I studied the evolution of human behaviour and psychology. During graduate school I was conducting anthropological research in Indonesia. I enjoyed my work, but at the same time the ‘replication crisis’ was growing and really getting to me. I had been in situations where I had seen questionable practices and had been involved in research that made me feel uneasy.

Honestly, I have been frustrated with how science is conducted since early in my career. During graduate school I was on the verge of leaving academia because I didn’t like the incentives. After travelling and fieldwork, I began working on a PhD proposal about testing ideas from signalling theory regarding honest and dishonest communication. But when I really dug into it, it felt somewhat trivial relative to issues surrounding reproducibility and scientific integrity. So, basically in the middle of my PhD work, I went to my advisor and said ‘I can’t do it, I need to focus on metascience’. Luckily, he was very understanding, I think partly because he shared my frustrations about academia and partly because he knew that he wouldn’t be able to convince me otherwise.

For my postdoc, I studied reward structures in various ways, for example theoretical modelling. My collaborators and I applied signalling theory from biology to academic publishing to see if we could change publishing incentives to prevent scientists from misleading journals about the quality of their submitted papers.”

Leo’s transition to Rathenau Institute and ERROR

In 2022, Leo had to stop Red Team Market due to a career change. “I was leaving academia and decided to go into data science consulting, which demanded a lot of time. Unfortunately, there was nobody else who could take the project forward.”

He didn’t stop because the concept itself was flawed, he says. The team finished some projects and had even made some profit, but this was not sufficient to hire someone on a full-time or even part-time basis to continue the project. Furthermore, the project had many aspects that still needed improvement, such as the quality control and the funding scheme.

In his new job as a senior researcher at the Rathenau Instituut, he is still actively working on improving science. “I study the Dutch scientific system so that policy makers and broader society can make better decisions about how to organise science”, he says. “One subset of research is error detection and correction, as well as creating incentives for fixing errors. Maybe it will turn out that Red Team Market can contribute to that goal.”

Furthermore, he is also part of the advisory board of ERROR, a new bug bounty program where authors can get their work reviewed. “Critics receive money for detecting errors. Also, if the authors’ papers survive the process, then they can also get a monetary reward”. The project is funded by a grant and has the additional goal of learning about where and what kind of errors occur. “It aims to incentivize people to be more critical and inspire policy makers to create more structural funding for these kinds of initiatives”.

JOTE Has a New Publishing Format: Registered Reports

Marcel Hobma — Sun, 17 Mar 2024 23:00:00 GMT

Starting from March 25, 2024, we will welcome submissions for registered reports (RRs) in our journal! We see the move to publishing RRs as part of a broader, cultural movement in academia towards more rigorous, responsible, and transparent research practice. We're proud to become another journal accepting the format.

The RR format allows researchers to submit their plans for a study for peer review prior to conducting the study (including data collection). When these 'Stage 1' research plans meet good standards for methodological and statistical soundness, and a logical linkage to existing theory, they will be granted in-principle acceptance (IPA) at JOTE. Providing the planned protocol is carried out faithfully (and any deviations are transparently, and clearly reported), the full 'Stage 2' study will be published with us.

Please note that the RR format does not 'put science in chains', nor suppress exploration, nor is it a 'prison' for your ideas, nor are they a panacea for all the problems with science. Put simply, RRs can be a powerful tool in helping authors avoid the pitfalls of hindsight and confirmation bias, and give us an opportunity to provide clear delineation between confirmatory and exploratory research in our research reports. They encourage detailed and transparent reporting, and are likely to lead to better planned studies.

Are you currently conducting original (meta-science) research - for example on the role of error, transparency, publishing culture or the reward-structures in science? Feel welcome to submit it to us in the RR-format - we look forward to seeing your Stage 1 protocol!

The RR-format will initially be included in our journal as a one-year pilot, and details of how this process will be followed at JOTE will soon be available in our author guidelines page. Until then, learn more on RRs from the Center for Open Science's explanation on the format here!

JOTE in Conversation: Michael Burke. "Why are publishing companies getting paid for our peer review?"

Marcel Hobma — Thu, 04 Jan 2024 23:00:00 GMT

After talking with Daniel Lakens about metascience, we now present the third instalment of our series ‘JOTE in conversation with researchers.’ In this series, we ask researchers how they experience 'failure' and 'success' within their research practices - whereby 'practices' can range from the research proposal and funding to experimental constraints and publication. How does failure arise in these different phases? And how do researchers in different fields and career stages deal with failure?

In this conversation, JOTE co-founder Stefan Gaillard talks about failure, peer review and interdisciplinarity with Michael Burke, professor of rhetoric at Utrecht University.

Has the academic view on failure changed throughout your career?

Absolutely. It was different when I started, and I am 55 now. Although I don’t think we’ve really talked that much about failure at the time, but it appears to be better hidden these days. Before, if a research project wouldn't work out and produce nothing to show, you'd still try to share that knowledge and even try to submit it to a journal. But these days you just don't bother and say “let's just forget it and move on”. Failure is more hidden because the system doesn't allow for it. Everybody has to be a winner – which is in itself preposterous.

Why does everybody have to be a winner? Is this tied to funding, you believe?

I think so. Departments I previously worked in were kind of well-balanced football teams. There were people who could score, good midfield players, defenders, keepers – everyone did their bit. Similarly, my departments had good teachers, good administrators, good researchers and good supervisors. But these days, everybody has to be a striker: everybody has to score goals. Imagine a football team with eleven players solely focused on scoring goals. It is nonsensical, but apparently this is supposed to work for academia.

Is there a particularly interesting instance of failure from your own career that you’d like to share?

I remember we did a large study on rhetorical strategies in the discourse of students in 2013. We took an old framework from Cicero, and subjected around 600 Dutch students to both qualitative and quantitative analysis. It took ages to squeeze something out, to get some result out of this huge project, and although it was very interesting, we ended up with nothing. So after a year we just abandoned it and never spoke about it again.

I think that had we told people about what we had done and why we had done it, the study might have been valued better. We spent a lot of time designing the study, had several outside experts contributing to the development of the design, and collected a lot of data – all of which ended up not being used. But we actually learned a lot. Methodologically but also about ourselves and how to handle disappointment. That we didn't do something with it is quite regrettable. Although the system asks: “Where is your publication? Why can't we find it in the best journal in your field?”, but the things we learned would not have been valued in this system.

I think this is part of a broader pattern in which failure has lost its importance. Years ago, I feel, researchers still saw falsification as a productive result of research. You try to disprove something, and in failing you actually succeeded by increasing the robustness of the theory. Nowadays there is no room for any of that. You have to produce something that is unique and fantastic all the time, and if you don't you might lose your job. But in reality, many scientific innovations and discoveries are preceded by a period with lots of failures and few publications.

Do you have any views on how that shift came about in the management?

I am not sure, shifts like these happen really quickly. When I was younger and still working towards becoming an associate professor, full professors were appointed based on their research output, but they were also expected to supervise PhD students and give keynote speeches at international conferences in their field. When I became a full professor in 2011, I was ready to accept this, but the reality didn't look like that at all! I had to bring in large grants for the university, try to get my name in the newspaper – or better, my face on television. So, before I knew it, the job of full professor changed to being some kind of academic celebrity and a money generator for the university.

This new role of the professor as money-generator can be related to changes in the academic publication system. It was probably Elsevier who was leading it. They are reputed to have bought this publishing model, or scheme, to fleece universities. I read a well-researched article in The Guardian about it. This model now generates 600 million a year for shareholders using public money. If I try to explain the model to anybody outside academia, they usually don't get it. And I'm not surprised, because I don't get it either!

So let me try to explain it: I'm employed by the university, the university gives me time to do research and write up my research, but it doesn't belong to me nor the university. Because I have to publish it in a journal, I don't get paid for it; I have to pay to publish it. And then the publishers refuse the university to access my research, so no colleagues or students can read it, and if the university wants it back, they have to pay a huge amount of money that goes up tremendously every year. The money then ends up being given to their shareholders. It is almost as if the government is giving millions of euros directly to private individuals. You might as well just cut out the academics and give it straight to the shareholders!

In doing so, it commercializes peer review. Peer review is the key to quality research and entails that your peers have critiqued your work and have demanded alterations and approved it. This happens for free, and that is fine; it is part of my job as an academic. But while both writers and reviewers are not getting paid, scientific publishing companies are currently getting paid for it, and that's very strange. What if we just publish our articles on the internet or some university's server, have the names of the people who review it next to it, and make it accessible to everybody around the world for free? Then the university won't have to give millions of euros to private individuals every year, and could invest more in teaching and research. Why are we not doing this currently? Surely this form of open science is a better way forward.

So you are suggesting that very substantial peer review contributions should actually count as co-authorship?

Yeah, why not? You are working together towards a final paper – although I suppose that no paper is really final, certainly not in the humanities – so why not recognize these contributions? If people are pointing out errors, failures in research designs and help find solutions, then I would consider them part of the team. If they are prepared to put their names next to your paper, it means they believe in it. I like that system.

Do you think that failure plays a different role in scholarship that is more focused on qualitative research?

I think that the concept of failure is different in qualitative methods because the outcomes in a qualitative study are less precise. Of course, you can go in different directions with your data in quantitative studies, but the expectations to come up with a precise result are higher. Quantitative research – and this especially happens a lot in the social sciences – needs to find some kind of effect in order to publish. When you find nothing in SPSS and you've tried all the drop-down menus, you've turned the whole research upside down, you are even inclined to change your hypotheses to fit something, to squeeze something out that you can actually publish.

A lot of academics don't like doing that, but when there is a system that says: “you will have to have an effect, and if not you will be a nobody, perhaps even a nobody without a job”, then you will do that, I think. So, I feel that not getting a result in quantitative research is punished more readily than in qualitative work.

I recently talked with some humanities scholars, and they told me that in the humanities, failure happens not during the execution of the study itself but in the application process. Proposals for humanities research are more often turned down than those for more social scientific work, they say.

Oh yes, sure! A lot of proposals are judged based on how they fit societal goals, like the United Nations' Sustainable Development Goals. If you are a humanities scholar who just works in the humanities and doesn't look over the walls of their own discipline, it might be hard to find a SDG that appeals to you. Maybe you can find one that appeals to you, but then you might get rejected anyhow because all humanities scholars are proposing studies aimed at the same goal.

I think that research should therefore be moving – and it is already doing so – towards an interdisciplinary and multidisciplinary approach. This system where you have humanities scholars applying for a humanities project in the humanities might be coming to its end. Since many societal problems have an interdisciplinary character, you also need people with specializations from across different disciplines to come together and find a solution.

Humanities scholars are absolutely essential within these teams. Without them there is no ethics, no effective communication and no borders or questions. There are many absurd stories about countries bombing other countries thinking they can solve local political problems, without anybody speaking the languages, and while totally disregarding the local culture and history.

Last question. How do you try to encourage more openness about failure in students or PhD candidates you supervise?

That's a good question. I do tell PhD students about the failure of other PhDs and my own failures as well. Writing a PhD is the most difficult thing you'll ever do in your life, so I use anecdotes and I tell them that it is important to fail. It's critical that you fail. The idea that everything you write will be an instantaneous success is preposterous.

If someone is working on a project at a technical university, they'll go through perhaps fifty prototypes before they actually get to the finished project, and every one of those prototypes can be seen as a step of failure – but these steps are crucial. You can't just go from model one to model fifty; you need those forty eight prototypes in between. Although this is clear in technical fields, for many other disciplines like the humanities it is expected to go from step one to step fifty in one go. And perhaps even skip step one as well. Instead of this, I would like to encourage a culture in which failure is shared and embraced. After all, failure is a stage in the road to success, a necessary stage that has to be valued as such.

National Science Communication Day: JOTE Presents on Values in Science

Marcel Hobma — Mon, 09 Oct 2023 22:00:00 GMT

Yesterday on October 9th, Stefan Gaillard (JOTE, De Nieuwe Utrechtse School), Megan Milota (UMCU, DNUS), Robert Dasović and Marcel Hobma (JOTE) organized a workshop on values in science, for the annual Dutch conference on Science Communication.

In early 2023, one of JOTE's members pointed out the existence of the annual Dutch Science Communication event in our cosy Whatsapp-group. "Isn't this a great thing for Stefan and Marcel?", Maura suggested. After all, Stefan studies promises and hypes within science communication for his PhD, and Marcel has a degree in Journalism. Furthermore, the Blog of Trial and Error itself can be seen as an example of science communication - our message of academic disruption and reflection is not only limited to an academic public, after all.

After several months of discussing, tinkering and online calls with the organisation of the event, we finally arrived at a concrete plan during the summer holidays. We asked a contribution from Megan Milota, who is assistant professor in Narrative Medicine, and enlisted the help of Robert Dasović to guide the interactive part of our session.

Coming from the perspective of ethics, history and philosophy of science, our group was in the ideal position to make our audience approach their subject matter from a novel way. Most attendees saw science as a rational process of truth-seeking, and in contrast to this we began setting out a more complete view of science: as a socially, politically and economically embedded process that contains many different values. We illustrated this view by letting the public interact with three cases.

First, it was my turn to introduce the literature on values in science by reflecting on an article I published about sex differences in clinical trials. If I would have emphasized the values in clinical research when I wrote this article, it probably would have led to more public debate and a better understanding of clinical research as a socially embedded process (for more, read my blogpost here). Next, Megan Milota introduced her research on how the use of AI in cancer diagnosis is being assessed. By making our audience reflect on a study, they realized that although using AI might lead to higher efficiency and less workload for doctors, the assessment of AI tends to neglect additional costs and the view and experience of the patient. Lastly, Stefan asked the public to take a step back and reflect on the theme of the conference (AI) as a whole: what values are embedded in this theme, and could it be that some topics in science communication receive inappropriate amounts of attention - maybe even amounting into a hype?

After one hour of discussing our cases, we had to let the audience go and enjoy the rest of their conference. Even though one hour was way too short for the interactive session we had in mind, we felt that our audience left the room with many critical questions, and a new perspective on science. Hopefully something that can be repeated next year!

Sex Differences and Clinical Research: a Reflection on Communicating About Values in Science

Marcel Hobma — Fri, 06 Oct 2023 22:03:00 GMT

This blogpost has been written as run-up to the Dutch National Science Communication Day, where JOTE-members Stefan Gaillard and Marcel Hobma, together with dr. Megan Milota and Robert Dasović present a workshop on values in science. The event is held on the 9th of October. For tickets and more information, please click here. Around three years ago, I stumbled upon a controversial and curious case of scientific dissent and decided to take it up as graduation project for my Bachelor’s in Journalism. At the time, several Dutch news media had reported about the shortage of women in clinical research on newly developed medicine, and warned that by being neglected by the pharmaceutical industry, women where disadvantaged and more likely to suffer from yet undiscovered side-effects. At first, I adopted the newspaper’s perspective and started investigating the participation of women in clinical studies, but when after two months neither evidence of structural exclusion, nor clear victims of under-researched drugs where anywhere to be found, I radically changed my approach: Could it be that the news reports where actually exaggerated?

Based on my quantitative research it sure appeared so: large scale clinical trials conducted in Europe included more women than men, and often looked at sex-differences in effects and side-effects. I started questioning the legitimacy and factuality of the newspaper reports, and found that they were driven on by an activist organisation. In an interview, the activists pointed out that women report way more side-effects from medicine than men, and that as a result, pharmacists should take sex differences as the main variable when studying effects and side-effects of new drugs. Ultimately, I adopted the perspective of the pharmacists instead: the two sexes are just one of the many relevant groups that can be researched – including children, elderly, diabetics, overweight people and more – and it is practically impossible to research a drug for every subgroup. My research culminated into an article published in the Dutch Medical Journal (NTvG), in which I sketched a dichotomy between the (objective) pharmacists and the (subjective) activists [1]. Only later, when during my Master’s in History and Philosophy of Science I joined a summer school on values in science, it dawned upon me that this frame might be unbalanced – and found that science communication as a whole could benefit from a greater emphasis on values in science.

The necessity of values in science

According to philosophers who study values in science, scientists and science communication experts should get rid of the value-free ideal - the idea that science can do without, or should be without values [2]. Instead, they argue that non-epistemic values play a crucial role in science, and raise a range of empirical studies and theoretical arguments to illustrate this point [3]. First, Helen Longino and colleagues have argued that many hypotheses, theories and study conclusions are underdetermined by their evidence [4]. Because a limited amount of evidence can never logically prove a study’s hypothesis or conclusion, there exists a logical gap that leaves room for non-epistemic value-laden assumptions to play an important role in the interpretation of evidence.

A similar gap appears when scientist or science communicator faces an (often unavoidable) risk of error in a study [5]. When there is a possibility of error in a study, how much evidence is needed to accept or reject a certain hypothesis? Values become especially relevant in answering this kind of question when being wrong can have important economical, ecological or health-related consequences. For example, the question on what amount of evidence is necessary before one can call a pesticide ‘safe’ and start applying it to crops, cannot be determined by empirical evidence or logic alone. Instead, researchers and policy makers need to make a choice based on their values: how much uncertainty is acceptable and what level of safety is deemed ‘safe enough’?

Next, philosophers have observed that many concepts and categories used in scientific research can contain values – most notably those concepts that seem to include many different definitions like well-being, intelligence or species [4]. Slight differences in the definition of a concept can contain positive or negative connotations, and have important consequences for the design and interpretation of studies. For example, when studying the effect of a medicine or food product on people’s wellbeing, it is more likely to arrive at a clear positive conclusion if one defines human well-being using specific biomarkers and intermediary endpoints, and harder when using a broader definition that includes emotional and social well-being.

Lastly, scholars have pointed out that the choice of research topic and direction are also necessarily steered by values. Again, only logic and scientific reasoning are not sufficient to determine a research agenda, as there is no such thing as an epistemically correct research agenda [5]. Values therefore play a crucial role during the whole research process - from the conception of the research question and design to the interpretation and dissemination of the results – and thus also have to be considered when writing or publishing about science.

Values and science communication

Authors within the field are predominantly concerned with the question of how to demarcate between ‘good’ values and illegitimate values, as well as finding ways to manage the latter [6]. Attempts to devise universal rules that can distinguish between these values have run into problems, and some authors now recommend to check on a case-by-case basis whether the role of values within a study is desirable or not [4]. A large share of the field, however, agrees that the values within scientific research should align with the public interest: it should reflect the needs of society, serve public-wellbeing or include democratic values. Consequently, they recommend that journalists and other communication experts should be transparent and scrutinize the values within science, and facilitate dialogue between the public and scientists. This would promote critical evaluation of values in science, align the values of scientists with that of the public, and also increase the public’s understanding of science as an essentially social process.

Nevertheless, the scholars note, a large share of science communication does not explicitly recognize the role of values in science and sticks to the value-free ideal. It portrays scientific research as a value-free truth-seeking activity in which social-political factors have no role. When values do play a role, it is often portrayed as a negative ‘bias’ that interferes with the truth-seeking process, for example in critical news reports. These articles might not recognize that science actually needs (public) values to function well, and unintentionally steer the public debate towards discussing scientific misconduct and the legitimacy of certain scientific claims. However, it has been noted that in many cases of scientific controversy it is not misconduct or disagreement about facts, but rather disagreement about values that at issue [5]. In these cases, it is more productive for public debates to focus communication efforts on the value differences between the researchers and the public.

However, some authors have also argued that transparency about values in science could also decrease people’s trust in science, while emphasizing the objective truth-seeking character of science might instead increase public trust [7]. Especially when publishing about controversial topics in which it is important to preserve the public trust, like climate change or vaccinations, being transparent about values and other social factors in the research process might actually be counterproductive. Some authors have even gone as far as to suggest using frames that are more likely to increase public acceptance of findings, while others argue that this could damage the credibility of science [4]. It is therefore recommended to choose very carefully how to be transparent about values. Communicating about values in science can have a range of different consequences, and can therefore be seen as a value-embedded act itself.

Values in clinical research

So what do these considerations hold for my article on the inclusion of women in clinical research? First, the current frame focuses on the possibility of scientific misconduct and conveys an unbalanced image of objective pharmaceutical researchers who are facing subjective activists – something that can be clearly seen in the last subheading: “Science or activism?”. The article thus upholds the value-free ideal, and does not recognize that values still play an important role in the decisions of the clinical researchers: what medicine are studied? Which illnesses are addressed? What methods are used and how many people and what kind of people are included?

Of course, there exist strict standards for clinical research that need to be complied before the local regulating agency will authorize the use and sale of a new drug, but there are still many ethical and economic values that influence the research process, and so, there is still a discussion to be had on these values. Even though my quantitative research found that enough women where included according to the medical standards, the article could perhaps have been used as opportunity to discuss these standards.

In retrospect, it might have been better to focus the article on the values of the two parties, and see where they clash in practice. The current frame focuses on the factuality of the activists’ claim, namely that women are underrepresented in clinical research, and offers the readers a clear-cut rejection of this statement. By writing about the value differences of the pharmacists and activists instead, the article could start an interesting discussion that raises questions and involve the public in the dialogue between the two parties. In addition, this approach could create more understanding for both perspectives – which could contribute to the prevention of politic polarisation – and offer the reader a better, more realistic view on the social character of clinical research.

References

[1]: Hobma, M. (2020). Worden vrouwen echt benadeeld in medicijnonderzoek. Nederlands Tijdschrift voor Geneeskunde. 2020;164:C4680

[2]: The field has roots in feminist studies on masculine values in scientific research. For a recent introduction, see the following work by Elliot, K. (2022). Values in Science. Cambridge University Press.

[3]: Within the field there exist multiple definitions of what values are. It is used to refer to the attitudes of researchers, but also defined as the interests that are being affected by researcher’s choices.

[4]: Elliott, K., (2022). Values in science. Pp.22-31

[5]: de Melo-Martín, I. (2019). The commercialization of the biomedical sciences:(mis) understanding bias. History and Philosophy of the Life Sciences, 41(3), 34.

[6]: Holman, B., & Wilholt, T. (2022). The new demarcation problem. Studies in history and philosophy of science, 91, 211-220.

[7]: John, S. 2018. ‘Epistemic Trust and the Ethics of Science Communication: Against Transparency, Openness, Sincerity and Honesty’. Social Epistemology 32:75–87.

Evaluative Measures or Game-Able Targets? Academic Competition Can Turn Even Qualitative Measures into Targets for Researchers

Marcel Hobma — Wed, 26 Apr 2023 03:58:00 GMT

Following decades of increasingly precise measurement of researchers’ scientific output and achievements, the fashion of quantifying science is losing its shine. In China, for example, the amount of published papers and citations have lost their role as key indicators of a researcher’s success – despite an expected decrease in publication rate and university rankings [1, 2]. The German research council proposed similar far-reaching reforms in their science evaluation; they want their research system to attach less value to research quantity and pay more attention to other academic activities like teaching and giving policy advice [3]. The Dutch council, taking it one step further, is removing all references to impact factors and citations from their application forms, and has now put the ‘personal narrative’ at the heart of their public funding distribution [4, 5].

The reforms seem to come in due time. Measures like citation indexes, numbers of publications and university rankings are widely criticized because they are biased and fail to accurately measure the social impact and scientific relevance of research [6-9]. More importantly, these measures tend to slowly turn into goals on their own, and lead to questionable and fraudulent practices among researchers that help them increase their rankings by ‘gaming’ the evaluation system [10-15]. Will the new quality-focused evaluation measures serve their promise and bring these issues to a halt, or will they eventually turn into gameable targets as well? I would like to suggest that the latter option can only be prevented if the harsh competition for research funding and academic jobs cools down.

Bias and Goodhart's law

Quantification started to gain hold over scientific research during the late 20th century [16-18]. The overwhelming growth of scientific publications and institutions created a demand for indexes and rankings, which was consequently enabled by new methods from the social sciences and – at a later point – increasing digitalization. Growing amounts of public funding and socio-economic expectations of science lead governments to impose a strict management- and evaluation system that guaranteed high research output and global competitiveness of national science. Since then, not a single scientific institution or agent has been left unquantified. Publication- and citation rates have become the yardstick for measuring the output of individual researchers and gained a critical role in job applications [19]. Journals were (and still are) ranked by their Impact Factor, which is based upon the amount of high-citation articles they publish, and universities are classed by the amount of prizes and highly cited researchers they can muster.

The drawbacks of this trend became only apparent in recent two decades and can be divided into two categories. First, many of the standards used to evaluate science have turned out to be unfit for measuring scientific or social relevance. Measuring citations, for example, might give a rough overview of one’s impact in the field, but citation count can be biased against narrow sub-disciplines, multidisciplinary research, articles in non-English languages and early-career authors [6, 7]. In addition, citation count often overlooks the use and references of academic works in books, reviews, online discussions, public policy documents and public science communication [8]. The Journal Impact Factor is based on this citation count and therefore inherits the same biases and blind spots, which can lead to a skewed ranking of journals [9]. Furthermore, most measures of the scientific and social impact of research do not account for mentoring abilities, teaching, public communication and other academic activities that are hard to quantify.

The second category consists of issues that arise when the measures become targets in themselves, and then shape the researchers’ culture and practices accordingly. When this happens, the measures lose their evaluative power, an effect also known as ‘Goodhart’s Law’: “When a measure becomes a target, it ceases to be a good measure” [10]. The high value attributed to publication count has for instance been linked to the practice of ‘salami-slicing’, where researchers redundantly split up an article in multiple publications [11], and it incentivizes researchers to commit questionable research practices like p-value hacking, eliminating inconvenient data or even rigging the peer-review process [12]. The emphasis on citation as a golden evaluation standard has comparable flaws. It forces young researchers to limit themselves to popular fields, topics and methods, and places a high risk - like losing one’s job - on epistemic original and innovative work that might perhaps take time before being appreciated or recognized [13, 14]. In addition, highly valuing citation counts are associated with increased rates of self-citations and lengthy reference lists [10]. Self-citation is also seen in the journals, where editors recommend and even oblige authors to cite from self-owned journals in order to raise their Impact Factor [15].

To sum up, the measures used to evaluate science are not only mediocre in measuring social and scientific impact, they are also becoming less effective in doing so because researchers are treating them as targets.

Novel measures, new targets?

Since these flaws of quantification became known to scientific community, there has been an explosion of new measures, rankings and indexes that ought to solve some of these shortcomings [18]. The widely used h-index, for example, aims to get a more balanced view of researchers’ achievements by filtering out researchers with extremely popular papers and favouring researchers who consistently publish well-cited articles – but it still preserves practically all other flaws [8]. Another popular initiative is Altmetrics, a program that maps the societal impact of academic articles by tracking their presence on social media, news outlets and blogposts. However, measuring social impact in this manner might bias articles on topics that coincide with public interests, like the health benefits of alcohol or puppies, and might be easily gameable by creating or buying false likes and mentions [20].

Could reforms like the ones recently suggested by the Chinese, German and Dutch research councils offer a solution? By also including teaching, mentoring, social-economic impact and the use of academic work in public policy and communication, the scope of the impact measured could be broadened. In addition, new quality-focused measures, like time spend on teaching or the number of Master’s thesis’s guided, could be harder to game – although this is not necessarily the case. The Dutch research council’s shift to base their funding on narratives could for instance lead to a competition that favours good story writers and give rise to agencies that specialize in writing attractive narratives [21]. The new evaluation methods might measure impact more accurately but the danger of these measures turning into new targets remains a looming threat.

It might therefore be more fruitful to ask the question why gaming practices arise and become mainstream in the first place. Why and how do measures become targets? The different suggested solutions and reforms don’t offer an answer, but instead only propose us new evaluation methods. As an alternative, I suggest we should look at the incentive structure of science and its relation with the high competitiveness within academia [12, 22].

Competition and the selection of questionable practices

It is widely recognized that the competition for research funding and academic positions has increased in the last fifty years [22]. The growing supply of PhD graduates is not matched in terms of public funding or available research and teaching positions, which intensifies the competition for these resources [22, 23]. Many graduates will be filtered out and eventually have to leave academia, which results in a strong selective pressure on norms and practices that help them score high on whatever evaluative measures are currently in fashion. Researchers who use methods or hold beliefs and values that help them secure many citations, publications or satisfy other measures, will be less likely to drop out than researchers who attach more importance to ethical or epistemic considerations [24]. Subsequently, the researchers that survive this selection process will eventually become the teachers and role models of the next generation of students and graduates. Over time, this can gradually and unintentionally alter the academic culture and normalize gaming practices like self-citation and salami-slicing, or even popularize bad methods that are more likely to find false positives.

If these questionable practices are to be countered, research institutions should not only adopt new evaluation methods, but also investigate and decrease competition in academia. For sure, new measures focused on research quality could broaden the scope of the impact that is measured and perhaps even lower academic competition, but changing only the measures will not be sufficient. Only when competition is lowered by increasing career certainty, funding possibilities and raising researchers' intrinsic motivation, the questionable practices that treat evaluative measures as targets will lose their advantage in the struggle for academic existence.

References

[1]: Huang, F. (2020). China is choosing its own path on academic evaluation. University World News.

[2]: Mallapaty, S. (2020). China bans cash rewards for publishing papers. Nature, 579(7798), 18-19.

[3]: Boytchev, H. (2021). Science council calls for ‘system reboot’ after pandemic. Research Professional News.

[4]: Woolston, C. (2021). Impact factor abandoned by Dutch university in hiring and promotion decisions. Nature, 595(7867), 462-462.

[5]: NWO. (2022). Declaration on Research Assessment. Dutch Research Council. From: https://www.nwo.nl/en/dora

[6]: Kumar, M. J. (2009). Evaluating scientists: citations, impact factor, h-index, online page hits and what else?. IETE Technical Review, 26(3), 165-168.

[7]: Thwaites, T. (2014). Research metrics: Calling science to account. Nature, 511(7510), S57-S60.

[8]: Ravenscroft, J., Liakata, M., Clare, A., & Duma, D. (2017). Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements. PloS one, 12(3), e0173152.

[9]: Neylon, C., & Wu, S. (2009). level metrics and the evolution of scientific impact. PLoS biology, 7(11), e1000242.

[10]: Fire, M., & Guestrin, C. (2019). Over-optimization of academic publishing metrics: observing Goodhart’s Law in action. GigaScience, 8(6), giz053.

[11]: Šupak Smolčić, V. (2013). Salami publication: Definitions and examples. Biochemia Medica, 23(3), 237-241.

[12]: Edwards, M. A., & Roy, S. (2017). Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition. Environmental engineering science, 34(1), 51-61.

[13]: Müller, R., & Rijcke, S. D. (2017). Thinking with indicators. Exploring the Epistemic Impacts of Academic Performance Indicators in the Life Sciences. Correction to earlier article.

[14]: Pardo Guerra, J. P. (2020). Research metrics, labor markets, and epistemic change: evidence from Britain 1970-2018. SocArXiv. January, 28.

[15]: Wilhite, A. W., & Fong, E. A. (2012). Coercive citation in academic publishing. Science, 335(6068), 542-543.

[16]: Sugimoto, C. R., & Larivière, V. (2018). Measuring research: What everyone needs to know. Oxford University Press. 7-14.

[17]: Lane, J., & Bertuzzi, S. (2011). Measuring the results of science investments. Science, 331(6018), 678-680.

[18]: Van Noorden, R. (2010). A profusion of measures: scientific performance indicators are proliferating--leading researchers to ask afresh what they are measuring and why. Richard Van Noorden surveys the rapidly evolving ecosystem. Nature, 465(7300), 864-867.

[19]: Petersen, A. M., Wang, F. & Stanley, H. E. (2010). Methods for measuring the citations and productivity of scientists across time and discipline. Physical Review E, 81(3), 036114.

[20]: Bornmann L. (2014) Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics. 2014;8(4):895–903.

[21]: Bronkhorst, X. (2022). Hoogleraar Hans Clevers waarschuwt voor “desastreuze” gevolven Erkennen & Waarderen. Digital University Magazine DUB.

[22]: Anderson, M. S., Ronning, E. A., De Vries, R., & Martinson, B. C. (2007). The perverse effects of competition on scientists’ work and relationships. Science and engineering ethics, 13(4), 437-461.

[23]: Rathenau Institute. (2021). Aanvraagdruk bij NWO, from: https://www.rathenau.nl/nl/wetenschap-cijfers/werking-van-de-wetenschap/excellentie/aanvraagdruk-bij-nwo

[24]: Smaldino, P. E., & McElreath, R. (2016). The natural selection of bad science. Royal Society open science, 3(9), 160384.

How 'Recognition and Rewards' in Dutch Academia Turned Metrics into Incentives

Martijn van der Meer — Mon, 27 Feb 2023 11:00:00 GMT

Setting the stage

Just before the Christmas break in 2022, the Advisory Council for Science, Technology, and Innovation presented an advisory letter on ‘qualities of science’ to the chair of the House of Representatives in the Netherlands. The document concluded that ‘there is no reason to expect that new ways of assessing academics would harm the international position of Dutch science.’ [1] It even celebrated pioneering efforts undertaken in the Netherlands to combine new quantitative and qualitative indicators to assess scientific quality under the banner of ‘recognition and rewards’ (in Dutch: ‘erkennen en waarderen’).

The presentation of the document marks the latest chapter of a heated, confusing, but certainly interesting debate about how science and scholarship should (not) be assessed. Those familiar with the discussion might know it as an international conversation stimulated by science funders, universities and policymakers. Yet, in the remarkable case of the Netherlands, the discussion left the protective walls of Dutch universities and entered the Dutch political arena – severely transforming the nature and scope of the discussion.

An overview of the trajectory of this debate helps unravel the confusing entanglement of the many seemingly related policy themes in discussions about research assessment. The connection between ‘recognition and rewards’ and ‘open science’, to give just one example, is not self-evident, especially to those opposed to both or either of these movements. And why criticism of reformative policies focussed primarily on the proposed abandonment of ‘quantitative’ and ‘objective’ indicators to assess quality might benefit from some explanation. From my attempt to explain the relationship between these themes, I learned that the Dutch debate about policies under the umbrella of ‘recognition and rewards’ slowly transformed into a technical discussion about the merits of quantitative versus qualitative indicators.

To me, this seems to be a missed opportunity. The past decade of discussing recognition and rewards reveals a fundamentally different view on the purpose of assessing scientific quality. Instead of finding out who is a great scientist and how this can best be measured, the discussion now centers around who meets policy goals and — only in the second instance — how this can be measured. Descriptive metrics have explicitly become ways of incentivizing researchers and steering research towards certain prescribed goals. Yet, what these goals are – what is prescribed – Is neither naturally given nor self-evident. When policymakers and administrators discuss which incentives to pick rather than the most effective metrics, the urgent question arises of what ‘we’ — citizens of a democratic, expertise-based society — expect from scientists and scientific research. With this blog, I intend to show that this side of ‘recognition and rewards’ appears to be neglected.

Act I – The problems of journal-level metrics within the academic community

On Sunday, 16 December 2012, during the Annual Meeting of the American Society for Cell Biology (ASCB) in San Francisco, a small group of prominent scientists and journal editors met to discuss ‘the scientific community’s obsession with the Journal Impact Factor (JIF).’ [2] This indicator is constructed by the number of citations a journal received in a given year for the total citable publications in the two preceding years. The resulting number suggests the popularity of articles in a periodical based on the average amount of citations every publication received in the past two years. The academics conveying at the 2012 ASCB observed that the JIF was not used to assess the popularity of journals but had become a tool for evaluating the merit of individual scholars during the recent decades. Using journal-level metrics as a proxy for academic quality in hiring, promotion, and funding decision, would result in a move away from the content of individual articles by individual researchers, they worried. [3] The conference panel, and the mail conversation that followed, resulted in what came to be known as the ‘San Francisco Declaration on Research Assessment’ (DORA) in 2013.

The authors of the DORA text point out that the Journal Impact Factor was not designed to assess the quality of researchers. Eugene Garfield (1925-2017), one of the pioneers of bibliometrics, is often credited with introducing the measuring principle behind journal-level metrics. [4] His version of the ‘Science Citation Index’ (SCI) added the ‘Journal Citation Reports’ (JCR) in 1975. This dataset included information retrieved from journal-to-journal citations and the possibility of comparing the popularity of journals based on the Journal Impact Factor. This information was marketed by the Institute for Scientific Information (currently owned by Thomas Scientific) to librarians. [5] Due to the combined rise of subscription fees, the number of new journals, and the relative decrease in science funding in the 1970s and 1980s, decisions on which journals to subscribe to became harder. [6] The JIF provided a seemingly substantiative and feasible solution to legitimise these decisions. According to Garfield, it helped librarians ‘counteract the inertia that too often prevails regarding journal selection.’ [7]

The measurement of the journal’s reputation based on the average citation every article receives seems objective because of its quantitative form. Yet, the JIF is problematic as the primary parameter for comparing institutions’ and individuals’ scientific output. Interestingly, the journal editors of reputable journals—generally academics with high standing in the field—were among the first to voice their concerns about the growing use of metrics. In 2005, Nature published an editorial acknowledging that their impact factor was influenced by only a ‘small minority of papers’. [8] This highlighted how the distribution of citations is highly skewed in most journals, meaning that the impact factor is a problematic predictor for the number of citations that publications receive. [9]

Another objection was voiced in an editorial by the editors of PLoS Medicine in 2006. They aimed to debate how editorial policies can manipulate the JIF openly. Strategies to increase citations included ‘encouraging authors to cite articles published in the journal’, ‘publishing reviews that will garner large numbers of citations’, and ‘decreasing the number of research articles published’. [10] These instances of ‘gaming’ are often explained as resulting from an indicator becoming a target, thereby foreclosing its ability to function as a good indicator. [11]

A third objection regards the discipline-specific nature of citation habits and the different average citations ‘high-impact’ publications receive. This potentially crowds out highly-specialized and less conventional fields receiving a lower amount of citations because they are not connected to big disciplines. Critics problematise using the JIF across disciplines without taking into account these differences. [12]

Within the Declaration on Research Assessment, these objections culminated into eighteen recommendations for funding agencies, institutions, publishers, organisations that supply metrics, and researchers. Its key message is directed to the discrepancy between the quality of individual research and journal-level indicators: ‘do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.’ [13] The original statement was signed by 75 academics. At the dawn of 2023, 22.674 individuals and organizations in 169 countries signed DORA.

Looking back at the five years that passed since launching the declaration, initiator Sandra Schmid observed how several funding agencies ‘instituted, strengthened, and/or made more explicit their guidelines to curtail the use of the JIF and to allow researchers to articulate the significance of their own work, through selected and annotated bibliographies.’ She also highlighted how since the presentation of DORA, researchers and journals started to develop and implement alternative measures that can indicate the quality of articles, disciplines, and individual researchers. Schmid was aware that these general developments may not be the immediate result of the declaration itself, but is still convinced that it at least contributed to an international debate on the problematic use of journal-level metrics. [14]

In more recent years, however, the focus of DORA changed. The 2018 strategic plan of the emerging community around DORA reveals a subtle yet fundamental change. Improving research assessment was now seen as ‘only a first step.’

Act II – Metrics as incentives in the arena of Dutch academic institutions

That next step required ‘changes in academic culture to ensure that hiring, promotion, and funding decisions focus on the qualities of research that are most desirable – insight, impact, reliability, and re-usability – rather than on questionable proxies.’ [15] This shift implies a fundamentally different interpretation of the goals that bibliometric indicators serve in the first place. The metrics are not necessarily aimed at measuring the quality of research output, but are instead interpreted as instruments for nudging scientists towards the most desired results.

Instead of only fixing a malfunctioning procedure, DORA now aimed to open a discussion on the rules of the game that scientists play. It reminds me of Goodhart’s law, stating that “when a measure becomes a target, it ceases to be a good measure,” but then operationalized intentionally. Bibliometric measures had not become mere descriptive indicators: they had explicitly become incentives. This interpretation of research assessment as a policy instrument during the last decade is clearly observable in the Netherlands.

A motley collection of reputable professors under the banner of Science in Transition had started to provoke and question the inward-oriented and metrics-obsessed research culture already in 2013. [16] Yet, the criticism of the DORA became institutionally embraced when the Alliance of Dutch Universities (VSNU) signed the declaration in 2014. [17] This formal and public endorsement preluded an official collaboration in 2019 with the Dutch Federation of Academic Medical Centers (NFU) and the two main science funders in the Netherlands: the Dutch Research Council (NWO) and the Dutch Health and Medical Research (ZonMW). In their 2018 press release, these institutional players announced developing and implementing a new system of ‘recognition and rewards’ in academia. [18]

The change that the VSNU, NFU, NWO, and ZonMW envisioned was threefold. Firstly, they wanted to innovate the research evaluation system to enable new ways of assessing the ‘quality and (societal) impact of research.’ This ambition was sealed by announcing the intended signature of DORA by science funders NWO and ZonMW in 2019. Secondly, these key institutions envisioned recognizing and rewarding ‘team science’ instead of ‘individual achievements’ in career paths. Finally, the collaboration aimed at in the ‘diversification of career paths’ by recognizing and rewarding individual researchers for investing time in ‘research teaching, knowledge valorization, and/or leadership.’

The coalition of universities, medical centers, and science funders not only announced to organize conferences, symposia, and round table discussions to discuss research assessment. They also aimed to change the criteria and guidelines used by review panels for funding, hiring, and promotion decisions. The VSNU, for example, announced to change its ‘strategic evaluation protocol’ (SEP) used for institutional accreditation and its ‘Job Classification System’ (UFO) to make explicit the expectations of academics and their direct institutional contexts.

The Dutch Research Council, to give another concrete example, announced to ‘incorporate expected quality and impact’ in the evaluation of researchers and funding proposals. [19] In the Netherlands, the discussion on research assessment did not solely refer to the problems of journal-based metrics that DORA initially set on the agenda. Instead, changing evaluation criteria became entangled with the answer to the normative question that formed the title of the kick-off meeting of ZonMW and NWO on ‘Recognition and Rewards’: ‘How do we envision the researcher of the future?’ [20]

An answer was formulated in a position paper called ‘Room for Everyone’s talent,’ drafted and endorsed by all Dutch universities, most research funders, and the Royal Netherlands Academy of Arts and Sciences (KNAW) in 2019. The document lists the priorities of these institutions very explicitly. Next to providing ‘academic education at the highest possible level,’ it listed ‘carrying out academic research’ and ‘having an impact on society’ as its key objectives. This required ‘leadership’ and’ sharing our academic research and education with society and making it accessible (open science).’ Moreover, the knowledge institutions and institutions stated that ‘Dutch science and academia is grounded in the principle of spanning the wide breadth of the knowledge chain. This ranges from fundamental, curiosity-driven questions to application and implementation—and back.’ [21]

To realize these changes, the document pleads for a ‘modernization of the system of recognition and rewards. This modernization should be designed to improve, in a reciprocal way, the quality of each of these key areas: education, research, impact, leadership, and (for university medical centers) patient care.’ [22] After all, ‘many academics feel there is a one-sided emphasis on research performance, frequently leading to the undervaluation of the other key areas.’

With reference to DORA, the position paper equates this narrow focus on research performance with the prominence of ‘traditional quantifiable output indicators (e.g., number of publications, h-index, and journal impact factor).’ This reliance on quantifiable indicators disrupts diversity and the societal impact of research and impedes the practice of open science. It is therefore essential to recalibrate and broaden the assessment system for research.’ [23] The position paper thereby marks another subtle but fundamental shift in research assessment: The key institutional players agree that quantitative bibliometric indicators are not only unsuitable for evaluating performance because they might incentivize undesired behavior, , but are instead inherently problematic.

That authors of the position papers were not merely playing with words, as became clear one month after publication. In December 2019, the Dutch Research Council announced implementing a ‘narrative academic profile’ for its ‘Vici-programme’—a funding scheme with grants of 1,5 million euros—after the ‘successful experimentation with the format in the Veni-scheme pre-proposal phase.’ Presented as an implementation of the 2019 position paper and the act of signing DORA, policy change requires applicants to provide ‘a narrative description of the candidate’s narrative profile.’ According to the Dutch Scientific Research Council, ‘this enables candidates to decide what is/is not important to mention in their CV.’ [24] Despite that, applicants must attach a list of ten items considered ‘key output,’ with a narrative explanation of why this is regarded as the researcher’s most important results. [25] In the pre-proposal form of the 2020 Veni-call, it becomes clear that applicants are instructed to ‘not mention H-indexes, impact factors, or any type of metric that refers to journal/publisher-impact.’ [26]

In a similar vein, Utrecht University—a member of the VSNU—announced in 2021 to formally abandon the impact factor in all hiring and promotion decisions. With respect to the new changes, ‘there are feelings of insecurity among young academics’ Paul Boselie states on behalf of the institution. ‘We feel that it’s a risk we are willing to take because we believe [the evaluation system] will change in the end.’ [27]

Act III – quantity versus quality in the public arena

Boselie was undoubtedly correct in expecting pushback. [28] The announcement that Utrecht University abandoned the Journal Impact Factor and NWO’s introduction of a narrative CV not only popularised Dutch attempts to change how academics are ‘recognized and rewarded’ at a fast pace. It also strengthened the perceived connection between policy reforms and general criticism of quantitative indicators. This criticism did not solely come from the ‘young academics’ that Boselie mentioned in his Nature interview. [29] The article in Nature inspired a heated debate on social media and in the form of published open letters.

The former president of the Royal Netherlands Society for Arts and Sciences and renowned cell biologist Hans Clevers felt the need to set up a special Twitter account to inject his strong concerns with the trend to implement alternative ways to evaluate research performance in the Netherlands. [30] ‘My university abandons quantitative output measurements and focuses solely on “collaboration” and “open science”.’ The cell biologist did not necessarily deny its relevance. Still, he objected that ‘this is unfortunately not quantifiable and therefore not objective.’ [31] As a result, Clevers stated, ‘the primary goal of science, the production of new knowledge, threatens to be no longer measured.’ [32] For the former president of the Royal Netherlands Society for Arts and Sciences, this felt like a step back in time: ‘at the start of my career, universities did not evaluate research performance. As a research intern, I worked at two places in Utrecht. Both groups had not published anything until then, and nobody seemed to care.’ [33] He remembered how in the 1980s, an evaluation method was introduced based on the number of publications per year. When researchers started gaming these assessment procedures, ‘the journal impact factor and citations became new criteria.

That system,’ according to Clevers, ‘worked quite well’. [34] The high-profile cell biologist already observed the first consequences of the new ‘recognition and rewards’ policies at Dutch universities and funding bodies: ‘this year, the positions of Dutch universities dropped sharply in the Shanghai rankings.’ [35] The Twitter thread received strong support in De Telegraaf, in a newspaper op-ed by Ronald Plasterk, former Dutch Minister of Education, Science, and Culture and professor of molecular biology. According to Plasterk: ‘Clevers is absolutely right that the most objective test of making discoveries would not be taken into account anymore in funding decisions.’ He even went one step further: ‘these forms of anti-intellectualism […] can harm the high quality of Dutch science.’ [36]

Memories of changes made to evaluation procedures in the 1970s and 1980s to determine quality based on quantitative and ‘objective’ bibliometric indicators—celebrated by Plasterk and Clevers for resulting in the top position of Dutch science—served as a central argument in a few open letters published as op-ed’s at Dutch news platform ScienceGuide. The letter titled: ‘threats to Dutch fundamental science endanger our welfare’ was written by cell biologist and associate professor Raymond Poot. ‘Dutch scientists are no longer assessed based on international, scientific, and measurable criteria, as it has been the successfully been done by the Dutch Research Council. These measures inspired by Open Science and DORA are partly removed and replaced by politically motivated criteria that can hardly be measured.’ In line with Clevers and Plasterk, Poot stated that ‘signing DORA by Dutch Universities and funders should be reconsidered’ and that ‘the Netherlands should stay in the top 5 of rankings based on international science indicators – if only for economic and societal reasons.’ After all, the Netherlands faced severe ‘socioeconomic and ecological crises.’ And according to Poot, this requires a strong knowledge economy rooted in top research. Against this backdrop, the preservation of quantitative ‘quality indicators’ and ‘selection mechanisms that assisted us for thirty years’ was essential to the author. [37]

Over a hundred prominent Dutch professors, including a Nobel Prize winner, a member of the Dutch Outbreak Management Team, and aforementioned former minister Plasterk, signed and endorsed the letter and accompanying report. A couple of days after publication, members of ‘Young Science in Transition’ and the Dutch ‘Young Academy’ published a response in favor of ‘Recognition and Rewards’ on Dutch science platform Scienceguide. In a piece titled ‘we have to abandon numeracy in science,’ the authors stated that ‘bibliometric indicators are easy and deceptively objective measures.’ Policies under the banner of ‘recognition and rewards,’ the authors claimed, ‘challenge us to evaluate each other’s publications on their quality of content, apart from quantity and venue of publications.’ [38] Both open letters show that in the Netherlands, criticism, and support of policy reform under the banner of ‘recognition and rewards’ became strongly intertwined with the use and abuse of quantitative indicators.

Act IV – The international position of Dutch science in the political arena

The criticism of policies in the name of ‘recognition and rewards’ entered a new phase when the Dutch national media outlets announced that Hans Clevers was about the leave the Netherlands to become the new director of Roche Pharma Research and Early Development on 2 February 2021. [39] The same day, he was invited to the Dutch talk show OP1 to explain his motivations. Introduced as a potential Nobel-prize laureate, Clevers presented his unanticipated ‘new challenge’ as resulting from a ‘hostile’ Dutch academic environment. In the talk show, Clevers voiced his irritation about the bureaucratic burdens of Dutch publicly-funded universities involved in the commercialization of biomedical inventions, or turning ‘public into private knowledge.’ [40]

Clevers added another dimension to this ‘academic environment’ in a much longer interview on a radio show hosted by Dutch media personality Jort Kelder about two weeks later. In the interview, later titled ‘Does science go down because of open science?’, he added his criticism on disbanding quantitative metrics intended to stimulate ‘open science’ to his sketch of an academic climate that did not support biomedical inventions. [41] A creative attempt to connect two personal irritations of the stem cell biologist Clevers. Stating that he feared a brain drain of talented young academics since it had become harder to obtain Dutch funding based on a CV stuffed with publications in high-impact factor journals. Assessing academics’ commitment to open science rather than their publications in HIF journals would damage the Dutch ‘international top position’. Clevers himself was not young anymore. Yet, his transfer to the pharmaceutical company could be considered an example of the point he tried to make.

The exodus of a ‘potential Nobel laureate’ because of a hostile academic climate, worries about a brain drain, and potential damage to the international top position of the Netherlands because of ‘open science’ presented new political stakes. In response to Clevers’ media performances, Hatte van der Woude, MP for the Dutch conservative-liberal party VVD, filed parliamentary questions for minister Robbert Dijkgraaf, a theoretical physicist, and – just like Clevers – a former president of the Royal Academy of Arts and Sciences. Dijkgraaf had just exchanged his post as director of the institute of advanced study at Princeton for a post as Minister of Education, Culture, and Science. Van der Woude asked whether Dijkgraaf was familiar with the radio show and signed letters and how he related to the trend of abandoning quantitative metrics and instead assessing scientists on their commitment to Open Science with qualitative criteria and ‘narrative CVs.’ She furthermore focussed on whether Dijkgraaf agreed that these reforms damaged the international position of Dutch science. [42] Dutch ministers are required to formally answer these questions on behalf of the Dutch government, and as such, these inquiries pushed the debate on reforming assessment to a pivotal edge. If the Minister agreed with the critics, reformative policies were about to suffer from a severe crisis of legitimacy.

Precisely the opposite happened. In straightforward language, Dijkgraaf replied that he supported the reforming assessment procedures under the banner of ‘recognition and rewards.’ He wanted to stimulate ‘open science’ as it matched his ambitions to strengthen ‘the interaction between science and society,’ as stated by the coalition agreement signed by Hatte van der Woude’s own VVD. Abandoning the ‘problematic’ Journal Impact Factor and the Hirsch Index, he explained with extraordinary precision, was in line with European trends. Such policy reforms did not harm the international reputation of the Netherlands, Dijkgraaf claimed. Instead, he noted: ‘the recent steps taken by Dutch universities enable Dutch science to strategically position itself even better.’ [43] Yet, the conservative-liberal MP was not entirely convinced: in the parliamentary committee on education, culture, and science, she demanded an ‘external and independent evaluation’ of the effects of new ‘recognition and rewards’ policies on the international position of Dutch science. [44] She even proposed an agency that had to carry out such an assessment: the Advisory Council for Science, Technology, and Innovation (AWTI).

The council’s advisory letter, titled ‘evaluating the qualities of science,’ was presented in December 2022 to the chair of the House of Representatives and concluded that they found ‘no evidence that new ways of assessing academics would harm Dutch science.’ The advisory council substantiated minister Dijkgraaf’s conviction that new ways of assessing academics reflect international developments. Most major science funders in Europe and the United States, AWTI stated, have underwritten the principles of DORA or the ‘Agreement on Reforming Research Assessment’ (RRA) from the European Commission, and some of these funders also stepped away from quantitative indicators, such as the H-index and the Journal Impact Factor. The advisory council described the pioneering position of the Netherlands within these trends, and highlighted the opportunity of shaping the international movement towards new ways of determining scientific quality. [45] Their advisory letter settled parliamentary worries about ‘recognizing and rewards’ and its connection to open science and substantiated the criticism of assessing academics solely on quantitative indicators. Ironically, political pressure inspired by public criticism of prominent scientists and the signatories of open letters published in the Dutch news platform Science Guide increased the legitimacy of using qualitative indicators as incentives for policy goals. Does this mean that ‘recognition and rewards’ has won?

Reprise: from technicalities to political deliberation

Looking back, it becomes clear how a political community of Dutch reformers united in their aim to change how research quality is evaluated, has successfully mobilized institutional and political actors over the past decade. The Dutch debate about reforming research assessment is fascinating in that it took shape along the lines of a perceived dichotomy of critics versus reformers. [46] More specifically, Dutch critics worried about abandoning quantitative indicators and the introduction of a ‘narrative-based CV.’ As such, the public and political discussion about ‘recognition and rewards’ centered on technicalities: the question of how research quality can be defined. And because critics failed in mobilizing political resistance to recent policy changes, it is tempting to frame the developments in this story as a victory of ‘qualitative’ perspectives over ‘quantitative’ ones.

Such a perspective overlooks the most impressive achievement underlying the process of making policies on ‘recognition and rewards’ uncontroversial. Namely, in the past decade, indicators to assess research(er) quality have become perceived as incentives to meet policy goals. This means that policymakers operationalized a cliché in the sociology of science: academics shape their behavior in adherence to how their performance is measured because they have an interest in how they are assessed. The political community that aim to reform how academics are ‘recognized and rewarded’ are not - at least not anymore - interested in the ‘best’ way to measure scientific quality or ‘excellence’, but in defining what scientists should do and finding indicators to assess whether scientists meet these goals.

However, what the goals of science as a collective and scientists as individuals are, is not naturally given: answering this question requires a fundamental and, indeed, a political discussion. From that perspective, the public and political discussion about recognition and rewards in 2022 is a missed opportunity. Before debating whether publications, high-impact factor journals, narrative CVs, citation counts (or whatever) are the best way to define quality, Dutch politicians, worried academics, or ‘the public’ could have discussed what they expect from academia and what values should serve as a point of departure. Is it international competitiveness? And if so, why is that necessary? Is it socioeconomic growth? Is it technology? Is it a well-informed and critical democratic society? The values and ideals underlying ‘recognition and rewards’ have become unclear as well as uncontested. If policymakers start introducing assessment criteria based on the presumption that these indicators incentivize academics to behave in specific ways, those representing Dutch citizens better deliberate what we, as citizens of a democratic expert-based society, want from our science. In the end, we should realize that ‘recognition and rewards’ is not about quantitative or qualitative indicators: it are the social and political goals of science that are at stake.

References

[1] Technologie en Innovatie Adviesraad voor Wetenschap, ‘Duiden van de kwaliteiten van wetenschap - Adviesraad voor wetenschap, technologie en innovatie’ (Adviesraad voor het Wetenschaps- en Technologiebeleid, 19 december 2022), https://www.awti.nl/documenten/publicaties/2022/12/19/index.

[2] Sandra L. Schmid, ‘Five years post-DORA: promoting best practices for research assessment’, Molecular Biology of the Cell 28, nr. 22 (1 november 2017): 2941, https://doi.org/10.1091/mbc.E17-08-0534.

[3] The PLoS Medicine Editors, ‘The Impact Factor Game’, PLOS Medicine 3, nr. 6 (6 juni 2006): e291, https://doi.org/10.1371/journal.pmed.0030291.

[4] Paul Wouters, ‘Eugene Garfield (1925–2017)’, Nature 543, nr. 7646 (maart 2017): 492–492, https://doi.org/10.1038/543492a.

[5] Vincent Larivière en Cassidy R. Sugimoto, ‘The Journal Impact Factor: A Brief History, Critique, and Discussion of Adverse Effects’, in Springer Handbook of Science and Technology Indicators, onder redactie van Wolfgang Glänzel e.a., Springer Handbooks (Cham: Springer International Publishing, 2019), 3–24, https://doi.org/10.1007/978-3-030-02511-3_1.

[6] Aileen Fyfe e.a., ‘Untangling Academic Publishing: A history of the relationship between commercial interests, academic prestige and the circulation of research’ (Zenodo, 25 mei 2017), 8, https://doi.org/10.5281/zenodo.546100; John B. Thompson, Books in the digital age: the transformation of academic and higher education publishing in Britain and the United States (Polity, 2005); Kimberly Douglas, ‘The Serials Crisis’, The Serials Librarian 18, nr. 1–2 (2 augustus 1990): 111–21, https://doi.org/10.1300/J123v18n01_08.

[7] Eugene Garfield, Journal citation reports (Citeseer, 1991); Larivière en Sugimoto, ‘The Journal Impact Factor’, 4.

[8] ‘Not-so-Deep Impact’, Nature 435, nr. 7045 (juni 2005): 1003–4, https://doi.org/10.1038/4351003b.

[9] Per O. Seglen, ‘Why the impact factor of journals should not be used for evaluating research’, Bmj 314, nr. 7079 (1997): 497.

[10] Editors, ‘The Impact Factor Game’, 0707.

[11] Mario Biagioli en Alexandra Lippman, Gaming the metrics: misconduct and manipulation in academic research (Mit Press, 2020), 1; Marilyn Strathern, ‘‘Improving ratings’: audit in the British University system’, European review 5, nr. 3 (1997): 305–21.

[12] Jerome K. Vanclay, ‘Impact Factor: Outdated Artefact or Stepping-Stone to Journal Certification?’, Scientometrics 92, nr. 2 (1 augustus 2012): 214, https://doi.org/10.1007/s11192-011-0561-0.

[13] ‘Read the Declaration’, DORA (blog), geraadpleegd 20 december 2021, https://sfdora.org/read/.

[14] Schmid, ‘Five years post-DORA’, 2942.

[15] ‘DORA Roadmap: A Two-Year Strategic Plan for Advancing Global Research Assessment Reform at the Institutional, National, and Funder Level’, DORA (blog), 27 juni 2018, https://sfdora.org/2018/06/27/dora-roadmap-a-two-year-strategic-plan-for-advancing-global-research-assessment-reform-at-the-institutional-national-and-funder-level/.

[16] Frank Miedema, ‘Science in Transition How Science Goes Wrong and What to Do About It’, in Open Science: The Very Idea, onder redactie van Frank Miedema (Dordrecht: Springer Netherlands, 2022), 67–108, https://doi.org/10.1007/978-94-024-2115-6_3; ‘Science in Transition Conference — KNAW’, geraadpleegd 21 december 2021, https://www.knaw.nl/nl/actueel/agenda/science-in-transition-conference.

[17] ‘Impact niet langer leidend’, ScienceGuide, 4 december 2014, https://www.scienceguide.nl/2014/12/impact-niet-langer-leidend/.

[18] ‘Instellingen en financiers gaan wetenschappers anders belonen en waarderen’, ScienceGuide, 26 november 2018, https://www.scienceguide.nl/2018/11/anders-belonen-en-waarderen/; VSNU e.a., ‘VSNU, NWO, NFU en ZonMw geven impuls aan verandering in het waarderen en belonen van wetenschappers’, geraadpleegd 21 december 2021, https://www.scienceguide.nl/wp-content/uploads/2018/11/Statement-Waarderen-en-Belonen-van-Wetenschappers_vDEF-2.pdf.

[19] ‘Instellingen en financiers gaan wetenschappers anders belonen en waarderen’.

[20] ‘‘Hoe maak je een systeem waarvan niet iedereen opnieuw gek wordt?’’, Mare Online, geraadpleegd 21 december 2021, https://www.mareonline.nl/achtergrond/hoe-maak-je-een-systeem-waarvan-niet-iedereen-opnieuw-gek-wordt/.

[21] VSNU e.a., ‘Room for everyone’s talent: towards a new balance in recognising and rewarding academics’ (The Hague, november 2019), 3, https://recognitionrewards.nl/wp-content/uploads/2020/12/position-paper-room-for-everyones-talent.pdf.

[22] VSNU e.a., 3.

[23] VSNU e.a., 4.

[24] ‘NWO Talent Programme | Vici - Health Research and Medical Sciences | NWO’, geraadpleegd 21 december 2021, https://www.nwo.nl/en/calls/nwo-talent-programme-vici-health-research-and-medical-sciences; ‘NWO | NWO voert narratief CV door in Vici-ronde 2020’, geraadpleegd 21 december 2021, https://www.nwo.nl/nieuws/nwo-voert-narratief-cv-door-vici-ronde-2020.

[25] ‘NWO | DORA’, geraadpleegd 21 december 2021, https://www.nwo.nl/dora.

[26] NWO en ZonMW, ‘Grant application pre-proposal form 2020 : NWO Talent Programme – Veni scheme’, 5, geraadpleegd 21 december 2021, https://www.nwo.nl/sites/nwo/files/media-files/EXAMPLE%20Veni%202020%20Pre-proposal%20Form%20with%20expanded%20-%20AES%20SSH%20ZonMw.pdf.

[27] Chris Woolston, ‘Impact Factor Abandoned by Dutch University in Hiring and Promotion Decisions’, Nature 595, nr. 7867 (25 juni 2021): 462–462, https://doi.org/10.1038/d41586-021-01759-5.

[28] ‘Banning Journal Impact Factors Is Bad for Dutch Science’, Times Higher Education (THE), 3 augustus 2021, https://www.timeshighereducation.com/opinion/banning-journal-impact-factors-bad-dutch-science.

[29] ‘Nieuwe Erkennen en Waarderen kan kwetsbare groepen schaden’, ScienceGuide, 27 juli 2021, https://www.scienceguide.nl/2021/07/nieuwe-erkennen-en-waarderen-kan-kwetsbare-groepen-schaden/.

[30] ‘Oud-KNAW-president ernstig bezorgd over Erkennen en Waarderen, maar minister enthousiast’, ScienceGuide, 8 november 2021, https://www.scienceguide.nl/2021/11/oud-knaw-president-ernstig-bezorgd-over-erkennen-en-waarderen-maar-minister-enthousiast/.

[31] Hans Clevers NL, ‘Mijn universiteit stopt kwantitatieve output meting en richt zich enkel op de thema’s “samenwerking” en “open science”. Belangrijk, maar dat zijn bijv mentorschap, good citizenship, talent en passie ook. Allemaal relevant, maar helaas niet kwantificeerbaar en dus niet objectief.’, Tweet, @HansCleversNL (blog), 7 november 2021, https://twitter.com/HansCleversNL/status/1457382003242897412.

[32] Hans Clevers NL, ‘Toevoeging van deze criteria aan het bestaande evaluatiesysteem zou uitstekend zijn. Maar E&W lijkt een andere weg te kiezen: het primaire doel van wetenschap, het daadwerkelijk genereren van nieuwe kennis, dreigt niet meer kwantitatief gemeten te worden.’, Tweet, @HansCleversNL (blog), 7 november 2021, https://twitter.com/HansCleversNL/status/1457385667789529094.

[33] Hans Clevers NL, ‘Nederland wil via ‘Erkennen en Waarderen’ het personeelsbeleid (onderwijs, wetenschap EN management) aan de universiteiten moderniseren via een radicale wijziging van de wetenschappelijke evaluatiesystematiek. NWO loopt hierin mee. Mijn grote zorgen hierover:’, Tweet, @HansCleversNL (blog), 6 november 2021, https://twitter.com/HansCleversNL/status/1457017307096625157.

[34] Hans Clevers NL, ‘Er kwam een evaluatiemethodiek: het aantal publicaties/jaar. Dit werkte redelijk, maar wetenschappers pasten hun publicatiestrategie aan: liever drie kleine papers dan een mooie. Journal impact factor en citaties werden toen het nieuwe criterium. Dit systeem werkte aardig goed’, Tweet, @HansCleversNL (blog), 6 november 2021, https://twitter.com/HansCleversNL/status/1457022284191969284.

[35] Hans Clevers NL, ‘Maar anderen doen dat wel: Dit jaar zakten de Nederlandse universiteiten fors in de Shanghai ranking, en enkele vielen zelfs uit de top 100. Deze ranking is niet op reputatie gebaseerd, maar exclusief op kwantitatieve output parameters. We kunnen daar onze schouders over ophalen.’, Tweet, @HansCleversNL (blog), 7 november 2021, https://twitter.com/HansCleversNL/status/1457386237946474499.

[36] ‘Anti-intellectualisme kan de Nederlandse wetenschap in gevaar brengen’, Telegraaf, 11 november 2021, https://www.telegraaf.nl/watuzegt/2016592458/anti-intellectualisme-kan-de-nederlandse-wetenschap-in-gevaar-brengen.

[37] ‘Bedreigingen voor fundamenteel wetenschappelijk onderzoek in Nederland brengen onze toekomstige welvaart in gevaar’, ScienceGuide, 9 december 2021, https://www.scienceguide.nl/2021/12/bedreigingen-voor-fundamenteel-wetenschappelijk-onderzoek-in-nederland-brengen-onze-toekomstige-welvaart-in-gevaar/.

[38] gastauteurs, ‘We moeten af van telzucht in de wetenschap’, ScienceGuide, 21 juli 2021, https://www.scienceguide.nl/2021/07/we-moeten-af-van-telzucht-in-de-wetenschap/.

[39] ‘Topwetenschapper Clevers stapt over naar farmaconcern Roche’, FD.nl, geraadpleegd 6 januari 2023, https://fd.nl/bedrijfsleven/1428836/topwetenschapper-clevers-stapt-over-naar-farmaconcern-roche; ‘Nederlandse wetenschapper Hans Clevers gaat onderzoeksafdeling Roche leiden’, 2 februari 2022, https://nos.nl/artikel/2415388-nederlandse-wetenschapper-hans-clevers-gaat-onderzoeksafdeling-roche-leiden.

[40] Hans Offringa, ‘Hans Clevers: “‘Wetenschap top in Nederland, aantal bedrijven buitengewoon armoedig’”’, Op1 (blog), 3 februari 2022, https://op1npo.nl/2022/02/03/hans-clevers-is-benoemd-tot-het-hoofd-van-de-onderzoeksafdeling-van-het-zwitserse-roche/.

[41] ‘dr Kelder en Co: Gaat de wetenschap ten onder aan open science? & Gezag in plaats van macht, alstublieft - 19 februari 2022 - NPO Radio 1 Gemist | NPO Radio 1’, 19 februari 2022, https://www.nporadio1.nl/uitzendingen/dr-kelder-en-co/28bd11da-a771-482b-b2bd-52045f2adaf9/2022-02-19-dr-kelder-en-co-gaat-de-wetenschap-ten-onder-aan-open-science-gezag-in-plaats-van-macht-alstublieft.

[42] Hatte Woude, van der, ‘De podcast “Gaat de wetenschap ten onder aan open science?”’ (2022), https://www.tweedekamer.nl/kamerstukken/kamervragen/detail.

[43] Cultuur en Wetenschap Ministerie van Onderwijs, ‘Antwoorden op Kamervragen over de podcast over wetenschap en open science - Kamerstuk - Rijksoverheid.nl’, kamerstuk (Ministerie van Algemene Zaken, 30 maart 2022), https://www.rijksoverheid.nl/documenten/kamerstukken/2022/03/30/antwoorden-op-kamervragen-over-de-podcast-gaat-de-wetenschap-ten-onder-aan-open-science. (My italics).

[44] ‘Verslag van een notaoverleg, gehouden op 11 april 2022, over hoofdlijnenbrief Hoger onderwijs en wetenschapsbeleid’, kamerstuk, 11 april 2022, https://www.tweedekamer.nl/kamerstukken/commissieverslagen/detail; ‘Motie Van der Woude/Van der Graaf over een eenduidige visie op maatschappelijke impact, waaronder valorisatie - Hoger Onderwijs-, Onderzoek- en Wetenschapsbeleid - Parlementaire monitor’, kamerstuk, 11 maart 2022, https://www.parlementairemonitor.nl/9353000/1/j9vvij5epmj1ey0/vls0qdwicbyg.

[45] Adviesraad voor Wetenschap, ‘Duiden van de kwaliteiten van wetenschap - Adviesraad voor wetenschap, technologie en innovatie’.

[46] ‘Verslag van een notaoverleg, gehouden op 11 april 2022, over hoofdlijnenbrief Hoger onderwijs en wetenschapsbeleid’, 4.

Not All That Shines Is Diamond: Why Open Access Publication Favors Rich Authors, Prestigious Universities and Industry-Funded Research

Marcel Hobma — Fri, 02 Dec 2022 12:00:00 GMT

In recent years, it has become increasingly common for researchers to publish their work in Open Access by paying article processing costs to the publisher [1, 2]. Before the digital revolution, academic publishing was mostly subscription-based and university libraries paid publishers at regular intervals for large bundles of journals. Every physical copy of a journal came with its own production and distribution costs, making Open Access an unrealistic pursuit. When academic research was digitalized and the costs of copying and disseminating research lowered dramatically, the Open Access movement gained momentum and at least four ways of Open Access (OA) publishing joined the old subscription model [3]. Authors can now publish their studies in subscription-based Green OA journals, which allows them to republish their work on large preprint servers such as ArXiv and in freely accessible institutional repositories managed by university libraries. A second option is to publish in full Open Access, peer-reviewed journals that rely on author-paid article processing costs to maintain a steady source of income. Diamond OA journals like the Journal of Trail and Error also publish in full Open Access, but don't charge the authors with any costs. Lastly, there exists the option to publish in commercial Hybrid journals that combine the subscription model with Open Access publishing.

Article processing costs allow researchers to publish Open Access articles in well-edited and prestigious journals, which is the main reason for authors and their funders to pay these costs. Open access is often portrayed as essential to the transparent and cooperative nature of science, but also aims to circumvent the high paywalls raised by commercial publishers that limit the access of research, and therefore facilitate the dissemination of valuable knowledge [4-6]. However, the promises and advantages of the author-paid funding mechanism also come with a downside in the form of publication bias. Not every author or institution might be able or willing to pay article processing costs if they are too high and this could lead to selective publishing practices that favour certain groups of researchers, institutions and research topics.

To analyse these possible biases, it is instrumental to look at the actual costs of publishing in Open Access journals. The highest article processing costs can be found in the commercial publishers’ hybrid journals, where the old subscription model is the norm and where researchers are offered the option to publish in Open Access from $1,000 to $5,000 [7, 8]. Since Open Access is not the primary publishing option in these journals, the publisher doesn’t rely on this stream of income and can ask relatively high prices – which results in only a small percentage of its authors choosing the Open Access option [1]. The lion share of Open Access research can therefore be found in the Gold and Diamond OA journals, which are published by academic societies as well as commercial publishers. The Diamond OA journals are mostly funded by the institutions themselves and don’t request any article processing costs, but a growing part of the OA journals are Gold and do demand a contribution from the author. The processing charges in these journals loosely range from $500 to $2000 [7, 9], and have been increasing steadily in recent years [1, 10]. Furthermore, it is important to note that the article processing costs of both types of journals do not represent the actual costs of processing and publishing articles, and are correlated with the publisher's market power [11], as well as the impact and prestige of the journal and the average amount of citations [7, 8, 10]. Specifically, journals with a higher Journal Impact Factor (JIF) and a registration in the Directory of Open Access Journals (DOAJ) tend to set significantly higher article processing charges [8, 12].

The high publication charges limit authors to choose their ideal journal if they lack the economic resources to pay these costs. In this case, they are restricted to enclosed subscription-based journals, or journals with lower impact that offer free or cheap Open Access. Since there are many systemic differences in the economic resources of researchers, the article processing costs can lead to different kinds of publication bias. On the largest scale, one can distinguish a geographical bias in which research and authors from developing countries are disadvantaged [2]. When publishing a study, the article processing costs are often covered by the research funder or the author’s research institution, and in Europe most research funders even oblige the author to publish in Gold or Green Open Access. However, in many other countries this is not the case. Studies show that 39% of the processing costs of research in low-income nations is funded by the authors [8], compared to 12% in Europe [1]. In addition, authors from these countries are less likely to publish in hybrid journals [13] and publish relatively more often in journals with lower impact factors [14]. Although some journals recognize this geographical inequality and offer fee waivers or discounts to authors from developing nations, they remain the exception and only a small percentage of authors benefit from these measures [15].

Inequality can also take place on the institutional level. Different research institutions and research funders can vary in the way they help authors fund publishing costs – if they fund them at all – and this can give certain authors and disciplines an advantage. In an American study, for example, it was found that researchers are more likely to publish in Open Access if they are male, more advanced in their career or employed at a prestigious university [16]. Similarly, a study on articles published in the field of Global Health studies shows that authors working at low-ranking universities are more likely to publish in subscription-based, closed journals, while authors linked to higher ranking universities and hospitals pay relatively more and higher processing costs [17]. This shows that researchers working for wealthy institutions are offered more resources to publish in expensive, high impact journals, which contributes to the development of unhealthy hierarchies between authors and between institutions.

Even at a larger disadvantage are authors that do not have a research-funding institution backing them at all, like retired researchers, students, researchers affiliated to certain universities of applied sciences, and especially young and mobile researchers who are in between jobs. In a Danish study [15] it was found that 30% of publishing authors in Danish journals fall within this category and therefore struggle with paying article processing costs. Since most publishers don’t offer grants or fee waivers to this group, the study suggests that article processing costs limit the academic freedom of these authors and create the risk that valuable research will remain unpublished.

Besides deepening the inequality among researchers, the selective publication practices also influence the contents of a discipline’s body of research. Well-funded organizations and industries have better access to (high impact) Open Access journals [13] and reach more people than research paid for with personal funds [8], but at the same time these institutions are often inclined to fund research into specific sub-disciplines and topics. Certain sub-disciplines, basic research, innovative views or controversial research might therefore become less visible in the field. Notably, this process can amplify existing biases and malpractices in industry-funded research. The industrial research agenda primarily contains topics that are beneficial to the industry itself while leaving other topics untouched, resulting in selective presence and gaps of knowledge. This is well-illustrated by Coca-Cola’s many research programs on health and physical activity, which obscure the role of calorie intake in childhood obesity and physical wellbeing [18, 19]. Additionally, industry-funded research in food and pharmaceutical sciences are known to have higher rates of sponsor-favoured outcomes than studies with other sponsors [20]. This can be explained by publication bias – industries are less likely to publish negative results – or confirmation bias, where the researcher is (unconsciously) biased towards finding positive results for its sponsor. Either way, when economic resources become a determining factor in whether a study will be published in a high-impact Open Access journal or not, these studies will gain a larger reach and help shift the field of knowledge towards the interest of industries and other resourceful organizations – and possibly away from public interests.

Lastly, one can wonder what article processing costs would entail for the publication of negative or non-significant findings, failed studies with methodological errors and failed replication studies. The publication of these studies is seen as an import part of combating the ongoing replication crisis in the social sciences and other fields [23]. However, since funding agencies generally don't see these studies as valuable and because they aren't directly helpful to a researcher’s career or citation ratings, the article processing costs could form an extra barrier to publish them in Open Access – and even prevent them from being published at all. The costs could therefore reinforce the well-known file-drawer problem, but it appears that this possibility hasn’t been investigated yet and clear evidence or counter-evidence is lacking.

Over the whole, publishing in Open Access adds transparency to fields of research and allows authors to spread their work more effectively, but high and variable publishing costs for authors might have unforeseen consequences. There is evidence that article processing costs give older, more resourceful male researchers and prestigious institutions an advantage over authors from developing countries and early-career authors that are not backed by strong institutions. Article processing costs increase the role of economic resources in access to high-impact Open Access journals, and can lead to stratification between researchers and among research institutions. Consequently, this process can amplify certain publication biases that favour topics and viewpoints that are backed by rich organisations and industries, and therefore distort certain fields of research and possibly steer scientific research away from public interests. Research into the range and impact of these effects is still sparse and should be expanded. However, current evidence already shows that it is essential to conceive of new ways to fund systemic Open Access publishing that offers a level playing field to all authors and institutions.

References:

[1]: Lund, A., & Zukerfeld, M. (2020). Corporate Capitalism's Use of Openness: Profit for Free?. Springer Nature.

[2]: Solomon, D. J., & Björk, B. C. (2012). A study of open access journals using article processing charges. Journal of the American Society for Information Science and Technology, 63(8), 1485-1495.

[3]: Bjork, B. C., & Korkeamaki, T. (2020). Adoption of the open access business model in scientific journal publishing: A cross-disciplinary study. arXiv preprint arXiv:2005.01008.

[4]: Davis, P. M., Lewenstein, B. V., Simon, D. H., Booth, J. G., & Connolly, M. J. (2008). Open access publishing, article downloads, and citations: randomised controlled trial. BMj, 337.

[5]: Eysenbach, G. (2006). Citation advantage of open access articles. PLoS biology, 4(5), e157.

[6]: Norris, M., Oppenheim, C., & Rowland, F. (2008). The citation advantage of open‐access articles. Journal of the American Society for Information Science and Technology, 59(12), 1963-1972.

[7]: Björk, B. C., & Solomon, D. (2012). Open access versus subscription journals: a comparison of scientific impact. BMC medicine, 10(1), 1-10.

[8]: Solomon, D. J., & Björk, B. C. (2012). Publication fees in open access publishing: Sources of funding and factors influencing choice of journal. Journal of the American Society for Information Science and Technology, 63(1), 98-107.

[9]: Ware, M., & Mabe, M. (2015). The STM report: An overview of scientific and scholarly journal publishing.

[10]: Björk, B. C., & Solomon, D. (2017). Developing an effective market for open access article processing charges.

[11]: Budzinski, O., Grebel, T., Wolling, J., & Zhang, X. (2020). Drivers of article processing charges in open access. Scientometrics, 124(3), 2185-2206.

[12]: Siler, K., & Frenken, K. (2020). The pricing of open access journals: Diverse niches and sources of value in academic publishing. Quantitative Science Studies, 1(1), 28-59.

[13]: Liyanage, S. S., & Raina Macintyre, C. (2006). Do financial factors such as author page charges and industry funding impact on the nature of published research in infectious diseases?. Health Information & Libraries Journal, 23(3), 214-222.

[14]: Ellers, J., Crowther, T. W., & Harvey, J. A. (2017). Gold open access publishing in mega-journals: Developing countries pay the price of western premium academic output. Journal of scholarly publishing, 49(1), 89-102.

[15]: Burchardt, J. (2014). Researchers outside APC-financed open access: Implications for scholars without a paying institution. SAGE open, 4(4), 2158244014551714.

[16]: Olejniczak, A. J., & Wilson, M. J. (2020). Who’s writing open access (OA) articles? Characteristics of OA authors at Ph. D.-granting institutions in the United States. Quantitative Science Studies, 1(4), 1429-1450.

[17]: Siler, K., Haustein, S., Smith, E., Larivière, V., & Alperin, J. P. (2018). Authorial and institutional stratification in open access publishing: the case of global health research. PeerJ, 6, e4269.

[18]: O’Connor, A. (2015). Coca-Cola funds scientists who shift blame for obesity away from bad diets. New York Times, 9(08), 2015.

[19]: Serôdio, P. M., McKee, M., & Stuckler, D. (2018). Coca-Cola–a model of transparency in research partnerships? A network analysis of Coca-Cola’s research funding (2008–2016). Public health nutrition, 21(9), 1594-1607.

[20]: Lundh, A., Lexchin, J., Mintzes, B., Schroll, J. B., & Bero, L. (2017). Industry sponsorship and research outcome. Cochrane database of systematic reviews, (2).

[21]: Fabbri, A., Holland, T. J., & Bero, L. A. (2018). Food industry sponsorship of academic research: investigating commercial bias in the research agenda. Public health nutrition, 21(18), 3422-3430.

[22]: Lexchin, J., Bero, L. A., Djulbegovic, B., & Clark, O. (2003). Pharmaceutical industry sponsorship and research outcome and quality: systematic review. bmj, 326(7400), 1167-1170.

[23]: Wiggins, B. J., & Chrisopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202.

Can Direct Replications Save Psychology From Its Conceptual and Measurement Issues?

Marcel Hobma — Fri, 28 Oct 2022 12:07:00 GMT

Since the reproducibility crisis in psychology and other sciences started a decade ago, direct replication has gained more and more traction as a valuable epistemic tool. Before the crisis, experiments in psychology were mostly aimed at building upon previous studies instead of verifying them [1]. Researchers generally didn’t see direct replications of experiments as valuable additions to the scientific process, and replications were regularly rejected by journals [2] [See 3 for an example]. This changed when researchers discovered that many psychological studies actually failed to replicate statistically significant results. Scientists generally believe that, for a phenomenon to be true, exact replications of an experiment should be able to consistently reproduce it. However, in psychology less than half of these replication attempts manage to do so [4]. Due to this lack of reproducibility, researchers started to doubt the existence of once prominent theoretical constructs like social priming [5] and ego depletion [6]. The direct replication became an invaluable tool for verifying study results and psychological concepts, and turned into an important yardstick for the overall health of the discipline.

Besides gauging the scope and severity of the reproducibility crisis, direct replications are also seen as one of its solutions. The lack of reproducible studies is most commonly attributed to questionable research methods, fraud, publication bias, small sample sizes and sloppy statistics [1]. By conducting exact replications of existing studies, reformers hope to increase the statistical power of earlier studies and cleanse the field from some of its bad apples. Other proposed solutions to the crisis, like promoting open data, increased statistical rigour, larger sample sizes and preregistration of studies, have also been implemented with the aim of improving replication efforts and success. However, by focusing mainly on direct replications and statistics, one might overlook other issues that underlie the crisis. Several philosophers and social scientists have pointed out that the discipline’s lack of validated measures and consistent theoretical concepts with a clear scope also contribute to the low reproducibility rates. These conceptual issues cannot be solved by mere methodological improvements, they argue, and even question the method of replication as a meaningful concept. Should reformers perhaps reconsider the dominant role of direct replications?

A first point of critique is that direct replications cannot possibly exist in practice. Researchers can never recreate the exact same situation as in the original experiment because they need a new sample of participants that haven’t experienced the experiment before. In addition, practically all replications necessarily take place at a later time than the original experiment [7]. Even within a few months, the norms, values, knowledge, meaning of words and phrases, and other cultural variables in the population could have changed, possibly diluting the results of the ‘direct’ replication. This might seem trivial and well-known to researchers, but it is important to consider how these subtle differences can have an impact on replication studies.

Because perfect replications are not realistic, researchers generally replicate only the factors that are relevant to the experimental setup and the theory that is being tested. However, and this is a second critique, what factors are deemed relevant or irrelevant to a specific experiment or theoretical concept depends on the judgement of the individual researcher. According to philosopher Uljana Feest [8], individual researchers often hold different notions of what theoretical concepts mean, what factors influence them and how they can be measured. These individual judgements are not always explicitly mentioned in psychological studies and can cause problems when researchers replicate an experiment. When a researcher conducts an original experiment, she will use her theoretical understanding of the issue at hand to describe all the factors that seem relevant to her. An experiment that tests the correlation between being outdoors in nature and intelligence, for example, could include the weather, season, and temperature of the test location, as well as a standardized IQ test and the demographics of the participants. However, this description might already harbour many assumptions that other researchers might not have. The researcher might assume that factors like the hour and day of the week, the population density of the test location, and the personal preferences of the participants are not relevant for her theory, and consequently fail to describe them. A second researcher may have completely different assumptions on how to understand and measure intelligence, and on what ‘being outdoors in nature’ might exactly constitute, which could lead to relevant differences in the replication of the original experiment.

A related problem is that many psychological concepts are vague and the methods that ought to measure them unvalidated. Psychology uses countless of tests and other methods to measure concepts like emotions, intelligence, skills, values and personality traits, but before one can use such a measure, one must confirm that the method works as intended [9]. If a researcher wants to study intelligence, for example, he should first clearly define what construct of intelligence he is going to address. Does he want to study emotional intelligence, spatial-temporal intelligence or the capacity for logical reasoning? Then, he has to find or create a measure that has been found to reliably represent the variation in the concept he wants to study – if the concept is quantifiable at all in the first place [10]. Only when both the concept and measure are clearly defined and validated, experiments can lead to valid and reliable results.

In reality however, many studies in psychology often seem to lack clear concepts and validated measures. A recent review by Flake and colleagues [11] shows that a significant number of studies in personality science and social psychology do not include evidence on whether their measures actually measure the construct they believe they are studying. According to Flake [12], these ‘questionable measurement practices’ are common in psychology: it is often not clear what exact constructs are used, how they are measured and why the researcher adopted these specific methods. In the last decades, more and more concepts and measures have been brought into existence by psychologists and it is theorized that this reflects the inability of the discipline to reject invalid measures and concepts [9].

The validity issues surrounding psychological concepts and measures underscore the gravity of the third critique of direct replications; which is that direct replications don’t inform the researcher about the mechanism that underlies the effect being studied [7 and 8]. Direct replications can be informative on whether a certain experimental finding is reliable, and tell us if a random error occurred in the original experiment, but they do not help develop the concepts or significantly increase the support for the underlying theory. An experiment could be poorly designed, and use vague concepts and invalid measures, but nevertheless succeed in replication. For example, by replicating the study on being outdoors in nature and intelligence, one could discover that there is a positive relation between these concepts, but it remains uncertain which element(s) of the experiment led to this effect. A direct replication might appear like the confirmation of an effect, but it obscures that our understanding of the actual concepts and mechanisms at work might still be wrong. Merely improving statistical methodology and increasing the replicability of studies is therefore not sufficient when solving the fundamental issues at hand.

The inability of direct replications to address conceptual and measurement issues is illustrated well by Lurquin & Miyake’s review of the literature on ego-depletion [13]. This theory hypothesizes that individuals have a limited resource of self-control, but researchers haven’t been able to reproduce the original studies that once discovered it. According to the authors, this is not something that direct replication and increased reproducibility alone could solve. They write that the literature on ego-depletion suffers from a ‘conceptual crisis’: the field has no clear definition of self-control and how it could be measured, and the tests that are used have not been independently confirmed to actually measure self-control.

If direct replications cannot tackle these conceptual and methodological issues underlying the reproducibility crisis, then how should they be addressed? Some authors, like Stroebe & Strack [7] and Crandall & Sherman [14] argue that we should shift our attention from direct to conceptual replications. In conceptual replications, a specific theory or concept is operationalized using a different measure or change in experimental setup, which could increase the confidence of the hypothesis tested and help establish the generalizability of the theory. However, conceptual replications are also criticized because of their susceptibility to publication bias and questionable research practices – and therefore linked to the origin of the replication crisis itself [1 and 15]. While failed conceptual replications don't offer researchers much new information, a successful conceptual replication is generally regarded as interesting and publishable because it confirms or extends an existing theory or effect. The difference in how the results of these studies are valued, can lead to selective publishing of positive findings and a lack of incentives to critically investigate existing studies and concepts. The resulting proliferation of many, poorly researched theories and concepts is further amplified by the high pressure of publishing in academia and the high researchers’ degree of freedom when designing conceptual replications, which makes these studies vulnerable to malpractice.

For Feest, the lack of clear concepts and ways to measure them makes it hard to value conceptual replications – regardless of whether they succeed or fail. A conceptual replication might succeed in achieving the same result as the original study, but the researchers wouldn't know what variables or mechanism could have led to this result. Conversely, a conceptual replication might fail because of the changes that the researchers made to the experimental setup, problems with the original experiment or because the effect studied is less generalizable or coherent as previously thought. Instead of replication, Feest argues that researchers should focus – and in practice they often do - on what she calls ‘exploratory work’. By critically probing the concepts and phenomena that are studied, researchers narrow down the exact scope of these concepts and weed out erroneous assumptions.

Feest’s suggestions seem to match with the recommendations made in the ego-depletion review. According to its authors, the local crisis surrounding this topic can be tackled if the researchers test their concepts and measures extensively, report more transparently on them, and consequently develop consensus on what self-control is and how it can be measured. Although direct replications efforts are still helpful in increasing the reliability of findings, it seems that they cannot solve the reproducibility crisis on their own: conceptual work and validating measures might very well play an equally significant role.

References

[1]: Wiggins, B. J., & Chrisopherson, C. D. (2019). The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology, 39(4), 202.

[2]: Makel, M. C., & Plucker, J. A. (2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304-316.

[3]: Aldhous, P. (2011). Journal rejects studies contradicting precognition. New Scientist.

[4]: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251).

[5]: O’Donnell, M., Nelson, L. D., Ackermann, E., Aczel, B., Akhtar, A., Aldrovandi, S., ... & Zrubka, M. (2018). Registered replication report: Dijksterhuis and van Knippenberg (1998). Perspectives on Psychological Science, 13(2), 268-294.

[6]: Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., ... & Zwienenberg, M. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546-573.

[7]: Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59-71.

[8]: Feest, U. (2019). Why replication is overrated. Philosophy of Science, 86(5), 895-905.

[9]: Schimmack, U. (2021). The validation crisis in psychology. Meta-Psychology, 5.

[10]: Michell, J. (2000). Normal science, pathological science and psychometrics. Theory & Psychology, 10(5), 639-667.

[11]: Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370-378.

[12]: Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456-465.

[13]: Lurquin, J. H., & Miyake, A. (2017). Challenges to ego-depletion research go beyond the replication crisis: A need for tackling the conceptual crisis. Frontiers in Psychology, 8, 568.

[14]: Crandall, C. S., & Sherman, J. W. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 93-99.

[15]: Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531-536.

Music and Mistakes: An Interview with Two Music Therapists on When Things Go Wrong

Stefan Gaillard — Fri, 21 Oct 2022 10:00:00 GMT

Without a chance meetup at a conference on music therapy, the book Breaking Strings: Explorations of Mistakes in Music Therapy might never have seen the light of day. Music therapists Avi Gilboa and Laurien Hakvoort found out they both wanted to explore the topic of mistakes in music therapy, and so they did. After years of hard work, they launched their book on the 20^th of September. The Journal of Trial and Error met up with them for an interview.

Let’s start with a cheeky question. Which mistakes did you yourselves make during the production of this book?

Laurien: “We lost time due to various reasons. I felt insecure about my English and my status in the worldwide community as a music therapist. This is why we really wanted to collaborate with colleagues from the very beginning, but at times we had trouble finding good matches. If we had started with just the two of us from the very beginning, it would have saved us time. But that has been a learning experience.”

Avi: “I would say there weren’t any big mistakes, but that there was a lot of wiggling about. The process was not linear at all. Sometimes we took a 180 degree turn, only to subsequently turn back again. We had different teams working on the book at different times. Some teams were great at one point, but then didn’t work out later on.”

Laurien: “With one team, for instance, we shifted from the idea of making two books, one theoretical and one with case studies, to just one book. Not everybody wanted to be an editor, some just wanted to be a chapter author. And then COVID hit, which was the last nail in the coffin. COVID made it almost impossible to meet physically, something which was really needed to co-author this sensitive topic. So, the composition of our editorial team changed.”

Avi: “In hindsight this was all part of the process, but in the moment itself it felt like a mistake. This ties into the question of: how do you define a mistake? There is always this temporal component; it depends on which point in time you look at an event whether it constitutes a mistake or not. Looking at the present, I don’t know if the book will be a success – I am uncertain about how it will be received by our professional community. We took design decisions which are not conventional and might be considered risky. So, for instance, we use strikethrough ~~like this~~, for words in parentheses, to give the impression that ideas are there but also not there, perhaps correct, perhaps not. This is a purposeful addition to our layout, but it might cause confusion, and readers might consider it unprofessional. So, maybe our unconventional design decisions will turn out to be a mistake in the future.”

What is the role of responsibility and liability with regards to mistakes themselves, exploring mistakes, and opening up about mistakes?

Laurien: “There were quite a lot of people who were willing to explore their cases with us. They describe one of the mistakes that they made. What we saw with all of them, which is typical for the music therapy community, is that we always feel responsible for our client’s wellbeing. That was a very important part of our concerns as editors, also when we were helping the authors with writing their case material. We had to make sure that even when a client reads about their own case, they should not be able to recognize it. Clients are vulnerable and they should be protected, for example by ensuring their privacy. On the other hand, you can learn from mistakes that you are going to make anyway.

Other mistakes you really can and should prevent. Like going too deep, using music which might trigger suicidal thoughts. The idea that music is harmless is something we wanted to dispel, because that is not the case. And if you think music is only beautiful and harmless, make music for yourself, make music in an orchestra, but don’t use it as a therapeutic intervention. That’s the message we want to bring across, there are boundaries and a music therapist should be informed, and you can be informed by mistakes that others make.”

Avi: “Let me add another point about mistakes and responsibility. If you start celebrating mistakes, you might start to idealize them. Of course you should not aim for mistakes. Mistakes might happen, and then the question becomes what to do with that. There should be a balance between two extremes. On the one hand, some music therapists have this attitude of pushing mistakes aside altogether. Pretending a mistake didn’t really happen, or if it did happen that it was a one-time thing. At this extreme you typically push away your emotions. On the other extreme music therapists have this attitude of just making mistakes and celebrating them. That’s also the wrong attitude because you are not taking responsibility for the mistake. Inside the word responsibility is the word response, so if you want to take responsibility over your mistakes you should give yourself the opportunity to respond to them. Develop the ability to look at the mistake and to respond to the emotions that surround it. Emotions are healthy as long as you don’t block them and they don’t block you.”

Laurien: “One other thing. We didn’t look at what clients felt were mistakes, or what the environment thought was mistakes. We only focused on what the therapists thought were mistakes. We had a lot of debates about what were mistakes. Some people said something wasn’t a mistake because they do it themselves. Well, sometimes that’s still a mistake. In the future, it would be great to have a book from the perspective of the client, or their environment. We chose to look at and present those mistakes that music therapists realized they themselves made. In addition, we didn’t look at mistakes that could have been made in verbal therapy, but at mistakes unique to the music and to music therapy. We wanted to reflect on the uniqueness of music therapy and find those specific cases where the uniqueness that caused the mistake, was shaped by the musical intervention or incorrect musical relations, things like that. So we really look at what is unique in the music therapy realm and how we can bring that to the profession to start discussing mistakes.”

How have the reactions to the book been so far?

Laurien: “We have had very positive reactions. The book is a much needed book, maybe it’s even something that we should have written earlier. Colleagues tell us they want to be in the next book, that they have cases too. Up till now the only criticism we have received is on the unconventional layout.”

Avi: “I think the idea was already out there, people always said someone should organize a conference, or a special issue about failed cases. With the exception of the German Musiktherapeutische Umschau, nobody ever did. But let’s not forget that typically, we would be getting mostly the positive reactions from people and that there may be other, more critical, reactions that – just like mistakes – remain in the shade, so to say.”

That’s an interesting situation which you describe. Why do you think there was such a need for more discussion on mistakes but nobody took it upon themselves to start that discussion?

Laurien: “The nice thing is that both of us have a scientific background, with support from our community. At the same time, we are not the hotshots of the profession, so we have the freedom to experiment, to do what we think is important. Also, Avi and I have a positive chemistry, and because we have that we could convince colleagues to actually contribute. I think we are both people who can put our own ego’s aside and for this topic that is very important. If you have to be this very high status person, it is sometimes more difficult to acknowledge that you have limitations than if you think: I can learn from mistakes. And for me that’s the baseline. It helped us to be tenacious and continue going.

And we were also aware that we wanted to claim a new field in a sense, so even though there was not a lot of literature on this topic, we were always afraid of someone else publishing before us. We started with this book all the way back in 2016, so we had good reason to be nervous about others publishing before us. And some people did. I had to rewrite a whole chapter on harm because by then publications about harm had already started to come out.”

Avi: “We brought this question about the timing of the book up in our focus groups. We had discussions with our fifteen authors whom we grouped into three focus groups. We discussed each other’s chapters and asked each other questions. One of the questions we asked them was: Is this the right time? Which is the reverse of your questions. And all of them answered “yes, it is time. The subject might seem peculiar or irrelevant to some, but it is time”. As we spoke, we realized that there were already fifteen music therapists who were willing to write a case chapter about their mistakes, so that is definitely an indication of an appropriate timing.

When we asked why it had not happened until now, some of the authors referred to the age and the status of the profession of music therapy in relation to its ‘older siblings’. Psychotherapy, for instance, already has a lot of articles and books on mistakes, failure, and error. So why does psychotherapy already deal with mistakes while music therapy doesn’t? Maybe it is because music therapy is younger and smaller. It is still trying to prove itself. When you’re still trying to prove yourself, you can’t go out and say look at my mistakes. You want to say look at me, and at my success rates!”

What role do emotions play in how we deal with mistakes? What were the cases which moved you?

Avi: “There were many cases which moved us very much. Because the therapists were so open and so courageous about these emotions and how vulnerable they were during or after their mistakes. So of course there were many cases. Because of these emotional dynamics, we decided to have frequent meetings with each of the authors. Though this is unusual for edited books, we understood this was a sensitive issue and we wanted to accompany the authors all along, not leave them alone with their emotions. Some of them were very harsh on themselves. We wanted to let them know it was ok, to experience such emotions. Some authors were circling around the issue, but not touching it. Probably because it was either too painful or because it appeared obvious to them. So we really needed to help them make the idea clearer. We never argued with them about what should be considered a mistake; this was always up to them to decide. Emotionally, we were trying to make it as easy as possible to express the mistake as clearly and communicatively as possible and to take these very difficult emotions and deal with them in a non-judgmental way.”

Laurien: “To add to that, one of the things we did was collect stories about mistakes from colleagues during the first initiatives on mistakes, so during conferences and music therapists’ meetings. One of those colleagues came in immediately the morning after a mistake happened, and was willing to share it with us, right then and right there. This therapist ended up writing about it a year later. So those were very valuable interactions, much more emotional than if we had invited people we had never met by just emailing them. That’s because of the personal connection.

One of the things we came across when investigating mistakes was the role of culture in how we deal with them. We invited people from Colombia, South Africa, China, Japan, Bahrain, and other countries. We got so many different types of mistakes, mixed stories. That’s because music is not really something universal, cultural aspects play a big role. Of course we all know what music is, like we all know what speech is, but the music that is suitable for one culture might not be suitable for another culture. Or the music that is suitable for one situation might not be suitable for another situation. So realizing that also brought a lot of new insights. The idea is not to generalize, but to expose as much diversity as possible.

So for me, the main emotion has been excitement. Of course you resonate with what a person tells, but the main emotion has been the excitement of realizing: wow, there is such a diversity and we have never actually delved into this. So this is really the first attempt and there will be much more to explore in the future.”

Avi: “Again there is this time factor, there is a dynamic to the emotions surrounding mistakes. A mistake can cause an emotional eruption, like a volcano. Admitting a mistake at that point can be very emotional, could lead to sadness or anger. As time passes, sometimes, the emotional ‘lava’ ‘solidifies’ and cools down, so it can more easily be touched. Many times we had colleagues tell us about mistakes they made ten or twenty years before. These years probably made it easier to talk about the mistakes. So you can imagine it’s very courageous to bring us a case right after it has happened.”

What do you think that your book can bring people from outside your field?

Laurien: “I really hope that the insights from the chapters on harm in music will trickle through into the musician community and the medical community. There are a lot of people who say things like: let surgeons listen to Mozart during surgery and everything will go well. There’s no reason to believe that. I hope our book can make people aware of what music can do and also what music cannot do. Music is not just harmless. If that was the case we wouldn’t need any education or training as music therapists. For me that is important information that should become more generally known.

Besides that, I hope we can show the diversity of music therapy. We have cases from geriatrics, children with autism, about nonverbal people, on psychiatric clients. There’s such a diversity of fields that we work in. Our book can make people aware of where we as music therapists can help, but again, also were we cannot help. That would be a great advantage outside our music therapy community.”

Avi: “I think it’s a sociological question. It depends on the person who looks at music therapists and what they are seeing. If they look at it in a sociological way, like at another nation or culture, and if they look at it in an open way, I think they can learn a lot. We have similarities with psychotherapy, some similarities with research fields, some with music and the arts, with medicine – some of us work in hospitals, with doctors and physicians. So, it depends on the openness of the person. Some people are very local and they like to look at themselves. For them music therapy and our book won’t say anything.”

What do you think of the tendency to instrumentalise failure and mistakes? Should they always be useful learning experiences or should there be room for mistakes that we cannot learn from?

Laurien: “Again, emotions play a big part in this. Somehow we don’t like to fail. I have worked in forensic psychiatry, and all those patients there have failed in a sense. Have failed society, have failed themselves. And you see different reactions and perspectives. You see people who become angry at themselves, people who become angry at others, people who become depressed, people who ignore the situation, people who avoid the situation, people who flee. You see very different reactions to mistakes, to how to deal with it. Somehow most people want to be good people, they even want to be perfect people. We all make mistakes but we all react differently to them. So I very much recognize this tendency to look for what we can learn from mistakes, to want to prevent them, to want to help others learn from them. It is not something to be ashamed of, but something we can learn from and which can help us set boundaries and limitations. But it’s always something we evaluate in comparison to what is right, to what is perfect, to what success is.”

Avi: “I would use the distinction we made in the first chapter, where we describe three approaches towards mistakes. There is the medical approach, the educational approach, and the explorative approach. The medical approach wants to extinguish mistakes, probably because the consequences are so severe – life and death. As long as this is the case, extinguishing mistakes is probably the best course of action. According to the educational approach, mistakes are considered an integral and necessary part of the process. Like in life, in therapy too, you have to go through mistakes. According to this you are not trying to extinguish mistakes, but you are trying to find them and work with them to better understand the process. If someone I supervise comes to me after a year of treating and continuously says that everything is going fine, I would tell them that something is probably going wrong. I try to tell them that when a mistake happens, we should celebrate. Because it will give you a lot of information. That’s the educational approach. The explorative approach is more neutral; you just look at mistakes from a perspective and try to analyze how they happen, when, and why.”

Do you have any views on how society in general handles mistakes and failures?

Avi: “In general societies are very suspicious of mistakes, they do not really provide much space for them. There are, of course, societies that support ‘mistake culture’, but it seems that the natural starting point is to suspect mistakes and to stay away from them. I would also add that music therapists are a hybrid, of at least two different professions: musicians and therapists, and that these professions have very different ways of looking at mistakes. Therapists usually say that mistakes are part of life – usually, not always – and part of therapy (the ‘educational approach’). With musicians, it depends on the musical culture. Some musical cultures do everything to get rid of mistakes (the ‘medical approach’) to the point where you are supposed to be perfect. Western classical music is such an example. You are supposed to strive for perfect (i.e. no mistakes), especially in the most strenuous situations such as on stage. This can then lead to problems like performance anxiety and stage anxiety. When such musicians come to study music therapy, we spot them very easily, because they really have a difficult time with mistakes, not only with musical mistakes, but also with therapeutic mistakes. With them, part of the process is to give them time to fuse into the ‘educational approach’ to mistakes.”

Is there anything else you’d like to add?

Laurien: “What I really like is the fact that ArtEZ Press allowed us to make mistakes. We got the time to do things incorrectly, and we had flexibility in our collaboration which was very nice. For the layout of the book Amir Avraham designed it from our basic ideas on how we wanted to discuss mistakes. In round tables, with room for mistakes, allowing the time to consider thoughts and mistakes, yet create room for humor and different perspectives. We worked together with students of the illustration department who made the illustrations for us. The whole reason the cover is orange, is that we asked students which color fits music therapy (green) and which color fits mistakes (red). Green and red together becomes brown, which is an ugly color. The color closest to brown was orange according to these students, so it became orange. So we had this flexibility, and because we got the freedom to experiment and to make mistakes, we felt more like we were actually playing with mistakes. I think it’s very important to have these platforms like ArtEZ Press or the Journal of Trial and Error, which is a wonderful platform, that allow people to make mistakes and to discuss mistakes.”

Who Is to Blame When We Fly Too Close to the Sun – Academia or the Individual?

Sarahanne Field — Fri, 10 Sep 2021 09:00:00 GMT

Error is part of the scientific process. Research studies are conceived, coordinated and conducted by humans, and, as such, there is always the chance for mistakes to be made somewhere along the line. Such errors are common and often mundane. For instance, an experimenter might have screwed up the randomization of participants to conditions when programming their study in Qualtrics. Variable names can easily be accidentally swapped around in data spreadsheets, especially if multiple people are working with the same Excel file and passing it around. If one starts using Python after first training in R, it’s not strange to mistakenly index the start of an array with 1 instead of 0, which has the potential to lead to incorrect statistical analyses. A diligent, enthusiastic student, new to qualitative research methods might ask leading questions and prompt interviewees too much, unwittingly allowing confirmation bias to cloud their final conclusions about the phenomenon of interest.

These are random errors. They occur irregularly, and without premeditation. Those committing them tend to be unaware that they have made a mistake until after the fact - if they end up finding out at all. Random error is not the only kind of error committed in science though.

Since the cases of high-profile researchers like social scientists Diederik Stapel and Jens Förster, and nutrition psychologist Brian Wansink, the scientific and lay-communities alike have become increasingly aware of another, more serious kind of error – that of misconduct and fraud. These kinds of errors are systematic, are knowingly committed, and are often covered up by the doer such that they may avoid detection. The good news is that most serious kinds of fraud involving data fabrication or manipulation are likely to be relatively rare in psychological science – less than 2% of the scientific population report to have engaged in such behaviour at least once, says Fanelli [1]. The bad news? 10-20% of researchers admit to knowing someone who has falsified data [2]. If trust is a key pillar on which the enterprise of science rests, how can we account for this violation; how can we as consumers of science trust what we read if potentially a fifth of it is made up? Perhaps most saliently, where can we place the blame and how can we fix it?

In light of such egregious violation of the community’s trust, it is tempting to blame dishonest researchers alone. One bad apple spoils the barrel, so the argument goes. This view treats the issue as one of moral failing—of bad people doing bad science—and has sparked initiatives like ‘rehab’ for offending researchers aimed at cleansing them of their scientific sins [3].

On the other hand, what if dishonest psychologists are not scientific heretics in need of penance, but rather psychology’s most devoted followers; those who respond rationally to a corrupt system. Put plainly, what if the academic incentive system in psychology is at least partly to blame for shoddy science. Let’s unpack this idea a little more, focusing on the very recent and popular case of professor of psychology, Dan Ariely.

Professor Ariely has been in the news lately, at the centre of an unfolding case involving allegations of data manipulation. At the end of August, 2021, blog site Data Colada run by Leif Nelson, Joseph Simmons, & Uri Simonsohn published an extensive post [4]. It detailed an analysis of the results of a 2012 article co-authored by Professor Ariely, and the ultimate judgment it communicated was that the data in that paper were fraudulent, and the article’s conclusions unreliable. The analysis, written by the Nelson, Simmons, and Simonsohn (the real whistle-blowers chose to remain anonymous), found computational evidence that the article’s data were fabricated in two ways: duplicated then altered, and possibly randomly generated. The data were highly uniform in their distribution (where one would expect that these kinds of data would be somewhat bell-shaped in distribution). The only explanation for such a finding, argue the Nelson et al., is that the data were simulated, rather than collected. Additionally, the data seem to have been tampered with at multiple timepoints in the experiment. It cannot be determined with certainty who manipulated the data, nor how the data were tampered with exactly. Suspicion has fallen on Professor Ariely, however.

This post shook the psychological community, due in no small part to the unfortunate irony that Ariely himself is a researcher of dishonesty, leading many to question why someone so acutely in touch with why people lie would lie himself.

Leaving aside whether Ariely is truly guilty of fabricating data or not (this is a question for the review boards), there are at least two issues to consider here. First, why do scientists commit fraud? An interview of Cristy McGoff, the former director of the research integrity office at the University of North Carolina, published by RetractionWatch in 2016, addresses this question [5]. The interview highlights two factors which interact to precipitate the behaviour our current society labels as fraud: personality traits and research culture. Or, to use the by-now cliché phrase, nature versus nurture. Of the interaction between these factors, McGoff says: “The environment of research advancement contains levels of ego, competitive behaviours, and the need to be respected both by peers and students. While those involved may not have started out in their field with these traits, it is in some way something that can be bred just by being within this culture daily and the modelling of these traits all around them.”

Although a toxic academic culture and perverse incentive structures are of course not the only factor contributing to fraud and misconduct, they play a big role. Every high profile researcher who stretched a story in a paper with ambiguous data, who has chased prestige by reviewing for a larger journal instead of a smaller one they may be better suited for, who exaggerated findings or plans for a grant, has contributed to the environment of ‘successism’, which, in part, has pushed researchers to engage in questionable research practices and, most extremely, fake data for publications. In other words, each scientist who finds themselves outraged at the Ariely story should look inward and recognize the parts of themselves contained within it. In a scientific community that is genuinely interested in knowledge-production and which does not prioritize novelty and impact above soundness and reliability, this would likely happen even less than it already does.

The second issue concerns integrity, and how it is socially constructed. In an excellent analysis of fraud as a sociotechnical construct, Park explores the idea that fraud can be framed as a “concept that is defined and judged in local contexts, rather than as timeless and universal…” [6]. If we think of fraud that way, says Park, we redirect attention from asking the question of who has committed the fraud to the question of “…what constitutes wrongdoing” (p. 396). We argue that wrongdoing in the context of academic misconduct is not only a burden to be placed at the door of the individual wrongdoer themselves, but also at the door of the community as a whole. Park asks whether it is even possible to have an isolated act of wrongdoing in labs today and within our current culture. He asserts that “to locate responsibility becomes a complicated and contested matter, which can reveal power relations within and between the labs”, and, we add, within the greater scientific community. Stretching a metaphor introduced by Park, we can compare academia to Daedalus in the Fall of Icarus; fraudsters to Icarus himself. Academic culture, like Icarus’s father Daedalus, is responsible for being on Crete in the first place (as a place of refuge after having been charged with murder) and provides the means to escape the island by flying. Icarus himself, however, pays the price: he falls from the sky into the sea, where he drowns.

Zooming out, we should beware of using Ariely and others as scapegoats. Rather than dwelling on the moral failings of individual researchers, we might reframe fraud and misconduct as a systemic problem we all ultimately contribute to as part of the scientific community. Firing Ariely and others who commit fraud won't do anything more than delay the next fraud or misconduct case. Until we address core problems in academia – the suppression of null results, emphasis on the "story" and not the substance of research findings, statistical misunderstandings and mishandlings, ‘pay-to-play' research, success-oriented funding bodies, etc. – we cannot expect that people will always play by the rules we construct. Only by making slow, trial-and-error science the norm, and working collectively to neutralize harmful academic culture can we come close to purifying science.

References

[1]: Fanelli D (2009) How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLOS ONE 4(5).

[2]: Stricker, J., & Günther, A. (2019). Scientific misconduct in psychology. Zeitschrift für Psychologie.

[3]: Cressey, D. (2013). ‘Rehab’ helps errant researchers return to the lab. Nature, 493, 147.

[4]: Nelson, L., Simmons, J., & Somonsohn, U. (August, 2021). Evidence of Fraud in an Influential Field Experiment about Dishonesty. Datacolada. Via: https://datacolada.org/98

[5]: McCook, A. (August 29, 2016). Who do scientists commit misconduct. Retraction Watch. Via: https://retractionwatch.com/2016/08/29/why-do-scientists-commit-misconduct/

[6]: Park, B. S. (2020). Making matters of fraud: Sociomaterial technology in the case of Hwang and Schatten. History of Science, 58(4), 393-416.

JOTE in Conversation: Daniël Lakens. Whose Fault Is it That Science is Irreproducible? And Who the Heck Are Metascientists?

Max Bautista Perpinya — Sun, 31 May 2020 20:29:00 GMT

After talking with Léa Roumazeilles, PhD candidate in Neurobiology at Oxford, we now present the second of our series ‘JOTE in conversation with researchers.’

In this series, we ask researchers how they experience 'failure' and 'success' within their research practices - whereby 'practices' can range from the research proposal and funding to experimental constraints and publication. How does failure arise in these different aspects? And how do researchers in different fields and career stages deal with failure?

This time, we spoke to Daniël Lakens, Associate Professor in the Human-Technology Interaction (HTI) group at Eindhoven University of Technology (TU/e). As a metascientist, Lakens' work consists of 'developing methods for critically reviewing and optimally structuring studies.' We talked with him about the reproduction crisis in psychology, and how practices, experimental methods, statistical approaches and funding affect the way errors are framed and published.

As I set the recorder to rec, I told Daniël: 'You don’t have to talk very loudly, the recorder is pretty good.' He responded, 'Alright, I’ll try.' We chuckled. More than the volume of his voice, Daniël’s words are loud and travel far because of his reflections on the status of science and the ethics of researchers may be poignant to some listeners. We hope this interview gives you something to reflect on.

What is metascience? How did it originate?

Metascience has been done for a long time, known as epistemology. However, metascience as we know it became very popular in psychology ten years ago. To help you understand, let me briefly explain what has happened in the field since then.

First, in psychology there was always an understanding that not everything was perfect but we didn’t really know how bad it really was, and it was easy to just to tell each other, ‘yeah, things are not perfect, but how bad can it be?’ It was extremely common, especially as PhD students, that we would tell each other, ‘we are flexibly analysing the data.’ It was a very convenient thing to say because you didn’t really have to think about how bad it actually was. At the same time, I remember my supervisor once told me, ‘yeah, this finding is not really reliable. We know this. We should have an independent committee that replicates this sort of finding, so that it becomes widely known in the literature.’ But nothing much was going on. That was sort of the state of the field.

Then in 2010, a paper came out that became really, really well known. This was a paper by Daryl Bem on precognition, the ability of people to predict the future. He did a classical task in psychology, with a variation. In the original task, you are presented with a picture on the left and on the right and press a button saying whether the stimulus is on the right or on the left. He flipped the task temporally, meaning that you had to press the buttons before they appeared, and he counted how often people guessed better than the guessing average. Apparently, most people could tell better than the guessing average. This was a nine-paper study that appeared in 2010 and circulated a lot around 2011. The editor said, ‘I don’t know what to do with this. This looks like any other psych paper that we have. Just because it’s a crazy topic and consists of nine studies, it is super convincing. I should accept this.’

After this, people had two choices: either to say that precognition is real since the evidence was extremely convincing, or that this indicated that there was a problem in the way that we worked. So many people opted for the second, unsurprisingly. From the moment the Bem papers came out, there were many responses of many different kinds, saying things like: statistical analysis is not good, it’s based on p-values and we should report base factors, etc. There were other strategies to understand what happened in Bem's studies. Then, some researchers (Simmons, Simonsohn, and Nelson, 2011) published what became a classic of the field, ‘false-positive psychology.’ First, they reported that they had played the song ‘When I am 64’ by The Beatles, and a control song, to people; they showed that those people listening to ‘When I am 64’ aged. They were older. And you were reading and said, ‘that’s impossible, that’s a nonsense finding.’ The authors said, ‘of course this is a nonsense finding. But this is not all that we did in the study.’ Then they added all the things that they left out, and their selective reporting became apparent: ‘we didn’t just ask this, we also asked this, this, and this. And we not only tested this, but also this, this and this. And if you have flexibility in your testing, then Type I error will inflate up to the point that it is really easy to find significant results.’ This paper made it very clear that the way that we work, and the answer to the question ‘how bad can it be’ was ‘well, it can be really, really bad.’ The important thing was that many people recognised the practices that this article described, and started thinking ‘okay, am I doing things the wrong way?’ People came around to the fact that methods were extremely important.

Parallelly, in the Netherlands there was the case of fraud by Diederik Stapel, who made up data. And that’s kind of boring in terms of the process, because he just typed random data. You are not supposed to do it, or so everyone agreed. So in the Netherlands you had these two things together, close in time. The Daryl Bem studies were the rational part, and the fraud case of Diederik Stapel was the emotional one. So it was the time when many people realised for the first time that there is a big issue with the way that we work, and we were all strongly motivated. We knew we had to do something about it. Now, the Netherlands is regarded by most people as one of the countries leading the change on these issues.

What is the role of journals in promoting good science?

They have a big role. Part of the key issues in disseminating tools for better science is educational. There’s a new journal - Advances in Methods and Practices in Psychological Science - that helps people to improve the way that they work in a very broad spectrum. There are papers about theory, measurement, concept formation, how we should collaborate and store data, informed consent - everything. All of this follows from each other: if you want more transparency you need to share your data and for this you need to get informed consent. But how do you do this for fMRI research where you scan people and have their face, for example? All sorts of practical and ethical considerations have come out of this, which has kept people busy for a very long time.

Another key development is the emergence of open access publications - especially the ones like PLoS One. For example, there was a classic case of a failure to replicate. It was a very highly-cited study about elderly priming, where people are presented subliminally with words related to the elderly and then they walk slower.

During a conference in 2009 or so, I remember talking to Stéphane Doyen, who was the first author on a paper that attempted to replicate it. He presented the study’s failure to replicate at the conference and I thought: ‘this is super cool that you’ve done this. Credits to you to try to replicate!’ I remember emailing him a couple of years later saying ‘hey, I was thinking about your study. Where is it?’ He said ‘yeah, unsurprisingly all journals rejected it’ because of course the original author was most likely one of the reviewers.

Then eventually it was published in PLoS One because it had an explicit rule: we don’t care about novelty, we care about the reliability of the findings. So there started to be outlets that gave people room to publish, for example, failures to replicate. This was an important development. There was a huge discussion about this finding; the original author wrote very harsh, impolite blog posts about it. It’s all very interesting to look back at. I’m sure he regrets it now but it tells you something about the state of the field: you couldn’t challenge a finding like this; people really thought that this was etched in stone but if we look at it now, these are studies with fourteen participants in each condition. C’mon.

There is a lot of innovation in how science is published, tell us about that.

There is a journal called Metapsychology that publishes about metascience. The journal itself is also a little bit of an experiment: it has open peer review and the submission is basically uploading your pre-print somewhere. It will take into account any comments that are made on the pre-print on social media or using Hypothes.is, which is a tool where you can annotate the web page and leave comments on the pre-print. It’s zero APC (author-processing charges) and they also have formats similar to what you try to do in your journal at JOTE; they have an ‘empty your file drawer’ format. This type of journal is not super popular yet but there are a couple of examples now. The problem is getting people to actually submit null results.

Do you know how positive publication bias (the file drawer problem) is being coped with at the moment?

In an indirect way, registered reports. Brian Nosek and I did a call for a special issue on this in 2012. Around the same time, Chris Chambers also proposed a similar idea, and he really pushed this at many journals - hundreds of journals now have it - where the decision to publish is made before the data is collected. It combats publication bias and it improves the quality a lot because the peer-review processes move to the front before the study is actually performed. This is a development that in practice works very well in reducing publication bias. Many people are trying it out. Not a lot of people, but some.

What about the role of funders, is it hard to get funding for replication research?

Not a lot is happening when it comes to funding. Sander L. Koole and I wrote a paper also in 2012 about the importance of rewarding replication research because who’s going to do a replication study when novel studies are so much more rewarded? And then when we wrote that paper I thought that we should at least try to reach out to somebody from NWO about this. We can just write a paper and put it in a journal but that seemed a bit boring. So we thought: if we want to be serious about this, we should send a letter to the NWO asking them to fund replication research. And so we did.

Initially they wrote a reply saying ‘we will fund replication research as long as it is innovative.’

[Silence]

Wait, what?

Chuckles, yes - I mean of course it’s not innovative. You’re just repeating something. So we thought, how can they officially reply like this? I think they didn’t completely understand. So I sent a letter back saying that basically this was destroying the foundations of science. If this is what you think then this is really problematic.

But even though their first response was odd, they’re actually a really nice organisation. They invited me over to talk about this topic and I tried to convince them that it would be a good idea to fund replication research. Crazy enough - it took them a long time, three years or something, but now they have a call to fund replication studies! It’s a pilot programme, so they’re evaluating it and taking a look at how they should extend it into the future, which is pretty nice.

A couple of other funders paid attention and thought: hmm okay, we should also take responsibility in making science more reproducible. This is good news, because I think we need to reserve a part of the fund to make it only available for replication research, otherwise novel research will always be valued more. The key point is that nobody will get a career by replicating other people’s work but we know that somebody needs to do it.

Do you think scientists behave ethically? Who in your opinion has the power to change the way science is done?

Anyone who would really follow the Code of Conduct in their lab nowadays would encounter pushback. If you ask them who’s responsible they might say ‘the head of my group is responsible’ or ‘I won’t get my PhD if I don’t do what my supervisor wants.’ In turn, the supervisor is going to say, ‘the dean of the faculty tells me that I have to publish this much, and do this and get these kinds of grants and so on. And how do I do that? By doing this kind of research.’ The dean is going to say ‘well, the university says that the government is forcing us to be such and such.’ So if you ask, ‘who’s responsible?’ The truth is that everybody is pointing towards each other but nobody is doing anything.

I’m interested in giving advice to the government or funders about these kinds of things. I don’t think it’s very popular amongst scientists to actually say people need to force us to do the right thing because they feel that it will make their lives more difficult. And yes, it will. But, honestly, I don’t care - I want to make my life more difficult. It is tax money that we’re spending and we should make sure that it is spent very efficiently.

What do you think is the role of metascience in all of this? Do you think this role of enforcing or making the rules is a job that metascientists could do?

Well first of all, will the general scientific community listen to metascience? No, of course not, because they will say ‘you’re just trying to get as much power as possible and do whatever your political agenda is.’ So I wouldn’t say that metascientists are supposed to determine these kinds of rules.

You should have a team of people who are interested in making advice for policy. I don’t think NWO should listen to just any single person or group of metascientsts because they are a skeptical bunch of people with very specific views about certain things and there are alternative viewpoints. Some people think we should focus on the big and the good stuff, for example.

I think metascientists can at least ask the questions that aren't asked. It’s amazing how bottom-up science organised itself over the last 500 years and how we keep things in place because ‘this is how it is.’ I think the role of metascientists is to ask: why is it like this? What are we doing? This doesn’t make sense.

For example, publication bias is a disaster for science. When I discuss it in workshops I don’t think people go home very happy and proud about being a scientist. I think they go home with at least some motivation not to contribute to publication bias too much in the future. This is the best I can achieve when I teach young researchers about this now.

Don’t you think that there is some sort of implicit bias that we are not able to see through?

Of course we will do things wrongly. This is a process where you try your best and in 50 years someone is going to say: look at these idiots. They thought they were improving Science in 2020. They missed this and this and this. That’s just the logical process in how we work.

You can’t justify everything. The real thing is to justify a little bit more than you’re doing now and ask yourself: why am I actually doing this? That’s the goal. I want to be able to stop following rules - especially norms.

The Four Horsemen of the Crisis in Psychological Science

Sean Devine — Wed, 25 Mar 2020 21:41:38 GMT

Psychological science is in a crisis. It’s been in a crisis for quite some time. For anyone doing research in psychology, or just paying attention to the field, this should come as no surprise. The most common statistical framework we use has been criticized since its inception (e.g., [1] , [2] , and [3]), many famous findings in psychology fail to replicate [4] [see 5 and 6 , for two examples], and much of the publication process incentivizes researchers to engage in questionable research practices (e.g., p-hacking, RUSS, HARKING, and so on) [7].

Like I said though, this is old news. Scientists have known about these problems for decades, yet improvements are slow to come, if they’re coming at all [8]. While issues of statistical inadequacy, replication failure, and poor methodology have dominated most of the discussion about methods reform in psychology, other important problems in psychological research have surfaced in its wake. In this post, I want to briefly introduce three lesser-discussed crises in psychological research that, along with the replication crisis, form what I call the four horsemen of psychological science. The goal of this post is chiefly to raise awareness of these issues and present them in a simple and condensed fashion. It is also to discuss how JOTE may act to help curtail some of these crises, a point which I raise at the end of the post.

White Horse: The Replication Crisis

The replication crisis is perhaps the best known crisis in psychology and science more broadly [9]. It has already been given much more detailed explanations than I can give here (see [10] for a review and assessment), so I will just summarize. Put succinctly, the replication crisis in psychology revolves around the fact that when findings are shown to be statistically significant in one study, they fail to be statistically significant in other studies that follow the same research protocol. This is an important problem for psychology, with an estimated two-thirds of studies failing to replicate [10]. The reasons for these failures are manifold, from institutional pressures to publish positive findings [11] to a misunderstanding and misapplication of statistics and statistical power [12].

Though it gets the most press, the replication crisis is neither the sole nor most widespread issue in psychology. While replication is a problem to be sure, it alone is more akin to a symptom of a broader disease than it is to an open stab wound.

Red Horse: The Measurement Crisis

To illustrate this last point, imagine we have a finding that perfectly replicates. That is, if the proposed research protocol is followed correctly, we will always find a statistically significant difference between measured scores on the dependent variable (DV) in one group vs. measured score on the DV in the other group (groups here representing levels of the independent variable; IV). If the replication crisis were the only thing psychological science had to contend with, all would be well here.

Unfortunately, there is more than replication to worry about in psychological research. As Flake & Fried [13 & 14 ] point out, when we replicate a finding, we assume that the construct underlying that finding was properly measured to begin with. For example, if we demonstrate that social rejection increases negative emotional state, we first have to assume that we can meaningfully measure and quantify (a) social rejection and (b) negative emotional state. However, as the authors demonstrate, researchers rarely give good reason to make such an assumption. Instead, researchers often engage in what Flake and Fried call “questionable measurement practices”, where they hide (or perhaps do not even question) the process by which their measures are chosen and their constructs are defined. Oftentimes, new measures are invented for the purposes of a single study, seemingly out of the blue, with little justification as to why this measure would be chosen over a previously established one. Even when a rationale is provided for these choices, measures are seldom rigorously validated and, when they’re purported to be, and appropriate statistics are sparsely reported [14] .

This so-called “measurement schmeasurement” attitude is a problem, because it means that even if a finding is replicable, we are still not sure exactly what is being replicated. It is for this reason that replication does not stand alone as the only crisis in psychological research. Indeed, it may not even be the primary one! After all, it is not very useful to replicate a finding that we can’t properly explain to begin with.

Black Horse: The Generalizability Crisis

Now imagine we have a field of research that is both highly replicable and properly measured. That is, we have a valid and reliable way of measuring our DV and IV and we can consistently demonstrate that the IV exerts a statistically significant effect on the DV across a variety of studies. Again, at first glance, it seems as though things are in good shape.

Enter the issue of generalizability. While it is important to combat both questionable research and measurement practices in experimental psychology, from the perspective of generalizability, these efforts are somewhat misguided [15] . Rather, the key issue researchers ought to focus on is to what degree their results match up with what they are supposed to predict in the real-world. To take our example from above, say that we choose valid and reliable measures of social rejection and emotionality and, using these measures, find a replicable effect across studies. Despite our efforts, we are still likely only using a small number of operationalizations of social rejection and negative emotion. These measures do not, however capture the broad and diverse experiences of being rejected—from divorce to being “swiped left” on Tinder—nor does it fully provide an account of negative emotions—from devastation to minor discomfort (for a similar argument by a leading figure in the field from which this example is taken, see [16] ). This remains the case even when results are replicable across many research contexts (so-called “conceptual replication”), because we are still talking about a narrow conceptualization of a psychological phenomenon that has no guarantee to apply to the myriad of real-world manifestations of said phenomenon. Furthermore, even when measures are properly validated, such that we can have strong confidence we are truly measuring negative emotion, psychological experiments can make few claims about how these emotions are experienced and elicited outside the lab and whether results extend to those situations. As Yarkoni [15] puts it: “The root problem is that when the manifestation of a phenomenon is highly variable across potential measurement contexts, it simply does not matter very much whether any single realization is replicable or not.” (p. 13).

Pale Horse: The Practicality Crisis

Finally, imagine we now have a research field that replicates, is validly measured, and generalizes to real-world phenomena. This is to say that we find some IV and DV relationship across a series of independent studies, we are confident our measurements properly capture the construct, and this relationship generalizes to a variety of real-world manifestations. Surely in this case our results are safeguarded from psychology’s crises?

Unfortunately not. Even if our results meet the criteria mentioned above, we have so far neglected a central question when doing research: “Who cares?” And when it comes to psychological research, the answer seems too often to be “no one much”. Berkman and Wilson [17] make this point more formally. They argue that, as a field, much of academic psychology has lost its sense of practicality. Theories in psychology live and die without much regard for how they could be applied to help improve society in a meaningful way. As a key point, the authors highlight the artificial distinction that’s been set up between “basic” and “applied” research. Underlying this distinction is the (incorrect) belief that “basic” research applies to some universal human figure, whereas applied research focuses on the specifics of a situation or group. However, for reasons we grappled with in the previous section, no research can in reality test such universal claims. All experimental samples are constrained by their components (the participants) and their contexts (the fact that they are experiments to begin with). In practice, these components are even more limited to WEIRD participants and contexts (Western, educated, industrialized, rich, and democratic) [18] . As such, the authors make the case that we should abandon this distinction and focus on researching topics that are of general interest to a group or society in order to produce work that stands to benefit that community.

From this perspective, our results may very well be valid, replicable, and generalisable, but to what end? If these results are born and die in the “ivory tower”, never to affect change in society, be applied by policy-makers, and ultimately help people, then what good do they serve?

Avoiding Apocalypse

Faced with these four horsemen, we might ask whether our sins have finally caught up with us; whether now the only options are to run to the hills or to brace for the oncoming stampede. I argue, instead, that now is the time for self-reflection and transcendence. When the replication crisis first hit, many scientists recognized that things needed to change. However, not much did. Perhaps this lack of change was due to the fact that, in many respects, the replication crisis was not an existential threat to psychological research. After all, the solution to the crisis was simple, albeit challenging: acquire more powerful samples, stop questionable research practices, gain a better understanding of statistics, promote transparency, and encourage replication [e.g., 19 ]. It’s business as usual, just a little different.

However, the crises I have outlined here—measurement, generalizability, and practicality—are not so easily dealt with. They require a fundamentally different outlook for how we conceptualise psychological science and its position within academia and society in general. Not only do these crises demand that the difficult methodological changes needed to ensure proper replication be in place, but they also require us, as scientists and consumers of science, to reflect upon what it is we value in psychology. Indeed, they require us to be philosophers as well as scientists.

It is for this reason I see JOTE as a step forward in combating the crises I have discussed in this post. While JOTE addresses the basic requirements of methods reform—publishing of null results, support of replication work, Open Science, and so on— it goes much further. It also opens a dialogue with thinkers from other fields who can contextualise the research that gets published. Commentators from sociology, philosophy, and anthropology are encouraged to discuss whether the research published in JOTE uses appropriate measures; whether it ought to bear any weight in broader discussions surrounding a phenomenon; and whether it is ultimately useful for society.

Certainly, JOTE alone cannot solve all the crises that I have outlined here. As I mentioned, doing so requires fundamentally rethinking how we conduct science. While I have focused on psychological science in this post, it is likely similar crises are ready to break in other fields of science, if they have not already. I believe that by uniting the scientific community and the humanities, JOTE is uniquely positioned to tackle these crises in a novel and necessary way that researchers from all areas could benefit from. Whether this approach works to curb the many worrying trends constantly bubbling up in psychological science and beyond remains to be seen. However, I argue that if any approach can begin the hard work of figuring out what it is we are all really doing when we engage in scientific research, it is one like JOTE’s, that unites thinkers across many field in transcending our limitations and collectively searching for truth in science.

References

[1]: Boring, E. G. (1919). Mathematical vs. scientific importance. Psychological Bulletin, 16, 335–338. doi:10.1037/h0074554

[2]: Berkson, J. (1942). Tests of significance considered as evidence. Journal of the American Statistical Association, 37, 325–335. doi:10.1080/01621459.1942.10501760

[3]: Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychological Bulletin, 57, 416–428. doi:10.1037/h0042040

[4]: Pashler, H., & Wagenmakers, E. J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528-530. doi: 10.1177/1745691612465253

[5]: Hagger, M. S., Chatzisarantis, N. L. D., et al. (2016). A Multilab Preregistered Replication of the Ego-Depletion Effect. Perspectives on Psychological Science, 11(4), 546–573. doi: doi.org/10.1177/1745691616652873

[6]: Klein, R.A., et al. (2019). Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement. Retrieved from: https://psyarxiv.com/vef2c

[7]: John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science, 23(5), 524-532. doi: 10.1177/0956797611430953

[8]: Replication Index (February 20, 2020). The Lost Decades in Psychological Science. Retrieved from: https://replicationindex.com/2020/02/20/the-lost-decades-in-psychological-science/

[9]: Ioannidis, J. P. (2005). Why most published research findings are false. PLos med, 2(8), e124. doi: 10.1371/journal.pmed.0020124

[10]: Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi: 10.1126/science.aac4716

[11]: Lilienfeld, S. O. (2017). Psychology’s Replication Crisis and the Grant Culture: Righting the Ship. Perspectives on Psychological Science, 12(4), 660–664. doi: 10.1177/1745691616687745

[12]: Tressoldi, P. E. (2012). Replication unreliability in psychology: elusive phenomena or “elusive” statistical power? Frontiers in Psychology, 3, 218. doi: 10.3389/fpsyg.2012.00218

[13]: Flake., J. & Fried, E. (2019). Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them. Retrieved from: https://psyarxiv.com/hs7wm/

[14]: Flake, J. K., Pek, J., & Hehman, E. (2017). Construct Validation in Social and Personality Research: Current Practice and Recommendations. Social Psychological and Personality Science, 8(4), 370–378. doi: 10.1177/1948550617693063

[15]: Yarkoni, T. (2020). The Generalizability Crisis. Retrieved from: https://psyarxiv.com/jqw35

[16]: Baumeister, R. (2020). Do Effect Sizes in Psychology Laboratory Experiments Mean Anything in Reality? Retrieved from: https://psyarxiv.com/mpw4t/

[17]: Berkman, E. & Wilson, S. (2020). So Useful As A Good Theory? The Practicality Crisis in Academic Psychology. Retrieved from: https://psyarxiv.com/h3nwd/

[18]: Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and brain sciences, 33(2-3), 61-83. doi: 10.1017/S0140525X0999152X

[19]: Kline, R. B. (2013). Beyond significance testing: Statistics reform in the behavioral sciences. American Psychological Association.

JOTE in Conversation: Léa Roumazeilles. How Does Failure Arise and How Do You Deal With It?

Max Bautista Perpinya — Thu, 05 Mar 2020 19:59:58 GMT

We are glad to publish this first interview as a set of conversations that we’ve been having with researchers from several fields and universities. These interviews revolve around how researchers experience 'failure' and 'success' within their research practices - whereby 'practices' can range from the research proposal and funding to experimental constraints and publication. How does failure arise in these aspects? And how do researchers in different fields and career stages deal with failure? These are the questions that guide our conversations.

For our first interview, we talked with Léa Roumazeilles, PhD candidate in Neurobiology at Oxford University. We spoke with her about the challenges and successes in researching her thesis subject: ‘Structure and function of the social brain in primates’.

What is your background?

I did a double undergraduate in Paris. I studied Biology at the Université Pierre Marie-Curie and History at La Sorbonne. For my Master’s I moved to the Université de Strasbourg where I studied at the Joint Master in Neuroscience.

And from there, how did you get where you are now?

My way into the PhD was through my research assistantship at the lab that I am at now. I was there for one year, and then I applied to several PhD research programmes within Oxford. I went through the interviews and was admitted to this laboratory. I have been doing my PhD now for almost 3 years.

Is it a paid position?

I have the University’s PhD fees paid by the Biotechnology and Biological Sciences Research Council (BBSRC). Regarding my personal funding, I did not get the full scholarship, but my supervisor was happy to pay with the laboratory money for the rest of my salary. The New College (founded in the 14th century) that I am part of here in Oxford also subsidises my salary.

What kind of research do you do?

The name of the project is ‘Structure and function of the social brain in primates’, so I am looking at a variety of primate brains and trying to understand how the differences in function and connectivity can explain differences in social abilities. I am studying what is called the ‘social brain network’.

Tell us a bit about your work with the animals: what is it like?

I study the functional side of the project with monkeys. I use the brain imaging tool fMRI (functional magnetic resonance imaging), which allows me to see the function of different brain regions while the monkeys perform tasks on a computer. These tasks resemble those that we do with human subjects. Basically, I am able to see what these regions do for each task. So fMRI is part of the functionality studies. For the anatomical research I use post-mortem brain samples of primates, and with those, I trace the connections between the regions.

You also handle animals, right? How do you get trained to work with the monkeys?

Here in Oxford, I had to take a course with a final exam, and that allowed me to be with the animals. But after passing the exam you still need to be trained and supervised. It took about 4 months of training as a research assistant to be able to be on my own with the monkeys.

Where does your research stand - is it applied or fundamental? Or is it perhaps neither?

I think it’s more basic science. My project is fitted within a larger evolutionary project, so in that way, my project aims to explore how our brains work and where they come from evolutionarily speaking.

Since I am working with monkeys, and because of our research question, I’d say that the application of what we find is quite far down the line. The closest to applications that I could think of would be in the field of autism research. Some of our models are similar to those models of autism.

Your research is not medically oriented, then?

No, not really. I am funded by the Biotechnology and Biological Sciences Research Council, which supports fundamental science. If I would do more applied medical research I would have to be funded by another council. They [the Research Councils] are very specific about what you can investigate given the funding that you have. In my case, that frames my research outside medicine, and within fundamental research.

Have you published some of your work already?

My name is on a couple of papers because I helped in some of the projects I was involved in. But I still haven’t published any major work. I am excited because I have just submitted my first author paper though!

On the topic of success, and failure - has something gone wrong during your experiments? How did you deal with it?

Well, actually, these past weeks it's been difficult because of that. I've been training the monkeys on a task for a bit more than a year. That's every morning for four hours, and this week I realised that they just won't learn this task. So I need to change the task. [laughs] Which is quite... Well, annoying.

The thing is, we designed this task to be very similar to a human task, and the monkey just don't get it. The behaviour that the monkeys have to do seems very easy for us, but they just won't learn it. And we were really not expecting that, so now we have to find ways of finding new ways of answering the same question.

You trained your monkeys for a year in this task. Is that normally how long it takes for them to learn a behaviour?

It depends a bit. My monkeys had learnt a very easy task just before. So it depends how long they've been trained beforehand, and the nature of the task. You can sometimes expect them to learn in six months.

What was this task exactly?

It was very simple. There were two objects on each side of a computer screen, and the monkey had two sensors (two buttons), one for each side of the screen. The two objects were the same, but only one had a reward - so you couldn't know which one was the one with the prize. At some point, there would be a 'cue' in the middle of the screen. In our case, it was the face of a monkey that would turn left or right and would give you good advice or bad advice on where the reward was. The idea was that the test monkeys would eventually learn which of the cue monkeys was giving bad advice or good advice.

This had not been done before, and we thought that it would work. It is known that monkeys can recognise facial expressions in other monkeys. But in our case, I think that they just can't infer that there are two types of advice. They couldn’t tell which one was giving good or bad advice.

And how did the monkeys realise that it didn't work?

Because they weren't following the advice at all. Normally they want to maximize the rewards, they are quite greedy... [laughs]. And these monkeys just couldn't.

What are you going to do now that the experiments didn't show what you expected?

Yes... [laughs and then silence] That's a good question. Since they can't learn the task, we have to change it completely. But the new experiments still have to fit within my PhD project and have to be close to the questions that we were asking before. Our project was part of a collaboration with another group doing the same tasks with humans, and so it would be good to keep collaborating and having comparable links with them. If we have the same or very similar tasks between our monkeys and their humans, we can then study the relationship between them.

Are you going to publish what you’ve found in the past year somehow?

Well, there isn't much that we could do with it. I don't have much data, just some behavioural data. I kept trying to change the task a little bit each time, trying to see if they would learn. So that means that I don't have enough data of one single task to report. I don't think that I have very usable data at the moment.

Also, I don't really know why they don't learn, and that is a big question that we should be answering. It could also be a lot of external factors, like changing the rooms of the tests, if they had surgery, etc.

It is true that it would be good to get this out there so that if someone else tries these tasks, they would know about it.

If you would know why it didn't work, would you go ahead and try to publish it?

If I wouldn't be constrained with time, I would consider it. The thing is, I need to be done with data collection in March, and I haven't really been able to get good data yet. My initial idea was that I would record their brain activity while they were doing the task, but since they couldn't learn the task, I have no measure of brain activity. And getting everything running will take time, so I also don't really have time to write a report on what has happened so far.

I see, writing a report does take time. Well, thank you for your time and good luck to you finding an experiment that does work!

Thank you!

Open Science Symposium Utrecht – Starting off with the Natural Sciences

Stefan Gaillard — Fri, 21 Feb 2020 15:05:44 GMT

On February 6, the Open Science Community Utrecht (OSCU) organized a symposium on Open Science at the Faculty of Science of Utrecht University. Max Bautista Perpinyà, student member of OSCU, visited this symposium to get an impression of the latest developments in Open Science for the natural sciences. Together with OSCU resident science journalist Stefan Gaillard he wrote an impression of the event. [This article appeared first on the OSCU website]

The first-ever symposium on Open Science at the Faculty of Science of Utrecht University achieved what these symposiums are for. They set the mood for new ways of doing science, and build community by providing not just an ethos, but also very specific tools for researchers of different domains.

As Open Science symposia go, they tend to be filled by an optimistic crowd, mostly researchers sharing their knowledge and teaching others how to share theirs. It was therefore refreshing to see not only researchers, but other important stakeholders of science. Library staff and faculty administrators are crucial in the process of making Open Science a reality, and their presence was much appreciated.

As Open Science symposia go, they tend to be filled by an optimistic crowd.

From the Faculty, Dean Isabel Arends encouraged the community to keep pushing forward the ideals of openness, and made the commitment of doing what is in the administration’s power to ‘facilitate’ them. And while ideals are inspiring and necessary, this symposium was overall informative and offered a variety of tools for us researchers to use.

The keynote address was by Erik van Sebille, oceanographer and chair of the ‘University and Society’ study group of the 2020-2024 UU Strategic Plan. Van Sebille showed his efforts concerning making the ‘workflow of scientific knowledge more open’. In the field of computational physics, GitHub is an extremely useful tool to work with the global research community. For example, his team’s code Parcels analyses the movement of particles (like the ‘plastic soup’) throughout the oceans, and is openly accessible (anyone can contribute by reporting bugs, improving it, using it…).

After the plastic soup in our oceans, physicist Sanli Faez provided the audience with another recipe. This time it was the ingredients for a healthy and tasty open community. With ‘the need of belonging’ as a principle of Open Science, Faez suggested that in order to move from a egocentric to a networked structure of science, we could look up to other successful communities, like Mozilla (check their extensive list of resources), and the Global Village Construction Set, with their grand vision of reconstructing civilisation with open source industrial machinery. In thinking about the recurring debates on recognition and reward in the new Open Science era, Faez recommended the work of second-wave feminist Jo Freeman in ‘The tyranny of Structurelessnes’, which gives specific suggestions for how necesary it is to establish a structure within a community, ensuring a fair and democratic allocation of responsability, power, and yes: credit. An important aspect in scientific practice, and which Open Science aims, first of all – to have conversations explicitly and transparently.

While in the computational world and physics departments sharing code is a well-established tradition, in the field of structural biology things look slightly different. Tzviya Zeev-Ben-Mordehai is working in a very competitive field and acknowledges that the fear of getting “scooped” prevents scientists from sharing their advances in the long process of decoding proteins’ molecular structures. In this field of biology, non-fair play seems to have itself a long tradition. Mordehai cited the widely-known case of Francis Crick and James Watson, who in the early 1950s got (somehow) hold of Rosalind Franklin’s ‘photo 51’, which helped them decisively in deciphering DNA’s structure in 1953. However, not all is bad news in the world of proteins and genes. Since early 1971 the WorldWide Protein Data Bank (wwPDB) has hosted the results of scientists throughout the globe working on establishing the 3D structures of proteins and nucleic acids. Moreover, Mordehai says, the wwPDB is curated, which ensures quality information. Important also is the fact that it is a requirement for the majority of journals that scientists first deposit their results in the database before heading for the press, which ensures proper recognition and open accessibility to the information.

Then, it was the turn of PhD student in molecular plant physiology Laura Dijkhuizen, who shared the progress of making their ongoing research more open. For their bioinformatic study of plant meta-genomics, Dijkhuizen uses tools such as e-labjournal, protocol.io and GitHub to make code openly available, but also to help them in project management tasks.

In short, the Open Science Symposium at the Faculty of Science at Utrecht University was a strong kick-off to the series of symposia that the Open Science Community Utrecht (OSCU) will be hosting this year. The organisation, location and speakers were all excellent – a promising indication of how the next six symposia might look like. Hopefully, OSCU can find a way to bring something interesting for researchers from all disciplines, while at the same time preventing too much overlap between the symposia. Regardless, bottom-up Open Science seems to have made a large breakthrough at Utrecht University.

Wetenschap, Wees Eens Eerlijk Over Jezelf!

Stefan Gaillard — Wed, 18 Dec 2019 15:53:10 GMT

As first seen in DUB, Utrecht University's magazine. The original article can be found here.

Vorige week presenteerde minister Van Engelshoven haar langverwachte ‘Strategische Agenda voor het hoger onderwijs en onderzoek’. In haar voorstel lijkt de minister direct te reageren op belangengroep WOinActie, die al maanden aandacht vraagt voor problemen in wetenschappelijk onderzoek en onderwijs. Deze problematiek erkent de minister inmiddels, en ze wil met haar strategische agenda in ieder geval meer erkenning voor de hoge werkdruk waaronder wetenschappelijk medewerkers gebukt gaan. Het is echter jammer dat het in haar plannen bij symbolische erkenning blijft.

De boodschap is duidelijk: besturend Nederland vindt de wetenschap zielig. De wetenschapper zelf heeft daar natuurlijk helemaal niets aan. Hoe zorgen we dan wél voor effectieve verandering? Volgens ons op twee manieren. Allereerst: protesteren. Daar moeten wetenschappers, studenten en docenten dan ook vooral mee doorgaan. Het is daarnaast noodzakelijk om kritisch te kijken naar wat de wetenschap zelf kan doen.

Succesfactor

In dat verband is de volgende observatie uit het rapport veel interessanter: hoewel wetenschappers een succesvolle wetenschapper als ‘individualistisch, competitief, prestatiegericht en zelfverzekerd’ omschrijven, zien zij zichzelf juist als ‘coöperatief’ en ‘betrokken bij collega’s’. Dat is tegenstrijdig: waarom typeren wetenschappers een succesvolle wetenschapper zo anders dan zichzelf?

De wetenschap heeft een verkeerd zelfbeeld. Wetenschappers proberen daaraan te voldoen en dragen dat beeld maar al te graag naar buiten uit. Het is geen verrassing dat we bij ‘wetenschappelijk succes’ moeten denken aan raketten, vaccins, Einstein en Newton. Dit beeld legitimeert het werk dat wetenschappers doen en aan dat ideaal wordt wetenschappelijk succes nog steeds gemeten. Het aantal publicaties waarin (een schaduw van) al dat zogenaamd baanbrekende onderzoek wordt gedeeld, en de tijdschriften waarin dit onderzoek verschijnt (liefst Nature of Science) bepalen wie ‘succesvol’ is. Deze succesfactor is doorslaggevend in het bepalen welk onderzoek hoeveel geld krijgt en welke wetenschappers de meeste kans op overleven hebben.

Heel veel falen

Als wetenschap inderdaad zou draaien om het doen van baanbrekende ontdekkingen, zou dat allemaal helemaal niet zo erg zijn. Het probleem is alleen dat wetenschap in de praktijk vooral draait om heel veel falen. Doodlopende onderzoeksagenda’s, mislukte experimenten en onbruikbare datasets behoren tot de dagelijkse wetenschappelijke praktijk. De onderzoeker die aantoont welke onderzoekspaden totaal onvruchtbaar zijn, is van essentieel belang voor een efficiënte wetenschap maar maakt geen schijn van kans in een cultuur waar succes op de verkeerde manier gemeten wordt. Wetenschappers houden die cultuur zelf in leven.

Je kunt er daarom voor kiezen succes anders te meten, zoals Van Engelshoven voorstelt in de nieuwe Strategische Agenda, maar misschien is dat niet genoeg. Hoe zou het zijn als wetenschappers beginnen met open en eerlijk zijn over wat wetenschap in de praktijk is? Wat als zij óók toegeven dat wetenschapsfinanciering voornamelijk mislukkingen faciliteert, dat die mislukking zelfs noodzakelijk is om de vragen te beantwoorden die in 2015 de Nationale Wetenschapsagenda vormgaven?

Wetenschappers zouden zelf kunnen beginnen met het publiceren van trial and error. Naast de zogenaamd significante succesverhalen, zou ook het proces van vallen en opstaan een prominente podiumplaats moeten krijgen. Als ze daar eenmaal aan gewend zijn, hoeven ze zichzelf – en de samenleving als geheel – niet meer zo voor de gek te houden. Als falen niet alleen onderdeel is van de dagelijkse wetenschappelijke realiteit, maar ook van de manier waarop wetenschappers zichzelf en elkaar beoordelen en zich presenteren naar de buitenwereld, verdwijnt die immens hoge prestatiedruk waaronder wetenschappers gebukt gaan ook.

The Open Science Summer School Experience

Alex Visser — Fri, 18 Oct 2019 10:09:36 GMT

When I first heard the term “open science” and realised what it meant, I thought, well, that sounds good, how hard can it be? It turns out that the answer to that question is: very. During the last week of August, the Utrecht University library had organised a summer school that dealt explicitly with open science and how it can best be incorporated into researchers’ workflow. Luckily, we were in attendance and - amazingly - we also managed to present our journal!

Because the summer school was dedicated to re-evaluating the research workflow in its entirety, we discussed every possible component of this, from preparation to literature review and analysis, to outreach. Perhaps unsurprising to you, but very surprising to me, each of these phases can be evaluated and improved upon by open science and this made the summer school an incredibly intense, but valuable experience. Another pleasant surprise was that several of the participants in the course had already heard of us and every one of them was incredibly enthusiastic about the ideas behind the journal! I realised we might have been preaching to the choir somewhat, but it was a great experience nonetheless.

The summer school provided a wonderful overview of current developments within the open science movement. It gave us tools to annotate our research papers in an open way (if you use Google Chrome, that is), explained the technicalities of open science publishing (and the copyright implications), and most importantly: it highlighted all possible lunches at the Utrecht Science Park.

Upon presenting our work to the Open Science audience, we received a lot of comments and were able to open the floor to a debate about the importance of negative results and publishing in various disciplines of science. Participants in the summer school were mostly geography and urban planning or psychology researchers, which meant there were many different perspectives to be discussed. It was fascinating to hear that in some universities and fields of study, publishing negative results is not an abnormality at all and negative results are already seen as valuable. This gave me hope that one day, we might be able to push for this mindset throughout all fields of science. If they can do it, why wouldn’t we be able to, right?

While the whole debate was interesting, and the audience posed many engaging questions, the best question we received was about the process of becoming a journal: how did we go about it, and how did we experience the whole thing? As you can see on this blog, we are doing it by keeping all of you perfectly updated on everything that’s going on with us, prioritizing transparency and the candid expression of our experiences. After all, if we aren’t open about that, we would not be the exceptional, innovative project that we are.

Are We Facing a Bigger Monster? The Journal of Trial and Error Visits Cambridge

Martijn van der Meer — Mon, 30 Sep 2019 11:44:06 GMT

As fall hits, we now look back and reflect about the beginning of summer, when all was excitement, when all the work of the previous year was put into public action, and our voices were being heard at several places in Europe.

This piece deals with something specific: our experience of presenting the Journal of Trial and Error in Cambridge at the Laboratory of Molecular Biology (LMB) Annual Graduate Student Symposium. In writing this, we hope to put some order to the information that we got from talking to scientists, as a form of take-home message, but also as a grander reflection on what we experience as key dynamic movements, and uncomfortable knots in the practice of science and the publishing of articles.

The story that was told

On July 10th, we arrived late at Cambridge train station, the sun had set, and the twilight truly gave us the sense of being in a town of tradition and history. It all was quiet, the few pubs and nightclubs seemed just a small obligated addition, the wandering tourists just a recent blister for the locals — what we felt was Cambridge’s essence was the intellectual brewing happening behind walls and the throwing of young minds into books and lab benches.

The conference started, and we wore our name-tags proudly and clowny. We, as a pure historian (Martijn), and a neuroscientist in the process of turning historian (Max), felt like personified anomalies of science: being in Cambridge, at the LMB research department (host of no less than twelve Nobel prize awards), at a structural and molecular biology symposium — and yes, we went there to talk about failure. As our manifesto hopefully makes clear: we didn’t and still don’t mean ‘failure’ in any demeaning sense. We are interested in the role of making mistakes at the bench and on the white board, and we think that errors are intrinsic and productive. This was the story we tried to tell.

After listening to talks, which Max could understand about 60% and Martijn about 10%, about hardcore cutting-edge biomolecular science, we intended to start our talk by doing what those reflecting on science can do best: ask annoying questions. With a first slide containing one simple sentence – “What is science?” – success was guaranteed. The audience, full of mainly PhD students from Cambridge, answered by sophisticated shouts about an “experimental method”, a “certain scientific scrutiny”, and “systematic observation”. Method, as became clear from the answers, demarcated real science. But what, then, is this method specifically?

From a historical point of view, a possible answer lies with those who first wrote explicitly about this methodological scrutiny. While being at Cambridge, we pointed dramatically at Francis Bacon (1561-1626) and his Rerum Organum (1620) in which he argued how natural philosophy should move away from dogmatic reasoning, while instead focusing more on gaining knowledge by means of empirical reasoning. A natural philosopher, as Bacon argued, should closely observe “the works of nature” by carefully producing ‘quality facts’. From deductive knowledge-gaining, to inductive knowledge-gaining, as philosophers would say.

However, in this inductive reasoning, communicating one's individual observations requires thorough and trustworthy communication, as Bacon was perfectly well aware of. As may be unexpected from the founding father of the experimental method, the Cambridge Scholar was very explicit on what should be communicated. In his Rerum Novanum, Bacon writes:

“First the materials, and their quantities and proportions; Next the Instruments and Engines required; then the use and operation of every Instrument; then the work itself and all the process thereof with the times and seasons of doing every part thereof., …

So far, it sounds really familiar to modern readers. But the quote goes on:

… then the Errors which may be committed, and again those things which conduce to make.”

For Utrecht historian of art, science and technology Prof. Dr. Sven Dupré – who pointed us in the direction of Bacon’s quote – this passage shows how the open endedness of the new experimentalist way of communicating, was crucial to the new way of doing natural philosophy. But for the purpose of this talk, we tried to surprise the audience by showing that the father of what we retrospectively call science, emphasized communicating failures as central element of doing good science.

Nevertheless, this line of reasoning enabled us to talk about another hero-of-science, we has been object of research in one of the most sophisticated and inspiring studies in history (and sociology of science): Robbert Boyle (1627-1691). In Leviathan and the Air Pump, Steven Shapin and Simon Schaffer famously explained the debate about the existence of a vacuum in terms of social context. They problematize the idea that by means of experiment the controversy between Robbert Boyle and Thomas Hobbes was not easily settled by the air-pump-demonstrates the former designed. After all, Boyle had to convince his peers that the natural facts he discovered could not be doubted. Therefore, demonstrations in front of peers, so that other natural philosophers could witness Boyle conclusions, were one of the most important rhetorical techniques of his experimental science.

Obviously, also natural philosophers who could not attend these demonstrations, should be convinced, albeit virtually. Schaffer and Shapin show in that regard how, again, communicating science effectively is the key for success. In line with Bacon, Boyle was very explicit in communicating errors in the process of science in the making. In 1660, he writes in his New Experiments what an experimentalist should do when his experimental set-ups malfunction. In case of a leaking air pump, for example, Boyle declares on page 494:

“. . . I think it becomes one, that professeth himself a faithful relator of experiments not to conceal such unfortunate contingencies.”

Shapin and Schaffer interpret this explicit writing about failed experiments as having a mainly rhetorical purpose: “On the one hand, it allayed anxieties in those neophyte experimentalists whose expectations of success were not immediately fulfilled. On the other, it assured the reader that the relator was not willfully suppressing inconvenient evidence, that he was in fact being faithful to reality.” Be it for rhetorical reasons, as in Boyle case, be it for normative reasons, as in Bacon’s case, communicating the messy process of doing science has a long history. How, we asked our audience, was that the case today?

The science of trial and error

Leaving the historical anecdotes behind, we went on to talk about the present. Our project, embracing the call for transparency of the movements of Open Science (open data, shared resources, collaborative work…), aims to bring into the light those mistakes that happen in the lab. We envision those trial and error processes as coming into two classes: methodological and conceptual.

Methodological mistakes that are of interest for us (and for working scientists as well) are those which tell something about widely shared methods, and that in revealing flaws, advance scientific communities by telling the do’s and don’ts of specific practices. During our talk we emphasised that those mistakes are not those trial and error processes limited to personal learning of the materials and methods; but rather, that there are mistakes showing the individual scientists better uses and limitations of techniques. When talking about these kinds of mistakes in during the coffee breaks in the conferences, scientists of all stages illustrated our principle with more concrete examples, ranging from optimal buffer concentrations to unsuccessful antibodies, from validated animal models to unfit fMRI sequences.

The impression we got from our Cambridge audience was that many labs try to improve and fine-tune a method (what is plainly known as “making it work”), and after months of trial and error, someone at a conference’s coffee break, or over a beer in a pub, tells them: “oh, yeah, that doesn’t work. Try that [input here your specific, favourite lab technique].” What came from the stories of these scientists, telling in excruciating detail their struggles at the bench, was a frustration that the information that they needed was already known. Any scientific niche has a handful of labs around the world, working with similar techniques and theoretical concepts. But not all these knowledge-forms, and especially the hands-on experimental details, are publicly available. Some of the fine-tuning experimental steps are only known by word-to-mouth, and although key factors, they are only known in some labs that are lucky enough to be at the apex of the field and at the frontier of the known.

There is some half-hidden, private knowledge that those labs that work with a specific technique or theoretical scope all seem to know. “Apparently, everybody knows that!”, many PhD’s shouted in pain at the conference. The younger these scientists where, the more baffled they seemed when realising that not all is out there in the open. “It wouldn’t take much to write that up, you know,'' they would comment. A few lines in a paper or a database would do, it would avoid them months of seemingly futile work, but more importantly, limit the amount of resources used, and give them time for “more important stuff.” Aside their personal frustration behind these stories, there seemed to be an underlying ethical worry, regarding the lack of efficiency in the use of (public) funds, and the lack of transparency of certain labs in communicating methodological details which are key to the advancement of science as a group enterprise. This dodgy area of “everybody knows that, but nobody says” in certain fields seemed to us to be directly connected to the current hyper competitiveness in science. Maintaining secrecy of techniques enables labs to keep the upper hand when it comes to publishing. It is all about a race to publication, not even anymore a race to discovery. Let alone, public discovery.

If revealing methodological mistakes (and solutions!) looked like a tough job to communicate in the age of competition, opening the possibility to disclose conceptual mistakes seem to touch upon something even more sensitive. And to begin with, perhaps the idea of ‘conceptual mistakes’ is not the best marketing strategy from our part. But let’s carry on. Our concept of ‘mistake’ isn’t as bad as it sounds. Mistake, error, failure or flaw is aimed to be understood within a dynamic process of trial and error, and as an inherent property of science. At the edge of ‘the known’ we find uncertainty, and so, what else can we expect but to make some conceptual mistakes in our hypothesis, more often than not? A hypothesis being conceptually flawed can only be thought of retrospectively, that is, after the fact of having experimented. By facing up to empirical data, hypothesis get support (positive results) or lack of it (negative results).

The problem arises, when most of the articles out there seem to mostly be comprised of supporting evidence. This problem is known as positive publication bias (you can read about it in a post, in detail in our manifesto, or in our brochure), and is the tendency to submit, accept, and publish preferentially statistically-positive (those that reject the null hypothesis)novel results. Understanding why there is such bias is a complicated task. Why are there more positive results? Are scientists proposing hypothesis ad-hoc? Are they waiting for the right results to appear? Are they fine-tuning methods and theories until they have all figured out with a higher level of certainty? Is there an inherent asymmetry between positive and negative results, which leads to a preference to the former? Is there an institutionalised pressure to disclose only novelty embedded in biomedical promises? What are the incentives to obscure negative results and reframe themes positively? What is the relationship between this publication tendency and grant applications? Is it a practical limitation of time, because negative results aren’t as interesting and writing manuscripts takes time? Or is it, as Francis Bacon suggested, a natural tendency of the mind: “It is the perpetual error of the human intellect to be more moved and excited by affirmatives than by negatives”?

Now, don’t let this avalanche of questions run you over. These are all partial explanations for why scientists publish virtually only statistically positive results. Publication bias is a swamp, and it’s foggy out there. These explanations, ranging from the most pragmatic to the most sociological explanations, are the inhabitants of this swamp. Perhaps the stagnant waters reek of decaying matter, perhaps there are too many flies, infesting bacteria, and invisible quicksandv. We have been trying to identify the dwellers of the swamp, that is, to distinguish what makes scientists conceal their mistakes (so we could change that). But in this search, we have found an even creepier creature. In talking to scientists we have progressively realised that we were standing on something sticky, large, invisible. This swamp is inhabited by a monster that all seem to know, but none seem to dare to awaken.

Getting to know the monster

The project of turning negative results into productive elements of the scientific process may seem like an innocent idea; but while researching on the topic, and especially while in the Cambridge symposium, we came across an intricate network of reasons, interests, institutions, and yes, power, that may complicate things for us. Our idealistic project of ‘making errors count’ early on got confronted by the interest of big publishers, the structure of funding bodies, and the reputation at stake of scientists. While we haven’t actually been confronted directly by these powers, we have been warned time and again that our project wasn’t a small addition to an already complex network of scientific communication; but rather, it seems to disturb (at least theoretically) how the whole thing is built. However, when one talks ‘power’ in science as an institution, one may lack the personal and very specific components of the relationships within the lab. Our experience in Cambridge precisely changed that — it illuminated us into what goes on behind the scenes. It showed us the face of the monster, a bigger monster than we thought we faced.

First of all, we are glad to say that our idea was generally met with enthusiasm. And especially coming from more junior researchers. PhD candidates told us about their experiences, their frustrations, and their disconformity on how things were sometimes run at their labs. The interest in our journal sprouted from their own confrontation with a system that praises novelty and success, and which assigns errors and negative results only a supplementary role. But young scientists seem to be enthusiastic to bring them to the forefront: “It’s a great idea!”; “we really need this!”

It then seemed as though all the ingredients were in place: young scientists believed in the idea, they thought it was worthwhile, productive and necessary. They had their share of examples of how they could contribute. However, while the ingredients were there, the heat was missing. When we proposed if they would be willing to publish an article based on their experimental work that went wrong, they would begin to step backwards. Why?

To understand this, one needs to bring into the mind the fact that laboratories are hierarchical institutions. A word that gets thrown in very often (in a combination of praise, fear, resentment, admiration, and conformity) is “PI”. The PI (Principal Investigator) is the boss; the head of the lab, the one guiding the line of research, allocating projects to people, discussing results and future experiments, and writing and proofreading articles for submission. Most PIs are not dictatorial, and do take into consideration the direction that PhD students and post-docs argue for in their research. However, PI’s do have the last word when it comes to what goes and what does not go into press. Grants are given in respect to the lab’s output (mostly the published articles), thus a tight control exists on what goes ‘public’. Funding bodies will look at the amount of papers and the ranking of the journals in which they are published, and since high-ranking journals typically reject those articles with statistically non-significant results, PI’s will not even try to spend time writing articles that (1) will not count much in receiving future grants, and (2) damage their credibility as a successful researcher.

It is interesting to know that publication bias is mostly a tendency to not submit negative/null results, meaning that these kinds of results will be filtered out mostly at the level of the individual labs (with the PI as the final authority), and not at the level of the editorial boards. Of course, editorial policies that encourage also submissions of statistically non-significant results (such as eLife orPLOS ONE) will help in reversing the publication bias tendency. However, the mere existence of these non-traditional journals seem to have little effect for the PI’s at high profile centers where competition is abysmal. At Cambridge, we heard often that although PhD candidates know of the existence of these journals, and are excited about initiatives like ours, they quickly say, “my PI is only interested in getting Nature’s and Science’s”. The one-word journals (like the previous ones and Cell) are high-ranking journals, and publishing in them will bend the balance of fortune (the grant-giving institutions) towards your lab.

PI’s not only write articles and propose which journals they should go to, but also guide their mentored PhD candidates in what experiments to follow, and how to allocate time and resources. Thus, the ‘power of PI’s’ is not so much direct repression to publish certain kinds of results (negative and null), but a general tendency towards the opposite direction: investing time in generating cutting-edge, unique, successful, grant-giving results. The hierarchical difference between PhD candidate and PI is manifested in the power of the latter in being the time-management of the former. The junior researchers we met at Cambridge were hard-working, devoted scientists, and although much impressed and enthusiastic for our project, they are time-constraint by deadlines and goals – while PI’s are marking the tempo.

In the days of the conference, we went from the “it’s a great idea” to “I don’t dare to be the one stepping up”. We now could translate the fear of PhD candidates to “step up” to their PI, in the more practical of “I don’t have the time”. The issue with negative or null results is that the experimenter will not typically assign the same level of certainty as if it was a positive result. A key question that was asked during our presentation was this: “how can you be sure that a negative is truly negative?” The idea behind this comment is that while a positive result is double and triple-checked, and thoroughly put to the test; negative results are not held to the same level of scrutiny. There seems to be an asymmetry when it comes to the direction of the results. A positive result (for example, protein X is involved with behaviour A) will be assessed through alternative testing method (for example, not only that removal of X leads to absence of behaviour A; but that genetic and physiological evidence points in the same direction, drawing a multilevel network of gene-protein-function-behaviour). However, a negative result (that protein Y is not involved in behaviour A) won’t be assessed so thoroughly. At least in the field of molecular biology, negative results tend to be taken more at face value than are positive results: “if it doesn’t work once, you try again; if it doesn’t work the second time, then that’s what it is.” Scientists won’t be spending time in detailing the ways a mechanism isn’t, and after all, the possibilities are endless: Y is not involved in A, neither in B, C, D…; and behaviour A is not caused by Y, Z, W…

And this is what we think is the key issue in making labs publish their negative or null results: while their content is relevant to the general scientific communities (especially in regards to reliability of meta-analysis), they are hardly published because they do not contribute to the individual laboratories. The bigger monster that we were facing (the power of PIs, time-management, scrutiny) makes us realise that we “are asking scientists to be altruistic, while scientific practice and publishing is a highly competitive race”, as one of the other speakers at the conference told us.

While this conclusion may leave anyone discouraged to fight a system of laboratories-publishers-funding bodies that encourages individualistic science, we do not want to stand passive and surrender. From the point of view of those sciences reflecting on sciences (history and philosophy of science [HPS], or science, technology, and society studies [STS]), we ought to reflect on this issues, and provide solutions to discuss, for example: the implications of a hierarchical organisation in laboratories. It is also on our shoulders (HPS, STS) the burden of proof to find fitting solutions for a situation in science, which is often criticised, and less often tried to improve. From the point of view of the Journal of Trial and Error, we aim to create a platform, where these kind of scholarly traditions, scientists themselves, and editors are put into contact directly in a forum where experimental science and multidisciplinary reflection take place. At Cambridge, it became clear that we are here to help fight the monster of science. Altogether.

Posthumanism, Lille & "New Forms of Science"

Stefan Gaillard — Tue, 17 Sep 2019 13:47:03 GMT

What does a congress on Posthumanism have in common with our journal? To be very honest, I had no idea at first. But, I was giving a talk there anyway, so I decided to ask whether I could give a short pitch on The Journal of Trial and Error. To my surprise, I was put on a panel together with Stefan Lorenz Sorgner and Francesca Ferrando, two giants in the field. The topic of the panel was “Works in the field and publication perspectives”, which actually matched the subject of my talk somewhat nicely.

While preparing my talk, I saw more and more parallels with Posthumanism that I could draw on. For example, a truly open peer review process would facilitate a wisdom of the crowds type situation, where the collective intelligence of multiple peer reviewers would exceed the sum of their parts (if this already sounds too vague to you, don’t visit the congress).

The day of glory finally arrived near the end of the weeklong congress. I went on stage before more than a hundred scientists, philosophers, artists and entrepreneurs and gave the pitch of a lifetime. Slightly hindered by the fact that I forgot to turn off the automatic transition between slides every 20 seconds, I still managed to woe the crowd. Pictures were made, notes were taken, good times were had.

Afterwards, I thought I would encounter the classic problem: enthusiastic silence, i.e. people that were very passionate about the project but unwilling or unable to contribute themselves. While this was mostly true, there were some – notable – exceptions. One assistant professor from Germany approached me and offered to bring me into contact with some South American colleagues of hers who are working on new ways of conducting science. Intrigued, I took her up on her offer. Most of her contacts would be out of office, but she told me that in September everything would be set in motion. Exciting times ahead…

Talk on Descartes Colloquium: Initiating a Journal for Trial and Error in Science

Martijn van der Meer — Fri, 06 Sep 2019 10:11:31 GMT

On the 16th of June, project leader Martijn van der Meer gave a lecture at the Descartes Colloquium for the History and Philosophy of Sciences and Humanities at Utrecht University. The audience contained philosophers, historians and sociologists of science.

Ladies and Gentlemen –

My name is Martijn van der Meer, and on behalf of Max, Stefan, Nayra, Jobke and Davide, I am proud to present you our idea and progress on the Journal for Trial and Error.

I am absolutely honoured that we – as a team of mostly first year HPS students – are invited at this stage to present you hands-on output of our Masters programme. We are both excited and scared. Excited, because this moment is the kick-off of a small promotional tour in Europe at several conferences, ranging from a biomolecular conference in Cambridge, to a posthumanist conference at Lille, and we consider this audience as our (future) peers. This is a safe space, so to say.

On the other hand, we are also a bit scared, because we consider this audience as our (future) peers. As I will show in a minute, we try to contribute to the scientific community in a constructive – not necessarily a constructivist – way. In that respect, we present ourselves as Master's students who aim to apply the academic knowledge we are intrinsically interested in. And as I have learned from many local histories of science, those who apply knowledge do more than often meet skepticism from peers. Nevertheless, we are standing here, presenting an idea which is the product of months of weekly discussions. And we are asking for feedback, criticism and help.

The next two months, Utrecht will be the global center of philosophy and history of science. Or – to put it even more idealistically – Utrecht will in the next couple of weeks, be the center of reflection on science. After all, besides arguing that history and philosophy of science is – as I personally think – one of the most beautiful academic endeavors one can think of, we are young and hopeful and – at this point at least – don’t have jobs as researchers. That is why we are still asking questions such as ‘How can we contribute to society?’, ‘what does this reflection we apparently do precisely mean?’, ‘How are history and philosophy connected?’ and – my favorite – ‘Do we as students of history and philosophy of science, actually like science?’

I think all five of us do. We believe in the value of producing scientific knowledge within the context of institutions such as universities and research centers. History and (some) philosophy of science has taught us that throughout history, scientific knowledge has been constructed in the context of social, political, cultural and institutional influences. Despite that some inspiring intellectuals have – based on the observation that scientific knowledge has been socially constructed – challenged and in some cases radically relativized the universal aspects of all scientific knowledge, we think that there is something unique about science. We optimistically understand science as bringing society forward and that reflection should help scientists to do so.

This is relevant in the context of our project, because we think that on this quasi-philosophical basis, some cliché conclusions in the world of history (and also philosophy) of science, can be relevant – even useful – for science today. Let’s call it a ‘constructive’ rather than ‘constructivist’ position. The project we are about to present, aims to transcend an attitude of throwing intellectual criticism over the fence of hard-working scientists.

In the context of our struggle with the question about how we can contribute to the scientific community we actually like and which fascinates us in a constructive way, the Journal of Trial and Error was born. At the Christmas colloquium last December – we presume you don’t remember: we heard that some of you actually fell asleep during our ‘pitch’ – we presented the idea as a thought-experiment. Well, let’s be clear, there is no finished issue yet. But we have started to believe more and more in our idea, our plans are developed, there is even a call for papers sent into the world: the Journal for Trial and Error is not an idea anymore, it is here to stay.

First, let’s turn to the three clichés in history and philosophy of science that we want to turn into constructive applications.

Picturing the development of scientific practice as a culmination of grand discoveries, done by Western-White-Male Scientists is an oversimplification – and most of the time full of wrong interpretations of what ‘actually happened’. Both scientists themselves, as well as philosophers, sociologists and historians of science have increasingly been highlighting the importance of science in the making. A more faithful picture of Science, the one of practices and fine-tuning methodologies, seems to be at odds with the unrealistic public image of big-discovery Science. After all, science is made by humans.
There is a gap between what is published and what is researched. We know positive publication bias pressures scientists to conceal methodological mistakes, while desdaining research containing negative findings. This often threatens proper interpretation. In the face of failed research —which we define as outcomes of Science that do not meet the initial aim of the individual researchers— scientists have two options at hand: not publishing or framing the results as productive by adding ad-hoc hypotheses in an inadequate way. We understand this as a consequence of the expectations of big-discovery Science and the publish-or-perish politics of this Science.
There is a replication crisis. Since scientists validate their results in terms of replicability, the present-day situation of unreplicable experiments is a serious problem. Debate on this replication crisis has focused on the misuse of statistics by scientists, on methodological carelessness, or theoretical inappropriateness.

Based on those three clichés, we want to address, market, think about and facilitate Trial and Error in Science, understanding it as the elementary process by which scientific knowledge is acquired.

Let’s be more specific with what I precisely mean with ‘trial and error’. On the one hand, trial and error refers to Methodological Errors in a practical sense. We believe that if those errors were more widely known, they would foster improvement in the understanding and application of techniques. Methodological Errors are here understood in a broad sense, those that go beyond the learning of the individual researcher and have an impact at the scale of the scientific community.

A nice example is the case of multiple comparison problem in fMRI scans. As you see can see in this picture, by means of fMRI, brain activity in a salmon is measured ––– the only problem is that this fish is actually dead.

As you might know, in MRI, the brain is divided up into units called ‘voxels’ –– of which there are over 40,000 in a typical scan. Most fMRi analyses treat every voxel independently, and test to see if each voxel is “activated” by a certain stimulus or task. So that’s at least 40.000 separate comparisons going on. As a consequence, there is a high probability that fMRI analysis provides, by chance, some weird results –– that explains this picture. Publishing this kind of research would be very welcome in our journal.

On the other hand, Trial and Error refers to Conceptual Flaws, arising from hypotheses being confronted with conflicting observations. When the initial hypotheses are inappropriate in the face of empirical evidence, scientists improve or reject theoretical frameworks by developing alternative theses aimed at increasing empirical adequacy. Not only hits (positive results), but also misses (negative results) are key to scientific progress.

Within chemistry for example, one important subfield is the optimization of the synthesis of chemical compounds - either by speeding up the process or making it cheaper. Unsuccessful efforts to do so often remains unpublished. As a consequence, many different unconnected chemists try the same optimization process to no avail, due to the fact that there is no available record of previous unsuccessful attempts.

In spreading the three lessons we’ve learned as good history and philosophy of science students, we think that we need to operate in the context of Open Science. But above all, we want to emphasize the necessity of dialogue. As proud children of the Descartes Centre, it may be of no surprise that we identify a highly specialized academic community, aiming to tackle and reflect on social and intellectual challenges in a frequently unproductive way.

Because of the scattered organisation of university departments and faculties, a constructive dialogue between different tribes of cutting-edge Science is missing. In the context of the earlier mentioned problems of a harmful public image of Science –––– the positive publication bias –––– the replication crisis –––– and inaccessible Science, the lack of communication has to be addressed even more urgently. In the face of these multifaceted problems, we need useful solutions for the future of Science.

So. How do we think to solve these problems by squeezing Trial and Error into scientific practice in the context of a scattered institutional context? What do we have to contribute?

We are going to launch a ‘scientific’ journal, so that we can publish Trial and Error in the form of methodological errors which have productive conclusions for the scientific community at large, and conceptual errors in the form of negative results.

That does not mean that we want to publish sloppy science. We think it is necessary that every published science article should be double-blind peer-reviewed by more senior scientists. We believe that in talking about errors, and taking this very seriously, scientists can learn about the do’s and don’ts of their methods and concepts. As well, because negative results are highly informative, this would help alleviate the issue of publication bias, and reframe the replication crisis.

We ask scientists to publish their experiences of trial and error in the form of a short article. This article should include the IMRAD structure (Introduction, Method, Results and Discussion), but also a ‘purpose’-section and a take-home message. An IPMRAD+T-structure, so to say. We want to experiment with this format, hoping that asking scientists about their initial purpose and asking to formulate a take home message, a constructive and intriguing format will come into existence.

However, as icing on the cake: on every published ‘science’-article, a subject specialist, or a philosopher, historian, anthropologist or sociologist of science will be invited to reflect, thereby answering and problematizing the question “what went wrong?”. This combination aims to ensure novelty and quality in our journal.

So, we aim to create an open access journal that contains pairs of science- and reflection articles, in which trial and error in science is published, and on which scientists and – especially – humanities scholars will reflect. That we understand as progress.

Now, I hear you wonder: Is this new? On the one hand, it is not. There do exist journals which aim to publish negative results, and of course, there are many journals in which reflectors of science can publish (for their own community). But we focus especially on young researchers, since only they are the hope for a change in Science, we believe. Therefore, we do take their work and ideas seriously. We aim to publish high quality work of early-career scientists, peer-reviewed and edited by more senior scholars. In concrete terms, that means that we at this point want to focus on articles based on Master's theses. But of course, we hope that this will be a slippery slope, leading to a new way of publishing in the future.

This focus on young scientists that believe change is necessary, has a pragmatic side as well. Of course, we don’t expect that there is any serious senior scientist risking his or her career by publishing work that did not meet his or her initial purpose (‘failure’?) in a journal without any academic reputation. We are not naive. However, we do hope that ambitious Master's students that cannot publish their research because they had to work with some faulty database or theorized an impossible experimental setting, will publish in our journal.

Of course, it is necessary that it looks like that we are serious. That’s why it is necessary that we are institutionalized. What are our traces?

We have written a Manifesto for Trial and Error in Science - on which this talk is based
We have a website: jtrialerror.com
We have a blog about our own activities on our website. After all, we are also part of science in the making and want to be transparent about it
We are institutionalized as ‘Open Science Initiative’
We have already some money from the Utrecht University Fund, being very supportive and enthusiastic
We have promotional material
We’ve sent a call for papers to several Master's programmes
We are even part of the UtrechtInc Student Entrepreneurship Incubator
We are going to give talks. We started in Strasbourg last week, got the stage at this very moment, will give a short talk at Woudschoten, are present at a post-humanist conference in Lille and are invited speakers at the GSA symposium of Molecular Biology at the University of Cambridge
There will be a newsletter, by which we can keep those being interested updated. Please sign up at this moment

We have sent a pre-call for papers, calling for abstracts from the 1st of June. Our planning is to start the call for papers on August 1st until the 1st of December. The peer-review and editing cycle will hopefully be finished at around the summer of 2020. We hope to choose about five suited articles. Because three of our project-members hold a Master's and have a network in neuroscience and psychology, our first issue will thematically (and pragmatically) be scoped as about ‘sciences of the mind’.

As I’ve said at the beginning of my talk: we are here to receive critical questions and suggestions. Of course, we hope you all like our idea. However, the worst thing that could happen at this moment, is that you all give us a tap on the back, say ‘good job’, and meet us with great silence.

Thank you.

Becoming an Open Access Online Journal

Stefan Gaillard — Wed, 12 Jun 2019 16:17:23 GMT

You would be surprised, but if you google ‘how to set up a scientific journal’, quite a decent amount of results show up. As if you would type ‘how to cook rice’ or ‘how to tie a tie’, the internet awaits your thirst for step-by-steps guides to solve practically everything.

If, like us, you are interested in setting up an open access online journal; you’ll find a variety of resources, tips, and fairly detailed expected timing and budget predictions. However, many of these guides assume some aspects that us, as junior academics-to-be, lack. The how-to articles on setting up the infrastructure needed for a journal have been for us a rough sketch of a path that we have been traveling mostly, and proudly, off-road.

What are the elements needed to set up the infrastructure of an online journal? And what are their necessary associated skills? This guide made by a Professor of Literature at the University of London identifies several key components. We've summarized it as follows, and have realized the skills necessary for each component. Sometimes we’ve had what it takes; but more often than not, what brought as forward was the policy of trial and error, and learning-as-we-go.

International Standard Serial Number

The first conundrum we faced was with getting an International Standard Serial Number (ISSN), an eight-digit serial number used to uniquely identify a serial publication, such as a magazine. To be able to get an ISSN, we first needed to have published several original pieces already, such as articles or books. However, one of the reasons for getting an ISSN in the first place was to show prospective authors that we were a serious journal. As a solution, we decided to notify our first contributors that the journal would get an ISSN as soon as the first few articles were published.

CrossRef Membership and the Digital Object Identifier

You have probably seen the DOI (Digital Object Identifier) in the articles that you read. It looks something like this: 10.1038/s41562-018-0522-1, where the prefix (10.1038) refers to the publisher (in this case, Nature), and the suffix (s41562-018-0522-1) is the unique handle referring assigned to the given publication (in this case, ‘A problem in theory’, a recent article definitely worth reading with an interesting take on the replication crisis in psychology). The DOI is the unique numerical code assigned to an article, conference proceeding, data set or other piece of academic work. It is a way to assure that referencing between publications remains functioning and stable. The DOI and CrossRef infrastructure helps build meta-data on the those publications and facilitates linkage and referencing between material of different publishers. CrossRef is a non-profit organisation, but their service comes at a cost, and the annual membership is $275.

Open Journals

The editorial process involved back and forth communication between editors, reviewers and authors. We decided to use the Open Journal Systems software, an open source tool where the whole editorial pipeline, indexing process and online publication are integrated. However, to truly belong to the Open Science Community, we needed to have an open access policy and a copyright and publication agreement that honoured the spirit of open science. Luckily, many such open access policies already exist, and in the spirit of openness, they are free to use. In addition, our copyright and publication agreement could be based on existing Creative Commons Attribution Licenses, ensuring that we could grant the authors the copyright while also enabling the free distribution of the articles.

Conclusion

Needless to say these are practical issues with a significant importance. However, the infrastructure of your journal won’t hold unless your concept is well developed and a team of dedicated experts is at work. The editorial board (editor-in-chief, editors and reviewers) plays the primary role in ensuring the quality and innovation of your journal. Soon, our new editor-in-chief, Davide Cavalieri, will write about the editorial pipeline in detail, highlighting the roles and responsibilities of each of the actors.

Open Science Community Utrecht

Stefan Gaillard — Mon, 03 Jun 2019 16:06:20 GMT

One of our first alliances was with the Open Science Community Utrecht (OSCU). Stefan was already a member, so it was relatively easy for him to come into contact with the founders of OSCU. Firstly, we asked whether they were willing to include a paragraph about us in their monthly newsletter. The founder of OSCU then came to us with the suggestion that we would become an OSCU initiative. OSCU initiatives are projects set up by OSCU members, which are featured on the website. This way, open science supporters can come together to support each other and work towards attaining goals together. The following text was put on the website:

“A few students at Utrecht University, one of them our very own member Stefan Gaillard, have launched a new journal. And not just a journal, but a journal dedicated to providing a platform for scientists who did not get the research results they wanted. The journal is appropriately called Journal of Trial and Error. Specifically, the journal will publish high quality research that failed to reject the null hypothesis. Articles will be double-blind peer-reviewed. In addition, the scientific articles will be matched with humanities articles that provide reflection from a historical, philosophical, or sociological perspective. Publishing research that failed to reject the null-hypothesis is important for battling publication bias and in turn the replication crisis. Being honest about the amount of research that comes to different findings, and how it comes to those, should be an integral part of Open Science.

The founders are looking for constructive criticism on their project and potential editors or peer reviewers. The first issue will be on the sciences of the mind, meaning psychology and neuroscience. Interested people can send a mail to info@jtrialerror.com.” In addition, we were mentioned in the May newsletter. We are very happy with the institutional support of OSCU. The founder also offered his help in distributing our first call for papers, which will launch this month (June 2019)!

Our First UtrechtInc Meeting

Stefan Gaillard — Tue, 21 May 2019 20:26:49 GMT

We must admit that we were a little nervous for our first UtrechtInc meeting, exactly one week ago. Our contact person, let’s call him John, was someone we had never met before and all his credentials we could find online were somewhat intimidating. However, he turned out to be a very nice guy. After some jokes about the Dutch language capacities of foreign PhD students and expats, we sat down to talk.

First, we explained the concept of our journal – again. Understandable, since John had only seen some rough outlines of our concept and had not been there during the pitch. Yet, we couldn’t help but feel like there had to be a more efficient system of informing every new contact person about the journal. If only we were famous; we could give just one interview for a big news outlet and everybody would know what our plan entailed…

At least John was very enthusiastic about the Journal of Trial and Error. Sometimes he was shocked about the current practices in science, which only fueled his desire to help us. That desire was more down to earth than ours though. Every time we suggested providing a service for free, we were met with skepticism. Ideals were nice, but having a startup that would survive the initial phase would be even nicer.

John gave us some concrete advice that we planned to incorporate into the execution of our plans for the coming weeks. Unless it was needed to gain certain funds, we wouldn’t become a registered institution just yet. This would save both money and manpower, two crucial resources for early ‘startups’ (the sole thing our whole board agrees on is how terrible this buzzword is). In addition, we would use the first call for papers as a prototype of sorts. We would treat it as seriously as any finished product, but also use it to inform ourselves how we could better the subsequent calls. We were slowly becoming entrepreneurs. Not to worry though. You can take the humanities student out of the humanities, but you can’t take the humanity out of the humanities student.

On How the JOTE Project Has Been Evolving

Max Bautista Perpinya — Wed, 01 May 2019 15:58:11 GMT

A few months have passed since the Journal of Trial and Error’s first public appearance at the Descartes’ Christmas colloquium. Since then, we’ve had several exciting developments.

We have thought long and hard about the journal’s aim and the specifics of its content. The idea of publishing negative results, steaming from the idea that “there is a difference between what is researched and what is published”; was in its infancy at the Descartes’ colloquium. We have followed this thought and found the match between ‘negative results’ and authors: Master's students. We have also developed further the idea of the ‘reflection article’, a humanities-based approach to science in the making in which authors reflect on 'failed' research projects or publications.

We have made efforts in reaching out and discussing the idea with scientists and humanities scholars. We have been in contact with several individuals and institutions throughout Europe, including master programmes and researchers. In June 2019 we will present our idea at the Gewina (Dutch and Belgium History of Science Society) Woudschoten Conference.

Furthermore, we are excited to announce that we have been invited to give a talk and share our ideas at the LMB-GGNB student symposium in Cambridge in July 2019. LMB-GGNB stands for the two organisers: the Laboratory of Molecular Biology (LMB) at the Cambridge Biomedical Campus; and the Göttingen Graduate School for Neurosciences, Biophysics, and Molecular Biosciences (GGNB) in Göttingen, Germany. We will be included in the ‘Global Collaborations’ session, were we aim to shed some light onto the issue of publication bias and the replication crisis as the pressing issues in scientific communication. We will propose our model of the journal as a form to alleviate and reflect on those issues.

As you can see, the different publics of these conferences are as as perhaps C.P. Snow described. And it is this gap that we aim to bridge.

And there is more exciting news: we have been accepted in the pre-incubation program of UtrechtInc Students, the incubation programme for upcoming students start-ups. Our initiative is to take our first steps to be consolidated as a formal organisation. In the following months we will further develop our business plans and marketing strategies, because good ideas need to be made public in just the right way.

Looking back, the Journal of Trial and Error has made some progress. The word is out, the website is running, researchers are interested, institutions are beginning to take a glance. Our initiative to construct a platform which remodels scholarly publication (of both science and humanities research) is optimistic and idealistic, and we are glad that we are taking our first steps into formalising this idea.

The Most Wonderful Time of the Year

Stefan Gaillard — Tue, 26 Mar 2019 21:44:53 GMT

Stefan Gaillard, our secretary, remembers promoting our Journal for the first time.

It truly was the most wonderful time of the year: the last Descartes Centre Colloquium of 2018. The Descartes Centre, of course, is a place of legend, it is right up there with El Dorado, the Vatican Archives, and Harvard Business School. The thing that sets the Descartes Centre apart is that it is the only one of the four that is a joint collaboration of all seven Faculties at Utrecht University. In their own words, the Descartes Centre ‘contributes to the social and political debate by critically reflecting on the foundations and the dissemination of scientific knowledge and the role of rhetoric in these processes.’ Exciting times indeed!

How does the Descartes Centre achieve these contributions in practice? Firstly, the Descartes Centre facilitates research into the humanities aspects of science by bringing scholars together and providing institutional support. Secondly, it provides education on the Bachelor and graduate level. At the Bachelor level, two optional minors are offered to Bachelor students from all faculties. At the graduate level, the History and Philosophy of Science Master ‘offers a unique opportunity to study the foundations, practices, and culture of the sciences and humanities from a historical and philosophical perspective’. This is the program most of us are following. Lastly, the Descartes Centre hosts several colloquia throughout the year, supporting research and education.

Of these colloquia, the December colloquium is a special one, semi-centered around Christmas and with the opportunity for students to give a talk as well. Because nothing screams Christmas more than launching a journal for failed science, we decided to give a talk. All four of us. It went as well as you might expect, which is to say… not too terrible! Less cynical people might even say it went quite well. Amidst talks about Santa Claus and Sinterklaas, Donald Trump and rakes, and the history and philosophy of wine, we steamed ahead to deliver the latest news on the replication crisis, publication bias, and awesome journals. Our talk was received with thunderous applause (as was every other talk that day), with sadly no time left for questions or criticism. Our newsletter sign-up was quickly filled with the information details of around four of our classmates who had also attended the colloquium (if you’re reading this, we appreciate you!).

Afterwards, there was plenty of time for us to toast our success with free wine, beer, and fancy food. We planned the next step in our nefarious plan to publish serendipitous science. Because at the end of the day, those are “failed” science findings: occurring or discovered by chance in a happy or beneficial way. Scientists often don’t see it that way due to the frustration surrounding “failed” science, but we - perhaps somewhat naively - hoped to change the perceived drawbacks and benefits of unexpected results.

By given our talk, we had hoped to gain feedback from the community and perhaps even some institutional support. We did not succeed in the former, but there was already some speculation that we might be able to get some help from the Descartes Centre. Our next step was to explore the options for structural support and to find a new platform to ask for critique. The latter we found at the Open Science Community Utrecht...

Humble Beginnings

Max Bautista Perpinya — Mon, 11 Mar 2019 19:25:00 GMT

From an Open Science conference to the founding of a Journal: Max Bautista Perpinyà, our treasurer, thinks back to those first days of JOTE.

Underlying our interest in establishing a humanistic science journal, was a need of constructing, not deconstructing, and by doing so, we wanted to positively contribute to the intellectual scene. We were all motivated by an optimistic sense of science and, though we were not blind to its pitfalls, wanted to create a space for scientists to self-reflect on their work, and for humanists to provide a theoretical ground for these reflections. Open Science, such a hot topic in 2018, was the spark needed for a few History and Philosophy of Science students to try, to essayer, to envision what the dialogue would be between the so called scientists and humanities scholars would look like. The “erring” seemed to us a fertile ground to start this conversation.

In our initial team we had an array of experiences and backgrounds that provided evidence for a need of reflection. Specifically, we had a team background in psychology, neuroscience, technology studies, and a historical interest in the social and medical sciences. The publication bias in psychology and neuroscience, that is the tendency to publish only hypothesis-confirming results, seemed to us clear examples of disciplines in need of reflection. Where does this bias arise from? Is it, as Francis Bacon advanced in 1621, “the perpetual error of the human intellect to be more moved and excited by affirmatives than by negatives” that prompted scientists to hide, or even not see, failures and mistakes? Was it then a cognitive tendency towards the positive, or could we bring the issue of publication bias towards a more down-to-earth explanation?

During the weeks following the Open Science lecture, we spoke with scientists of different disciplines and found evidence for a perhaps a more practical consideration. We identified some causes, such as the lack of motivation to publish “failed experiments”. Writing about an experiment that did not go as planned – or just writing an article in general – is still time consuming. After all, why would I, as a scientist, go the extra mile and spend a week or two writing, structuring, and proofreading an article about an experiment which “failed”, when I already know where the cause of the failure is (or maybe I don’t even know, so what is then the conclusion?) and expose myself to the discredit of my peers? The reluctance of the scientist to publish errors seemed like a fair explanation to us, but also a fence to jump over in our ambition to create an open space to talk about failure.

We considered other options and asked ourselves: who, if anyone, actually writes about experiments in a structured manner, with fairly advanced scientific understanding and legitimacy, independently of the results? The answer seemed clear to us: Master's students! From our experience, Master's students perform internships in labs for a brief period, normally below a year, where they carry out experiments under the supervision of PhD’s, post-docs or heads of departments. The brevity of the experimentation period may, as many Master's students can confirm, be a reason why the experiments go awry. Indeed, many of these early researchers spend large amounts of time setting up the experimental procedures, carrying out controls, improving the methodology… That is, for a few months they try and err. And those experiences, those detailed reports on failure and success, finally take the form of a Master's thesis. Although they do engage with theoretical implications and many times they do test relevant hypothesis, it is common for Master's thesis to be quite specific on methodology. Ask any Master student, they will tell you how for endless months used a technique that “didn’t work” and painfully tried to reconstruct it in a coherent way in their final written report. Even if they reach the hypothesis-testing stage, many are the cases were the students “don’t get the results they wanted” (scientific lingo for failing to reject the null hypothesis).

In a series of meetings, first around a beer at a wooden table in a brown Utrecht bar, then followed by early morning coffees in the cafeteria of the library, we arranged what seemed to be a good idea. We formed a board for, as we initially called it: “the project fails due to lack of failure”. Though admittedly the roles weren’t much delineated in the first months (and still aren’t) because the brainstorming was still ongoing and the tasks not fully defined, having a chair (Martijn), secretary (Stefan), treasurer (Max), design (Nayra) and legal (Jobke) board members gave us legitimacy and structure to our mission. Our own experiment had begun.

The Birth of a Journal

Martijn van der Meer — Sat, 09 Mar 2019 10:00:00 GMT

Chair Martijn van der Meer recalls the Open Science meeting that inspired the birth of the Journal.

The Open Science talk we attended on that very November 23rd was delivered by Dr. Katja Mayer, a sociologist working as a postdoc at the School of Governance of the Technical University of Munich, where she investigates practices of Open Science in computational Open Science and big data. At the point of writing this post (the end of February 2019) I don’t remember that well which topics Mayer’s presentation precisely considered, but her message was loud and clear: science needs to be as democratic as possible; the contemporary practice of “publish or perish” was a very, very bad thing; it appears to be impossible to reproduce a lot of scientific results. Mayer pointed out that the combination of the publication bias and the reproduction crisis may lead towards bad science and can be solved by the (slightly elusive) concept of ‘Open Science’. A short discussion after her talk made clear that it was very hard to change the institutionalized systems based on evaluating scholars on the amount of citations and publications and the power of academic journals which earn money by subscriptions. According to Mayer, the question is not whether or how we should get in touch with Baconian ideals of science, but who can provoke ‘the’ transition in the right direction.

She answered her rhetorical question by pointing at her audience. According to Mayer, only young researchers can make change happen. I began to wonder what historians and philosophers of science can do in this revolution against established academic institutions. As reflectors of science, we are not so much interested in the ‘best’ scientific method, ‘good’ results or ‘good’ ways of doing science. We like to emphasize how science – as many scientists are most of the time very well aware of – is not always (if at all) a linear story of success. We like to give attention at how scientists cannot always produce the desirable data, because doing research is hard and frustrating. We, as historians and (most of the time) philosophers, like to think of science as a practice.

After the talk, Max, Stefan, Jobke, and I began discussing to which extent the problems Open Science aims to solve play(ed) a part in our short academic experience. Max told how, as he experienced it, his thesis ‘failed’, because his results were not publishable: they only showed how the databases he had to use were unusable. While drinking one of the free beers, we had a fruitful and passionate discussion about the simple idea that publishing papers in which the hypothesis remains unanswered - papers with ‘bad results’ so to say - could potentially meet the goals of Open Science. Would this not meet the standards of the publication bias in a more doable way? Is it not thinkable that the publication of bad-practices is very productive? And, the biggest question of all: what if we, as idealistic young science enthusiasts, started such a journal ourselves?

Open Science Talk

Martijn van der Meer — Sat, 02 Feb 2019 18:10:00 GMT

How an Open Science meeting inspired the birth of a Journal, by our Chairman Martijn van der Meer.

Our story begins in November, ten weeks after the start of the History and Philosophy of Science (HPS) Master's program in Utrecht. After a course in philosophy of science, which was basically an introductory course in the analytic tradition of philosophical reflection of science, and a historiographical course on the history of the natural sciences, we had gotten a taste of what the rest of the program could be. Although there might be a (fundamental) difference between philosophical and historical reflections on science, both approaches have in common that they take science as their object, without necessarily participating in a practical or theoretical sense.

What is the purpose of such a program? A good question. Scientists are not necessarily waiting for criticism or vague reflections from people who don’t know ‘the technical stuff’ – something I learned from living as an enthusiastic history student in a house full of honours students in biology and theoretical physics. This in itself seems to be a good reason: historians and philosophers like to reflect on science, thus bridging the gap between those two cultures. But what do we reflect? ‘Reflecting’ sounds nice and intellectual, as it connects nicely to the rhetoric of defending the humanities against cynical (and stupid) questions about its relevance or usefulness. Humanist reflection leads to ‘create cultural value’ or is even ‘necessary for our democratic society’, and yet, it may surprise the beginning Master students in History and Philosophy of Science that scientists appear to be able to survive without their reflection. Are we, as young, naive and ambitious students, left with our own fascination for meta-reflections of science; for example emphasizing the Copernican revolution to show that we are justified in finding it interesting?

I’m not going to answer this question. Hopefully, we – authors of this blog – will find some answers to those very real problems over the course of our academic development. After all, let’s face it: besides feeding our hungry minds with cool stuff on dead and almost-dead scholars, at some point we have to make a living.

Why then, am I elaborating on these cliché and annoying criticisms on the relevance of philosophy and history? Because it provides a good explanation of why five students, while protecting themselves against the pressure of pursuing their intensive academic careers, invest time in initiating a Journal of Trial and Error. It may explain why we (accidentally) attended a colloquium on a very trendy and 21st century discussion about science, in which also (or even) scientists themselves participated. A topic about “ just science done right”.

The story of our Journal of Trial and Error starts at November 23rd 2018, a quarter past three in the afternoon, at the University Hospital in Utrecht, where we attended a colloquium on – hold your horses – Open Science.