Commentary: Running the Risk

Keywords: serendipity, paradigm, cultural differences

Commentary by Samantha Copeland & Stuart Firestein

Koen Beumer’s narrative provides an interesting example of how surprise and expectations are inter-related in research, particularly when we branch into areas about which we may have assumptions but don’t have knowledge. A particularly rich context for this kind of surprise is in international empirical research, as illustrated in the case of Beumer’s visit to Kenya: Taking oneself into a completely new context and truly engaging with what is happening there (with an ’open mind’, one might say) will often result in surprise. This surprise works both ways, however – something commonly missed in examinations of serendipity stories. Beumer expresses surprise at what he saw in Kenya during his fieldwork, but we would propose that he is also surprised at the depth to which he had held assumptions he thought he had been careful to keep fluid. This proposition comes from the context of the research itself: The study expected to find differences in approaches, and thus the researcher was open to new experiences during the fieldwork (indeed, the purpose of fieldwork is to step out of one’s own perspective while gaining the perspective of the subjects of one’s study). But the kinds of differences found were clearly unexpected insofar as they were unprepared for—the tools of analysis at hand were insufficient for the contrast and comparison that was planned.

In brief, an observation was made that was undeniably in conflict with assumptions. Importantly, the expectations expressed by the choice in methods revealed deeper assumptions about common elements that would be found in all (cultural) responses to risk. As Beumer notes, the ’conventional response’ was to keep the assumptions and try to bring other cultures into pace with Beumer’s own. Notably, however, the presence of ’risk talk’ in both Dutch and Kenyan contexts was found not to signify a shared conception of risk. Therefore, the science had to go deeper, upending expectations about what is common among cultures. The unexpected observations required evaluations of cultural attitudes toward risk, as they supposedly lay deeper than anticipated and were expressed in policy and regulations.

In situations like this, a new paradigm is needed—one must ’wonder’ about possible alternatives. When Robert Merton first described his ’serendipity pattern’ as an empirical method in the social sciences, he noted that the "unanticipated, anomalous and strategic datum exerts a pressure for initiating theory" (my emphasis; Merton, 1948, p. 506). The observation, or the thing observed, is not passive but directly calls on the researcher, insisting on a change in their theory to address what they are seeing. Both in the case of Beumer’s Kenyan visit and in the example that Merton draws on to illustrate the serendipity patten, prevailing assumptions about social causes are shown to be mistaken, new factors within that society are shown to have causal relevance, and social responses must be re-explained. We might call this both deconstructive and constructive in relation to the social theory, particularly in the case given by Beumer, where the call to action was answered by a change in approach toward inclusivity and greater understanding of both researcher and the observed culture.

Not only in science, but in other situations, we see such shifts in paradigms caused by serendipitous encounters with each other which highlight differences in expectations and cultures (Copeland, 2023). Beumer’s anecdote is a particularly good example of how scientific and ethical expectations of others can be interdependent, and thus how the shift in one (our expectations about how societies in general develop) can generate a shift in another (our expectations about how different societies think about values in relation to each other). A question that remains, however, is how great a role chance plays in these instances of ethical insight gained serendipitously. Could another researcher have had the same insights, in the same situation, as Beumer and his colleagues did? What might be the conditions for preparing oneself to shift to a new paradigm, and can they be created, for instance by insisting on fieldwork as part of engaging with cultures different than one’s own? Unexpected and valuable insights not only into the culture of study but also into one’s own assumptions and biases are the ultimate goal in cross-cultural, transdisciplinary, and ethical training and practice. Given cases like these, serendipity is potentially a valuable tool for generating these insights (Copeland & Ross, 2024).

But ’finding out afterward’ that something is not what one assumed raises a new issue, likely relevant to serendipity stories in general. This is the problem of post hoc findings and the concerns raised by those who feel there is a ’replication crisis’ in contemporary science. The concern is that, if you go into an experiment with one hypothesis and fail to prove it, the resulting data should not be used to advance an alternative hypothesis – rather, experiments are designed to produce specific kinds of data, and thus introduce bias into their results. These inherent biases limit the wider use of the data collected. In its worst form, repurposing data to advance a hypothesis different from the one the data were collected for is called p-hacking, the dangers of which the field is well aware of. But it is questionable whether this reflects common understandings of science. Additionally, there is value in thinking about what one’s surprising observations might mean. In cases of serendipity and in cases like those discussed here, reflection on one’s results and what they can mean is a legitimate change of perspective, resulting from new and unexpected data. As I remarked earlier, in science, "revision is a victory" (Firestein, 2012). One might say that hindsight is as important as serendipity in breaking through our basic assumptions, whether about each other or about what the scientific method can produce.

Reply by Koen Beumer

On the problem of post hoc findings

Samantha Copeland and Stuart Fierstein provide some fascinating reflections and raise interesting points about serendipity in scientific investigations. In this response, I would like to zoom in on the question of what conditions might help in preparing oneself to shift to a new paradigm.

Based on my experiences in Kenya, Copeland and Fierstein argue that one helpful approach to maximize the openness to serendipitous findings, may be to insist on fieldwork that makes the researcher engage with cultures different than their own. I would like to add that other features of the research design may enable or constrain openness towards serendipitous findings and contribute to paradigm shifts.

I found Copeland and Fierstein’s observations about ’the problem of post hoc findings’ especially helpful in exploring the topic of serendipity. They follow Merton in his observation that serendipitous findings can contribute to paradigm shifts by exerting pressure on the initiating theory (Merton, 1948). As such, receptiveness to the unanticipated, the anomalous, and the unexpected should be cherished and promoted, even if this receptivity may upset existing norms within a given paradigm. At the same time, however, we may also want to impose certain limits to such receptiveness due to what they call ’the problem of post hoc findings’: the problem that if an experiment is designed for testing one hypothesis, the data generated in that study cannot always be used to advance an alternative hypothesis, as this could introduce bias into the results.

The well-known example of p-hacking that Copeland and Fierstein mention illustrates this problem especially well. In p-hacking, a dataset is turned inside and out until something statistically significant is found. While this could be described as a strategy that is highly receptive to serendipitous findings, we can understand why this is generally not accepted due to the likelihood of such findings being spurious.

It is easy to imagine less fraudulent but equally problematic cases of the problem of post hoc findings. In the case of risk research, if one were to formulate a hypothesis about the carcinogenic effect of a substance, then one would usually select a specific type of rat in which such carcinogenic effects can be optimally tested. Using these rats, researchers may, for example, also observe anomalous eating behaviors. While certainly an interesting finding, this post hoc result may not provide evidence that is sufficiently robust to confirm or deny a hypothesis on effect of the substance on eating behavior. The effects may not have been systematically or comprehensively observed, and it may well be that effects on eating behavior require entirely different animal models.

This example helps us to understand how the research design may impose limits on the ability to take serendipitous findings seriously. Conversely, we can also imagine research designs that allow for a wider scope of ’post hoc findings’ to be taken seriously, without falling into the p-hacking trap. Ethnographic approaches and methods such as semi-structured qualitative interviews are, for example, specifically designed to be more attentive to the unexpected. When compared to structured survey techniques using multiple-choice questions, semi-structured interviews allow participants to articulate their views in their own terms, thereby broadening the scope of unanticipated and anomalous phenomena that can be captured. This was also pivotal in collecting data on ’risk talk’ in Kenya. Similarly, this method also offers more flexibility in redirecting the data collection towards anomalous findings. While in highly structured survey techniques the data collection is restricted to questions that have been determined beforehand, semi-structured interviews allow the researcher to decide on the spot to spend more time and questions on specific themes as these emerge during the interview. Only because of this flexibility, could we systematically collect data about the ’risk talk’ in Kenya.

The problem of post hoc findings offers a useful lens to explore the conditions for preparing oneself to shift to a new paradigm. This problem highlights the necessity of thoughtful research design to allow for serendipity while maintaining rigor. Such intentional design, in combination with engagement with cultures unknown to the researcher and sufficient preparation to accept surprising data through ethical and transdisciplinary training, positions us to best capitalize on serendipitous findings.

Want to continue the discussion? Comment on PubPeer! Click here to visit this article.

References

Copeland, S. (2023). Doing ethics, and the possible. Possibility Studies & Society, 1(4), 427-435. https://doi.org/10.1177/27538699231210907

Copeland, S., & Ross, W. (2024). Serendipity. In F. Darbellay (Ed.), Elgar Encyclopedia of Interdisciplinarity and Transdisciplinarity (p. 656). https://doi.org/10.4337/9781035317967.ch103

Firestein, S. (2012, July 12). Doubt is good for science, but bad for PR. Wired Magazine. https://www.wired.com/2012/07/firestein-science-doubt/

Merton, R. K. (1948). The bearing of empirical research upon the development of social theory. American Sociological Review, 13(5), 505-515. https://doi.org/10.2307/2087142