Post Hoc versus A Priori Exploratory versus Confirmatory Discovery versus Legitimation
Philosophers of science make a distinction between the context of discovery and context of legitimation (e.g. Duhem, 1905).
Traditionally, we don’t care where an idea comes from: E.g. Kukelé’s discovered the (ring) structure of benzene from a daydream about a snake swallowing it’s tail.
Traditionally, methodologists focus on the context of legitimation and ignore context of discovery. For traditional methodologists the main focus is on the truth status of some claim, not where the claim comes from.
In recent years, social scientists have used the terms exploratory and confirmatory (instead of discovery and legitimation). The aim of exploratory research is to use empirical observations to formulate new theories or hypotheses. The aim of confirmatory research is to test a theory.
Notice that theories in exploratory research emerge post hoc, whereas in testing-oriented research, the theories are a priori.
(Huron doesn’t like either of the terms “legitimation” or “confirmatory” since we never “legitimate” or “confirm” an idea. Currently, in social sciences methodology, the terms “exploratory” and “confirmatory” research are popular. The terms “testing” or “prophetic” would be appropriate instead of confirmatory. The terms post hoc and a priori are perhaps clearest.)
Notice there are an infinite number of theories and hypotheses. Obviously, we can’t test them all. Which should we test first? The order in which we address theories/hypotheses implies some prioritizing.
It is entirely legitimate to ask why we should give priority to testing one theory or hypothesis over another. We should ask “whose” theory? “whose” hypothesis?
We could well imagine that white English-speaking males have their own preoccupations, and so their research tends to reflect the concerns of white English-speaking males.
By way of example, Western medical research is strongly biased toward dealing with diseases endemic in the developed West—such as heart disease, cancer, and diabetes. However, the most common debilitating disease in the world is malaria. A quarter of a billion people are infected with the malaria parasite each year, and the resulting infection is permanent.
We need to guard against researcher bias—not simply in how we observe and interpret phenomena—but also in the choice of the phenomena we elect to investigate.
While it is understandable that researchers will tend to focus on issues that touch them personally, there is much to be said for taking a broader perspective.
Since historically, methodologists regarded the context of legitimation as much more important than the context of discovery, it followed that testing or confirmatory research was held in high esteem and exploratory research was held in low esteem.
However, one of the most important functions of exploratory research is to alert us to new phenomena, different ways of thinking about a phenomenon, and ultimately promising theories and hypotheses.
Non-quantitative research methods like reconnissance, descriptive, ethnographic fieldwork, etc. offer important tools for reducing egocentric bias related to theories and hypotheses. The medical researcher who casts a wide net is more likely to recognize the importance of malaria—even if the researcher has never encountered a person who suffers from malaria.
So here is Huron’s interpretation of the argument between quantitative and qualitative research: We have something to learn from both qualitative and quantitative approaches. As traditionally practised, both approaches have their blind spots.
Quantitative research:
Research driven solely by a priori hypothesis testing can blind researchers to novel insights and relationships which await discovery by vigilant observers. Quantitative research doesn’t pay sufficient attention to the creative discovery of new ideas.
Constant reliance on a strictly quantitative approach can lead researchers to become poor observers.
Rarely recognizes the existence of egocentric bias in the choice of hypotheses or theories to test.
Finally, things that are difficult to measure are in danger of being ignored, or lead to claims that they don’t exist. (This is known as the positivist fallacy, which we’ll discuss further later).
Qualitative research:
Tends to ignore hypothesis testing so the conclusions are less reliable.
Double-use data is rampant in qualitative research.
Since most qualitative research doesn’t invite failure, researchers tend to become overconfident (they are never wrong). Because the researcher is never wrong, the researcher tends to place excessive trust in his/her intuitions.
We need both approaches: exploratory methods to expand our horizons and alert us to ideas or phenomena we haven’t considered, and testing-prophetic methods to keep us honest and humble.
There is room for a “division-of-labors” approach, where one group of researchers specialize in exploratory research (generating lots of interesting ideas and theories to be tested), while a second group of researchers engage in solely confirmatory research (where ideas are formally tested). In fact, this division of labors is evident in modern physics. However, it is probably better that researchers are trained and become experienced in both approaches. Each approach helps empower the researcher to do good things.
References
Scott E. Page (2007). The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton: Princeton University Press.