Huron’s Research Slogans
Motivated by truth, with no hope of Proof
There is no inductive proof. We are not in the business of proving something to be true. We would love to know the truth (if that exists), but we understand that we could never be sure of the truth, even if we had it. The best we can hope for is that what we observe is consistent with our theories.
The best research invites failure.
Give the world an opportunity to tell you that you’re wrong. (This is the essence of good research.)
We invite failure by testing predictions.
Test an idea by making a prediction, and then determine whether the observations are consistent with the prediction.
We recognize failure by drawing a line in the sand.
In order to make failure obvious, establish a criterion in advance that says, “If the evidence doesn’t cross this line, then I’ll admit failure.” In statistics, the line is referred to as the confidence level.
Aim not to be right, but to be not not right.
Instead of establishing The Truth, our more modest aim is to be not obviously wrong. When our observations turn out to be consistent with our hypothesis, we don’t claim that we are right; instead the observations suggest that our hypothesis may not be wrong.
Test hypotheses by operationalizing terms.
Translate all of the terms in a hypothesis into concrete things you can observe or measure. We can’t directly measure concepts like “sadness.” We have no choice but to measure things using imperfect rulers.
Operationalize, but don’t essentialize.
All concepts are inherently enigmatic and fuzzy. Terms like “melody,” “listen” or “note” can never be pinned-down. It is impossible to provide comprehensive definitions or grasp the essence of some concept. We are forced to approximate or estimate concepts through operational definitions — but don’t confuse the operational definition with the concept itself, and don’t assume that all concepts are “real.”
Compare, compare, compare.
Contrast a “treatment” condition with one or more “control” conditions.
The rhetoric of science is the rhetoric of prophecy.
People are most impressed when someone accurately foretells the future. Science is a form of rhetoric whose persuasive power resides in the testing of predictions. The rhetorical power of science comes not from scholars assembling evidence, but from scholars testing predictions.
Hindsight is 20/20.
Most things seem obvious in retrospect (hindsight bias). When the results aren’t obvious, humans are enormously gifted at coming up with explanatory accounts. We can make up a story for just about any set of data. Post hoc theories don’t have the same plausibility as a priori theories. The true test is making up the story first (i.e., prediction)! Prefer theorizing first, then collect (or examine) your data.
Reductionism is a method, not a belief.
We simplify problems, not because we believe problems to be simple, but because we believe problems to be complex. Restricting our gaze is a useful strategy for discovery.
Don’t try to explain the whole world at once.
Manipulate one variable at a time. Seek simplicity, even as you distrust it.
Generalize, but don’t universalize.
When presenting your results, frame them narrowly rather than broadly.
Avoid chronic hypothesislessness.
Exploratory and descriptive studies are important, but you can’t invite failure without testing predictions.
Beware of the post hoc theory.
The scholar who only offers theories after looking at the evidence is a scholar who is never wrong. Post hoc theorists don’t allow the world to tell them when their ideas are problematic.
From Question to Theory to Conjecture to Hypothesis to Protocol.
Start with a question, propose an explanatory theory, derive a conjecture, refine the conjecture into a hypothesis, then operationalize the terms of the hypothesis into a protocol. The protocol provides an action plan for how to carry out the research.
No causation without manipulation.
Causality cannot be inferred unless you manipulate one of the variables. The Experiment is the only type of study in which it is possible to infer causality. Correlational studies don’t allow us to discount the possibility of a “3rd variable.”
Don’t get stuck with sticky data.
Seek data independence. Ideally, each piece of data should be gathered from a different source. (Collecting independent data is another way of minimizing the effect of unknown third variables.)
The law of large numbers does not apply to small numbers.
Pay attention to sample sizes. The smaller the sample size, the greater the variability.
Always debrief.
Listen carefully to what people say about their experiences. Look for ways in which participants are misunderstanding the instructions. Be vigilant for possible demand characteristics — where the participant forms an idea about the experiment that confounds the results.
Make friends with a statistician.
Before you collect any data, talk with a statistician. Describe what you are planning to do and listen carefully to the advice. Statistical consultants are thrilled when people come and talk with them before the data are collected. Take advantage of their expertise.
Correct for multiple tests.
Each statistical test increases the likelihood of making a Type I (false positive) error. Repeating an experiment also increases the probability of a Type I error.
If you torture the data long enough it will confess to anything.
It’s not true that “statistics can prove anything,” but it is possible to manipulate data in ways that deceive yourself and others. Create a data analysis plan. Do not exclude outliers, normalize data, set conditions for excluding participants, or introduce post hoc tests without some principled prior reasoning. Be honest in reporting the analyses you carry out. Don’t hide multiple tests.
The plural of anecdote is not data.
Anecdotes are not randomly sampled and so are not representative. Anecdotes are easily recalled when they go against expectations. For example, some people have died in automobile accidents because they got tangled in a seat belt — even though seat belts save many more lives than they end. Be wary of anecdotal or informal evidence.
Randomize stimuli for each participant.
Minimize the effect of primacy (the first thing heard) and recency (the last thing heard) by providing a different order for each participant.
Write the paper first.
Begin by imagining what story you hope to tell. Writing down your story (in detail) will help you see what work is needed in order to make the story compelling. Most importantly, writing a detailed story first better invites failure.
Nurture your passion.
Your most precious resource in doing research is your own passion. Have fun. Don’t squander your enthusiasm — feed your passion.
Make time to think.
If you are always engaged in “busy work” then you are not a scholar. The essence of the contemplative life is leisure. Schedule time to do nothing but think. Chronicle your thoughts using a research diary. Busy work makes you tired, but thinking makes you smarter.
Choose projects from a long menu.
Don’t start with the first project you think of. Make a list of at least 10 projects before you begin. From your list, choose the project that looks to be the most exciting (nurture your passion), the most likely to produce positive results, and the one that appears to be the quickest to do.
Grow your projects.
Since research can fail, it is better to fail with small projects than with big projects. Big failures will dampen your passion. Start with small projects, and then pursue bigger projects as you gain experience and confidence.
Knowledge is communal.
Scholarship is a group activity. Read. Learn from the experiences of others. Ask questions. Read. Don’t hoard the fruits of your own labors. Share your own experiences. Read.
There are no competitors, only collaborators.
Resist the temptation to view other researchers as competitors. We all benefit from each others’ efforts. Treat everyone as a teacher from whom you can learn, and as a student you can nurture.
Don’t be afraid to add your own slogans.
In doing research, we learn not just about the topic of interest, but we learn better ways of doing research. Be prepared to make mistakes. Keep abrest of the latest literature on methodology, and be viligant for insights you may be able to contribute.
SOME SUPPLEMENTARY SLOGANS:
Splurge to converge.
Seek converging evidence. The most compelling theories are ones that have been tested in a variety of ways. Pursue alternative ways of operationalizing the terms of a hypothesis. Test different musical samples and different groups of participants.
Be skeptical, but not nihilistic.
Be skeptical, but don’t let your skepticism paralyze you from doing research.
Over-deliver and Under-conclude.
Aim for modesty and competence: over-deliver on evidence while understating your conclusions. Offer more evidence than your readers think necessary, and express your conclusions in a more tentative or circumspect manner than your readers expect.
Triumphalism is unscholarly.
What most displeases critics of empirical research is the smug, gloating, self-satisfied, triumphalism of science. Aim for modesty, even as you make your points. Remind critics that most of the unguarded and triumphalism language is the product of journalists, populists, and other non-researchers. The strength of empirical methodology is precisely that its practitioners choose methods that enforce intellectual humility.
Collaborate.
Musicians understand the value of group effort. The same thing holds with research. You can’t be a known-it-all, and you shouldn’t pretend to be. Seek out one or more collaborators — people who have similar interests, and (hopefully) different skills. You will learn more from working with someone else than from some book. Make sure you bring something to the table — something that makes it attractive for other people to work with you.
It’s never too late to quit.
Some problems are just plain hard. Some projects drag on far too long. Patience, dedication and stamina can be admirable qualities, but they can also stand in the way of using your time productively.