The value of high quality qualitative research

Here’s an interesting paper (Greenland & Moore, 2021) that used our (Fugard & Potts, 2015) quantitative model for choosing a sample size for a thematic analysis. The authors also had a probability sample – very rare to see in published qualitative research.

Key ingredients: they had a sample frame (students who dropped out of open online university courses and their phone numbers); they wanted a comprehensive typology of reasons for drop out and suggestions for retaining students; and they could complete each interview within an average of 15 minutes (emphasis on average: some must have been longer).

Here are the authors’ conclusions:

“This study’s research design demonstrates the value of using a larger qualitative probability-based sample, in conjunction with in-depth interviewer probing and thematic analysis to investigate non-traditional student dropouts. While prior qualitative research has often used smaller samples (Creswell, 2007), recent studies have highlighted the need for more rigorous sample design to enable subthemes within themes, which is the key purpose of thematic analysis (eg, Nowell et al., 2017). This study’s sample moved beyond simple thematic saturation rationale, with consideration of the level of granularity required (Vasileiou et al., 2018). That is, 226 participants had a 99% probability of capturing all relevant dropout reason subthemes, down to a 5% incidence level or frequency of occurrence (Fugard & Potts, 2015). This study therefore presents a definitive typology of non-traditional student dropout in open online education.”

It’s exciting to see a rigorous and yet pragmatic qualitative study.

References

Fugard, A. J. B. & Potts, H. W. W. (2015). Supporting thinking on sample sizes for thematic analyses: A quantitative toolInternational Journal of Social Research Methodology, 18, 669-684. (There’s an app for that.)

Greenland, S. J., & Moore, C. (2021). Large qualitative sample and thematic analysis to redefine student dropout and retention strategy in open online education. British Journal of Educational Technology.

On methodology

“I did not make a deliberate decision to adopt a particular methodology: I had the good fortune to work alongside gifted colleagues from backgrounds in different disciplines, and their various techniques seemed to be producing results. With hindsight, I should describe how one learns from both experiments and intelligent software in terms of the distinction that philosophers draw between the correspondence and the coherence theories of truth. An assertion is true according to the first theory if it corresponds to some state of affairs in the world; true according to the second if it coheres with a set of assertions constituting a general body of knowledge. Experiments provide information about correspondence with the facts, but they exert a dangerous pull in the direction of empirical pedantry, where the only things that count are facts, no matter how limited their purview. Computer programs provide information about the coherence of a set of assumptions, but they exert a dangerous pull in the direction of systematic delusion, where all that counts is internal consistency, no matter how remote from reality. Give up one approach and you turn into a Gradgrind, the teacher in Dicken’s novel Hard Times, whose only concern is with the facts; give up the other and you become an architect for the Flat Earth Society. Those, at least, are the dangers.”

From the prologue to Mental Models by Philip Johnson-Laird