Mr Justice Mostyn vs. vague, rhetorical applications of theory

A court case (GM v Carmarthenshire County Council [2018] EWFC 36) has ruled that a social worker’s “generalised statements, or tropes” based on attachment theory are not admissible evidence.

The full judgement by Mr Justice Mostyn has interesting thoughts on the valid application of theory and balance between theory and observation.

“… the local authority’s evidence in opposition to the mother’s application was contained in an extremely long, 44-page, witness statement made by the social worker […]. This witness statement was very long on rhetoric and generalised criticism but very short indeed on any concrete examples of where and how the mother’s parenting had been deficient. Indeed, it was very hard to pin down within the swathes of text what exactly was being said against the mother. […] [The social worker] was asked to identify her best example of the mother failing to meet L’s emotional needs. Her response was that until prompted by the local authority mother had not spent sufficient one-to-one time with L and had failed on one occasion to take him out for an ice cream. […] A further criticism in this vein was that the mother had failed to arrange for L’s hair to be cut in the way that he liked.”

There is also a detailed section on attachment theory:

“… the theory is only a theory. It might be regarded as a statement of the obvious, namely that primate infants develop attachments to familiar caregivers as a result of evolutionary pressures, since attachment behaviour would facilitate the infant’s survival in the face of dangers such as predation or exposure to the elements. Certainly, this was the view of John Bowlby, the psychologist, psychiatrist, and psychoanalyst and originator of the theory in the 1960s. It might be thought to be obvious that the better the quality of the care given by the primary caregiver the better the chance of the recipient of that care forming stable relationships later in life. However, it must also be recognised that some people who have received highly abusive care in childhood have developed into completely well-adjusted adults. Further, the central premise of the theory – that quality attachments depend on quality care from a primary caregiver – begins to fall down when you consider that plenty of children are brought up collectively (whether in a boarding school, a kibbutz or a village in Africa) and yet develop into perfectly normal and well-adjusted adults.”

Much to discuss!

From methods to goals in social science research

Note. This is quite a ranty blog post – especially the first two paragraphs. Readers may therefore wish to read it in the voice of Bernard Black from the series Black Books to make it more palatable. You may also be interested in this short BMJ comment.

History_of_Screaming.jpg

Onwards…

Many of the social science papers I read have long jargon-heavy sections justifying the methods used. This is particularly common in writeups of qualitative studies, though not unheard of in quantitative work. There are reflections on epistemology and ontology – sometimes these must be discussed by doctoral students if they are to acquire a degree.

There is discussion of social constructionism, critical realism, phenomenology, interpretation, intersubjectivity, hermeneutics. “But what is reality, really?” the authors ponder; “What can we know?” Quantitative analysis is “positivist” and to find or construct meaning you need a qualitative analysis (it is claimed).

Although I love philosophy, most of this reflection bores me to tears and seems irrelevant.

I think many differences between methods are exaggerated, clever-sounding –isms are fetishised, grandiose meta-theories concerning the nature of reality are used to explain away straightforward study limitations such as poor sampling. I bet some researchers feel they have to reel off fancy terminology to play the academic game, even though they think it’s bollocks.

But there are different kinds of research in the social sciences, beyond the dreary qual versus quant distinction as usually discussed. Might it be easiest to see the differences in terms of the goals of the research? Here are three examples of goals, to try to explain what I mean.

Evoke empathy. If you can’t have a chat with someone then the next best way to empathise with them is via a rich description by or about them. There is a bucket-load of pretentiousness in the literature (search for “thick description” to find some). But skip over this and there are wonderful works which are simply stories. I love stories. Biographies you read which make you long to meet the subject are prime examples. Film documentaries, though not fitting easily into traditional research output, are another. Anthologies capturing concise, emotive expressions of people’s lived experience. “Interpretative Phenomenological Analyses” manage to include stories too, though you might have to wade through nonsense to get to them.

Classify. This may be the classification of perspectives, attitudes, experiences, processes, organisations, or other stuff-that-happens in society. For example: social class, personality, goals people have in psychological therapy, political orientation, mental health problem, emotional experiences. The goal here is to impose structure on material, reveal patterns, whether it be interview responses, answers on Likert scales, or some other kind of observation. There’s no escaping theory, articulated and debated or unarticulated and unchallenged, when doing this. There may be a hierarchical structure to classifications. There may be categorical or dimensional judgments (or both, where the former is derived from a threshold on the latter), e.g., consider Myers-Briggs or the Big Five personality types. Dimensions are quantitative things, but there are qualitative differences between them.

Predict. Finally you often want to make predictions. Do people occupying a particular social class location tend to experience some mental health difficulties more often than others? Does your personality predict the kinds of books you like to read. Do particular events predict an emotion you will feel? Other predictions concern the impact of interventions of various kinds (broadly construed). What would happen if you voted Green and told your friends you were going to do so? What would happen if you funded country-wide access to cognitive behavioural therapy rather than psychoanalysis? Theory matters here too, usually involving a story or model of why variables relate to each other.

These distinctions cannot be straightforwardly mapped onto quantitative and qualitative analysis. As we wrote in 2016:

“Some qualitative research develops what looks like a taxonomy of experiences or phenomena. Much of this isn’t even framed as qualitative. Take for example Gray’s highly-cited work classifying type 1 and type 2 synapses. His labelled photos of cortex slices illustrate beautifully the role of subjectivity in qualitative analysis and there are clear questions about generalisability. Some qualitative analyses use statistical models of quantitative data, for example latent class analyses showing the different patterns of change in psychological therapies.”

People often try to make predictions without using a quantitative model. Others use quantitative approaches to develop qualitatively different groups. Cartoonish characterisations of the different approaches to doing social (and natural) science research stifle creativity and misrepresent how the research is and could actually be done.

 

A farcical proposal for mental health outcomes measurement

If you’re going to develop a questionnaire for something resulting in a total “score” — quality of life, feelings, distress, whatever — you’ll want all of the questions for one topic to be related to each other (as a bare minimum). This questionnaire probably wouldn’t be very “internally consistent”:

THE GENERAL STUFF QUESTIONNAIRE

  1. How often do you sing in the shower?
  2. What height are you?
  3. How far do you live from the nearest park?
  4. What’s your favourite number?

You won’t be able to do much with the result of summing answers to those together to a total score.

This one would:

THE RELIABLE FEELINGS QUESTIONNAIRE

  1. How do you feel?
  2. How do you feel?
  3. How do you feel?
  4. How do you feel?
  5. How do you feel?
  6. How do you feel?
  7. How do you feel?
  8. How do you feel?
  9. How do you feel?
  10. How do you feel?

However, you might wonder if questions 2 to 10 add anything… (So internal consistency isn’t everything.)

There are many ways to test the internal consistency of questionnaires, using the answers that people give. One is to use a formula by Lee Cronbach called Cronbach’s alpha. Answers run from 0 to 1. Higher is better (but not too high; see the second example above).

In England, it is now recommended (see p. 12 of Mental Health Payment by Results Guidance) to use scores on a “Mental Health Clustering Tool” to evaluate outcomes. I think there are at least two problems with this:

  1. It’s completed by clinicians. It’s unclear if service users even get to know how they have been scored, never mind to what extent they can influence the process.
  2. The questionnaire scores aren’t internally consistent.

The people who proposed the approach write (see p.30 of their report): “As a general guideline, alpha values of 0.70 or above are indicative of a reasonable level of consistency”. Their results: 0.44, 0.58, 0.63, 0.57. They also refer to previous studies showing that this would always be the case, due to “its original intended purpose of being a scale with independent items” (p. 30). So, by design, it’s closer to the General Stuff Questionnaire above: a list of “presenting problems” to be read individually.

Not only are clinicians deciding whether someone has a good outcome (are they really in the best position to decide?), but the questionnaire they’re using to do so is rubbish — as shown by the very people proposing the approach!

Undergraduate psychology students wouldn’t use a questionnaire this poor in their projects. Why is it acceptable for a national mental health programme?

Some claims psychology students might benefit from discussing

  1. It’s okay if participants see the logic underlying a self-report questionnaire, e.g., can guess what the subscales are. It’s a self-report questionnaire — how else are they going to complete the thing? (Related: lie scales — too good to be true?)
  2. Brain geography is not sufficient to make psychology a science.
  3. Going beyond proportion of variance “explained” probably is necessary for psychology to become a science.
  4. People learn stuff. It’s worth explicitly thinking about this, especially for complex activities like reasoning and remembering. How much of psychology is the study of culture? (Not necessarily a criticism.)
  5. Fancy data analysis is nice but don’t forget to look at descriptives.
  6. We can’t completely know another’s mind, not even with qualitative methods.
  7. Observation presupposes theory (and unarticulated prejudice is the worst kind of theory).
  8. Most metrics in psychology are arbitrary, e.g., what are the units of PHQ-9?
  9. Latent variables don’t necessarily represent unitary psychological constructs. (Related: “general intelligence” isn’t itself an explanation for anything; it’s a statistical re-representation of correlations.)
  10. Averages are useful but the rest of the distribution is important too.

Those who want to study what is in front of their eyes

Wise words from Colin Mills:

“I’m seldom interested in the data in front of me for its own sake and normally want to regard it as evidence about some larger population (or process) from which it has been sampled. In saying this I am not saying that quantification is all there is to sociology. That would be absurd. Before you can count anything you have to know what you are looking for, which implies that you have to have spent some time thinking out the concepts that will organize reality and tell you what is important.”

“… the institutionalized and therefore little questioned distinction between qualitative and quantitative empirical research is, to say the least, unhelpful and should be abolished. There is a much bigger intellectual gulf between those who just want to study what is in front of their eyes and those who view what is in front of their eyes as an instantiation of something bigger. Qualitative or quantitative if your business is generalization you have to have some theory of inference and if you don’t then your intellectual project is, in my view, incoherent.”

More thoughts on qualitative/quantitative research

All attempts to capture another’s phenomenological experience, either in a relatively bottom-up manner, through unstructured discourse (“qual”?) or more top-down through a questionnaire (“quant”?) get stuck eventually. You still can’t really know what it feels like to be the other.

Giving people a chance to go outside standardized questions makes it more likely an important experience will be reported. But we all have similar experiences; a lot can I think be gained by trying to capture the commonality. Basic questions can be answered like how many people (report) feel(ing) a particular way, how frequently, and how many of those enjoy, can cope with, or are bothered by the feeling. Simply knowing this population-level information can be helpful at an individual level.

The “quant” end is as subjective as the “qual” end of research. Data needs interpretation and the stats doesn’t know how to do that. Two people presented with the same ANOVA can and often do come to different conclusions as they think about the context around a study.