“The tendency of empiricism, unchecked, is always anti-realist; it has a strong tendency to degenerate into some form of verificationism: to treat the question of what there is (and even the question of what we can – intelligibly – talk about) as the same question as the question of what we can find out, or know for certain; to reduce questions of metaphysics and ontology to questions of epistemology.”
—Strawson, G. (1987, p. 267)
Strawson, G. (1987). Realism and causation. The Philosophical Quarterly, 37, 253–277.
Minkowski posited a structure of four dimensional spacetime which treated time as another dimension like the three dimensions of space. “From now onwards space by itself and time by itself will recede completely to become mere shadows and only a type of union of the two will still stand independently on its own.”
One way to think about spacetime, known as eternalism or the block universe, is that everything that ever happened, is currently experienced by someone as happening (at a point in spacetime), or is going to happen, has a location in this 4D structure.
This challenges the idea that we have free will, since our future selves already exist somewhere in the 4D structure. But free will left us long before anyway if we assume that all events have causes (see Spinoza’s, 1677, Ethics Part 2 Proposition 48; or Strawson, 1994, on the Basic Argument). It is apparently very difficult to defend the possibility of free will.
Each point in spacetime depends on others close by (potential headaches here caused by quantum entanglement), and the dependencies stretch back to the Big Bang, which is down one end of the structure. For “dependencies” think information, passed between the points in spacetime, and satisfying the weird geometries of general relativity.
Time is ordered according to these dependencies. (And because of something about entropy which makes zero sense to me.)
Consciousness is part of the physical world, located in points in spacetime alongside bodies (and there is a lot of conscious stuff about according to panpsychism). Regions of spacetime containing someone from birth till death have millions of conscious experiences along the time dimension, each one frozen in time.
But if each conscious experience at each point in spacetime is frozen, how come it feels to us that the world is moving…?
The feeling of time passing is produced by us using perceptual memories, passed along the causal chain with everything else, and a comparison of these memories with whatever is perceived at a given location in spacetime.
It’s a hyper-complicated variation on how we recognise change in a graph of something changing over time, even though the graph itself is stationary. The graph has a record of what happened and we can use it to infer change.
So, although each conscious experience is frozen in time, it can feel like that conscious experience is moving if it is different to a perception – or series of perceptions – in memory.
If you have no perception, no memory, no way to compare memory representations, or if there is no information passed between two points in spacetime, then you have no sense of a passage of time between those two points.
This is a very rough sketch of how you could have a feeling of movement in an instant at a point in spacetime. BUT it doesn’t explain how all those instants are piped together. There would be millions of instants across spacetime, not a journey between them.
Possible way out: “some approaches to these questions center on the idea that an experience of something temporally extended is itself temporally extended. The experience itself takes time to unfold – in fact it takes as much time as the process that it is an experience of. Arguably, this makes it easier to understand the nature of temporal experience. We no longer have to ask how it is that multiple individual experiences, each succeeding the other, can add up to an experience of succession. Instead, we recognize that the fundamental experiential unit is itself temporally extended, and use this to explain how there can be an experience of a temporally extended content.” See Deng (2019).
“The positivist picture of the structure of scientific theories is now widely rejected. But the underlying idea that scientific theories are primarily designed to predict and explain claims about what we observe remains enormously influential, even among the sharpest critics of positivism.” (p. 304)
“Phenomena are detected through the use of data, but in most cases are not observable in any interesting sense of that term. Examples of data include bubble chamber photographs, patterns of discharge in electronic particle detectors and records of reaction times and error rates in various psychological experiments. Examples of phenomena, for which the above data might provide evidence, include weak neutral currents, the decay of the proton, and chunking and recency effects in human memory.” (p. 306)
“Our general thesis, then, is that we need to distinguish what theories explain (phenomena or facts about phenomena) from what is uncontroversially observable (data).” (p. 314)
Bogen, J., & Woodward, J. (1988). Saving the phenomena. The Philosophical Review, XCVII(3), 303–352.
‘A mechanism is one of the processes in a concrete system that makes it what it is—for example, metabolism in cells, interneuronal connections in brains, work in factories and offices, research in laboratories, and litigation in courts of law. Because mechanisms are largely or totally imperceptible, they must be conjectured. Once hypothesized they help explain, because a deep scientific explanation is an answer to a question of the form, “How does it work, that is, what makes it tick—what are its mechanisms?”’ (p. 182; abstract)
‘Consider the well-known law-statement, “Taking ‘Ecstasy’ causes euphoria,” which makes no reference to any mechanisms. This statement can be analyzed as the conjunction of the following two well-corroborated mechanistic hypotheses: “Taking ‘Ecstasy’ causes serotonin excess,” and “Serotonin excess causes euphoria.” These two together explain the initial statement. (Why serotonin causes euphoria is of course a separate question that cries for a different mechanism.)’ (p. 198)
‘How do we go about conjecturing mechanisms? The same way as in framing any other hypotheses: with imagination both stimulated and constrained by data, well-weathered hypotheses, and mathematical concepts such as those of number, function, and equation. […] There is no method, let alone a logic, for conjecturing mechanisms. […] One reason is that, typically, mechanisms are unobservable, and therefore their description is bound to contain concepts that do not occur in empirical data.’ (p. 200)
‘Even the operations of a corner store are only partly overt. For instance, the grocer does not know, and does not ordinarily care to find out, why a customer buys breakfast cereal of one kind rather than another. However, if he cares he can make inquiries or guesses—for instance, that children are likely to be sold on packaging. That is, the grocer may make up what is called a “theory of mind,” a hypothesis concerning the mental processes that end up at the cash register.’ (p. 201)
Bunge, M. (2004). How Does It Work?: The Search for Explanatory Mechanisms. Philosophy of the Social Sciences, 34(2), 182–210.
Realist evaluation (formerly known as realistic evaluation; Pawson & Tilley, 2004, p. 3) is an approach to Theory-Based Evaluation that treats, e.g., burglars and prisons as real as opposed to narrative constructs; follows “a realist methodology” that aims for scientific “detachment” and “objectivity”; and also strives to be realistic about the scope of evaluation (Pawson & Tilley, 1997, pp. xii-xiv).
“Realist(ic)” evaluation proposes something apparently new and distinctive. How does it look in practice? What’s new about it? Let’s have a read of Pawson and Tilley’s (1997) classic to try to find out.
Open any text on social science methodology, and it will say something like the following about the process of carrying out research:
Review what is known about your topic area, including theories which attempt to explain and bring order to the various disparate findings.
Use prior theory, supplemented with your own thinking, to formulate research questions or hypotheses.
Choose methods that will enable you to answer those questions or test the hypotheses.
Gather and analyse data.
Interpret the analysis in relation to the theories introduced at the outset. What have you learned? Do the theories need to be tweaked? For qualitative research, this interpretation and analysis are often interwoven.
Acknowledge limitations of your study. This will likely include reflection about whether your method or the theory are to blame for any mismatch between theory and findings.
Add your findings to the pool of knowledge (after a gauntlet of peer review).
Loop back to 1.
Realist evaluation has similar:
It is scientific method as usual with constraints on what the various stages should include for a study to be certified genuinely “realist”. For instance, the theories should be framed in terms of contexts, mechanisms, and outcomes (more on which in a moment); hypotheses emphasise the “for whom” and circumstances of an evaluation; and instead of “empirical generalisation” there is a “program specification”.
The method of data collection and analysis can be anything that satisfies this broad research loop (p. 85):
“… we cast ourselves as solid members of the modern, vociferous majority […], for we are whole-heartedly pluralists when it comes to the choice of method. Thus, as we shall attempt to illustrate in the examples to follow, it is quite possible to carry out realistic evaluation using: strategies, quantitative and qualitative; timescales, contemporaneous or historical; viewpoints, cross-sectional or longitudinal; samples, large or small; goals, action-oriented or audit-centred; and so on and so forth. [… T]he choice of method has to be carefully tailored to the exact form of hypotheses developed earlier in the cycle.”
This is reassuringly similar to the standard textbook story. However, like the standard story, in practice there are ethical and financial constraints on method. Indeed the UK government’s evaluation bible, the Magenta Book (HM Treasury, 2020), recommends using Theory-Based approaches like “realist” evaluation when experimental and quasi-experimental approaches are not feasible. (See also, What is Theory-Based Evaluation, really?)
More than a moment’s thought about theory
Pawson and Tilley (1997) emphasise the importance of thinking about why social interventions may lead to change and not only looking at outcomes, which they illustrate with the example of CCTV:
“CCTV certainly does not create a physical barrier making cars impenetrable. A moment’s thought has us realize, therefore, that the cameras must work by instigating a chain of reasoning and reaction. Realist evaluation is all about turning this moment’s thought into a comprehensive theory of the mechanisms through which CCTV may enter the potential criminal’s mind, and the contexts needed if these powers are to be realized.” (p. 78)
They then list a range of potential mechanisms. CCTV might make it more likely that thieves are caught in the act. Or maybe the presence of CCTV make car parks feel safer, which means they are used by more people whose presence and watchful eyes prevent theft. So other people provide the surveillance rather than the camera bolted to the wall.
Nothing new here – social science is awash with theory (Pawson and Tilley cite Durkheim’s 1950s work on suicide as an example). Psychological therapies are some of the most evaluated of social interventions and the field is particularly productive when it comes to theory; see, e.g., Whittle (1999, p. 240) on psychoanalysis, a predecessor of modern therapies:
“Psychoanalysis is full of theory. It has to be, because it is so distrustful of the surface. It could still choose to use the minimum necessary, but it does the opposite. It effervesces with theory…”
Power (2010) argues that most effects in modern therapies can be explained by transference (exploring and using how the relationship between therapist and client mirrors relationships outside therapy), graded exposure to situations which provoke anxiety, and challenging dysfunctional assumptions – for each of which there are detailed theories of change.
However, perhaps evaluations of social programme – therapies included – have concentrated too much on tracking outcomes and neglected getting to grips with potential mechanisms of change, so “realist” evaluation is potentially a helpful intervention. The specific example of CCTV is a joy to read and is a great way to bring the sometimes abstract notion of social mechanism alive.
The structure of explanations in “realist” evaluation
The context-mechanism-outcome triad is a salient feature of the approach. Rather than define each of these (see the original text), here are four examples from Pawson and Tilley (1997) to illustrate what they are. The middle column (New mechanism) describes the putative mechanism that may be “triggered” by a social programme that has been introduced.
Poor-quality, hard-to-let housing; traditional housing department; lack of tenant involvement in estate management
Improved housing and increased involvement in management create increased commitment to the estate, more stability, and opportunities and motivation for social control and collective responsibility
Three tower blocks, occupied mainly by the elderly; traditional housing department; lack of tenant involvement in estate management
Concentration of elderly tenants into smaller blocks and natural wastage creates vacancies taken up by young, formerly homeless single people inexperienced in independent living. They become the dominant group. They have little capacity or inclination for informal social control, and are attracted to a hospitable estate subterranean subculture
Increased burglary prevalence concentrated amongst the more
vulnerable; high levels of vandalism and incivility
Prisoners with little or no previous education with a growing string of convictions – representing a ‘disadvantaged’ background
Modest levels of engagement and success with the program trigger ‘habilitation’ process in which the inmate experiences self-realization and social acceptability (for the first time)
Lowest levels of reconviction as compared with statistical norm for such inmates
High numbers of prepayment meters, with a high proportion of burglaries involving cash from meters
Removal of cash meters reduces incentive to burgle by decreasing actual or perceived rewards
Reduction in percentage of burglaries involving meter breakage; reduced risk of burglary at dwellings where meters are removed; reduced burglary rate overall
This seems a helpful way to organise thinking about the context-mechanism-outcome triad, irrespective of whether the approach is labelled “realist”. Those who are into logframe matricies (logframes) might want to add a column for the “outputs” of a programme.
The authors emphasise that the underlying causal model is “generative” in the sense that causation is seen as
“acting internally as well as externally. Cause describes the transformative potential of phenomena. One happening may well trigger another but only if it is in the right condition in the right circumstances. Unless explanation penetrates to these real underlying levels, it is deemed to be incomplete.” (p. 34)
The “internal” here appears to refer to looking inside the “black box” of a social programme to see how it operates, rather than merely treating it as something that is present in some places and absent in others. Later, there is further elaboration of what “generative” might mean:
“To ‘generate’ is to ‘make up’, to ‘manufacture’, to ‘produce’, to ‘form’, to ‘constitute’. Thus when we explain a regularity generatively, we are not coming up with variables or correlates which associate one with the other; rather we are trying to explain how the association itself comes about. The generative mechanisms thus actually constitute the regularity; they are the regularity. The generative mechanisms thus actually constitute the regularity; they are the regularity.” (p. 67)
We also learn that an action is causal only if its outcome is triggered by a mechanism in a context (p. 58). Okay, but how do we find out if an action’s outcome is triggered in this manner? “Realist” evaluation does not, in my view, provide an adequate analysis of what a causal effect is. Understandable, perhaps, given its pluralist approach to method. So, understandings of causation must come from elsewhere.
Mechanisms can be seen as “entities and activities organized in such a way that they are responsible for the phenomenon” (Illari & Williamson, 2011, p. 120). In “realist” evaluation, entities and their activities in the context would be included in this organisation too – the context supplies the mechanism on which a programme intervenes. So, let’s take one of the example mechanisms from the table above:
“Improved housing and increased involvement in management create increased commitment to the estate, more stability, and opportunities and motivation for social control and collective responsibility.”
To make sense of this, we need a theory of what improved housing looks like, what involvement in management and commitment to the estate, etc., means. To “create commitment” seems like a psychological, motivational process. The entities are the housing, management structures, people living in the estate, etc. To evidence the mechanism, I think it does help to think of variables to operationalise what might be going on and to use comparison groups to avoid mistaking, e.g., regression to the mean or friendlier neighbours for change due to improved housing. And indeed, Pawson and Tilley use quantitative data in one of the “realist” evaluations they discuss (next section). Such operationalisation does not reduce a mechanism to a set of variables; it is merely a way to analyse a mechanism.
Kinds of evidence
Chapter 4 gives a range of examples of the evidence that has been used in early “realist” evaluations. In summary, and confirming the pluralist stance mentioned above, it seems that all methods are relevant to realist evaluation. Two examples:
Interviews with practitioners to try to understand what it is about a programme that might effect change: “These inquiries released a flood of anecdotes, and the tales from the classroom are remarkable not only for their insight but in terms of the explanatory form which is employed. These ‘folk’ theories turn out to be ‘realist’ theories and invariably identify those contexts and mechanisms which are conducive to the outcome of rehabilitation.” (pp. 107-108)
Identifying variables in an information management system to “operationalize these hunches and hypotheses in order to identify, with more precision, those combinations of types of offender and types of course involvement which mark the best chances of rehabilitation. Over 50 variables were created…” (p. 108)
Some researchers have made a case for and carried out what they term realist randomised controlled trials (Bonell et al., 2012; which seems eminently sensible to me). The literature subsequently exploded in response. Here’s an illustrative excerpt of the criticisms (Marchal et al., 2013, p. 125):
“Experimental designs, especially RCTs, consider human desires, motives and behaviour as things that need to be controlled for (Fulop et al., 2001, Pawson, 2006). Furthermore, its analytical techniques, like linear regression, typically attempt to isolate the effect of each variable on the outcome. To do this, linear regression holds all other variables constant “instead of showing how the variables combine to create outcomes” (Fiss, 2007, p. 1182). Such designs “purport to control an infinite number of rival hypotheses without specifying what any of them are” by rendering them implausible through statistics (Campbell, 2009), and do not provide a means to examine causal mechanisms (Mingers, 2000).”
Well. What to make of this. Yes, RCTs control for stuff that’s not measured and maybe even unmeasurable. But you can also measure stuff you know about and see if that moderates or mediates the outcome (see, e.g., Windgassen et al., 2016). You might also use the numbers to select people for qualitative interview to try to learn more about what is going on. The comment on linear regression reveals surprising ignorance of how non-linear transformations of and interactions between predictors can be added to models. It is also trivial to calculate marginal outcome predictions for combinations of predictors together, rather than merely identifying which predictors are likely non-zero when holding others fixed. See Bonell et al. (2016) for a very patient reply.
The plea for evaluators to spend more time developing theory is welcome – especially in policy areas where “key performance indicators” and little else are the norm (see also Carter, 1989, on KPIs as dials versus tin openers opening a can of worms). It is a laudable aim to help “develop the theories of practitioners, participants and policy makers” of why a programme might work (Pawson & Tilley, 1997, p. 214). The separation of context, mechanism, and outcome, also helps structure thinking about social programmes (though there is widespread confusion about what a mechanism is in the “realist” literature; Lemire et al., 2020). But “realist” evaluation is arguably better seen as an exposition of a particular reading of ye olde scientific method applied to evaluation, with a call for pluralist methods. I am unconvinced that it is a novel form of evaluation.
Bhaskar’s critical realism emphasises a distinction between intransitive and transitive objects. I think the easiest way to see how the distinction works in social science (as opposed to say, geology) is as follows. Find all the social theorists and make them and their books and journal articles vanish. The things that are left are intransitive objects, e.g., people and social institutions likes banks and governments, and all the things they do even though no theorists are around to observe. The things that vanish with the theorists are all the transitive objects – the fallible accounts of how the various intransitive objects “work”.
It should be recognised that the theorists and their theories are intransitive objects too and theories influence social life, e.g., through the pop psychology jargon people use when they talk to each other. Also everyone theorises, not just professionals. But let’s not get tied up in knots.
Ontology is about the kinds of things that exist, including material and abstract “things” like numbers. Cruickshank (2004) argues that ontology is defined in two different ways by critical realists. Sometimes it refers to all the things, knowable and not, in the intransitive sense. Other times ontology refers to critical realists’ theories of what there is – these theories are transitive objects. But reducing what there is to what is known (philosophically) about what there is commits what Bhaskar called the epistemic fallacy – one of the key fallacies critical realists are trying to help us avoid.
Cruickshank concludes that Bhaskar shoots himself in the foot by making critical realist theories of ontology inevitably commit the epistemic fallacy (Cruickshank, 2004, p. 572):
“The problem though is that in defining the epistemic fallacy as the transposing of questions about being [ontology] into questions about knowing, Bhaskar has defined the said fallacy so broadly that any reference to what we know of reality (which may well be knowledge claims with a high degree of veracity) must commit this putative fallacy. Indeed the only way to avoid this fallacy would be to step outside knowledge to ‘see’ reality in itself.”
It’s a challenging debate, aiming for precise understandings of concepts like ontology and exploring the possibilities and limits of philosophical reasoning, but it seems unhelpful for the day-to-day work of doing social science.
Perhaps more helpful – and bigger than critical realism – is to emphasise the role of creativity in doing science. We can’t just go out and rigorously observe reality (whether social life or the cosmos) and somehow perceive theories directly. Although rigorous observation is important, science involves speculating about what might be out there and then working out what evidence we would expect to see if we were correct or if plausible alternative theories were correct.
My favourite analogy comes from cryptanalysis. We can systematically analyse letter and word frequencies in cyphertexts to try to spot patterns. But it helps to guess what people might be trying to say to each other based on something beyond the ciphertext, and to use those guesses to reduce the search space of possible encryptions.
Cruickshank, J. (2004). A tale of two ontologies: An immanent critique of critical realism. Sociological Review, 52, 567–585.
The core idea of Jenkins’ norm-relevancy account of gender is that someone’s gender is defined in terms of the gender norms they experience as relevant – whether or not they comply with those norms (there’s a lot more to the account than this – forgive me and please read the original). I’m not sure if this is enough to define gender; however, I think it’s an interesting idea for how people might decode their gender. Jenkins uses a crisp classical binary logic approach. This blog post is an attempt to explore what happens if we add probabilities.
Note that I’m simply using Bayesian networks because they do the sums for me. The direction of the arrows below is not meant to imply causation. Rather, the idea is from the assumption that someone is a particular gender, it is straightforward to guess the probability that a particular gender norm would be relevant. The Bayes trick then is to go in reverse from experiencing the relevance of particular norms to decoding one’s gender.
Let’s get started with some pictures.
The network below shows the setup in the absence of evidence.
The goal is to infer gender and at present the probabilities are 49-49% for man/woman and 1% for non-binary. That’s probably too high for the latter. Also I’m assuming there are only three gender identities, which is false. But onwards.
Each node with an arrow leading into it represents a conditional probability. The table below shows a conditional probability probability distribution defined for one of the norm-relevancy nodes.
So, in this case if someone is a man then this norm is 80% likely to be irrelevant; if someone is a woman then it is 80% likely to be relevant; and if someone is non-binary there is a 50-50 split. I’ve set up all the nodes in this pattern, just flipping the 80% to 20% and vice versa depending on whether a norm is for men or for women.
The idea then is to the use the Bayesian network to calculate how likely it is that someone is a man, woman, or non-binary based on the relevance or irrelevance of the norms.
I have not yet mentioned the Spaces node top left. This is a convenient way to change the prior probabilities of each gender; so in LGBT spaces the prior probability for non-binary raises from 1% to 20% since there are likely to be more non-binary people around. This also captures the intuition that it’s easier to work out whether a particular identity applies to you if you meet more people who hold that identity. See the picture below. Note how LGBT is now bold and underlined over top left. That means we are conditioning on that, i.e., assuming that it is true.
But let’s go back to cisgendered spaces.
Suppose most (but not necessarily all) of the male norms are experienced as irrelevant and most (but not necessarily all) of the female norms are perceived as relevant. As you can see below, the probability that someone is a woman increases to over 90%
Similarly, for the converse where most male norms are relevant and most female norms are irrelevant now the probability that someone is a man rises to over 90%:
Now what if all the norms are relevant? Let’s also reset the evidence on whether someone is in a cis or LGBT space.
The probability of being non-binary has gone up a little to 4%, but in this state there is most likely confusion about whether the gender is male or female since they both have the highest probability and that probability is the same.
Similarly, if all the norms are irrelevant, then the probability of non-binary is 4%. Again, it is unlikely that you would infer that you are non-binary.
But increasing the prior probability of non-binary gender, for instance through meeting more non-binary people in LGBTQ+ spaces, now makes non-binary the most likely gender.
To emphasise again, there are many more varieties of gender identity here and an obvious thought might be that gender nonconforming but still cis man or woman could apply – especially if someone views gender as closely coupled to chromosomes/genitals. I think it’s also interesting how the underdetermination of scientific theories can apply to people’s ruminations about identity given how they feel and what other evidence they have.
The situation can also be fuzzier, e.g., where the difference between one of the binary genders and non-binary is closer:
We don’t have conscious access to mental probabilities to two decimal places, so scenarios like these may feel equiprobable.
So far we have explored the simple situation where people are only aware of three male norms and three female norms. What happens if we had more, but kept the probability distributions on each the same…? Now we’re tip-toeing towards a more realistic scenario:
Everything works as before for men and women; however, something different now happens for non-binary people. Suppose all the norms are experienced as irrelevant (it works the same for relevant):
Now the most probable gender is non-binary (though man and woman are still far from zero: 24%).
This is true even in cis spaces:
Finally, there’s another way to bump up the probability of non-binary. Let’s go back to two gender norms, one male and one female. However, set the probabilities so that if you’re a woman, it’s 99.99% probable that the female norm will apply (and similar for men and male norms). Set it to 50-50 for non-binary. Now we get a strong inference towards non-binary if neither or both norms are relevant, even in cis spaces.
That’s as far as I have got for now. Interim conclusion then:
It is possible to view norm-relevancy through probabilities and as a sort of Bayesian self-identity decoding process.
When there is a small number of norms and (say) 80% chance of a norm being relevant for a particular binary gender, the prior probability of non-binary has a big impact on whether someone decodes their gender that way.
As the number of norms increases, it is easier to infer non-binary as a possibility.
Additionally, if there are only a few norms, but the probability that they apply for men and women is very high, then seeing them as all relevant or irrelevant is strong evidence for non-binary.
So, you have pledged allegiance to the big four critical realist axioms (Archer, et al., 2016) – what next?
Here are some ideas.
1. Ontological realism
What is it? There is a social and material world existing independently of people’s speech acts. “Reality is real.” One way to think about this slogan in relation to social kinds like laws and identities is they have a causal impact on our lives (Dembroff, 2018). Saying that reality is real does not mean that reality is fixed. For example, we can eat chocolate (which changes it and us) and change laws.
What to do? Throw radical social constructionism in the bin. Start with a theory that applies to your particular topic and provides ideas for entities and activities to use and possibly challenge in your own theorising.
Those “entities” (what a cold word) may be people with desires, beliefs, and opportunities (or lack thereof) who do things in the world like going for walks, shopping, cleaning, working, and talking to each other (Hedström, 2005). The entities may be psychological “constructs” like kinds of memory and cognitive control and activities like updating and inhibiting prepotent responses. The entities might be laws and activities carried out by the criminal justice system and campaigners. However you decide to theorise reality, you need something.
2. Epistemic relativity
What is it? The underdetermination of theories means that two theorists can make a compelling case for two different accounts of the same evidence. Their (e.g., political, moral) standpoint and various biases will influence what they can theorise. Quantitative researchers are appealing to epistemic relativity when they cite George Box’s “All models are wrong” and note the variety of models that can be fit to a dataset.
What to do? Throw radical positivism in the bin – even if you are running RCTs. Ensure that you foreground your values whether through statements of conflicts of interest or more reflexive articulations of likely bias and prejudice. Preregistering study plans also seems relevant here.
3. Judgemental/judgmental rationality
What is it? Even though theories are underdetermined by evidence, there often are reasons to prefer one theory over another.
What to do? If predictive accuracy does not help choose a theory, you could also compare them in terms of how consistent they are with themselves and other relevant theories; how broad in scope they are; whether they actually bring some semblance of order to the phenomena being theorised; and whether they make novel predictions beyond current observations (Kuhn, 1977).
You might consider the aims of critical theory which proposes judging theories in terms of how well they help eliminate injustice in the world (Fraser, 1985). But you would have to take a political stance.
4. Ethical naturalism
What is it? Although is does not imply ought, prior ought plus is does imply posterior ought.
What to do? Back to articulating your values. In medical research the following argument form is common (if often implicit): We should prevent people from dying; a systematic review has shown that this treatment prevents people from dying; therefore we should roll out this treatment. We could say something similar for social research that is anti-racist, feminist, LGBTQI+, intersections thereof, and other research. But if your research makes a recommendation for political change, it must also foreground the prior values that enabled that recommendation to inferred.
The big four critical realist axioms provide a handy but broad metaphysical and moral framework for getting out of bed in the morning and continuing to do social research. Now we are presented with further challenges that depend on grappling with substantive theory and specific political and moral values. Good luck.
Archer, M., Decoteau, C., Gorski, P. S., Little, D., Porpora, D., Rutzou, T., Smith, C., Steinmetz, G., & Vandenberghe, F. (2016). What is Critical Realism?Perspectives: Newsletter of the American Sociological Association Theory Section, 38(2), 4–9.