I’ve moaned a bit about (what felt at the time to be a religion of) “effect size”. Recently Thom Baguley has published a paper on the topic, comparing standardised effects measures, which involve scaling with respect to the sample variance, with simple effects measures, which are expressed in the original units of measurement.
Baguley reviews some of the problems with standardised measures, all related to factors affecting sample variance. In general he advises reporting simple effect sizes, and preferably with confidence intervals. If you really want to use standardised measures, for instance to compare conceptually similar measures on different scales, then he advises against reporting absolute and “canned” judgements like “small”, “medium”, and “large”, arguing instead in favour of descriptions about the relative size of effects.
I like his Tukey quote:
“… being so disinterested in our variables that we do not care about their units can hardly be desirable.”
It does seem odd to focus on, e.g., how much variance is explained rather than actually characterising the nature of relationships between variables.
Baguley, T. (2009). Standardized or simple effect size: What should be reported? British Journal of Psychology, 100, 603–617.