Smallest Worthwhile Change: Individual vs Group
If you haven’t been living under a rock over past few years, you must be familiar with Will Hopkins work on magnitude-based inferences (MBI). One of the basis behind MBI is defining smallest practically meaningful change, or smallest worthwhile change (SWC). Together with Typical Error (TE) of a test, SWC is very needed in judging real individual changes of athletes (e.g. “Is Athlete’s B improvement in vertical jump height of 6 cm “real” given know TE of 2 cm? (which is variable/random error of the test)” ).
On the individual level SWC is defined either with the help of (a) domain knowledge (e.g. asking coaches what they think SWC is), (b) measurement error (if TE is 3cm, then SWC cannot be less than say 2 x 3cm, or 6cm, otherwise we are not very certain it is real change, rather the measurement fluke), or (c) using other athletes and using 0.2 x between athletes SD (or using 0.3 x individual SD, if we track athlete over multiple periods of trials).
What is problematic here is the automatic use of SWC defined on the individual level, to judge magnitudes of changes of the group level. For example, if the individual SWC is 2.5cm for the vertical jump, why it has to be the same for the group effects (for example mean change between pre- and post- samples).
In the video below I am using R simulation to brainstorm one idea using the “common language” effect size (CLES) approach is judging SWC on the group level. With this video, I wanted to start a conversation on this topic, and by no means it is extensive. Please let me know if you have any feedback or ideas on this very topic.
My colleague Ivan Jukic and myself are working on a pre-print paper for PeerJ on the topic of (bootstrap) magnitude-based predictions (bMBP), which is more prediction-based rather than estimation-based approach for judging validity, reliability and effects/changes quite similar to CLES. In bMBP, besides MBI, we utilize an approach quite similar to CLES when judging effect size using a random athlete, SWC, and TE. I Will let you know when this paper is available. Watch this space
- Website mentioned in the video: http://rpsychologist.com/d3/cohend/
- Download R source code HERE
Responses