CONSUMER'S GUIDE: HOW TO EVALUATE METHODOLOGY
OF THERAPY STUDIES
When comparing the differential effectiveness of therapies (Rx)
the following points are relevant:
1. Mini-Experiment: Comparison must assure that any observed Rx
changes are due to the active ingredients of the intervention.
Therefore we must be able to rule out alternative explanations of
the observed differences and/or changes. This process, of being
able to rule out rival alternative hypotheses, is referred to as
establishing the "internal validity" of
the study.
2. Intentional (i.e., designed to be effective) factors: Identify the factors
can cause "change".
A. Hopefully the specific effective intervention
B. Providing correct (effective) dosage
C. Having control over active ingredients
D. Trained therapists
3. What extraneous (i.e., nonplanned for) factors cause change?
A. Subject selection biases
B. Nonrandom assignment to treatments (Note: very important!)
C. "Maturation" effects
D. Extra (collateral) therapy input (e.g., readings, inspirational groups)
E. Regression to the mean effects (Law of Large Numbers)
F. Practice effects from test taking
G. Failure to assign therapists randomly to therapies
H. Nonspecific effects
4. What should a good study include?
A. Control group of some kind (Note: issue of ethics here)
B. Available treatment manual (what was actually done!)
C. Subjects representative of some defined population
D. Independent evidence that Rx manual was followed
E. Nonreactive pre- and post-measures indicating change
F. Follow-up; are the effects long lasting?
G. Adequate sample size
H. Clearly stated hypotheses
I. Corrections for making many statistical comparisons within
the same study
5. Case studies, per se, cannot provide evidence that a therapy
is better than no therapy.