CONSUMER'S GUIDE: HOW TO EVALUATE METHODOLOGY
                                             OF THERAPY STUDIES

When comparing the differential effectiveness of therapies (Tx) the
         following points are relevant:

1. Mini-Experiment: comparison must assure that any observed Rx changes are due
to the active ingredients  of  the intervention. Therefore we must be able to rule out
alternative explanations of the observed  differences and/or changes. This process
of being able to rule out rival alternative hypotheses is referred   to as establishing
the "internal validity" of the study.

2. What intentional (i.e., designed to be effective) factors can cause "change"?

   A. Hopefully the specific effective intervention
   B. Providing correct (effective) dosage
   C. Having control over active ingredients
   D. Trained therapists

3. What extraneous (i.e., nonplanned for) factors cause change?

    A. Subject selection biases
    B. Nonrandom assignment to treatments (Note: very important!)
    C. "Maturation" effects
    D. Extra (collateral) therapy input (e.g., readings, inspirational groups)
    E. Regression to the mean effects (Law of Large Numbers)
   F. Practice effects from test taking
   G. Failure to assign therapists randomly to therapies
   H. Nonspecific effects

4. What should a good study include?
   A. Control group of some kind (Note: issue of ethics here)
   B. Available treatment manual (what was actually done!)
   C. Subjects representative of some defined population
   D. Independent evidence that Rx manual was followed
   E. Nonreactive pre- and post-measures indicating change
   F. Follow-up; are the effects long lasting?
   G. Adequate sample size
   H. Clearly stated hypotheses
   I. Corrections for making many statistical comparisons within the same study

5. Case studies, per se, cannot provide evidence that a therapy is better than no therapy.



 
 Return to Section III
  Back to Evaluating Outcome