Section V: OUTCOME EVALUATION

Section V: OUTCOME EVALUATION

Overview
I. Process vs. Outcome Distinction
A. Process – describing events within sessions associated with change
B. Outcome --effectiveness of therapy (TX ): A pre- post-change
II. Measuring Process Variables
A. Measures of change in--

Aspects of client verbalizations

Depth of emotion

Therapist interventions and consequences therefrom

B. Compliance with model

Did therapist do X, Y, or Z as stated in the treatment manual?

Someone rates session for therapist behaviors

III. Measuring Outcomes
A. Pre-, Post-, Follow-up

Controlling for repeated measures?

Intervening "treatments"?

Deterioration effect (does anyone get worse?)

B. Distinguishing between what is taught and what is therapeutic

People can learn what you taught them, but is that therapeutic ?

I can learn to spin around but what condition does that improve?

C. Ecological or External validity
    Who's responsibility is it to show that the skills taught in treatment
    ARE effective (i.e., the right ones) for bringing about a better
    relationship ?

D. Sources of information
From whom do we get reports of improvement?
Clients, Therapist (?), Outsiders

IV. Threats to Internal Validity ("How shall I count the ways?")

Review your "Consumer’s Guide to Evaluating Therapy Studies"

Non-random assignment to treatments

Differential therapist experience

Small n's per treatment

Inconsistent application of treatment elements

Failure to consider maturation effects

Attrition rates in treated group

Etc. etc. etc.

V. What IS the marital outcome variable?

First lecture of course, again!
Who decides? Who can tell?
Review assessment weapons stockpile!
What assessment devices do we have to index outcomes?

VI. Assessing Outcome Reliability

A. Box Score Approach

Count the number of studies that show any improvement

B. Meta-analysis: what it is and isn't

Meta-analysis techniques review the literature for studies that compare

treatment to a no treatment (a control condition). The studies are rated

as you did in HW-1; that way quality is controlled. Instead of counting

studies which show improvement, the idea is to express numerically

how much improvement there was (relative to a control standard) using

a comparison statistic that allows us to compare studies of different

measures and scales. (That way the units are made equivalent.)

Express change relative to a control group in standard units:

Effect Size = Tx mean – Control mean

S.D. Control

Note: This is the familiar z-score, where we subtract two mean scores

and divide by a standard deviation. The larger the result the more

the TX mean differs from the Control mean in terms of distance from

the control mean. (Also a measure of effect size ES.)

C. Calculating Effect Size
Based on area under the normal probability curve, use the formula:

(.50 + Z area ) * 100 = % Better Than

EG. Some Sample Values of ESs (based on values from a table of the normal

Probability curve):

If ES = Then % Better than=

.40 66%

.50 69%

.60 72%

.70 76%

.80 79%

.90 82%

1.00 84%

1.10 86%

Pre- Post- Comparisons: Technical Issues

I. Making Pre- Post- Comparisons

A. Therapy effectiveness means change from Pre to Post on some MEASURE

B. Real Change vs. Noise
Depends upon the Reliability of the Measure

C.When is a Measure reliable?

When repeated testings yield same result
When test is long enough to sample the domain adequately

E.g., 3 items to measure satisfaction vs 50 items

When the Standard Error of measurement is small

D. Standard Deviations (SD's) and Standard Errors (SE's)

SD = variability around the mean for a single sample
SE = variability of sample of MEANS around the "true" mean;

the SE sets the boundaries for any given score.

The larger the SE the more likely the real score falls within a

very wide window. SE is given by the formula--

SE =

Thus, when N is large SE gets smaller!

E. All of this is to get to this point: we want to express outcome success of
    individuals. How many people in the treatment group showed significant
    improvement? Improvement over what? Where they started from;
    how much is significant? If their change score is divided by the SE of the
    measure (e.g., DAS) we can determine whether they changed significantly.

                Reliable Change Index = RC = Post - Pre
                                                                      SE
         Note: SE in this case is the standard error of measurement, which is
                    estimated by the equation

where r_xx = the reliability of the test, and SD = the sigma of test mean.

The change is relative to error units! We can now count the percentage
of people who improved significantly.

F. Clinical vs. Statistical Significance
       It is possible to show statistically significant change based on the R-C score,
       but this may be different from clinically significant change. A person (couple)
       may move from a low DAS score of 75 to 85, pre- to post-therapy. But 85 on
      the DAS is still within the distressed range for that test!

G. Also note that we have the option of reporting individual spouse scores
        or couple scores. Couple scores would be the avergae of each person's
        score. But what to do when one person is 85 and the other is 110? Is the
        average meaningful in this case since it yields a score of 97.5, which is
        almost "normal" although one person is clearly distressed.

Use of Control Groups and Ethical Issues

I. Alternatives to Control Groups

          A. Treatment on demand (TOD): control over control group
          B. Minimal contact agreements
          C. Non-specific controls

II. Ethical Issues in Marital Therapy (AAMFT Ethics)

A. Who is the client?

B. Issues of confidentiality

C. Therapist values vs. client values

D. Therapist quality assurance issues

Back to Main