Evaluations of General Education Mathematics Programs

Recent studies comparing the effects of various mathematics programs on students' mathematics performance are summarized below. The programs achieving the greatest effects are characterized by validated features of quality instruction. For more information regarding these features of quality, see the web page of the National Center to Improve the Tools of Educators..

Conventions used in selecting the studies and reporting the results. Only evaluations based on comparison (i.e., control) groups of students are included. All reported differences between groups are statistically significant (i.e., greater than expected by chance alone). The term "substantial" is used to describe differences that are not only statistically significant, but also large in magnitude (e.g., performing at the 45th %ile versus the 20th %ile), and therefore assumed to reflect an important academic advantage for participating students.

Each entry summarizes information about:

Contents

Last modified: February 8, 1996.

Tarver and Jung, 1995

Target: 1st and 2nd Grade Mathematics

Major Finding: Implementing Connecting Mathematics Concepts (CMC) resulted in better mathematics performance than Math Their Way combined with Cognitively Guided Instruction representing recommended teaching practices of the National Council of Teachers of Mathematics (NCTM).

Subjects: 119 students who entered 1st grade in an elementary school in a midwestern suburb were assigned to 1 of 5 classesmended practices. Students who were still enrolled in the school at the end of 2nd grade served as the CMC and comparison groups respectively.

Duration: 2 years

Measures: The mathematics subtests of a nationally norm-referenced achievement test, the Comprehensive Test of Basic Skills (CTBS), were administered in September of year 1 to determine initial equivalence of groups, and were administered in May of year 1 and March of year 2 to determine differences in outcomes between groups. An experimenter-constructed math attitude survey designed to reflect the NCTM affective goals (e.g., learn to value mathematics, become confident in their ability to do mathematics, etc.) was also administered in May of year 1 and in March of year 2.

Results: Pretest biases favored the comparison group. Overall, CMC students outperformed comparison students on the CTBS by the end of 1st grade. By the end of 2nd grade CMC students performed substantially better than comparison students on all subtests. The percentage of students performing below grade level by the end of 2nd grade decreased substantially more for the CMC group than for the comparison group. The percentage of CMC students achieving ceiling scores at the end of 2nd grade far exceeded the percentage of comparison students (i.e., 20% versus none). Further, the CMC students' attitudes toward mathematics, as indicated by the attitude survey reflecting NCTM affective goals, were at least as positive, or more positive, than the comparison students.

Confounds: Due to the ceiling effect on the CTBS for CMC students at the end of 2nd grade, the reported magnitude of the difference between the groups may be a substantial underestimate of the true difference in mathematics achievement.

Discussion: The CMC program was more effective in meeting the needs of diverse learners and reaching NCTM goals than the practices recommended by NCTM (Math Their Way with Cognitively Guided Instruction). Both high and average performing CMC students learned to apply problem solving strategies and reason mathematically, as indicated on the Concepts and Applications subscale of the CTBS, to a greater extent than the comparison students. The program implementers and evaluators emphasized the importance of these CMC program features: (a) flexible instructional grouping that allowed the classroom teacher to provide supplementary instruction to lower performing students, (b) explicit explanations of mathematical concepts, of connections among concepts, and of problem solving strategies, subsequently reinforced with supplementary manipulative activities, and (c) application activities designed to ensure generalization and use of basic understandings.

Data Source: Tarver, S. G. & Jung, J. S. (1995). A Comparison of Mathematics Achievement and Mathematics Attitudes of First and Second Graders Instructed With Either a Discovery-Learning Mathematics Curriculum or a Direct Instruction Curriculum. Effective School Practices, 14(1), 49-56.

Contact person: Tarver@mail.soemadison.wisc.edu

Materials Source: Call 1-800-843-8855 (SRA/Mc-Graw-Hill) for further information on the Connecting Math Concepts program.

You may return to the table of contents or continue browsing.



Brent and DiObilda, 1993

Target: 1st and 2nd Grade Mathematics

Major Finding: Implementing Connecting Mathematics Concepts (CMC) resulted in better mathematics performance than the modified Holt Math Series, 1991. The Holt curriculum was modified and supplemented to teach specifically the objectives tested by the statewide CTBS test.

Subjects: All students entering 1st grade in one school served as the CMC group, and all students entering 1st grade in another school served as the CTBS-aligned Holt group. Both schools were in similar neighborhoods in Camden, New Jersey. The 23 CMC students and 27 Holt students who were still enrolled in their respective schools at the end of 2nd grade were considered stable. 76 CMC and 63 Holt students who were enrolled in their respective schools only during year 2 of the study were considered mobile.

Duration: 2 years

Measures: Results of the mathematics subtests of 2 nationally norm-referenced achievement testsined at the end of year 2. The MAT was only administered to stable students.

Results: Overall stable CMC students performed substantially better than stable CTBS-aligned Holt students on the MAT (i.e., 76th %ile versus the 50th %ile), and they performed as well as stable CTBS-aligned Holt students on the CTBS (i.e., 88th %ile and 87th %ile). Mobile CMC students performed better than mobile CTBS-aligned Holt students on the CTBS.

Confounds: A district-wide initiative to raise test scores on statewide tests provided all teachers in the district, except the selected CMC teachers, with training, supervision and assistance in teaching the specific set of objectives represented on Form U Level D of the CTBS, using the Holt Math Series. The CMC teachers received preservice and inservice training in delivering lessons and monitoring student progress and used prepublication versions of CMC materials.

Discussion: The magnitude of the difference in performance between groups on the MAT (i.e., within the above average range of the national distribution for the CMC group versus the average range for the CTBS-aligned Holt group) is particularly impressive given that the evaluation was conducted in an area that represents the highest percentage of children living in poverty in the nation.

Data Source: Brent, G., & DiObilda, N. (1993). Curriculum Alignment Versus Direct Instruction: Effects on Stable and Mobile Urban Children. The Journal of Educational Research, 86(6), 333-338.

Contact person: George Brent, 1-609-256-4738.

Materials Source: Call 1-800-843-8855 (SRA/Mc-Graw-Hill) for further information on the Connecting Math Concepts program.

You may return to the table of contents or continue browsing.



Vreeland et al., 1994

Target: 3rd and 5th Grade Mathematics

Major Finding: Implementing Connecting Mathematics Concepts (CMC) resulted in better mathematics performance than Addison-Wesley Mathematics (AW, 1985), a traditional mathematics curriculum.

Subjects: 2 classes of 3rd grade students (except for 10 students instructed in the next lower level of the curriculum by a different teacher) and 1 class of 5th grade students in a low socio-economic status (SES) area of Kalamazoo, Michigan, served as the CMC groups. AW groups included 1 class of 3rd grade students and 1 class of 5th grade students in the same school, 2 classes of 3rd grade students and a class of 5th grade students in a high SES school.

Duration: 1 year

Measures: Mathematics subtests of 2 nationally norm-referenced testsgram and the AW program.

Results: Overall the CMC students in both 3rd and 5th grade performed substantially better than AW students in the same school and also outperformed students in the high SES schools on the ITBS, the KTEA, and the Problem Solving test. AW group performance declined in grade 3.

Confounds: CMC teachers used a daily point system to reinforce participation and independent work.

Discussion: The CMC program was initially introduced to the school by a school psychologist who recommended implementation to a 5th grade teacher seeking assistance the year prior to the study. Based on the results of that pilot and the teacher's enthusiasm, implementation was expanded to the larger study described here. All the CMC teacherstively. In particular, CMC teachers reported these program benefits: (a) high student success rates, (b) increased academic en-gagement, (c) increased sophistication of problem solving skills, and (d) improved student confidence. The impressive empirical results of the study, along with teacher and parent support, led to administrative support to continue and expand CMC implementation to 85% of the students the following year. Despite its success, the district subsequently terminated the program.

Data Source: Vreeland, M., Vail, J., Bradley, L., Buetow, C., Cipriano, K., Green, C., Henshaw, P., Huth, E. (1994). Accelerating Cognitive Growth: The Edison School Math Project, Effective School Practices, 13(2), 64-70.

Contact person: Michael Vreeland, 1-616-343-1165, 5323 Rugby, Kalamazoo, MI 49008.

Materials Source: Call 1-800-843-8855 (SRA/Mc-Graw-Hill) for further information on the Connecting Math Concepts program.

You may return to the table of contents or continue browsing.



Wellington, 1994

Target: 1st and 4th Grade Mathematics

Major Finding: Implementing Connecting Mathematics Concepts (CMC) resulted in equivalent mathematics performance for 1st grade students, and substantially superior performance for 4th grade students, than a traditional basal mathematics curriculum.

Subjects: 1 class of 1st grade and 1 class of 4th grade students in each of 8 elementary schools in a suburb of Philadelphia served as the CMC groups. All other 1st and 4th grade students in the district served as the comparison groups.

Duration: Fall through February

Measures: CMC criterion-referenced placement tests were used as pretests to determine initial equivalence of groups. Teacher-developed criterion-referenced posttests consisting of items common to both curricula were used to determine differences in outcomes between groups.

Results: 1st grade CMC students performed as well as comparison students on both the pretest and the posttest. 4th grade CMC students performed lower than comparison students on the pretest and substantially higher than comparison students on the posttest. The initial difference between the groups in favor of the comparison students was reversed such that differences in outcomes strongly favored the CMC group. In one school all 4th grade CMC students receiving Chapter 1 services in mathematics no longer qualified for those services by the end of the year.

Confounds: Although all CMC teachers volunteered for the project, their willingness to implement CMC with integrity varied substantially.

Discussion: Based on the results of this study, the CMC implementation was expanded to districtwide adoption in grades 1 through 8 in the following year. The mean proportion of students performing above district-set criterion on district-developed tests in the spring of that year increased to 90% from 62% in the spring of the previous year. Two years later, the district terminated the program.

Data Source: Wellington, J. (1994). Evaluating a Mathematics Program for Adoption: Connecting Math Concepts, Effective School Practices, 13(2), 70-75.

Contact person: John Wellington, 1-610-853-4580.

Materials Source: Call 1-800-843-8855 (SRA/Mc-Graw-Hill) for further information on the Connecting Math Concepts program.

You may return to the table of contents or continue browsing.



Grossen and Ewing, 1994

Target: 5th and 6th Grade Mathematics.

Major Finding: Implementing the Systems Impact Direct Instruction videodisc mathematics programs (DIV), Mastering Fractions, Mastering Decimals and Percents, Mastering Equations, Roots, and Exponents, Mastering Ratios and Word Problems, and Mastering Informal Geometry, resulted in better mathematics performance than Scott Foresman's 1991 Exploring Mathematics programmenting the teaching practices recommended by the National Council of Teachers of Mathematics (NCTM).

Subjects: 58 fifth-grade students from a public elementary school in Boise, Idaho were stratified based on high, medium, and low school performance (previous teacher's evaluation) and then randomly assigned to 2 equivalent groups for treatment. Each treatment group included 2 talented-and-gifted students and 5 students with disabilities.

Duration: 2 years

Measures: 6 tests were used to evaluate differences in mathematics performance between groups. At the end of the 1st year, 2 criterion-referenced measures (1 from each of the 2 programs) were taken. At the end of the 2nd year, 2 problem-solving measures that were developed to align with the different goals of both programs

Results: Overall the performance of the DIV group was higher than the NCTM group. The DIV group scored higher on both year 1 measures, including the measure from the NCTM program. The DIV group also scored higher on the algebraic problem solving measure. There were no differences on the measure of nonroutine problems. All differences on the subscales of the standardized measures favored the DIV group. Closer analyses showed that the DIV curriculum benefited high-performing students as well as low-performing students.

Confounds: The DIV treatment also included the use of less-structured application activities that reviewed and maintained earlier taught skills, similar to the types of activities used in the NCTM treatment. A survey of parents indicated that children in the NCTM group did an average of 3 hours of mathematics homework each week. The DIV children did no homework but occasionally showed their in-class work and discussed what they were learning with their parents instead.

Discussion: The NCTM program emphasized nonroutine problems but was not more effective in achieving that goal. The DIV emphasized instruction in generalizable problems and was more successful than the NCTM program in achieving that goal.

Data Source: Grossen, B, & Ewing, S. (1994). Raising Mathematics Problem-Solving Performance: Do the NCTM Teaching Standards Help? Final Report, Effective School Practices, 13(2), 79-91.

Contact person: BGrossen@oregon.uoregon.edu

Materials Source: Call 1-800-221-1274 (BFA Phoenix Film) for further information on System's Impact Direct Instruction videodisc programs.

You may return to the table of contents or continue browsing.



Kitz and Thorpe, 1994

Target: Adult Remedial Mathematics

Major Finding: Implementing the Systems Impact Videodisc program Mastering Equations, Roots, and Exponents (videodisc) resulted in substantially better mathematics performance for adult remedial students than Macmillan's 1991 Intermediate Algebra textbook (text).

Subjects: 26 students in a summer program for learning disabled adults who met the Mastering Equations, Roots, and Exponents placement criteria served as subjects of the study. Students were randomly assigned to either the videodisc or text group.

Duration: 6 weeks

Measures: 2 criterion-referenced testsson textbook test bank items were used to determine differences in outcomes between groups.

Results: The groups were initially equivalent on the videodisc placement test, and the videodisc group performed below the text group on the textbook pretest. At the end of 6 weeks the videodisc students substantially outperformed the text students on both the videodisc and textbook posttests.

Confounds: 2 extensively trained instructors taught the text group while 1 instructor, relying only on the directions and suggestions in the videodisc teacher's guide, taught the videodisc group. Approximately 1 hour per week of instructional time for the videodisc group was used to relate videodisc topics to textbook topics.

Discussion: Program implementers and evaluators emphasized the importance in 3 videodisc program features in achieving impressive results: (a) sufficient distributed practice, (c) curriculum-fostered initial success and generalization, and (c) monitoring and individualized remediation when necessary.

Data Source: Kitz, W., & Thorpe, H. (1994). The Effectiveness of Videodisc and Traditional Algebra Instruction with College-Aged Remedial Students, Effective School Practices, 13(2), 76-78.

Contact:Kitz@vaxa.cis.uwosh.edu

Materials Source: Call 1-800-221-1274 (BFA Phoenix Film) for further information on System's Impact videodisc programs.