|
This is a summary of assessment materials from the NSF conference mentioned below. The
first page is an overview of the questions discussed and the methods suggested to answer
the questions. The subsequent pages are elaborations of the methods suggested for each
question.
Examples of Inquiry-Based Assessment Practices Developed and Used by
Science, Math, Engineering &Technology (SME&T) Faculty
Prepared by Susan Millar (smillar@engr.wisc.edu)
of the UW-Madison LEAD Center
for the NSF Conference
Shaping the Future of SMET Education: Managing Reform, Assessing Success, Harnessing
Technology"
Lincoln, Nebraska, May 28-30, 1998
(Condensed by Diane Ebert-My 11/28/98)
Examples Listed by Type of Assessment Question and Unit of Analysis
A. Methods Focused on "What are students
understanding?"
1. To help students during the semester the course is taught
Weekly reports (Physics-Etkina)
Minute Paper (from Cross and Angelo)
ConcepTests (Chemistry-Ellis/NT; physics-Mazur)
Challenge problems (Math-Emerging Scholars "guide on the side")
Student self-evaluations (Biology-Ebert-May)
2. To help design a better course/major over the longer run
Oral assessment (New Traditions "Chem 110 Study")
Concept retention into subsequent semester (Chem intro sequence)
Standardized exams (Math-Emerging Scholars/Chemistry-New Traditions)
B. Methods Focused on "How are students reacting to
course strategies?"
1. To help students during the semester the course is taught
Faculty-designed Mid-course Surveys (Freshman Engineering Design)
2. To help design a better course/major over the longer run
Student Assessment of Learning Gains (Chemlinks/ModularChem Consortia)
C. Methods Focused on, "Are our courses having a positive
effect on students’ interest in/attitudes toward the SME&T disciplines?"
To help design a better course/major over the longer run
Pre/post attitude surveys (Physics-Duncan)
Lecture attendance/student retention within course (Chemistry-New Traditions)
Note: At points in this document, the FLAG (Field-tested Learning Assessment Guide
for Science, Mathematics, Engineering, and Technology Faculty) website is mentioned. This
site is located at <http://newtraditions.chem.wisc.edu/FLAG/nt-flag.htm>.
WEEKLY REPORTS
Developed by faculty in physics
Introductory physics, Euginia Etkina, Rutgers University
A paper in which Etkina describes this method appears on the FLAG (highly
recommended.)
Learning Goals
Foster students’ ability to:
summarize what they have learned;
become aware of and concentrate on the key concepts in the course;
understand what they do not yet understand; and
develop the capacity to pose meaningful questions.
Assessment Questions
How can I immediately learn what kinds of difficulties my students are experiencing
while learning new material (so as to adjust the level of difficulty and my teaching
methods to match student needs and thus maximize their learning), while simultaneously
helping them achieve my goals for their intellectual development?
Inquiry-based Assessment Method
Students answer the following three questions in graded weekly reports:
a) What have you learned in class this week?
b) What is still unclear?
c) What I will ask you tomorrow about the material taught today?
Outcomes
Students:
improved their writing abilities;
became more aware of what physics concepts they did and did not really understand;
provided the instructor timely information on how well the course strategies are working;
provided questions that the faculty member could use to launch topics for subsequent class
meetings, and
framed questions for approximately 65% of the material on which the faculty member planned
to test them.
Instructor:
found the course more rewarding to teach.
MINUTE PAPER
This is a simple technique that has been used by many SME&T faculty.
Described in Angelo, T.A., and K.P. Cross. (1993). Classroom Assessment Techniques: A
Handbook for College Teachers. Jossey-Bass, San Francisco.
Goals:
Foster students ability to synthesize and integrate information and ideas;
Develop their ability to concentrate and listen, improve study skills and habits,
Foster dialogue about course concepts between instructor and members of a class.
Assessment Question
How can I learn what students are taking away from each class session in a way that
uses only a bit of class time, and allows me to hear from everyone?
Inquiry-based Assessment Method
At the end of the lecture, ask students to write answers to the following two questions
on a half-sheet of paper:
What was the most important thing you learned during this class?
What important question remains unanswered?
Outcomes
Faculty member learns what concepts students are/are not understanding and adjusts
teaching methods accordingly;
Students develop learning skills and competence in course material.
"CONCEPTESTS"
Developed by faculty in chemistry, physics
New Traditions Chemistry Project, headquartered at UW-Madison (see FLAG);
Arthur Ellis (http://www.chem.wisc.edu/~concept/CTinfo.html)
and
Eric Mazur (http://galileo.harvard.edu)
Goal
To foster peer learning while also getting students to realize that they hold certain
assumptions about how the world works which are not supported by evidence;
To actively used scientific principles presented in lecture to understand real-world
problems.
Assessment Question
How can I learn how well students are understanding my lecture, while also using the
power of peer learning to foster student understanding of the meaning of scientific
principles?
Inquiry-based Assessment Method
Conceptual questions are posed in the lecture room along with a few possible answers.
Students vote on the possible answers, then try to persuade their neighbors in the lecture
room that they are correct, and finally vote again.
Outcomes
This strategy fosters learning from peers and develops students’ analysis skills,
while also providing the instructor with immediate feedback as to how well the class is
following the lecture.
STUDENT SELF-EVALUATIONS
Developed by faculty in biology
Introductory biology for non-majors, Diane Ebert-May, Northern Arizona University.
A paper in which Ebert-May describes this method appears in the Synthesis and
Proceedings of "Indicators of Success in Postsecondary Science, Mathematics,
Engineering and Technology Education: Shapes of the Future,"
NISE Forum, February 1998 (contact the NISE at www.wcer.wisc.edu/nise/), and will appear on the
FLAG.
Goals
Help students develop the capacity to evaluate their own understandings and abilities;
Help students integrate the various concepts presented as the course moves along;
Motivate students to become more deeply engaged in biology; and
Help students improve their writing and communication skills.
Assessment Question
How can I achieve my goals for student learning while also obtaining substantive
feedback to guide changes in my teaching activities?
Inquiry-based Assessment Method
Ask students to write reflective statements during the course on particular course
strategies (such as use of cooperative groups or concept maps).
Require students to complete a reflective end-of-course survey, using the web. The survey
asks students to rank (on a 5-point scale) how well they accomplished each course goal,
and to comment on what in the course influenced them to choose each ranking.
Outcomes
Most students believed they accomplished each course goal to a moderate degree,
Comments helped both students and faculty understand why and how they students
accomplished the course goals.
PERFORMANCE ON ORAL ASSESSMENT EXAMS
Developed by faculty in Chemistry
Analytical Chemistry ("Chem 110"), John Wright, UW-Madison
Wright and LEAD researchers have written articles about Chem 110, plus LEAD has produced
various reports. These are listed on LEAD Products. Some material appears on the FLAG.
Goal and Teaching Strategies
The goals for this large analytical chemistry course for first year undergraduates are
to:
Improve students’conceptual, computational, laboratory, critical thinking, problem
solving, and group skills;
Improve students’ attitudes toward learning chemistry and each other;
Foster deep comprehension levels;
Develop habits of mind needed for successful problem solving;
Course strategies include:
an absolute grading scale (announced at the beginning);
a student board of directors appointed to oversee all aspects of the course;
students read and analyze research papers;
interactive large lecture techniques;
spreadsheet programs used for homework and lab problems;
cooperative exams complement traditional exams;
open-ended lab projects replace many standard lab experiments, with the grading based on
written reports and oral discussion of the project results
Assessment Question
How can I establish that the use of inquiry-based learning strategies ("structured
active learning"-SAL) results in habits of mind that successfully characterize
problem solving?
Inquiry-based Assessment and Evaluation Methods
We conducted a formal comparative study, possible because two sections of the course
were taught during the same semester, one by an instructor (who used a responsive
lecturing (RL) approach combined with very challenging homework problems, quantitative
exam questions, well-defined lab experiments and one open-ended lab project), and the
other by John Wright (who used the "structured active learning" (SAL) strategies
described above). External evaluators judged that both sections represented best practice
for each method. The assessment strategy was to use oral exams of all students in both
sections, conducted by 25 faculty from diverse disciplines related to chemistry. Each
assessor designed their own exam, examined 8-10 individual students who were from the same
class octile (as determined by incoming performance). Half the students were from each
section; information about student section was kept from the assessors. Assessors ranked
students and also completed a questionnaire on each student. Because external funds for
evaluation were available, third party (LEAD) evaluators also used qualitative and survey
methods in combination with the oral assessment project. The LEAD analysis produced two
case studies, as the patterns of social interaction and student and instructor experiences
emerging from interviews, observations and open-ended survey responses were too different
to sustain a comparative analysis. Comparative statistical methods were used to analyze
faculty assessor ranking and numeric questionnaire data.
Outcomes
Interview data showed sharp differences in the nature of the learning interactions. The
SAL students’ learning characterized by student-student interactions. Most SAL
students reported that they valued the challenge of solving open-ended problems and
acquired greater self-reliance. The RL students’ learning was characterized by a
focus on the lecturer as the authority for knowledge. They reported a strong sense of
accomplishment in mastering the material using the lecturer’s step-by-step problem
solving and mathematical modeling methods.
Statistical analysis of the faculty assessor ranks revealed very significant
differences in the ranks, with the SAL students ranked higher for all octiles, and the
largest differences seen for the bottom and middle octiles. Similar differences were found
in the faculty questionnaire responses pertaining to the students’ overall
competence. Analysis of the faculty assessors’ criteria for ranking their students
plus other data indicates that those assessors who were looking for
"meta-awareness" in student performance (did the students self-correct, have a
variety of perspectives, understand the larger context surrounding a particular problem,
relate theory and practice?) had the greatest differences in rankings between sections.
This finding indicates that the major reason for the large differences in assessment of
student competence was the thinking process that students displayed during their oral
exams.
Interestingly, although a large proportion of the faculty used meta-awareness as their
primary criterion for judging competence, faculty comments during interviews with
third-party evaluators showed that these same faculty did not use this criterion when
designing exams for their own courses.
CONCEPT RETENTION INTO SUBSEQUENT SEMESTER
Developed by faculty in Chemistry
Introductory and Analytical Chemistry ("Chem 109 and 110"), Art Ellis & John
Wright, UW-Madison
An article on this research is being written. Contact the LEAD Center.)
Assessment question
Do students taught using inquiry-based methods are more able to retain course concepts
into the next semester?
Inquiry-based Assessment Methods
Assess students at the beginning of the inquiry-based (SAL) section of Chem 110 in
order to determine if there is an average differences in performance between those who had
an inquiry-based approach to Chem 109 and those who had a traditional approach to Chem
109. Oral exams similar to those conducted in the Chem 110 Oral Assessor project (see
above) were conducted, and the traditional exercises, exams, homework, lab experiments,
and computer projects completed in the very early part of the semester were graded.
Outcomes
The results of the oral assessor exams were similar to those for the Oral Assessor
project (see above), with significantly higher ranks for students who had an inquiry-based
approach to Chem 109. However, the performance of the two groups on the written exercises,
exams, and other work was identical. Interestingly, at the end of the semester, a second
series of oral exams was conducted, and the differences that were found at the beginning
of the semester had largely vanished. The students who had not had the inquiry-approach to
Chem 109 acquired the skills during the semester that equalized their performance on oral
exams.
STANDARDIZED EXAMS
Goal
Determine if students taught with inquiry-based teaching methods perform differently on
standardized and/or traditional exams than students taught with traditional methods.
Inquiry-based Assessment Method
Administer standardized or traditional exams to matched samples who did and did not
have reformed course.
Outcomes
In most cases (Emerging Scholars, see "Challenge Problems," above, is an
exception) where this exercise has been conducted, there is no significant difference in
performance between students who have had inquiry-based and traditional courses. The
conclusion to be drawn is that the inquiry-based methods "do no harm," when
judged in terms of the performance criteria that inform traditional and standardized
exams, and that additional examinations that are informed by performance criteria aligned
with the inquiry-based methods need to be administered in order to assess the intended
outcomes of the inquiry-based methods.
FACULTY-DESIGNED MID-COURSE SURVEYS
Developed by faculty in Engineering
Freshmen Engineering Design course, team-taught by 7 cross-disciplinary engineering
faculty, UW-Madison
(For more information, see reports listed in LEAD Products.)
Goal
Each semester, get freshman engineering students to design and build solutions to
genuine engineering problems posed by a community or industry group in order to introduce
them to the work that actual engineers do, to give them experience with teamwork, and help
them understand why they need the conceptual tools presented in their mathematics and
science courses.
Assessment Questions
Is this semester’s design problem working for the students?
How are our course materials and other teaching strategies working?
Are the lab-groups functioning well?
Inquiry-based Assessment Method
With assistance from a LEAD evaluator, faculty designed a survey with closed-ended
questions (responses made on a 5-point scale) and open-ended questions. The survey
analysis was performed by outside evaluators, but could have been performed by
upper-division student assistants.
Outcomes
Survey responses indicated that students:
were frustrated by the lecture sessions,
did not find the homework and handouts of much value,
loved the design work in the labs and wanted more lab time, and
enjoyed working in teams.
This information allowed faculty to make informed mid-course corrections. However, this
technique requires quite a lot of analysis time, and students learning gains as a result
of completing the survey are minimal.
STUDENT ASSESSMENT OF LEARNING GAINS SURVEY
Developed by two NSF chemistry consortia; well-suited to all SME&T
disciplines
Designed and tested by Elaine Seymour and Joshua Gutwill (evaluators) and many faculty in
the ChemLinks and ModularChemistry consortia. See FLAG.
Assessment Question
How can we obtain an assessment of the effectiveness of a course overall, as well as of
the specific features of the course which is based on student learning gains, rather than
on student judgements of faculty performance?
Inquiry-based Assessment Method
This instrument is intended to elicit students’ estimates of how much they gained
from the class, and to relate their gains to particular aspects of the class pedagogy. It
aims to take the focus off the teacher, per se, and avoids critiques of pedagogy that are
not directly related to student estimations of what they gained. The particular focus (on
what students gained from the class) of this assessment tool arises from a strong
finding from student interviews on ten campuses. Seymour found that asking students what
they "liked" or "valued" about their classes, or how they evaluated
their teachers was far less productive than asking them what they had specifically gained.
Seymour and Gutwill hypothesize that students can meaningfully answer questions about
their own gains, whereas what they "like," and how they judge faculty in their
role as teachers is much less reliable. This proposition is supported by faculty interview
data in which faculty express mistrust of the numeric data (if not the written comments)
provided by many institutional classroom evaluations. Faculty also expressed concern that
departmental evaluation of their efficacy as classroom teachers (for tenure and promotion
purposes) is largely based on the scores of instruments that ask the wrong types of
questions. Additionally, those faculty engaged in pedagogical innovation find themselves
at special risk of lowered evaluation scores in the early stages of course redesign partly
because traditional classroom evaluations give students insufficient opportunity to
estimate the value added to their learning by new class features.
Outcomes
Analysis of a sample of students’ responses to open-ended questions from student
classroom evaluations used at four participating campuses revealed that students’
comments focused not on the teacher’s professional performance, but on how much they
did, or did not, gain from the class. Seymour and Gutwill found that the totals for
students comments on traditional evaluations (for both reformed and comparative classes)
were broadly 50% positive and 50% negative. By contrast, in both the reformed and
comparative classes, students gave clear indications about what they had
"gained" (in understanding, skills, approach to learning and to the subject)
from the various aspects of their classes.
PRE/POST ATTITUDE SURVEYS
Developed by faculty in Chemistry
Astronomy for non-majors, Douglas Duncan, University of Chicago.
(see http://www.aas.org/education/challenges.html)
Goal and Teaching Strategies
Goals
Encourage a sense of awe and appreciation for topics investigated in modern
astrophysics.
Develop a sense that some topics in astrophysics are so interesting that a student will
want to follow them on their own (i.e., affect the student’s attitude about science).
Encourage student understanding of the scientific method and practice in its use.
Help students realize that they hold assumptions and concepts about how the world works
that are not consistent with scientific evidence.
Give teaching assistants and faculty instructor opportunities to improve our own teaching
skills.
Give students a moderately comprehensive introduction to the topics and results of modern
astrophysics.
Give students experience with quantitative reasoning.
Primary teaching strategy:
Weekly Challenges. A weekly challenge is an experiment which is set up in class
every Tuesday 20 minutes before the end of lecture. The experiments chosen have results
that are not intuitive. (E.g., on refraction: Students are told that the index of
refraction of Pyrex and Wesson oil are the same, and asked what would happen if a Pyrex
stirring rod is immersed in oil.) Student are told to form into small teams of 3-4
throughout the lecture hall and predict what will happen. The experiment is not performed
on Tuesday. At the beginning of class Thursday, the predictions are collected and the
experiment is performed.
Assessment Question
How do I know if this course is improving my students’ attitudes about science?
Inquiry-based Assessment Method
Beginning- and end-of-course attitude surveys.
Outcomes
Student response on question about whether they like science was 15% at beginning of
term. When asked if the course changed their view of science, 88% said yes. Typical
comments (verbatim) were, "Yes. It is not just useless data or memorization, it is
something that I can apply to everyday life." "It made me realize my instincts
weren't always true." "Things that seem so apparent, such as the reason for the
seasons, turn out to be a little more complex than expected." My favorite comment
was, "My parents asked me if something was wrong, because I was starting to talk
about science at home, something I never did [before]."
LECTURE ATTENDANCE/STUDENT RETENTION WITHIN COURSE
Developed by faculty in chemistry
New Traditions Chemistry Second Semester General Chemistry, Clark Landis, John Moore,
Earl Peace, UW-Madison
Assessment Question
Students who are acquiring a greater appreciation for the methods and processes of
science will show greater interest in my course. How can I demonstrate to myself, my
colleagues, and the NSF that my students are more interested in the course as a result of
these new strategies?"
Inquiry-based Assessment Method
Determine if a higher percentage attend lecture and are retained in the course.
Outcomes
Notably higher class attendance and course retention rates were found.
| 
