Chapter 7: Assessment of IT-asssited PBL

| Home | Week by week outline |Readings & Bibliography | Reading Assignments | Homework Assignments | Assessment |

Moursund, D.G. (1999). Project-based learning using information technology (Selected Chapters) Eugene, OR: International Society for Technology in Education.

The materials that follow are from a next-to-final version of the above named book.

| Home | Week by week outline |Readings & Bibliography | Reading Assignments | Homework Assignments | Assessment |

Moursund, D.G. (1999). Project-based learning using information technology (Selected Chapters) Eugene, OR: International Society for Technology in Education.

The materials that follow are from a next-to-final version of the above named book.

Preface

Chapter 1: Introduction and a PBL Example

Chapter 2: Overview of IT-assisted PBL

Chapter 3: Some PBL Lesson Topic Ideas

Chapter 4: The Case for PBL

Chapter 7: Assessment in IT-assisted PBL

Appendix C: Overview of Problem Solving

References and Resources

 

Chapter 7: Assessment in IT-assisted PBL

An noted in Chapter 2, two of the defining characteristics of IT-assisted PBL are that both the content and the assessment be authentic. Authentic assessment is substantially different than traditional assessment that is based on objective and short answer questions.

This chapter focuses on authentic assessment of IT-assisted PBL. Successful implementation of authentic assessment requires education of the key stakeholders as well as developing authentic curriculum and instruction (NFIE, 1997). This means that students need to learn about assessment and their roles in the assessment process.

Portfolios are often one component of authentic assessment. Increasingly, portfolios make use of digital storage; such a portfolio may be called an electronic portfolio.

Authentic Content and Assessment

Two of the defining characteristics of IT-assisted PBL given in Chapter 2 are that the content/purpose and the assessment be authentic. Some teachers are quite comfortable in using both traditional assessment tools and authentic assessment tools. Others will find that authentic assessment has a challenging learning curve. This chapter is designed to help teachers climb that learning curve.

While general goals for education and for IT in education are relatively widely understood and accepted (see Appendix B), the same cannot be said for assessment. Should we have nationwide testing? Should graduation from elementary school, middle school, or high school be based primary on student performance on exams? Should the exams be objective and short answer, or should assessment be authentic and performance-based? Are the testing methods valid, reliable, and fair (equally fair to male/female, different minority groups, and other major categories of test takers)?

Assessment is a complex field, and almost everybody has an opinion as to what should be done. In recent years, ideas such as authentic assessment, performance-based assessment, and portfolio assessment have received a lot of attention. At the same time, many school districts and states have placed renewed emphasis on the traditional paper and pencil tests (some, of which, are now administered by computer, perhaps in a computer-adapted testing mode (Applied Measurement in Education, 1992; Barrett, 1994; Brewer, 1996; Educational Leadership, 1992: Fogarty, 1996; Meng and Doran, 1993; Rothman, 1995; The Computing Teacher, 1994; Wiggins, 1993; Wiggins, November 1993; Wiggins, 1996-97). Some states (such as Oregon, with its emphasis on a 10th grade Certificate of Initial Mastery and its 12th grade Certificate of Advanced Mastery) have combined traditional and authentic assessment at the statewide level and for college entrance.

The teacher doing assessment of an IT-assisted PBL lesson needs to think about the purpose of the assessment. This will help to shape the evaluative information that will need to be gathered and how this evaluative information will be used in the assessment.

There are many different stakeholders in an educational system. Examples include students, educators, parents, taxpayers, school boards, and the government. These various stakeholders have differing views on the goals or purposes of assessment. Three common purposes for assessment in education are:

  1. To obtain information needed to make decisions. Different stakeholder groups often have different information needs and make different types of decisions based on the assessment information received. Assessment designed to fit the needs of students (arguably, the most important stakeholders) may be quite a bit different than assessment designed to meet the needs of teachers or of government officials.
  2. To motivate the people or organization being assessed. It is often said that student assessment drives the curriculum. Student success on state, national, and international tests serves as an affirmation to students, teachers, school administrators, and other stakeholders. This motivates teachers to "teach to the test" and motivates to students to orient their academic work and studies specifically towards achieving well on tests.
  3. To emphasize accountability of students, teachers, school administrators, and the overall educational system. For example, a school district's educational system might be rated on the percentage of its students that graduate from high school or how well its students do on college entrance exams. Poor student performance may lead to major changes of administration and instructional strategies in the school district.

Sometimes the purpose of an assessment can be categorized on a scale such as given in Figure 7.1. This scale runs from "Very Low Stakes" to "Very High Stakes." The "stakes" may be for a student, but they may also be for the teacher, school, or school district.

Figure 7.1. "Stakes" scale.

The student-stakes are modest as a teacher wanders purposefully around the classroom, watching students work in groups to on a project. The teacher makes mental and written notes (perhaps using a hand-held personal digital assistant) about activities of individuals and the groups.

The student-stakes are likely to be somewhat higher on major projects that engage students over a period of weeks. A significant portion of a student's grade may depend on producing a newsletter that is carefully researched, designed, written, and desktop published.

Assessment can be for really high stakes. For example, a particular college may require that applicants score above a specified level on an entrance exam. No matter what the student's previous record of achievements, failure to achieve above this specified level on the test results in not being admitted to the college. This is certainly high student-stakes assessment.

Both the students and the teacher should have a clear understanding of the level of the stakes in a particular assessment. That is one aspect of authentic assessment.

The Lesson Goals Need to Be Assessed

Assessment in IT-assisted PBL needs to be closely aligned with the goals of the lesson. These goals may be much broader than the goals in a more traditional lesson plan.

Both a traditional lesson and an IT-assisted PBL lesson will have content or product goals. In the IT-assisted PBL lesson, the content or product goals are apt to be interdisciplinary. They will, of course, include IT goals as well as goals of students learning to work individually and on a team in designing and carrying out a project. Thus, the content/product goals may be more complex and varied than in a traditional lesson.

An IT-assisted PBL lesson typically has a number of process goals. Examples include cooperative learning, collaborative problem solving, self and peer feedback, and learning to function in a community of scholars. In IT-assisted PBL, all such goals are made explicitly clear to the students, as well as how they will be assessed.

As compared to a traditional lesson, an IT-assisted PBL lesson tends to have more goals and more varied goals. Thus, assessment tends to require more careful planning in advance of lesson implementation.

Feedback and Assigning Grade

Learning requires feedback, A learner has many different sources of feedback, including self, peers and other students, teachers, parents and other adults, teaching materials (for example, an answer key or in computer-assisted learning), and so on.

In many cases this feedback is not connected to formal evaluation and grading of a student. This is important to keep in mind in an IT-assisted PBL lesson. Both feedback (to facilitate learning) and assessment (part of the teacher's job, and providing important feedback to the teacher) are essential. Careful consideration needs to be given to each.

From a teacher point of view, there are three common phases to the evaluation of an IT-assisted PBL activity.

  1. Formative evaluation. Formative evaluation is designed to provide feedback while the student is still working on the project. This allows both the student and the teacher to make mid-project corrections. The teacher may make use of some of the formative evaluation information in a final assessment, but may chose not to do so.
  2. Summative evaluation. Summative evaluation is carried out after the project has been completed. A teacher might decide to base the project assessment purely on information gathered in the summative evaluation phase. However, a final assessment might also give considerable weight to the processes carried out in the project, such as accomplishing a project's milestones on time and the quality of intermediate products.
  3. Portfolio evaluation. A portfolio is a collection of work samples. Typically, a student and teacher work together to decide which work samples will go into the student's portfolio. During a school year, a large number of items may be collected for use in the school year portfolio. Then, some of them will be added to the student's long term portfolio. The assessment of a student's overall work in high school might be based on an evaluation of a portfolio based on work done in high school.

In an IT-assisted PBL lesson, both the students and the teacher need to understand these three categories of evaluation and their possible roles both in assessment of the project and possible assessment for other purposes, such as meeting graduation requirements, getting into college, or getting a job.

Overview of Evaluation

An IT-assisted PBL lesson is assessed by gathering a variety of evaluative data and information and then analyzing the evaluative data and information to produce an assessment. The evaluative data and information will be both quantitative and qualitative. It will cover both the processes students carry out during the project and the final product, presentation, or performance.

Figure 7.2 contains a brief summary of the major areas in which one typically gathers evaluative data and information. We will briefly discuss each of these three major areas.

Evaluation Area
Comments

1. Subject area content goals of the project

Often the project content is interdisciplinary. Usually, different students are emphasizing different aspects of the content area.

2. IT knowledge and skill content goals of the project

While a project may have IT as its primary goal, usually increased IT knowledge and skills is a secondary goal. Usually, different students will make considerably different uses of IT in a project.

3. Learning to do a project when working:

A. Individually

B. In a group

Some projects will be done individually, and some will be done by teams. In a team project, there will typically be both individual and team processes and components of the final product. Cooperative learning and collaborative problem solving are both very important in PBL.

Figure 7.2. Evaluation areas for IT-assisted PBL.

Subject Matter Content Area

Suppose that a history teacher is doing a unit on the Civil War in the United States. Students are to do a project that focuses on some important aspect of the Civil War and the historical time in which it occurred. The students are allowed to work individually or in small teams. The teacher observes:

It is easy to see that students will not all be learning the same thing. Moreover, the chances are that many students will gain knowledge that their teacher does not have.

However, all students will produce a product, presentation, or performance. The teacher has considerable knowledge about what constitutes a high quality product, presentation, or performance and in many cases is qualified to evaluate the student work. In special cases, the teacher may need to ask for help from other teachers or outside experts.

Moreover, as the history teacher makes continued use of PBL over a period of time, the teacher will acquire baseline data on the quality of product, presentation, or performance that can be expected from students. The teacher will gain increased knowledge and skill about how to evaluate this type of student work. One of the values of PBL is that it helps to create a good learning environment for the teacher.

Presentations and performances are typically done for an audience, and the audience may be the whole class or a still larger group. A product (for example, a written document) also lends itself to sharing with a group. Thus, all students in a class can share in the products, presentations, and performances of their fellow class members. They will learn from the content of these activities. They can learn to evaluate these activities and to provide constructive feedback. This is an important aspect of a PBL lesson.

In the Civil War example, the varied nature of the content areas being studied by the students precludes the use of traditional objective tests. Of course, the teacher may also assign some general background reading and other learning tasks to the entire class, and test them over these materials. This would lead to a student's grade being based on a combination of traditional and authentic assessment.

IT Knowledge and Skills

Some elementary schools have a computer specialist who gives students their first formal instruction in IT. However, in many elementary schools the regular classroom teacher is responsible for helping students to develop their initial IT knowledge and skills. The teacher may have to give explicit instruction in use of basic hardware and generic software tools.

This issue is complicated by the steadily increasing number of students who are learning about IT at home. Such students may enter school with a high comfort level in use of a microcomputer, CD-ROMs, and the Web.

Basic knowledge and skills about IT can be taught in a didactic manner. A teacher wants all students to know how to use a digital camera and a scanner to put graphics into a word-processed document. The teacher can do a whole class demonstration and then provide individualized help as students try to apply what has been covered.

An alternative approach is to make use of peer instruction. One-on-one instruction might be provided by students in the class who have previously learned to use the equipment. Alternatively, the teacher may teach a small number of students how to use the camera and scanner, and have these students serve as one-on-one teachers for the rest of the students in the class. Research strongly supports this approach. It is effective for the learners and it is good for the students doing the teaching.

One of the goals in an IT-assisted PBL lesson is for students to increase their IT knowledge and skills. Thus, you (a teacher who is most likely not an IT expert) must do evaluation and assessment on this IT learning.

There are three general areas in which you might gather evaluative information in IT.

  1. Students helping other students to learn IT. Here, you can observe students helping each other. You can see which students are in demand as helpers. You can see which students are frequently asking for help--perhaps repeatedly on the same topic.
  2. Students learning IT on their own, from fellow students, at home, and so on. If a student's IT knowledge and skills are significantly better than might be expected from the level of instruction provided by the school, you can assume that the student is learning in other ways. Actual levels of knowledge and skills can be assessed through both pencil and paper tests and through hands-on tests. In addition, you can observe a student making use of the IT facilities and gather data about the student's performance skills.
  3. Students demonstrating IT knowledge and skills in their products, presentations, and performances. You can observe the results of student use of IT. In addition, you can look at work samples. This is discussed more in the paragraphs that follow.

When you look at a printed desktop published document, you cannot discern all of the underlying IT knowledge and skills the student has in the desktop publishing area. You may not be able to tell if the student is using the word processor like an electronic typewriter, or whether the student has gained the knowledge and skills of the desktop publication field. For example, you cannot tell whether the student is using a first line indent or a tab to indent at the beginning of a paragraph. You cannot tell if the student is following the desktop publication rule of one blank space after an end of sentence punctuation mark. You cannot tell if a student has learned to use "styles" in doing word processing. It is necessary for you to examine the student's work on a computer.

If you are just learning to make effective use of IT, the chances are that you have not had much formal instruction in such areas as desktop publication, hypermedia design, design of effective interactive communication between a person and a computer, Web design and implementation, carrying out effective searchers on the Web, and so on. Thus, you are not well prepared to evaluate the details of this type of student work.

As you observe students doing IT work and look at their products, you will gradually see that some approaches are more effective and efficient than others. Gradually you will gain the knowledge and skills to help your students learn these aspects of IT, and you will learn how to evaluate the work your students are doing. Professional development--and reading some of the literature--can be very helpful. While there are many aspects of IT that can be learned on the job by observing students at work and working with students who have particular knowledge and skill in this area, there are other aspects that require formal instruction and study. IT is a large and challenging field!

Learning How to do a Project

Figure 7.3 is a Project Planning Table from Chapter 5. Such a table provides a brief summary of some aspects of planning and carrying out a project.

Description
Resources
Timeline
Milestones

Task 1

-

-

-

Sub task 1.1

-

-

-

Sub task 1.2

-

-

-

Task 2

-

-

-

Etc.

-

-

-

Figure 7.3. Project Planning Table.

As young students first begin to learn to carry out projects, you will do most of the project planning for them. It is not easy to visualize the sequence of steps needed to complete a long project. It is not easy to estimate how long it will take to carry out various tasks and sub tasks. It is not easy to organize a team and to allocate individual task to the various team members.

With instruction and practice, however, students will learn to take more and more responsibility for defining the topic, developing an outline of the tasks and sub tasks, setting goals and milestones, deciding on needed resources, allocating the resources that are available, and completing a complex project in a timely and high quality manner. With instruction and practice, students will learn to work individually and in a team to do complex projects

Each project that you assign should be designed to give your students the opportunity to maintain and improve their project skills. This means that in each project, you and others will need to gather evaluative information and provide feedback that will help students to get better at doing projects.

For example, suppose that you have decided that defining milestones will be emphasized in a particular project. Assume that this is a project in which there is considerable variety among the students as to the specific project they are doing. Each student will need to allocate their own time and other resources. Each student will need to set milestones and when the milestones are to be accomplished. An initial milestone is to describe the project to be done and to complete a Project Planning Table. You can analyze a student's project description and Project Planning Table. You can provide written feedback, or you can discuss the project and planning table with the student. This feedback may lead to modifications in the project to be carried out and/or in the Project Planning Table.

The Project Planning Table is a useful evaluation instrument. You can expect a student to provide you with regular reports that are based on this table. These reports may include draft copies of various components of a project. They may include an analysis of barriers that are being encountered and how they are being overcome. They may include a student's self-assessment of how the work on the project is going. If a team of students are working on a project, each student can be expected to provide feedback on how well they think their fellow students are doing on project components that are they are specifically responsible for.

Rubrics (Scoring Criteria)

One of the most important aspects of authentic assessment is that the students have a full understanding of the assessment criteria. Part of the learning that needs to go on in authentic assessment is for students to learn to understand the assessment criteria, learn to assess themselves, and learn to assess their fellow students.

A rubric is a scoring tool that can be used by students (for self assessment), peers (peer assessment), teachers, and others. It lists important criteria applicable to a particular type or piece of work. It also lists varying levels of possible achievement of the criteria. Figure 7.4 gives a very general purpose, six level scoring rubric. The next section in this chapter presents an adaptation of this general-purpose rubric to IT-assisted PBL.

Level
Brief Description

1: Emergent

Student displays few, if any, of the rudimentary knowledge and skills that are expected.

2: Limited

Student displays rudimentary knowledge and skills, but often requires substantial individual help and guidance.

3: Developing

Student displays a minimally adequate level of the expected knowledge and skills.

4: Capable

Student displays a functional, adequate level of the expected knowledge and skills.

5: Strong

Student displays a high level of the expected knowledge and skills.

6: Exceptional

Student displays an outstanding and creative/innovative level of the expected knowledge and skills.

Figure 7.4. A general-purpose six level rubric.

The general-purpose rubric of Figure 7.4 needs more detail before it can be used in a particular assessment situation. For example, suppose that you want to assess the writing component of an IT-assisted PBL lesson. Clearly, the meaning of the six points will be different for 3rd grade students than it will be for 12th grade students. The levels have to be defined relative to what can reasonably be expected of students at a particular grade level.

In writing and in all of the traditional subject matter areas, there is considerable information about what can be expected of students at various educational levels. In IT, however, this knowledge is just beginning to be developed. The National Educational Technology Standards discussed in Appendix B. are reflective of these emerging guidelines. The performance indicators for the various grade levels are designed to show a much higher expectation for older students.

Rubrics have been developed for many different curriculum areas and lists of these have been published (Brewer 1996). While each Wide scale implementation of such rubrics has been accompanied by extensive research on their effectiveness as well as on the nature and extent of teacher education needed for their effective use. Several conclusions are:

Sample Information Technology Rubric

The next six subsections are the six levels of general-purpose rubric for assessing student use of IT tools. The information in these six levels is quoted from the Oregon Educational Technology Consortium (October, 1997) Web site <http://www.oetc.org/projects/bl.htm>.

Level 1--Emergent Technology User

Level 2--Limited Technology User

Level 3--Developing Technology User

Level 4--Capable Technology User

Level 5--Strong Technology User

Level 6--Exceptional Technology User

A Specific Desktop Publication Example

This section contains a partial example of a scoring rubric that might be used in a unit of instruction in which students are learning to design and desktop publish a newsletter. The main emphasis in the instructional unit is on students learning to use information technology tools. However, there is also some emphasis on students learning to communicate effectively while making use of these tools.

This unit of instruction is different than a writing unit in which the main emphasis is on effective written communication, with only a minor emphasis on desktop publication of the resulting written document. Remember, assessment must be aligned with curriculum and instruction. Scoring rubrics that fit a unit emphasizing information technology will, necessarily, be different than scoring rubrics for a unit emphasizing written communication.

In this particular unit of study, the students are studying a number of principles of design for an effective newsletter. The assumption is that they are already skilled in using a word processor, scanning and editing graphics, and printing documents.

The rubric shown in Table 7.5 contains only part of the items that would be used for this newsletter project. We have not included any items having to do with the quality of the written content. This particular example makes use of a six-level scale on each rubric item. Four-level scales are also commonly used. The intent is that the different levels form an equal interval scale.

It is common to make use of a Likert-type scale with an even number of levels for a rubric item. This forces the assessor to place the work into an "above the middle" or "below the middle" category.

It is evident that it takes considerable learning on the part of students and the teacher to make effective use of these rubric items. The assessment, curriculum, and instruction are interwoven. The curriculum and instruction will include the examination of a number of different desktop-published newsletters. Students will practice assessing these newsletters, their own newsletters, and the newsletters of their fellow students.

Desktop Design and Publication

The student work displays:

  1. no evidence of understanding and making use of this principle.
  2. limited evidence of understanding and use of this principle.
  3. developing understanding and use of this principle.
  4. capable understanding and use of this principle.
  5. strong and creative understanding and use of this principle.
  6. exceptional and highly creative understanding and use of this principle.

Principle
Level
(Circle one)
Comments and suggestions

Banner: communicates well; attracts and holds reader.

1 2 3 4 5 6

-

Text: readable; limited number of typefaces; right amount of text.

1 2 3 4 5 6

-

Grid and alignment: clear and consistent pattern of use.

1 2 3 4 5 6

-

White space: not too little or too much; used well, not trapped.

1 2 3 4 5 6

-

Unity: text, graphics, and design work together.

1 2 3 4 5 6

-

Visual scan: directs reader to important elements.

1 2 3 4 5 6

-

Etc. Etc. Etc.

1 2 3 4 5 6

-

Figure 7.5. Part of a rubric for a student newsletter project.

Electronic Portfolios

Portfolios and portfolio assessment have long been an important part of assessment in the graphic arts. Similarly, students of music can record their performances, and students of dance and theater can videotape their performances.

In recent years, the ideas of portfolio and portfolio assessment have spread to other disciplines. A portfolio might contain writing samples that illustrate changes in writing skill over a period of time. A portfolio might contain samples of science lab write-ups or written tests from any discipline. A student's hypermedia project can become a component of a portfolio.

Students are now carrying out projects and developing products that can only be adequately represented and used on a computer. Interactive multimedia "stacks" and Web pages provide excellent examples of this new form of product. Student work done using a computerized music synthesizer, science simulations, and sophisticated mathematical software all support the need for electronic portfolios. Thus, increasingly portfolio items are created or transferred into digital format so that they can be stored in a computer readable medium. Such a portfolio is called an electronic portfolio. Electronic portfolios have considerable advantages over other forms of portfolios in terns of ease of editing, making copies, and portability. However, viewing an electronic portfolio requires appropriate hardware, software, knowledge, and skills. Moreover, over a period of time, hardware and software change so that an electronic portfolio may no longer be readily usable.

Final Remarks

There are many challenges to a teacher first learning to do authentic assessment of IT-assisted PBL. For both the teacher and for the students, this will be a learning process. It is a process in which the teacher and students can work together and learn from each other.

Activities

  1. What are your personal feelings about authentic or performance based assessment versus traditional assessment? From your personal experiences and point of view, present arguments for and against each approach, and then summarize your current position.
  2. Select an IT-assisted PBL lesson topic. Develop both content (product) and process assessment rubrics that would be suitable for this lesson.

 

| Home | Top of page | Week by week outline | Readings & Bibliography | Reading Assignments | Homework Assignments | Assessment |