The eInstruction Methodology for Academic Progress (MAP): A Formative Assessment Tool and Approach for Raising Standards of Achievement

Tuesday January 18, 2005

Darrell Ward,PhD

Introduction

Classroom activities drive the learning process which ultimately impacts student academic progress that is measured by state or district testing against standards. Black and Wiliam, (1998) present arguments that formative assessment is at the heart of effective teaching while Sander's research identifies the importance of the teacher in academic progress of students (Sanders, 1999). This article summarizes the evidence supporting the major learning gains that can be achieved utilizing formative assessment innovations and identifies some major problems involved in the implementation of these innovations. Finally, an approach supported by simple handheld technology is described which leverages the value of formative assessment in the classroom while empowering both teachers and students throughout the process. This approach called MAP - Methodology for Academic Progress - is presented and coordinated to documented research on classroom teaching and learning. The MAP approach, supported by simple and seamless technology, is demonstrated as an efficient and effective tool for integrating formative assessment activities into classroom activities.

The Argument for Formative Assessment as a Tool to Raise Standards and Improve Student Academic Progress

Black and Wiliam (1998) define assessment to include all activities that teachers and students undertake to get information that can provide feedback which can be used to modify teaching and learning activities. The assessment becomes formative when that information is actually used to adapt the teaching process to meet the student needs. The assessment information can come from a variety of sources and include: Classroom interactions among the teacher and students Observation of student performance via Discussion Homework Worksheets Tests There is substantial evidence that formative use of assessment information will positively impact academic progress of students. Black and Wiliam (1998) reviewed 20 studies where formative assessment innovations were used and all of the studies show that strengthening the practice of formative assessment produce significant and often substantial learning gains. These studies ranged over age groups from 5-year-olds to university undergraduates, across several school subjects, and over several countries. This research demonstrated that by employing formative assessment innovations into the classroom, a typical effect size gain of .4 to .7 was achieved with low achievers benefiting the most, thus reducing the achievement gap.

Classroom questioning, appropriately used is one example of formative assessment, and has been thoroughly researched by Thalheimer (2003). This research shows that learning gains of 150% or more can be achieved based on effective use of questioning with appropriate feedback mechanisms.

A very substantial study over a six year period was undertaken by Massen, Poulis, Robens and Gilbert (1998) based on very simple formative assessment integration into large lecture environments. Their research showed that the active formative assessment increased the mean pass rate of individuals exposed to it and reduced the variability between the achievements of different students.

This substantial body of work makes it obvious that teaching and learning should be a very interactive process and that formative assessment is a foundation for effective teaching producing student academic progress.

Using Formative Assessment Effectively and Efficiently

The challenge for the classroom teacher is to capture the assessment information in a form that can be evaluated and utilized to adapt and meet the student needs. For example, collecting homework from 30 students with 15 homework activities each will produce 450 individual items for inspection and evaluation by the teacher with potentially many adaptations and directions for each of the students - a daunting task to achieve on a regular basis for any teacher.

It is obvious that attention to details on assessment information such as the homework assignment above by providing detailed feedback to each student on their work and direction/correction for their benefit will deliver tremendous learning gains. However, the practical applications of such efforts become counterproductive if attempted on a regular basis as teachers quickly burn out and leave the teaching profession (Texas Center for Educational Research, 2000). The education enterprise has consistently added administrative burdens to the classroom teacher reducing, not enhancing, the time available for attention to such innovations as expanded formative assessment.

Basically, the human resources required to deliver effective formative assessment must be balanced with the return on that investment. Teachers must obtain a good return on their investment in instructional pedagogies or they will quickly discard these innovations.

Black and Wiliam (1998) recognize the problems with investment of time and return on the investment as they suggest that a small subset of teachers be identified to lead the way with practical changes to increase student academic progress. The fragile environment of the teacher must be carefully examined before instituting substantial change.

The utilization of technology has the promise of supporting both aspects of the formative assessment that become keys to teacher acceptance and implementation. The efficiency of capturing and evaluating assessment information via the personal computer which resides within classrooms provides an enormous opportunity to support the classroom with a tool that can impact with effectiveness, formative assessment. Many programs have been created and implemented with simple scanners attached to the PC allowing for efficient collection and evaluation of student performance. This first step has provided exciting systems such as those provided by Renaissance Learning to support Math and Reading at the Elementary level. Students utilize self-assessment and scanners to move through materials for portions of their instructional sequences.

eInstruction has taken an approach for integrating formative assessment into the daily instructional activities with a seamless integration of questions into classwork periods, homework capture, self-assessment activities, team activities and exam activities utilizing simple handheld devices which empower each student in the class.

The eInstruction MAP Approach

eInstruction has utilized computer technology, simple student handhelds, robust software for capture, feedback and analysis of student performance and progress. The resulting system is called the Classroom Performance System (CPS) and is a vital part of the MAP program as it is the engine for the academic progress.

Black and Wiliam (1998) assert that underlying the various approaches applied to classroom teaching and learning are assumptions about what makes for effective learning -- in particular the assumption that students have to be actively involved. The eInstruction MAP approach is to optimize student engagement and activity as formative assessment instructions are utilized. While maximizing that aspect, the effectiveness of the learning environment, MAP provides efficient capture and reporting tools to empower the teacher to focus on the fundamentals of academic progress in the classroom - formative assessment activities. The following encapsulate the major building blocks of the MAP featuring seamless formative assessment activities: Active involvement, interaction and feedback by students utilizing simple handheld remotes managed by the CPS system Continuous formative assessment with interactive feedback based on assessment information captured and recorded in real-time during all instructional class work activities Regular self-assessment using objective paper-based instruments capturing student results with immediate feedback via response pads and projected computer screen Regular formative assessment applications of homework and exams, again capturing the student information via simple handheld devices Regular team activities focused on serious work with time for team analysis of their responses, yet again all information is captured for use in adapting the instruction to the needs of the students

All of the above are key ingredients of the approaches for improving formative assessment as outlined by Black and Wiliam (1998). Additionally, the various questioning activities and the resulting positive learning benefits are fundamental to the approaches suggested by Thalheimer, (2003).

The student is an active participant in this pedagogy. Mazur (1996) proposes using a pedagogy that has students working in pairs or small groups to evaluate an answer or solution to a problem and then providing that via a spokesperson for the team using the response units. MAP actively promotes setting of goals for the class and working, as a team, to achieve those goals quantitatively during an instructional session. All of this is designed to promote ownership, active involvement and total student empowerment.

The specifics of the eInstruction MAP program are provided in Appendix A as a guideline for improving formative assessment.

Conclusions

Key items for academic progress which improves standards performances by students are identified and revolve around formative assessment. These include: Active student engagement Teacher consideration from an efficiency and effectiveness perspective Seamless integration of formative assessment into the class Return on investment by the teacher Teacher empowerment to deliver on academic progress with formative assessment tools Student empowerment to take ownership of their academic progress The research is clear on what can be done to advance student academic progress, the MAP approach creates a tool for achieving the progress and it merely needs to be applied.

Education is currently at a crossroads needing to implement bold innovations to achieve academic progress for all students. Research shows that formative assessment is a bold innovation to stimulate the progress. MAP is a current tool for implementing formative assessment innovations in an efficient and effective manner in the traditional classroom environment, seamlessly empowering teachers and students in the process.

Appendix A: The MAP Program

Research on the learning benefits of questions has shown that the learning environment can be improved by as much as 150% by the application of various questioning approaches. The use of appropriate questions have proved to (1) engage the learner (2) focus the learner on specific objectives (3) help the learner practice retrieval and application of information to answer questions (4) provide opportunities for feedback allowing students to understand why they missed specific questions (5) provide opportunities for repetition within the learning process. The eInstruction MAP program is designed to maximize the classroom learning environment with continuous and seamless formative assessment. It consists of the following four fundamental components.

Daily Instructional Activity

We recommend, as a general guideline, that 8-12 appropriate objective questions be integrated into every one hour learning activity. The break down of the questions, assuming the maximum of 12 questions is as follows: Prequestions (4) 2 review questions over previous materials to provide retrieval and repetition practice 2 prequestions to direct attention to the session's learning objectives Inserted questions (5) Questions for retrieval and repetition purposes Postquestions (3) Questions for retrieval and repetition practice (spacing occurs which improves learning benefits).

Bi-weekly Instructional Activity

On a bi-weekly basis, it is recommended you incorporate a fun team activity into the learning environment. eInstruction has developed a CPS team activity called "There It Is!" to support a simple and fun retrieval/repetition activity. Basically, each team has a set of objective questions in hard copy format numbered from 1 to some reasonable number, let's say 30. Each question has a set value and a bonus value. Prior to the game, the teacher will input a key for the game providing the correct answer for each question.

The teacher initiates each round of the game. CPS will randomly select a question number from the 30 questions and assign that question to a team. Each team will immediately identify that numbered question from their hardcopy of the questions, process that question and respond with their answer. When all the teams have provided an answer, the teacher will end the round and the scores will be shown.

Of course, teams that missed the question will not receive any points. The teams that get the correct answer will receive the point value for correct answers. The first team to get the correct answer to their question, on their first attempt, will receive the bonus points.

The game proceeds on that basis until a predetermined number of rounds have been completed. If there is a tie, then a sudden death can proceed to determine a winning team.

This activity integrates fun, competition, teamwork, repetition, retrieval, spacing and delayed feedback into the learning environment. The questions would be focused on the learning points of the previous 2-3 days. Additionally, delayed feedback can be accomplished by revisiting questions, if any, which were missed by any team. A very active, fun and rich formative assessment environment is represented by this activity as well as our Challenge Board team activity and more to come in 2005.

Weekly Instructional Activity

On a weekly basis, the MAP identifies a self-paced classroom activity with immediate feedback. This is a student self assessment activity. This is diagnostic in nature with practice on retrieval, repetition and spacing. Again, CPS supports this by coordinating a teacher-entered key to a set of objective questions in hardcopy format. It is recommended that a relatively small number of questions be used and coordinated to the key learning points of the week. Approximately 10 - 15 questions are reasonable to support this activity.

Each student proceeds to answer the questions. As an answer to a question is entered the student is provided feedback in the following manner. If the answer to the question the student is attempting is correct, CPS will blink the student's response pad number and then move immediately to the next question. The feedback is positive - the answer entered was the correct answer.

On the other hand, if the answer provided is incorrect, CPS will blink the student's response pad number, but remain on the question. The feedback indicated that the answer entered was incorrect and another answer must be entered until the correct answer is provided. At that time, the student is positioned on the next question and ready to enter an answer for that question. Teachers may choose to have students comment on the paper as they answer each question, indicating why they choose an incorrect answer and what led to the final selection of the correct answer. This produces ownership and self-assessment which are components of formative assessment instruments.

Although the feedback is very minimal, correct or incorrect, the student can eventually attempt every question and identify the correct answer to every question. This can provide the basis for additional work for the student to understand the basis for the correctness of a question. Additionally, upon completion of this activity the reporting system will identify any problem areas by standards allowing for a delayed feedback session to occur with teaching strategies immediately available to correct misconceptions.

Bi-Monthly Instructional Activity

Twice a month, we suggest a more formal 30-50-question test. This test can be constructed in the same manner as the Bi-weekly activity with a hardcopy for the student and a CPS teacher-constructed key. The key benefits of spacing, repetition, retrieval and delayed feedback are applicable with this activity.

Notice the delayed feedback approach recommended here. As the student undertakes the questions, no feedback is provided other than the fact that CPS has received the student's answer. Since the results are electronically captured there should be no reason why the feedback for the activity cannot be provided within a day or two of the activity providing a formative assessment instrument.

Many times activities like these require so much teacher work in the mundane grading phase that the results of the activity are not available for several days or perhaps a week. Research shows that delayed feedback beyond a "reasonable" time of a day or so provides no learning benefit thus the assessment moves from a formative assessment category to a summative assessment category.

References:

Boston, Carol (2002). The concept of formative assessment. Practical Assessment, Research & Evaluation, 8(9).

Black, P. and Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80 (2): 139-148. (Available online: http://www.pdkintl.org/kappan/kbla9810.htm.)

Massen, C., Poulis, J., Robens, E., Gilbert, M. (1998). Physics lecturing with audience paced feedback. American Journal of Physics, Vol 66, May 1998.

Mazur, E. (1996). Peer Instruction: A Users Manual. Prentice Hall, 1996.

Sanders, William F. (1999). Teachers, Teachers, Teachers. Blueprint Magazine.

Texas Center for Educational Research. The Cost of Teacher Turnover. 2000.

Thalheimer, W. (2003). The Learning Benefits of Questions. A Work Learning Research Publication (www.work-learning.com.)