By Darrell L. Ward, PhD

Research on the learning benefits of questions has shown that the learning environment can be improved by as much as 150% by the application of various questioning approaches. The use of appropriate questions have proved to (1) engage the learner (2) focus the learner on specific objectives (3) help the learner practice retrieval and application of information to answer questions (4) provide opportunities for feedback allowing students to understand why they missed specific questions (5) provide opportunities for repetition within the learning process. This paper will identify how these techniques can be simply integrated into the CPS empowered classroom.

Finally, providing relief to teachers from an administrative perspective is merely "icing on the cake", with respect to the research results of the methodology. Teacher turnover and morale have been directly related to increased administrative duties that take away from what they are trained to do - teach the students in the classroom. CPS is a proven tool for immediate and permanent reduction and relief of these administrative burdens.

The CPS- Empowered Classroom

The CPS classroom is characterized by the following:

• A one computer classroom - the teacher's computer
• All students are empowered with a simple durable response pad to communicate with the teacher's computer
• A software interface supporting:
  • Simple questioning capabilities for involving all students by verbalizing questions
  • Feedback mechanism for providing students with absolute knowledge that their answer has been received
  • Feedback mechanism denoting class results for each question
  • Absolute capture of individual student results
  • Delivery and capture of student performance on traditional paper tests

In this environment, each student, regardless of age and background, can easily and without risk of embarrassment, deliver their answers to objective questions. These questions can be:

• Constructed by the teacher "on the spot" and verbalized to the class
• Written on the board by the teacher
• Referenced by the teacher from any printed materials
• Constructed by teams of students and presented to other teams of students in a collaborative mode
• Pre-developed by the teacher and projected to the class from CPS

A large television monitor and/or a data projector connected to the teacher computer can support the environment visually.

There are currently around 7000 such CPS-empowered classrooms. CPS classrooms reside in all 50 states within the United States and several foreign entities, including the United Kingdom, Puerto Rico, Canada, Australia, Singapore and the Netherlands. The CPS-empowered classrooms range from kindergarten classrooms in the K-12 environment to graduate classrooms in the higher education environment.

The Foundation of an Interactive Learning Environment - Student Engagement

Clearly, no learning will take place if students are not engaged and motivated to learn. Research by William Sanders (Sanders, 1999) in support of the Tennessee Department of Education has identified the most important aspect of academic progress is the teacher. CPS is a major teacher tool with the following important characteristics to support the teacher:

• CPS provides a tool for the teacher to positively and simply engage all students in the class at any point in the instructional process
• CPS provides both immediate and delayed feedback to the teacher and/or the students in the class
• CPS reduces the cost to the teacher of analyzing short-term and long-term results of classroom interactivity
• CPS supports the repetition of core learning concepts

Certainly, if Sander's research is accepted, any tools that provide teachers more time to devote to student academic progress are extremely important. CPS addresses this by supporting the research documented to improve the learning environment as well as reducing the teachers' relatively non-productive time accommodating the various administrative activities.

With the key component in academic progress being the teacher, arming the teacher with CPS helps to make student engagement during the instructional process become the objective. CPS achieves that by providing each student with a simple and wireless handheld, the CPS response pad. This pad is non-threatening, robust, and is in use (at the time of this white paper, more than 250,000 CPS response pads are in this market) from pre-K through graduate education.

The engagement of the student on a continuous and no-risk basis clearly changes the classroom paradigm. CPS supports the teacher with a simple interactive mechanism for achieving classroom interactivity for all students in the classroom. This classroom environment assumes a teacher computer, CPS software, wireless handhelds for each student, and a recommended but optional projection system consisting of a video projector or TV. This mechanism, in its simplest form, is as follows:

• The teacher delivers an appropriate question to the class by verbalizing the question, projecting it on an overhead, writing it on the chalkboard or flipchart, or projecting it via computer software such as PowerPoint or CPS.
• The teacher provides objective answers from which the students may choose their answer.
• Students answer with their handheld CPS pads with affirmation on the computer projected screen or with verbal affirmation by the teacher if no projection is available.
• The results are typically provided to the teacher with the number of responses for each answer option shown next to the option or with a histogram. Normally, CPS will deliver the percentage correct for the question along with the cumulative percentage correct for the session.

Since students have no risk of embarrassment with respect to their individual answers, they are very motivated to participate. Many CPS-empowered teachers will provide ownership to the students by working with them to set a cumulative scoring goal for the class and then tracking their class progress in relation to that goal. If the class has an achievable learning session goal and incentive to reach that goal, student ownership of learning can be provided to every group learning session.

Results by Ryan Delaney (2002) in Ohio, where student ownership of learning has been an integral part of the process, have been very impressive in the K-5 environment. The student academic progress of several schools employing the concept of goal setting and data collection based on student engagement is shown in Appendix A.

Learning Benefits of Questions and CPS

Research over a long period of time has documented the best practices and benefits of questions as a learning tool. Of course, the assessment tool for utilizing questions and answers to evaluate student knowledge has been prevalent for centuries. This paper will summarize the learning benefits of this research while coordinating the basic research results to corresponding CPS tools.

Engaging the Students

Thalheimer (2003) has summarized almost a century of research on the learning benefits of questions. First and foremost, any question is of no benefit to the learners and teacher unless the learner processes the question and participates in the learning activity. Learners who are motivated to answer questions are more likely to learn than those that are not motivated (Frase, 1971; Frase, Patrick & Schumer, 1970). CPS provides motivation for every student to answer individually while also providing a group incentive for processing and answering questions.

Individuals are motivated to answer as CPS captures a complete record of their performance. Additionally, peer pressure is applied in a subtle manner, as a large majority of the class will typically answer every question.

Group incentives can lead to improved individual motivation because the learners will want to contribute to the overall success of the group. With a quantitative and real-time feedback tool present in the classroom, student motivation to participate is heightened. The teacher can announce "if our group cumulative score for this learning session is above 80, we will have no homework tonight" providing individual and group incentives to participate. Since the cumulative data is continuously available, everyone can strive to reach or stay above the group goal with all learners actively participating.

Feedback, Interactivity and Learning

With CPS, feedback is available at every learning opportunity. Clearly, feedback seems intuitive as a factor to improving the learning environment and all teachers' strive to create an interactive environment. In fact, research validates the importance of feedback in producing learning benefits (Bangert-Drowns, Kulik, Kulik, and Morgan, 1991).

The work of Mazur (1995, 1996), with his Peer Review approach in Physics, attempts to deliver questions to the class that will create a 50% correct response (Mazur, 1998). Thus, corrective feedback is provided to approximately half of the class. Guthrie (1971) claimed huge learning improvements when it is provided for incorrect answers or in a corrective manner.

Question Repetition

Boyd (1973) showed that multiple questions on a single learning point improved learning results over single questions by 29%. CPS can assist on the evaluation of the need for question repetition since the results become available in real time.

Clearly, when the results of a question are overwhelming positive, one can decide that an additional question may not be necessary. On the other hand, if the results are "marginal" from the teacher's perspective, then, after corrective feedback, an additional question is most likely in order.

Learning Benefits

Thalheimer (2003) has quantified the learning improvements as follows:

1. Delayed feedback produces better learning than immediate feedback by 10 to 25%
2. These numbers represent research on both non-inserted prequestions and learning objectives.
3. When initial repetitions are compared to no repetitions, the representative minimum is 30%, and the maximum is 110%. When subsequent repetitions are compared to fewer repetitions, the representative minimum is 15%, and the maximum is 40%.
4. Although questions inherently utilize some spacing, this effect may be small because one repetition with minimal spacing may not produce a large result, and certainly not the 5 to 40% improvements typically created by spacing. To be conservative, spacing improvements are not used in the overall calculation

eInstruction's Methodology for Interactive Data Driven Instruction

With the basic results provided above, eInstruction has developed an instructional approach or methodology that attempts to maximize the learning benefits of questions. This methodology is encapsulated in eInstruction's Interactive Data Driven Instruction model (Ward, 2003) and supported by the CPS empowered classroom.

Daily Instructional Activity

We recommend, as a general guideline, that 8-12 appropriate objective questions be integrated into every one hour learning activity. The break down of the questions, assuming the maximum of 12 questions is as follows:

• Prequestions (4)
  • 2 review questions over previous materials to provide retrieval and repetition practice
  • 2 prequestions to direct attention to the session's learning objectives
• Inserted questions (5)
  • Questions for retrieval and repetition purposes
• Postquestions (3)
  • Questions for retrieval and repetition practice (spacing occurs which improves learning benefits)

With CPS as an active supporting mechanism, real-time instructional decisions can be implemented based on the data that is collected and provided during this process. Since the data is collected in real-time, this data can drive or direct the instruction in real-time to support repetition, retrieval, and spacing decisions for additional questions that may be used.

Bi-weekly Instruction Activity

On a bi-weekly basis, it is recommended you incorporate a fun team activity into the learning environment. eInstruction has developed a CPS team activity called "There It Is!" to support a simple and fun retrieval/repetition activity. Basically, each team has a set of objective questions in hard copy format numbered from 1 to some reasonable number, let's say 30. Each question has a set value and a bonus value. Prior to the game, the teacher will input a key for the game providing the correct answer for each question.

Each round of the game is initiated by the teacher. CPS will randomly select a question number from the 30 questions and assign that question to a team. Each team will immediately identify that numbered question from their hardcopy of the questions, process that question and respond with their answer. When all the teams have provided an answer, the teacher will end the round and the scores will be shown.

Of course, teams that missed the question will not receive any points. The teams that get the correct answer will receive the point value for correct answers. The first team to get the correct answer to their question, on their first attempt, will receive the bonus points.

The game proceeds on that basis until a predetermined number of rounds have been completed. If there is a tie, then a sudden death can proceed to determine a winning team.

This activity integrates fun, competition, teamwork, repetition, retrieval, spacing and delayed feedback into the learning environment. The questions would be focused on the learning points of the previous 2-3 days. Additionally, delayed feedback can be accomplished by revisiting questions, if any, which were missed by any team.

Weekly Instructional Activity

On a weekly basis, the methodology identifies a self-paced classroom activity with immediate feedback. This is diagnostic in nature with practice on retrieval, repetition and spacing. Again, CPS supports this by coordinating a teacher-entered key to a set of objective questions in hardcopy format. It is recommended that a relatively small number of questions be used and coordinated to the key learning points of the week. Approximately 10 - 15 questions are reasonable to support this activity.

Each student proceeds to answer the questions. As an answer to a question is entered the student is provided feedback in the following manner. If the answer to the question the student is attempting is correct, CPS will blink the student's response pad number and then move immediately to the next question. The feedback is positive - the answer entered was the correct answer.

On the other hand, if the answer provided is incorrect, CPS will blink the student's response pad number, but remain on the question. The feedback indicated that the answer entered was incorrect and another answer must be entered until the correct answer is provided. At that time, the student is positioned on the next question and ready to enter an answer for that question.

Although the feedback is very minimal, correct or incorrect, the student can eventually attempt every question and identify the correct answer to every question. This can provide the basis for additional work for the student to understand the basis for the correctness of a question.

Bi-Monthly Instructional Activity

Twice a month, we suggest a more formal 30-50-question test. This test can be constructed in the same manner as the Bi-weekly activity with a hardcopy for the student and a CPS teacher-constructed key. The key benefits of spacing, repetition, retrieval and delayed feedback are applicable with this activity.

Notice the delayed feedback approach recommended here. As the student undertakes the questions, no feedback is provided other than the fact that CPS has received the student's answer. Since the results are electronically captured there should be no reason why the feedback for the activity cannot be provided within a day or two of the activity.

Many times activities like these require so much teacher work that the results of the activity are not available for several days or perhaps a week. Research shows that delayed feedback beyond a "reasonable" time of a day or so provides no learning benefit.

Teacher Productivity in the CPS-Empowered Classroom

Clearly, the more data that is collected from instruction and assessment activities, the more time and effort is required to the grading and evaluation of student performance. Teachers must carefully balance the capture and evaluation resources against the actual time delivering instruction and imparting knowledge. A technology that produces an abundance of quality student data from the instructional and assessment process while delivering more time for teachers to focus on actual instruction is a compelling technology. CPS does precisely that as it seamlessly assists the instructional process while reducing teacher time spent on the mundane grading and analysis activities.

The methodology described above delivers a tremendous return on investment with respect to the time of teachers. Since all aspects of the methodology outlined above are based on existing teacher objective materials, the efforts to coordinate CPS answer keys to these materials are very minimal.

The returns on those minimal efforts are significant and include the following:

• Complete reporting on student performance
• Standards-based reporting
• Immediate and delayed feedback to students and teacher
• Repetition on key learning points
• Retrieval of information in a variety of scenarios
• Spacing of questions to maximize learning
• Grading, grade entry and grade reporting electronically
• Availability of student performance data to district administrators seamlessly

Summary

The empirical data supporting the use of questions as a learning tool is overwhelming. With that as a basis, this white paper demonstrates, from both a productivity and instructional perspective, how CPS supports and extends the learning environment. The foundation of the learning benefits of questions is the actual engagement of the students. This paper demonstrates the value and a methodology for continuous engagement of students in the learning environment.

An eInstruction methodology with CPS support for each application is presented which consists of: