IBM Research

IBM Study Proves Use of Student Response Systems Increases Attentiveness

The attached was presented by Harold M. Horowitz, Ph.D., Program Director of Educational Technology, IBM Corporation at the Sixth Conference of Interactive Instruction Delivery for the Society of Applied Learning Technology (SALT).

This Paper describes the Advanced Technology Classroom at the IBM Corporation Management Development Center, and the application of interactive student response units along with the educational developments derived.

Student Response Systems:

Interactivity in a classroom environment
Harold M. Horowitz, Ph.D.
Program Director Educational Technology
IBM Corporation
Corporate Education Center
Thornwood, New York 10594

ABSTRACT

This Paper describes the research and development activities in educational technology which preceded the installation of the new Advanced Technology Classrooms at the IBM Corporate Management Development Center. Specifically, the application of interactive student response units is presented along with the educational benefits derived. The impact of this interactive capability on the design and development of courses is described in terms of goals, question categories and potential for improving instructional design methods. Finally, traditional and interactive classroom environments are compared based on students’ reactions and retention scores.

Background

During the past three years, the Corporate Management Development Center (MDC) Located at IBM Corporate Headquarters in Armonk, New York has served as a test-bed in experimenting with new concepts for brining technology into the industrial classroom. The culmination of these efforts resulted in the development of the Advance Technology Classrooms which are being used by IBM today for management training at corporate headquarters. This paper focuses on a major component of this new classroom - the student response system. This system enables each student to participate by responding to questions during the learning process. This interactive process was designed to increase the students’ attentiveness, aid in individual knowledge discovery and increase retention of key learning points.

Training Environments

IBM’s training policy and practice requires for all new managers to attend a one week course at corporate headquarters within three months after appointment. The topics taught include IBM’s history and values, personnel practices, leadership, communication, performance planning, reviews, counseling, appraisals, compensation, delegation and employee relations. The students come to the class with varied experiences and represent virtually every function, occupation and IBM location in the United States. The primary objective of this training is to provide these new managers with insight on how to manage employees within the spirit of IBM’s beliefs and value system. This training objective is accomplished by using lectures, case studies and discussions which are taught by instructors who have been identified as future "high-potential" managers. These appointments to the school are made by each executive areas and consist of a two year chair assignment.

IBM has always placed special emphasis on Management Development and the requirement for continual management training on an annual basis has contributed to the conclusion found in numerous surveys which report that IBM is among the best managed companies in the world. This high standard for education and quality of instruction at the Management Development Center created a difficult environment to initiate research and experimentation searching for innovative ways to improve the process. Indeed, as this effort began, doubts were raised about the importance of studying the existing MDC classroom environment which, in the minds of many both inside and outside the company, already represented the leading edge in industrial education.

Traditional Classroom Observations

During a six month time period from January through June 1984, detailed observations were made of management training for newly appointed managers. A typical class consisted of one-hundred students divided into five color groups of twenty students each. Each color group was assigned a separate classroom for interactive training and the entire class met in a large "main tent" on occasion for special presentations which warranted little or no interaction. The findings reported in this paper were based on data collected in "live classroom" courses by observation without any intervention in the learning process. The following four questions guided the data collection process:

How was class time spent by instructor and students?

What was students’ attentiveness and apparent interest?

What was the nature of students’ questions?

How effective was the use of audio/visual aids?

In order to decrease unexplainable variability of the findings, one specific course was selected which was taught every week by multiple instructors. This was a six hour course hundred key learning points which were covered in approximately 300 minutes of productive class time excluding breaks. This course was well designed and considered among the very best taught at the school. It consisted of about 100 key learning points. A key learning point was defined as a specific interpretation of policy and/or practice which the student was expected to know and remember when back on the job. The course material was presented using visual aids including approximately seventy transparencies (foils), numerous flip charts and handouts. A laboratory exercise was also included to enable the students to apply and practice the learned material and a video tape was used for presenting a case study to demonstrate the process using role models. The bottom line of the observations of this our course was – the instruction was effective on the whole and did achieve the objectives as stated in the syllabus. But basic questions about possible improvements in productivity and effectiveness still needed answers including whether technology could enhance the process. The following five generic observations were cited as having potential for improvements to this traditional classroom environment:

Observation 1: Visual Aids For the most part, visuals were busy, difficult to read and predominately consisted of bullets of associated word phases. These "word charts" received only periodic glances by students and did not appear to capture and hold their attention. However, when clear, conceptual visuals, diagrams and model representations were used, the students appeared to be more interested in the visuals and more attentive to the accompanying explanations by the instructor.

Observation 2: Logistics At some point in each class observed, the instructor experienced difficulties with the logistics of using multimedia, adjusting lighting levels and locating transparencies properly on the projector. These logistical difficulties resulted in nonproductive time in the classroom and while these situations were normally handled in a humorous vein, they were frequently disruptive to the continuity of the learning process.

Observation 3: Time Management Allocated course time did not always permit all the key learning points to be covered if the instructor did encourage students’ questions and comments as part of the learning process. Ratio of instructors’ presentation/lecture time to students’ question and answer time varied considerably among instructors.

Observation 4: Student Interaction Participation was not evenly distributed among students. In a typical class, between 10 and 20 percent of the students dominated the discussion, i.e., these vocal students asked the most questions, offered most of the unsolicited comments and were more likely to volunteer to answer the questions posed by the instructor. The remaining 80 to 90 percent of the students contributed only occasionally to the discussion unless specifically asked to do so by the instructor.

Observation 5: Attentiveness Students’ apparent interest and attentiveness while course material was presented tended to decrease during pure lectures which did not encourage student participation and increased as the instructor served more as a facilitator/enabler who encouraged students towards interaction and participation.

Time Allocation

How time was spent in the classroom was very much a function of the teaching instructors were observed teaching the same course – two who favored facilitation mode. While every instructor used both styles at times, there appeared to be a dominate style which was used more frequently. Three major categories of classroom activities which were derived from these styles were identified:

Lecture (Tell) The prepared verbal and visual presentation communicated by the instructor which included the theory behind the key learning points and associated explanations.

Dialogue (Share) The unplanned or unstructured discussion by the instructor and the students which covered related material and was designed to amplify the understanding of the key learning points and interpret their application on the job.

Question and Answer (Query) The specific questions and answers generated by both the instructor and students on key learning points and other related and non-related points. This Q & A normally expanded the application of the lecture and dialogue to specific situations, resolved confusion and provided some indicators to the instructor about the level of understanding by the students.

Time Management vs. Attentiveness

Lecture, Dialogue and Q & A activities were observed and a comparison was made between different instructor styles. A dichotomy was uncovered between efficient time allocation and student attentiveness. In summary, lecture style used time more effectively in terms of covering required material and learning points in the allotted time. Facilitation style tended to require more time but appeared to sustain the students’ interest and attentiveness.

Time Management When an instructor primary used a lecture style without encouraging questions and comments by the students, virtually all of the learning points were covered in the allotted time. However, the facilitator had more difficulty in completing all required material in the allotted time. In a typical class, the facilitator required 10 to 15 percent more time to cover the same number of learning points.

Figure 1 represents a composite example of time allocation for the instructors who primarily used a lecture style. Approximately half of the available class time was used lecturing and the remainder was used for dialogue and Q & A. Approximately ¼ of the time was spent with Q & A but for every three questions, the instructor originated two and the students just one.

On the other hand, the facilitator spent less time lecturing and used about 43 percent of the time for Q & A. The distribution of origination of questions between the instructor and students was about even as indicated in composite example shown in Figure 2.

Attentiveness While determining whether or not a student is exhibiting attentive behavior is subjective and judgmental, an attempt was made to compare the classroom environments based on instructor’s teaching styles. Body language signals were categorized using head, eye, hand, leg and sitting positions to depict possible attentive or non-attentive behavior. (1) & (2) Observations were made in a class of twenty students in 5 minute intervals and an instant determination was made for each student. While this determination has substantial potential for erroneous conclusions for any one student, consistent criteria was used for each student and the composite findings showed an interesting contrast between teaching styles.

The observations summarized in Figure 3 compared lecture and facilitation styles in terms of resulting student attentiveness. At the beginning of each class, most students exhibited attentive behavior which diminished rapidly within 20 minutes – after which the average number of students who exhibited apparent attentiveness stabilized. However, each student exhibited both attentive and non-attentive behavior at different observations and the recording of individual student’s fluctuating attentiveness was not performed as part of this study.

In order to compare teaching styles, an index was established to represent a composite of the area under the attentiveness curve. An index of 100 indicates attentiveness of every student at every observation point. In the lecture style, this index was 47 or just under half of the class. This index of attentive behavior increased to 68 for the class taught with facilitation style.

Short – Term Retention The students were tested anonymously to determine their understanding of the key learning points which were taught using both teaching styles. The results of testing showed that while facilitation required additional class time for the same number of learning points, the average short-term retention (3 days) was about 19 percent higher using facilitation when only considering those learning points which time permitted to be covered adequately. This finding corresponds to the 22 percent increase of average scores reported by Paul Macali in 1981 as the result of using Socratic questioning methods.

Technology in the Classroom

The initial observations were analyzed further in different training environments. The conclusion reached was that technology could play a vital role in focusing attention, supporting interactivity, improving logistics and fostering the facilitation style of an instructor.

An experimental classroom was completed during the summer of 1985 which incorporated promising technologies to address the problem areas found during the investigation. Observation 1 (Visual Aids) was addressed by replacing foil transparencies and slides with computer-generated graphics which focused on conceptual visualization rather than "word charts" . Observation 2 (Logistics) was resolved by incorporating an IBM PC/AT coupled with a plasma panel display podium into the classroom. The keyboard was eliminated and replaced by a podium. The IBM PC manages all the classroom logistics for the instructor including graphics, audio/video, lighting levels and automatic switching to connect any required component to the video projection system.

The resulting "Advanced Technology Classroom" has been successfully used for management training and is currently under evaluation in other diverse educational functions within IBM. Further descriptions of this new classroom concept can be found in papers by Levine, Garwin and Shappert and Vadas. The primary purpose of this paper is to report on one major aspect of technology brought into the classroom, namely the student response system which in conjunction with conceptual visuals and quality instructional design of courses addressed the opportunities described in Observations 3,4 and 5.

Interactive Classroom Concept

The design concept of the Advanced Technology Classroom was guided by the premise that the learning process could be improved if the visuals and the instructional design of courses would expand students’ interest and self-discovery through a high level of interactivity. Strong support for this basic concept for effective teaching is found in the Paideia Proposal which calls for radical reform of basic schooling in the United States. The interactivity aspect of this design concept has its roots in Socratic teaching principles which encourages questioning and active participation as the keys to more productive and enlarged learning.

The research and implementation by the Amherest H. Wilder Foundation in conjunction with the St. Paul School District in using a key board for each student to support group instruction in a classroom setting provides further evidence for merits of this interactive concept.

Student response systems

Could higher levels of student learning and retention be achieved by providing each student with a device to facilitate the Q & A process and would increase the students’ involvement and interaction in the classroom environment? To answer this question a student response system was incorporated in the classroom to enable each student to respond to questions during the learning process and to become a more active participant in the process.

Goals of Student Response

The concept of incorporating student response units (or keypads) for use by each student in the classroom was conceived and guided by five goals:

1) Student Activity Stimulate the active processing of data information, ideas, viewpoints and beliefs at the same time as the learning is taking place. The opportunity for participation and contribution should be available equally to all students.

2) Communication Create an environment in the classroom where differences in answers and opinions as a group can be observed and discussed immediately upon tabulation while keeping each student’s specific response anonymous.

3) Learning Desire and Commitment Provide students with frequent indicators of both individual and class learning progress which include comparisons with peer groups, previous classes and demographic subgroups – to encourage positive effects of self-assessment and competition among students.

4) Customized Instruction Provide the means for both preplanned questioning and ad-hoc questioning including the opportunity for students themselves to initiate the solicitation of class responses.

5) Data Collection Capture data on student responses divided into demographic categories to facilitate course revisions, to provide input to students on demographic positions; and to provide information for personnel research into critical topic areas.

Hardware

The experimentation with alternative student response systems was conducted using products manufactured and distributed by Reactive Systems, Inc. The Student Response Unit (keypad) which was selected for implementation in the classroom was a device comprised of ten keys for data entry and five function keys as illustrated in Figure 4. An LCD readout screen for eight numeric digits is included to enable the student to receive confirmation of the data input. The function keys provide capability to: 1) clear the screen, 2) answer more than one question at a time, 3) recall last answer, 4) ask for help and 5) send the student input to the system. up to 128 response units are cabled to an interface box using telephone-type wire. The interface box is connected to an RS 232 asynchronous serial communications port on a PC.

Software

Software which is resident in the PC polls, the response units, tabulates results and presents graphics simultaneously to the students and instructor.

The software provided by Reactive Systems which polls, tabulates and creates graphical results is called Instant Feedback. The instructor activates keypads and tabulates results by pressing appropriate function keys on a PC. There are four primary modes in this program: Instructor, Quiz, Reports and Roster. The Instructor mode enables the presenter to activate the keypads, poll the audience and display results as bar graphs. The Quiz mode is used for answering multiple (batched) questions. Reports are printed which summarize all responses by question for both individuals and the entire class. These reports include student names (if input in the Roster menu mode).

While this program supplied by Reactive Systems was effective in our early experiments, the complex requirements for student response in the Advanced Technology Classroom necessitated the development of a special purpose program which could run under the control of a Command Processor in the PC. This program activates and polls keypads from a remote control device, accounts for the different types of questions asked, includes paraphrasing on the resulting graphs, stores and retrieves demographic data for comparisons, and creates specially designed output displays on the presentation screen in the classroom.

Categories of Questions

Experience with using student response capabilities in the classroom has yielded an array of categories and types of questions which can be asked of the students for keypad input during the learning process. Listed below are examples of the categories and types of questions used to elicit input from the students during a course:

Yes/No or True/False This is the simplest of all question types which asks for a positive or negative response to a statement, situation or condition. Typically, this type of question elicits opinions, viewpoints and experiences with respect to key issues. Also an answer to a stated question can be presented for the student to either agree or disagree with the accuracy of the answer. Yes/No type responses can either be asked one at a time or in pairs where two questions are related. In the latter case, both graphical results are shown on the same screen for comparison.

Multiple Choice This question consists of a stem followed by alternative response choices. Typically , the stem consists of a question or an incomplete sentence and the alternative answers range from three to ten. The student selects the best answer or the best ending from the alternatives or "None of the above" is used if all other alternatives are incorrect.

Mean Numeric Entry This type of question requires the student to answer with a specific numeric value. Usually, there is no correct answer in this category but rather a range of answers which are of interest to the class along with a computed mean. The output display presents the mean, upper and lower limit as well as the historical mean from previous classes. An example question would be "How long have you been with the company?"

Correct Numeric Entry This type of numeric response is targeted towards eliciting a specific correct answer to a stated problem. Each student’s answer is compared against the predetermined correct answer and the percent of the students who obtained the correct answer is shown on the output display along with historical data of previous correct answers in other classes. This type of question requires more time for response than any other because some thought is required before entering answer. An example of a Correct Numeric Entry type question would be "what is the mean of the following set of numbers: (10, 22, 38, 5, 25). If 15 out of 20 students specified the correct answer, the output display would provide the correct answer (which is 20) and would indicate that 75% of the class provided the correct answer.

Rating Scale 1 to N This category allows the student to express his or her feelings or opinions about a particular situation or topic. A statement asks for a rating from 1 to n (where n usually ranges from 5 to 10). Each alternative numeric response is assigned a condition such as very poor, poor, average, good, very good in a graduated scale. The student selects the rating which best represents his or her reaction and the result is shown as a vertical bar chart containing the collective responses. A variation of this category is the "Consensus/No Consensus" which takes the distribution of responses and computes the mean and standard deviation. Consensus is assumed if the standard deviation is less than a predetermined value (such as 1.5 or 2). An example question would be "On a scale of 1 to 5 (where 1 is poor, 3 is average and 5 is outstanding) how would you rate the manager’s handling of this personnel situation?" If mean was 4 and the standard deviation was within the tolerance previously established, the output would indicate a consensus among the class respondents.

Group Response

When group activities are determined to be an effective method to stimulate learning of a particular topic, the Group Response variation is very effective. In this approach, students are assigned to one of two, three or four groups and the tabulation of results compares the collective summary totals from each group on the output display. Group Response Sequences further stimulate interest by promoting healthy competition among groups. The "Game" environment creates a peer pressure to participate and the desire to win encourages higher levels of attentiveness in order to provide correct answers and contribute to the success of the Group. Group Response is used in one of two approaches- On-line Class & Offline Breakout.

On-line Class variation can be used in conjunction with any of the previously described keypad sequences. At the beginning of a class, the instructor establishes groups either arbitrarily or for a specific purpose. Examples of group segmentation are gender, birth location, occupation, division, time in the business, etc. Once this determination is made, each student enters his or her affiliation with a specific group via the keypad. Whenever responses are tabulated and displayed, group results are shown in addition to the total class results. The rankings of the groups in terms of score achievement are determined by the system at the end of the course or at key milestones.

Off-line Breakout variation is used when groups meet separately to discuss issues and to derive a consensus which is reported back to the class. The group responses are tabulated and reported as each spokesperson enters the group’s consensus. However, all students are encouraged to remain active participants by providing their individual answers which may be different from the consensus opinion. The extent of minority opinions can be observed in the results and the resulting discussion includes all members of the entire class rather than predominately the spokesperson for each group. The rankings of the groups plus the individual scores are tabulated and both group winners and individual winners are determined by the system.

RESULTS

Pilot Classroom

The installation of an initial student response system in a pilot classroom did stimulate student interest beyond just a "Hawthorne Effect" but it did not make the dramatic differences in the classroom environment as was predicted. An analysis revealed that this was due to the fact that the instructors were unfamiliar with the system, the system was used too infrequently and the questions were almost all "multiple choice" and the students would find the correct answer about 90 percent of the time. Another finding was that when there were more than four alternative choices, too much time was spent by the students in the selection process.

Prototype Classroom

Based on the experienced gained in the pilot classroom, a prototype Advanced Technology Classroom was built. The graphical presentation and interactivity processes were improved and the results were very positive. The relevance and quality of the student response questions were enhanced and soliciting student responses more timely and frequent, i.e., at least once every 15 to 20 minutes. In addition, keypad questions were inter-weaved into the overall instructional design of the course and a variety of student response categories were used beyond just multiple choice. Thought provoking questions stimulated the students’ desire to seek self-discovery and provided them with the opportunity to compare their answers to those of the rest of the class and their peer groups. The subsequent discussions explored issues at greater depths and encouraged participation from a much larger percentage of the students. Classroom time was used more productively and substantially more learning points could be covered in the same allocated time. The technology assisted the facilitator to complete all the learning points within the allotted time and in some cases the total number of learning points were increased by 20 to 30 percent.

Student Attentiveness and Retention

When the same criteria to measure students’ body language was applied to classroom with student response units, the index of apparent attentiveness was found to be 83 as shown in Figure 5. In this classroom environment, the facilitator used the keypads to solicit student responses six times during the 90 minutes of instruction and the interactive process tended to peak students’ interest and attentiveness which would then decrease somewhat until the next opportunity for the students to respond. The test scores were higher in this environment – from the 19 percent improvement reported for the facilitation style to 27 percent when this style was coupled with the student response system.

Student Reactions

The students were asked to express their reaction to the interaction and feedback provided by the student response system. A scale of 1 to 7 was established to compare the conventional Q & A approach to the student response unit approach. A value of "4" was designated to represent an equal attitude and feeling between the two classroom approaches. A rating of 1 would be a strong vote for conventional Q & A while a rating of 7 would be a strong vote for the new student response system. Ratings in between these values would be graduated towards those indications but to a lesser degree. The results were a 6.6 out of 7 in favor of student response systems.

Conclusions

Based on the experimentation and findings described in this paper, interactive classrooms which use student response capabilities have been shown to improve the learning process and this concept should be explored further as we look for technology’s role in the "classroom of the future" for both industrial and public education. However, much additional research is required beyond the limited studies presented here. Within IBM, the Advanced Technology Classroom concept is being currently expanded into other learning environments beyond management development to determine its suitability and needs for enhancements in hardware and courseware to address new educational requirements.

Most seem to agree that education is the key to the future of our society. Unfortunately, the classroom suffers from technical neglect and a lack of creativity which would enhance the instructor teaching capabilities. While technology has provided our society with vast improvements in quality of life and productivity during this century, the classroom has not yet been a prime benefactor of technical innovation and ingenuity. most of today’s educational research is focused on interactive video and related self-learning concepts but the classroom requires some special focus and attention since it will likely remain our primary educational delivery system for many future generations. This paper suggests that computer supported interactive video and related self-learning concepts but the classroom requires some special focus and attention since it will likely remain our primary educational delivery system for many future generations. This paper suggests that computer supported interactive classrooms could enhance learning by supplying the teacher with relatively inexpensive technology.

About The Author

Dr. Harold Horowitz is the Program Director of Educational Technology with International Business Machines at Corporate Headquarters. He is currently responsible for research and development in instructional applications for the Advanced Technology Classroom, satellite remote training and individualized learning. Prior to this assignment, he was the principal architect in the design and development of the Advanced Technology Classroom. Dr. Horowitz has spent over 31 years with IBM and served as an engineer, systems analyst, operations research analyst, project manager, program director and educator. He holds a Bachelors of Electrical Engineering degree, a Masters of Business Administration in Operations Research and a Ph.d. in Educational Research. He has served as a lecturer at the University of Maryland and as an Adjunct Professor in the graduate program at the University of Connecticut.

References

McConnell, Charles R., Learn To Read Nonverbal Trainee Messages. Training. May 1978.

Pease, Alan, Signals: How To Use Body Language. New York: Bantam Books. 1984.

Micali, Paul J. The Power of the Questioning Approach, Training. March 1981.

Levine, James L., Garwin Richard L. and Schappert, Michael S., An Electonic Podium fo rthe Classroom. 1987 International Symposium Digest of Technical Papers.

Vidas, Judith E., Interacive Videodisc for Management Training in a Classroom Environment. Eighth Annual Conference on Interactive Videodisc in Education and Training, August 20-22, 1986.

Adler, Mortimer J., The Paideia Program: An Educational Syllabus. New York: MacMillan Publishing Company, 1984.

Robinson, Steven L., Technology and Group Instruction: A Communication and Management Tool for Teachers. 29^th International Conference of the Association for the Development of Computer-Based Instructional Systems. November 1987.