Since 1998, the Office for Domestic Preparedness (ODP), as part of the Department of Justice, has been mandated to enhance state and local capabilities to respond to Weapons of Mass Destruction (WMD) terrorism. To meet this mandate, the ODP Training Strategy was developed through the combined efforts of ODP, the National Domestic Preparedness Consortium (NDPC), and other federal, state, and local entities.
State and local authorities have long relied on established approaches (such as classroom instruction and practical exercises) to train First Responders. However, as the threat of domestic terrorism increases and the demands on Responders intensify, a more distributed and flexible training model is needed to guide future efforts. The training model must be agile enough to address dynamic requirements quickly, and robust enough to reach a large, diverse, growing audience.
For these reasons, ODP is pursuing a "Blended Learning" approach to provide modular training content in a variety of mediums (including, but not limited to, traditional, Web-based, computer-based, and Video Teletraining) to keep pace with current needs. By balancing the technological advances in Distributed Learning with traditional training methods, Blended Learning will improve support for First Responders in a number of ways as illustrated in Section 1.2.
Since 1998, ODP has conducted a thorough analysis of the WMD training landscape to support the development of a comprehensive Training Strategy. Ten distinct disciplines were identified as responding to WMD incidents: Emergency Management Agencies, Emergency Medical Services, Firefighters, Governmental Administrative, Hazardous Materials Personnel, Law Enforcement, Public Health, Health Care, Public Safety Communications, and Public Works. In a survey of subject matter experts in each of these disciplines, 152 tasks were identified as comprising the universe of necessary tasks to be performed prior to, during, and immediately after WMD incidents.
The survey found that ODP has accommodated or is accommodating 73 percent of the tasks unique to WMD, with 6 percent within the purview of sister agencies and 21 percent not accommodated by existing training or current development efforts (See Figure 1, page 2). The survey also found that the 21 percent of the tasks not accommodated by existing training (32 specific tasks), were complex and involved coordination among and between disparate agencies and organizations, or the management of activities within agencies. Though these complex areas require higher levels of the cognitive domain, The ODP Training Strategy acknowledges that they are gleaned from more basic level tasks, and urges training development in these areas.
Figure 1. Summary of Tasks Accommodated by ODP Training
Along with validating the instructional soundness of the WMD curricula, The ODP Training Strategy presented the following conclusions and goals:
The strategic planning process is important to current and future curricular development for critical, sensitive areas such as WMD;
Traditional training methods are appropriate for most tasks, the preferred location for most tasks was "on site" in the agency receiving training, and the preferred methods for providing training and performing evaluations was "projects and exercises";
There is a clear and present need for the standardization of expectations and performance measures for tasks associated with the response to WMD threats;
It is critical for ODP to maintain its position of prominence in facilitating the training efforts of each of the 10 key disciplines;
ODP should be the coordinating force at the federal level to "build capacity" of the local and state agencies and organizations, in a collegial fashion;
ODP should develop those courses needed to accommodate tasks which are not covered by existing training or influence disciplines to expand or bridge the existing training to accommodate the tasks.
Using the findings and conclusions of the Training Strategy as a foundation, ODP then published the Emergency Responder Guidelines to help provide the Responder community with a baseline understanding of the training content necessary to respond to acts of WMD terrorism safely and effectively. Though not provided as official regulations, the guidelines presented an integrated compilation of responder skills, knowledge, and capabilities. In effect, it served as a practical guide to the required WMD subject matter for first responders, course developers, and trainers.
This document is intended to serve as a practical guide for achieving the organizational goals identified in The ODP Training Strategy for improving WMD training through Blended Learning. This includes the necessary components, approaches, and considerations for delivering effective and consistent First Responder training.
The approach defined in this document recognizes existing ODP training processes (including Grants, Training and Technical Assistance, and Exercises) and recognizes the need to balance successful, existing instructional methods with the appropriate mix of Blended Learning technologies to:
- Increase the quality, consistency, and accessibility of training (any time, any place)
- Maximize content sharing and reuse among sister organizations
- Reduce classroom time by providing prerequisite learning materials via alternative media
- Increase training effectiveness and throughput by institutionalizing best practices
Blended Learning will ultimately provide the means to sustain First Responder performance levels over time, as seen in Figure 2 below.
Figure 2. Performance Impact of a Blended Learning Strategy
This document is not intended to be prescriptive. It includes concepts and tools to assist ODP and its training partners in performing analyses and reaching their own well informed training and performance support decisions. The remaining sections of this document describe these concepts in greater detail.
The primary audience for this document is ODP's training partners - consisting of members of the National Domestic Preparedness Consortium (NDPC) and others who develop and deliver training content to First Responders. This group will be responsible for communicating the strategy and overseeing the implementation within their organizations. Further descriptions of these partners and the role they play are presented in the ODP Stakeholder Panel document.
1.5 Strategy for Blended Learning
1.5.1 Overview
The overall goal of the ODP Training Strategy is to ensure domestic preparedness. A strategy supporting this goal must be:
- Comprehensive and integrated - to include all ODP program components that are related to training, training requirements, and human performance (e.g., Training, Exercises, and Equipment and Technical Assistance).
- Performance-based - to improve effectiveness in real-world crisis situations. A performance-based, integrated strategy would enable real-world results from exercises and incidents to drive training content and priorities.
- Efficient - able to leverage precious resources including facilities, funding, and expertise, to achieve the most return on the investment.
- Consistent - aligned with the goals of the ODP Training Strategy and Emergency Responder Guidelines as described in Section 1.1.
This section outlines a strategy designed to meet these requirements. It is important to note that this strategy builds on the existing ODP Training Strategy and highlights the importance and effectiveness of the training centers that provide unique, hands-on exercises and experiences. Unfortunately, traditional methods alone cannot meet the dynamic requirements of the First Responder community. Alternate instructional delivery methods can more efficiently and effectively meet these requirements. Combining these alternative instructional methods with traditional delivery constitutes the "blended" approach.
The ODP Strategy for Blended Learning involves three major baseline components:
- Guidelines
- Models and Best Practices
- Tools
These components are illustrated in Figure 3.
Figure 3. Baseline Strategy for Blended Learning
- Program inputs would include all information, resources, content and requirements for the design and development of a training course.
- Guidelines are established and agreed upon by the Blended Learning Stakeholder Panel and are codified in Style Guides that provide guidance on the design concept, interactivity, media and graphics, ADL SCORM conformance, and 508 compliance for courses.
- Models and Best Practices are provided through the Analysis, Design, Development, Implementation and Evaluation (ADDIE) model and the Style Guide that illustrate lessons learned in courseware production.
- Tools such as the Delivery Methods Analysis Tool are provided to guide the determination of appropriate delivery methods for courses or modules, in a consistent manner.
- Outputs of this model are just-in-time courses or other learning products that are consistent, effective, efficient, and of high quality.
As identified in The ODP Training Strategy, "resources must satisfy several million individual responders in need of initial training and sustainment training." To help relieve the pressure on faculty and facilities caused by this challenge, it is important to identify some means of increasing student throughput by exploring the possibility of moving content out of traditional classroom delivery, converting it to an alternative media, and delivering it at the Responders' work or home site.
This document views these alternative instructional methods as support systems that:
- Enable large numbers of trainees to meet awareness-level training objectives at minimum cost. ODP training is provided to First Responders from ten professional areas nationwide. This potentially heterogeneous trainee population constitutes a challenge for instructor-led training.
- Better prepare trainees to take advantage of instructor-led training experiences. Trainees lacking prerequisite knowledge are likely to require remediation. Meanwhile, more knowledgeable and experienced trainees have little to challenge them as others are brought "up to speed."
- Enhance the effectiveness of instructor led training. Blended Learning can enhance the efficiency and effectiveness of the classroom experience by ensuring that all attendees arrive at the training site with a baseline of knowledge, accurate and realistic expectations, and be ready to learn, allowing instructors to offer a more productive and robust training experience in a shorter time frame. At training centers, this increased efficiency can result in more time available for valuable hands-on experiences and exercises. Furthermore, some elements of Blended Learning media (graphics, animations, and video clips) and stand-alone content (reviews and practice exercises) can be used within the classroom to further enhance the training experience.
- Reinforce and sustain the knowledge and skills acquired through instructor-led, hands-on training experiences. Blended Learning can also reinforce and sustain what was learned in the classroom. Most people forget about 80% of what they have read within two weeks. Even intensive training experiences decay over time. To reduce that decay, the learning that takes place at training centers can be further reinforced through alternative distance delivery after students return to their duty stations.
Several considerations are important in defining a Blended Learning approach. These considerations are described in the following sections.
1.5.2.1 Right Media, Right Job
The many alternative delivery and support methods provide rich choices for delivering knowledge-based content. However, the array of available media and delivery can be confusing. To make sense of these alternatives, it is useful to categorize them using the following criteria:
- Purpose - are they best suited for communication, or training, or both?
- Time dependency - are they scheduled (synchronous) or available on-demand (asynchronous)?
- Interactivity - do they provide high levels of student interactivity, opportunities to respond, or minimal student involvement?
Using these criteria, various Blended Learning methods are summarized in Section 1.6 (page 9). To help Training Partners select the right media, a media selection tool is included in Section 1.6.1.3.3 (page 15) using specific combinations of audience, content, and instructional objectives.
1.5.2.2 Delivering a Consistent Message
Maintaining the consistency of information presented in training is an important consideration when designing WMD training for inter-agency use. Introductory material that establishes a common foundation for subsequent course-work is ideally suited for pre-work training. Because the knowledge, skills, and experience of new students are highly variable, alternative self-paced delivery of pre-work content will enable students to take control of their own learning progress, to "test-out" of sections of training, while others remediate at their own pace until confident. Better-prepared students with more consistent entering knowledge can increase the efficiency of classroom instruction, further improving input.
1.5.2.3 Performance-Based
Instruction can involve two different types of learning: knowledge acquisition and practice to proficiency. Knowledge acquisition involves learning new facts, ideas, concepts, processes, and skills. Practice to proficiency involves turning learning into experience by applying already-learned knowledge and skills in realistic settings, until a prescribed level of proficiency is attained.
Most training, and much of the content in the First Responder curriculum, involves the acquisition of new knowledge and skills. However, for many aspects of WMD training, acquiring new skills and knowledge, while essential, is not sufficient. When knowledge must be applied in complex real world settings that involve situation assessment and decision making under stress, there is no time to stop and try to recall what was learned. Learning must be internalized through practice - sufficient enough to assure high levels of proficiency and retention. Currently, the integration and application of what was learned in the classroom is provided through practice and exercises that are used for student evaluation.
The instructional effectiveness of First Responder training is largely determined by the ability of users to:
- Retain and recall the most important knowledge and information taught in the curriculum
- Effectively apply the skills acquired during exercises when confronted by challenging real-world situations
To be effective, students need more opportunities to integrate and apply their knowledge and skills in realistic situations. Field exercises that provide such opportunities are often too labor-intensive and time-consuming to provide enough practice for skills to become "automatic". Simulation is an excellent candidate for providing integrative contexts and practice opportunities for students to apply their newly acquired knowledge and skills until attaining acceptable levels of proficiency. Fully internalizing critical knowledge and skills is an extremely important training objective for First Responders who must later face situations that do not afford them the luxury to stop and try to recall what to do next.
Over the next two years, the use of PC-based Part-Task Simulation will become far more common for training First Responders. The notion of Part-Task Simulation was derived from flight training applications and the military. It is an acknowledgement that simulation cannot capture all of the complexities and nuances of complex tasks in the real world. They generally provide environments that simulate a critical part of a larger task, or enable the practice of a sub-set of the skills required in the real world. Military uses of networked simulations for mission rehearsal and the training of unit commanders, such as the Marine Corps Small Unit Leader Non-Lethals Trainer, are moving to civilian applications.
1.5.2.4 Reinforcement
Newly acquired information is often quickly forgotten unless it is associated with already meaningful knowledge and constructs. Reinforcement training aids in the retention of information by restating the information in a variety of contexts. As an integrated part of the First Responder training program, reinforcement training, in the form of a technology-enabled media, would be used to summarize pre-work content and present exercises for applying content learned in the classroom.
1.5.2.5 Refresher/Job Performance Aid
Crisis situations require finely honed skills and split-second responses. Refresher training is a means for responders to keep their skills sharp, and their knowledge current, throughout their careers. Much of the content developed for refresher training, along with computerized or Web-based simulations, could be provided to responders at their job sites, and made available on-demand. In addition, opportunities to package information in the form of Job Performance Aids (JPA) could be identified and distributed. A JPA, in the form of field-use cards or Personal Data Assistant (PDA), would be instantly available to the responder whenever needed to summarize critical procedures that would be needed to accurately assess a situation and save lives.
1.5.2.6 Mapping Available Technologies to Support Training Objectives
A well-designed Blended Learning Solution makes valid and justifiable decisions about when to use technology and when to involve instructors and exercises. By adopting a Blended Learning distribution model, trainers can incorporate the classroom as only one of several ways to deliver learning and development to First Responders.
There are many technologies that can be used to support a Blended Learning approach to training (e.g., training technology and communications technology). While most media are flexible and can be used for a variety of teaching functions, some media provide a better representation of objects, facts, ideas, processes, etc., and have great potential to develop learning skills. Choosing the technology that best supports the training goal involves consideration of several factors:
- Which media are likely to appeal to the learners?
- Do the learning objectives suggest a particular media?
- Is the technology needed to support these media available?
- How should each of the preferred media be used, and how much should each be used (for example, for what portion of the student's time)?
- What is the cost for designers, instructors, and learners in money, time, and flexibility-to use these media?
- Would less expensive media be sufficiently effective?
- Do the chosen media offer variety of stimulus and activity?
- How can the media be combined for maximum effect?
The Blended Learning Strategy will help to establish common standards and processes for development where possible, addressing delivery in an environment where infrastructure constraints vary widely. With the high degree of variability in available First Responder technology, training planning decisions will in many cases need to be based on educated guesses about the best delivery means when employing more than one technology strategy for a given course (e.g., putting a course online and offering a CBT version simultaneously). A description of these technologies can be found in Section 3.0.
1.5.2.7 Instructional Systems Design and the ADDIE Model
Instructional Systems Design (ISD) is the most widely used methodology for developing new training programs. This approach provides a step-by-step system for the evaluation of students' needs, the design and development of training materials, and the evaluation of the effectiveness of the training intervention. There are many ISD models, but almost all are based on the "ADDIE" model - Analysis, Design, Development, Implementation, and Evaluation, as illustrated in Figure 3 below. Each step has an outcome that informs the subsequent step.
Figure 4. The ADDIE Model of Instructional Design
Adoption of this model will promote best practices and establish a consistent methodology for the selection and use of appropriate tools and media in a Blended Learning environment. The details of this model and how it can be applied are addressed in Section 1.6.
This section presents the ADDIE instructional design model as the foundation for a Blended Learning approach. It focuses on the critical phase of analysis to provide the strategy for the remaining phases. Finally, this section explains the purpose of the ADDIE model which is to incorporate the results of the analysis and provide a foundation for the remaining phases.
A specialized approach is necessary with Blended Learning to select, design, and develop training courses. Content should be examined and assessed from the unique perspective of strategies, concepts, and tested practices.
In this section, the five stages of the ADDIE model are mapped to specific tools and processes that will enable the training partners to link available Blended Learning technology to specific training objectives. The goal is to find the most appropriate solution for the end user community. While specific methods and tools may differ among alternative media development, the phases of the ADDIE model are present in each.
1.6.1 Analysis
The Analysis Phase is the first phase of any instructional design project and the most critical phase in the implementation of a blended approach to training. Through a proper analysis, the "Who, What, Where, Why, and By Whom" is determined for the training program. The following processes are included in the Analysis phase:
- Curriculum or Needs Assessment
- Course Outline
- Delivery Methods Analysis
1.6.1.1 Curriculum or Needs Assessment
The first step in the Analysis process is to decide if a problem can be solved with training by performing a needs assessment. A needs assessment is the process of determining the need to improve performance in a specific area and to what extent the performance should be improved. The determination is based upon the discrepancy between desired and actual performance.
The needs assessment identifies the characteristics of the target group of learners. Learner characteristics include knowing the requirements of the target groups' job in terms of the background knowledge and skills. The information obtained regarding these characteristics will assist in defining the needs and constraints of the course.
The second step of Analysis is to decide the goal of the instruction and the content that will be included. This decision may be determined through a process called task analysis. Typical steps in a job task analysis include determining prerequisites, identifying job functions, identifying tasks within each function, and determining the task as procedural, process or principle based. The outcome of this process is the identification of the skills and knowledge that will structure the training program.
Figure 5. Sample Task Analysis
The next step examines the identified learning areas to define specific and measurable objectives. These objectives will determine the skills, knowledge and situations that must be developed and will provide a framework for the content. Higher-level skills and knowledge should be identified wherever possible so that understanding and problem-solving abilities are developed in learners.
A key output of the analysis phase is a detailed course outline. The course outline is the foundation of the complete course and guides the designer to organize content considering knowledge of the target audience, the course goals and objectives, and any resources materials, including information obtained from subject matter experts (SMEs).
Course content is organized into a detailed, logically sequenced outline considering knowledge of:
- The target audience
- The course goals and objectives
- Any resource materials, including information obtained from SMEs
To ensure a complete and logical sequence of instruction, there must be a grouping of related objectives and activities, and a sequencing of teaching activities. Objectives should be sequenced according to their learning level hierarchy so that the instruction begins with developing knowledge and then progresses to applying that knowledge after it has been understood (skills). Learners can then be taught how to apply the knowledge and skills to a variety of situations.
Once the objectives have been sorted into learning levels, sequencing of content can proceed according to a logical structure, such as:
- Proceeding from what learners know to what they do not know
- The logical or historical development of a subject
- Proceeding form concrete experiences to abstract reasoning
1.6.1.3 Delivery Methods Analysis
A specialized plan of action is necessary to select, design, and develop training courses for Blended Learning. It is advisable to assess course content from the unique perspective of strategies, concepts, and tested practices. Two essential activities used in a delivery methods analysis are a Training Delivery Strategy Analysis and a Media Selection Analysis.
The first step in a Delivery Methods Analysis is to conduct a Training Delivery Strategy Analysis. This procedure examines key elements of course content and assists in the determination of which course modules may be appropriate for Blended Learning delivery. As specific modules are nominated for possible alternative media delivery, each module is examined with the media selection analysis tool. This procedure examines general and specific criteria for each delivery technology in order to identify those technologies most appropriate for delivery of the specific module.
1.6.1.3.1 Training Delivery Strategy Analysis
Elements considered include outputs from the needs assessment and task analysis, as well as the following requisites:
- Audience Composition - The number of participants, their job d uties, knowledge, experiences, and learning styles constitute a profile by which better decisions can be made about course development and delivery. Analyzing and using this profile to inform the development and delivery process will maximize learner understanding, retention, and application.
- Course Goals and Objectives - To effectively present training information, increase knowledge and understanding, and promote skill building, those who develop training must be able to clearly express what they expect participants to learn.
- Course Modules - Course modules represent relatively self-contained components of associated information that can stand alone as a lesson or session, but are part of a larger topic. Modules segment information so that it is easier for learners to grasp and retain.
- Course Compression - Course content is streamlined and divided into modules containing only content that is specific to that module's learning objectives. Module content is then restructured/reorganized, to ensure a consistent flow of information. As a result of this process, course conversion may experience a 20-50% compression of content.
- Course Prerequisites - Considerations are made for previous knowledge, experience, or training that is needed before the course is taught.
- Course Participant Preparation - Course preparation is pre-course materials (i.e., questionnaires, exercises, self-assessments, or video tapes) or a pre-course learning event (i.e., audioconference or online conference) that the learner is required or recommended to complete before attending a course.
- Course Follow-up - Considerations are made for addressing learner needs after the course has been delivered (i.e., providing clarification, increasing understanding, and supporting the transfer of knowledge to job performance).
- Course Assessment - Feedback and evaluation data reflecting participant satisfaction and learning that supports the continuous improvement of a Blended Learning program.
1.6.1.3.2 Media Selection Analysis
The selection of appropriate media and technology for a specific course module is an essential element of effective and efficient use of available learning technologies in a blended approach. Often the same content can be delivered effectively in more than one medium. Therefore, the process of media selection is one of identifying the most appropriate media alternatives for a specific situation. Media selection analysis must evaluate general and specific criteria, including instructional, student, and cost aspects for each delivery technology to identify those technologies that are most appropriate for delivery of the specific module.
There are many technologies that can be used to support a blended learning approach to training. The following table provides an overview of the major delivery and support technologies available today.
Table 1. Delivery Technologies
Delivery Technology Description Instructor-Led Training (ILT) The ILT delivery method traditionally takes place in a classroom. The instructor provides knowledge or information to students in a systematic manner by presenting information, structuring learning experiences, and managing group discussions and activities. Interactive Video Teletraining (IVT) IVT is defined as a one-way video, two-way audio, training technology; includes the capability for students to ask questions of the instructor and to respond to other students. Video Teleconferencing (VTC) VTC systems are two-way communication systems that offer both audio and video from local and remote sites. Electronic Performance Support Systems (EPSS) EPSS provides on-demand access to information that a participant may need to perform a specific task. EPSS also provides a decision support system that enables students to identify the appropriate action required for a particular set of conditions within their working environment. Computer Based Training (CBT) and CD-ROM Interactive instructional experience between a computer and a participant where the computer provides the majority of the stimulus and the student responds, resulting in progress toward increased skills and knowledge. Web-Based Training (WBT) On-demand electronic training stored on a server and accessed across a network. It is delivered over public and private computer networks and displayed by students on a Web browser. Student access is asynchronous, and self-paced. Collaboration Environments (Synchronous Web-Based Training) Software and services delivered over the Web that enable synchronous e-conferencing, text chat, audio, video, document and application sharing, whiteboards, presentations. Table 2. Support Technologies
Support Technologies Description Synchronous Online Tools (SOT) These tools allow synchronous communication over the Web and include a variety of products. Some of these products are stand alone with just one capability such as text chat. Other products may bundle these tools with additional features such as whiteboards, session recording, polling/voting, presentation and application sharing. Audio Conferencing An electronic meeting in which students in different locations use telephones or audio conferencing equipment to interactively communicate with each other in real time. Audiographics Audiographics combines audio conferencing with personal computer text and graphics, allowing both voice and data to be transmitted to remote sites. Typically, a site consists of audio conference equipment, plus a large screen that serves as an electronic whiteboard. This system allows two-way data exchange between instructor and students. Asynchronous Online Tools (AOT) These tools allow asynchronous communication over the Web using tools such as threaded discussion groups, email, listservs, and search engines. This asynchronous tool is generally no longer used as a sole delivery method, but should be considered as a supplemental tool for effective Blended Learning, especially when prepared in electronic format for dual distribution. Video On-Demand and Web Casting Video content that can be delivered real time via the Web or downloaded to network servers. Students can access the system, get the specific information, data or instruction they require at whatever time of day or night they wish. Video Tape A method of capturing learning content on tape for viewing on-demand. It can be used as the sole means of content or as part of a blended approach to training. It is often used to capture real time event and is a useful tool because it captures an accurate image. 1.6.1.3.3 Advantages and Limitations of Delivery Media
An analysis of available technologies must include a thorough examination of the advantages and limitations that each present within the current training environment. Traditional courses cannot be transported to blended delivery without significant modification. Courses conversions will need to take advantage of the special characteristics of that medium, such as high levels of visualization and interactivity, and minimize the limitations of the medium, such as no visualization (audio conferencing), or the inability to see the students (one-way video/two-way audio).
The following sections address the advantages, limitations and instructional strategies for the major media listed.
1.6.1.3.4 Instructor Led Training (ILT)
Advantages - ILT is a traditional method of training that is comfortable for both students and instructors. It provides social interaction with immediate feedback, can be used with variably sized audiences, and can "blend" a variety of media into instruction that tailors content to the group or can be adjusted by the instructor while in progress. Students are removed from the work environment so that they can focus on the content free from distractions.
Instructional Strategies - The following instructional strategies can be implemented in a face-to-face environment:
Strategies
- Lecture
- Guest Speaker
- Large Group Discussion
- Small Group Discussion
- Cooperative Learning
- Peer Teaching
- Brainstorming
- Case Studies
- Panel Discussion
- Interview
- Field Experiences
- Lab Experience
- Role Playing
- Simulation
- Demonstration
- Drill and Practice
- Discovery
Many educators today are advocating the use of "blending" technology with face to face training to reinforce skills and knowledge, provide different avenues for students to explore, and to address different learning styles in the classroom, among others.
Limitations - While removal from the work environment may be beneficial for learning, prolonged absence from the workplace may impose hardships for work sites that must function for extended time periods without essential staff. Also, time and expenses for students and trainers to travel to "exotic" locations to attend training may be difficult to manage, and scheduling of training may not meet the needs of everyone who needs instruction. ILT moves at one pace, cannot provide enough time for individual feedback, and does not account for individual rates and styles of learning. It relies heavily on instructor knowledge and cannot promise a consistent delivery for all students trained by other instructors. Most importantly, training usually ends at the completion of class time and learning often lacks transfer to the workplace.
1.6.1.3.5 Interactive Video Teletraining
Advantages - IVT has the ability to simulate the classroom environment, can incorporate a variety of media, and can ensure a consistent delivery of content across geographical boundaries to a much larger audience in a shorter period of time. It can also provide high levels of interaction and immediate feedback to questions despite the distance between instructor and students. Travel costs for both students and instructors are reduced and less time is needed away from the job site for training.
Instructional Strategies - The following strategies often work well in an interactive video teletraining event:
Strategies
- Lecture
- Guest Speaker
- Panel Discussion
- Field Experiences
- Demonstration
It is important in this media to limit the content coverage to 3 to 4 key points per 50-minute tele-lesson (ideal time frame). Developers and instructors should strive to visualize as much as possible with pictures, graphics, video clips, word pictures, and other visualization techniques. Industry experts suggest three times as many visuals for interactive television as for traditional teaching. Video clips should be used, if available, to illustrate and reinforce key points. Activities or exercises should involve students at least 30 percent of the teleclass time.
Limitations - IVT requires the availability of a satellite broadcast infrastructure with studio uplinks and receiving site downlinks, therefore access may be limited to some learners (though ODP and its training partners have made a significant investment in their existing satellite infrastructure). Satellite time is expensive, and although it can reach vast audiences, large time zone differences may require some learners to participate at inappropriate times. Special training of the instructor is necessary to handle the technology, as well as a large staff to handle the broadcast equipment. Most IVT networks offer only one way video of the instructor and if the program is not well designed, becomes little more than a talking head with little or no interaction, making it unsuitable for content that requires deep processing of content.
1.6.1.3.6 Video Teleconferencing (VTC)
Advantages - Although VTC can provide two way video, it has many of the other advantages (and disadvantages) of IVT. Less expensive than IVT, it can simulate the classroom environment, incorporate a variety of media, and can ensure consistent delivery of content. It can be highly interactive and offer immediate feedback. Travel costs for both students and instructors may be even less than for IVT if equipment is available at the job site.
Instructional Strategies - The following strategies often work well in a Video Teleconferencing environment:
Strategies
- Lecture
- Guest Speaker
- Large group discussion
and brainstorming- Case study
- Panel Discussion
- Interview
- Lab experiences
- Demonstration
- Drill and Practice
Although instructors may need additional training, many of the strategies implemented in traditional training work well here. Videoconferencing can allow students to meet and confer with others across the country. Instructors find it easy to include lecturers, subject matter experts and guest speakers in classroom instruction. Two-way video facilitates collaborative learning via discussion, brainstorming, and group projects. Optional equipment such as VCRs, video scan converters, document cameras and allow instructors to include video illustrations, display PC application screens and "zoom in" on objects for classroom discussion.
Limitations - VTC may present access problems if the necessary equipment is not available locally. Delays due to compression and decompression rates of video may result in video and audio out of synchronization, distracting learners. As in IVT, large time zone differences may require some students to participate at inappropriate times. As with IVT, careless or ineffective design makes it unsuitable for content that requires deep processing of content.
1.6.1.3.7 Electronic Performance Support Systems (EPSS)
Advantages - EPSS can provide invaluable job aids by simplifying tasks and processes and providing just in time access to accurate information and to refresher training. The elaborate help system reinforces knowledge through practical exercises and immediately validates students' abilities to perform the task, empowering them to be more self-sufficient.
Instructional Strategies - The interface of an EPSS application determines to a great extent if students will use that application or not. Interface components include the position of elements on the screen, appearance and color, screen size, texture, sound, visual effects such as animation, and metaphors that accurately represent the real-world or real tasks that students perform. Good interface design meets the following criteria:
- The screen architecture (structure and navigation) must be consistent within an application.
- Screen elements must be placed so they are easy to find and use.
- Screen real estate usage must ensure that "clutter" is minimized.
- Menus and data input forms must be intuitive and easy to use.
Limitations - EPSS, built for performance support, is by its nature not suitable for initial training. Students will need initial training on the topic, as well as training to become comfortable with the EPSS tool functionality. In addition, the EPSS tool is not structured in a linear fashion and does not capture student input, so there is no easy method to measure whether students are using it or how often. Finally, the tool usually provides no method to effectively evaluate learning.
1.6.1.3.8 Computer-Based Training (CBT)
Advantages - CBT can provide a consistent delivery to audiences; can be easily sent to a large audience by mail or distributed via network; and can accommodate individual time schedules. With its large capacity for sound, video, and graphics, concepts are presented in their best form, motion or action is shown in video or animation, sound is presented as sound, and assembly or construction is shown as video or animation and practiced with hands-on simulations. Students can control the pace of instruction, receive immediate feedback to enhance learning and can "opt or test out" of lessons already mastered or repeat lessons to build competency. Production can be cost effective if large numbers are produced.
Instructional Strategies - CBT often employs the following strategies:
Strategies
- Lecture
- Guest Speaker
- Case Studies
- Computer-assisted
Instruction (CAI)- Lab Experience
- Role Playing
- Simulation
- Demonstration
- Drill and Practice
- Tutorials
- Discovery
When multimedia is high quality and carefully selected and created, CBT is an extremely useful teaching tool. It is important, however, that all media be carefully chosen for its educational value. Media for the sake of media can become irritating to students and become counterproductive. The ability to accommodate media rich files makes CBT an excellent tool for simulations, role-playing, and drill and practice that may facilitate experiential training.
Limitations - Development costs may be high unless a large audience is to be trained. Distribution efforts may require additional resources to track mailing and ensure all students have the latest version. Media content cannot be modified or updated easily and may require an upgrade of hard hardware (sound card, speakers, memory, graphics card). Often used as a self-study media, students may feel isolated and unmotivated to complete training.
1.6.1.3.9 Web-Based Training (WBT)
Advantages - WBT can provide consistent delivery to extremely large audiences and is considered an excellent training media for development of cognitive objectives as well as attitudinal objectives. WBT can accommodate students with disabilities by use of assistive technologies. It can be designed to support small group work and collaboration. Multi-purposed (training and resource), it can restrict access if necessary to authorized users, is easy to update, engaging if well designed, and can offer students the opportunity to remediate materials until proficient or "opt out" of content they have already mastered. Content can be linked and learning tracked within Learning Management Systems and "modularized" into small units of instruction suitable for assembly and reassembly into a variety of courses.
Instructional Strategies - WBT has become an extremely effective media using a variety of instructional strategies including the following:
Strategies
- Demonstrations and Simulations
- Drill and Practice
- Tutorials
- Games
- Story-Telling
- Role Playing
- Discussion and Debate
- Field Trips and Lab Experience
- Case Studies
- Lecture
- Guest Speaker
- Cooperative Learning
- Simulation
- Discovery and Research
As with other Distributed Learning media, a development strategy must be in place for writing Web-based instruction to ensure students are actively engaged and will be able to meet specified learning objectives. Designers must weigh a number of considerations including effective Website interface development, graphics and rich media development, opportunities for both asynchronous and synchronous learning environments, and the complexities of learning management systems (LMS) and Learning content management systems (LCMS), including interoperability and usability standards and considerations (e.g., SCORM and Section 508).
Limitations - WBT is technology dependent, requiring faster Internet connectivity and a slightly higher degree of computer literacy than basic PC knowledge. The interface is not ideal for extensive text and must be well designed to avoid skimming information rather than going in-depth. Development requires specialized knowledge, software, and equipment, with development time often running between 200 to 300 hours per one hour of finished product. Bandwidth limitations may restrict the use of high-end visuals and sound.
1.6.1.3.10 Synchronous Web-Base Training (SWT)
Advantages - SWT provides live interaction with instructors, experts and other students and provides an efficient way to disseminate information to dispersed audiences. Additional graphics, video, and sound can enhance content and learning. It can be easily updated, even "on the fly" and is considered an excellent media to achieve both cognitive objectives and performance objectives. SWT supports collaboration, breakout sessions, and can be very engaging if well designed. Students unable to attend a class can access a recorded text of the session after class.
Instructional Strategies - Many of the instructional strategies that work well in the classroom can also be adapted for SWT:
Strategies
- Lecture
- Guest Speaker
- Small Group Discussion
- Cooperative Learning
- Peer Teaching
- Brainstorming
- Case Studies
- Panel Discussion
- Interview
- Simulation
- Demonstration
While some synchronous tools support streaming video, equipment and bandwidth requirements usually prohibit two-way video of instructor and students. While much of the visual content can be shared with the class through graphics, application sharing and Web surfing, designers and instructors must consider the fact that students may only be able to discuss their views and interact via the audio connection.
Limitations - Synchronous training by nature is less flexible than asynchronous, requiring students to be available when the class is held. This type of Web-based delivery is also technology dependent, requiring Internet connectivity and student familiarity with application software. In addition, firewall issues within student organizations may make connection difficult or prohibit connection entirely. Development, although relatively easy for trained providers, must be well designed to engage remote students. Too often, content is limited to simple text in PowerPoint slides with few other techniques to engage other learning styles. Bandwidth limitations usually restrict the use of high-end visuals and sound as well as limit class size to between ten and fifteen students. If too many students access the session at different speeds, students may have to wait for others to view displayed screens or post remarks, lowering the motivation to continue.
1.6.1.4 Support Technologies for Distance Delivery
Several technologies, both old and new, can supplement the media chosen as the main delivery method, adding flexibility and sophistication to instructional design. The following media are not recommended as sole or primary delivery media, but should be examined for their ability to support and enhance a blended approach that increases interactivity, addresses learning styles, motivates students and provides an engaging and effective training outcome.
- Print Materials - This asynchronous tool is generally no longer used as a sole delivery method, but should be considered as a supplemental tool for effective Distributed Learning, especially when prepared in electronic format for dual distribution. Courses that "blend" this technology into training solutions may use the media to reach students who may not have Web access (such as Responders out in the field or in remote locations), or may simply need to provide supplemental materials in both print and electronic forms. Print materials can provide inexpensive representations of charts, graphics, and text to supplement instruction, and can provide pre-training or refresher materials as well as effective job aides. This cost effective solution can be used in role-playing, drill and practice, case studies, as well as discovery instructional strategies.
- Audio Conferencing - While students might have difficulty remaining engaged for course material delivered entirely via this media, audio conferencing can be a valuable support technology if students need to receive modification or updates to course content quickly, or feel the need to interact with instructors or experts and other students to clarify content previously delivered. This synchronous technology can also be a valuable media for asynchronous delivery of content, providing opportunities for brainstorming, role-play, and discussion that might not be available in the media selected for course content delivery.
- Audiographics - By combining audio conferencing with text and graphics provided by technology, both voice and data can be transmitted to remote sites. This support technology provides synchronous communication between instructor and students at a relatively low cost and can serve to transmit the complex images, animation and sound that other blended media cannot. This synchronous tool can be very interactive, easily updated, and provide even the novice instructor with local control of the learning environment.
- Videotape - Because of its limitations, videotape (often used in the past as a sole means of delivery content) is today considered better implemented as part of a blended approach to training. Low tech and user friendly, it is often used to capture real time events with accurate images. It can effectively convey a motivational or emotional message and is a useful tool for modeling behavior.
- Video on Demand and Web Casting - With many of the characteristics of videotape, delivering live or delayed versions of sound or video broadcasts via the Web may not be considered appropriate as the sole media for delivery of content, but is considered extremely useful as part of a blended approach. By providing students with Web access to pre-recorded training content, they can access the system and get the specific information, data, or instruction they require when needed. Like a VCR or DVD player, important sections of the program can be paused and watched repeatedly if necessary or skipped at the student's discretion.
- Synchronous Collaborative Online Tools (SCOT) - While not a delivery medium, these simple tools allow students to communicate and share files and graphics and can support blended solutions in variety of ways. Examples of SCOT are chat programs such as AOL's Instant Messenger and whiteboard software such as that included in Microsoft's NetMeeting. SCOT provides real time, immediate interaction for all students. They can be used by instructors to provide after class access for follow up discussion and Q&A or to conduct one-on-one private conversations with students to encourage or modify behavior. Students can also use the tools to share ideas about class content and collaborate on class projects.
- Asynchronous Collaborative Online Tools (AOT) - Asynchronous Support Technologies such as e-mail, threaded discussion groups, and listservs, allow students to post and reply to comments regardless of time or location. As with some of the synchronous tools, AOT can help instructors encourage or modify student behavior as well as support communication between students. Those self-conscious about disabilities or differences need not reveal it to the group. Multiple topics can be discussed concurrently with contributions organized by topic. Students can retain discussion messages and take time to mull over ideas, check references, and refer back to previous postings before making a comment.
1.6.2 Design
Once the blended solution is determined in the Analysis phase, the Design phase creates the "blueprints" for the instructional experience. In this phase, the instructional designer plans the elements of instruction, such as instructional objectives, assessment strategies, course outlines, design documents, style guides and storyboards to describe the presentation of content, practice activities, and feedback mechanisms.
The following processes are included in the Design phase:
- Design Document
- Style Guide
- Storyboards
During the design phase, the following questions are addressed:
- How will the content of the learning material be structured -
Sequencing, presentation, and reinforcement of the content will rely on grouping of related objectives and activities. Dividing the course content into lessons and determining the best order in which to present those lessons are the most challenging parts of the ISD process. The designer must have a clear understanding of the content in order to clearly interpret and communicate it to the desired audience. This process begins in the Analysis phase when developing the course outline. The outline is then finalized in the Design phase.
- How will learning be assessed -
Assessment measures the competence or capability of students in terms of whether or not they have realized the objectives. Feedback is important because students need to assess the adequacy of their responses and learn from the assessment experience.
- What resources and strategies will be used in the instruction -
Teaching resources and activities should be chosen to complement the learning ou tcomes.1.6.2.1 Design Document/Video Treatment Script
The purpose of the Design Document (sometimes referred to as a Video Treatment Script in synchronous training such as IVT, VTC and Synchronous Web Training) is to create a roadmap to use throughout the course design and development process. It provides a skeleton of the course content and documents the detailed expectations. A Design document typically includes the following items:
- Course structure and flow
- Definition of assessment and instructional strategies
- Interactivity guidelines
- Technical specifications
- Estimation of the course length
- The prototype objective
- Final detailed course outline
A style guide sets the standards for the design, development, and production of training courses. It provides tools and best practices and sets the foundation for the final four phases of the ADDIE model. Establishing a consistent style provides uniformity across courses and ensures clear guidance and high quality materials. The style guide also increases course development efficiency by avoiding repeated design, development, and evaluation of key courseware elements.
A style guide for Web-based development will be provided as an Appendix item in the final version of this document.
Storyboards are the blueprints of an interactive courseware design and development process. They provide a combination of text and graphics or graphic descriptions that convey all the necessary information about the delivery of course content. Storyboards describe in detail all images, animations, movie segments, sound, text and navigational paths. The more complete, detailed, and accurate they are, the fewer the assumptions, questions, delays, confusion, and errors that occur later, during the costly Development phase. An accurate content outline, course design document, and style guide are essential to the success of the storyboard design.
Development is the production phase of the ADDIE process. During development, the plans defined in the Design phase become reality. In this phase, the instructional designer is concerned with issues such as using the instructional materials to facilitate and support learning. Interactivity is also a key concern. Designers and developers work to incorporate the visual design of the instructional materials to best support and facilitate learning and determine appropriate interactions that are creative, innovative, and engage students to explore further.
Two tools that assist the development of content are templates and authoring tools. Templates are documents with a preset format prescribed and defined by the Style Guide. They are used as a starting point for content design so that the format does not have to be recreated each time it is used. Authoring tools are software applications that aid in the creation of content for a particular delivery media, such as Macromedia Dreamweaver MX and Flash MX. Again, interoperability and usability considerations must be threaded throughout the development process.
In the Implementation phase, the course is tested and deployed for use by the desired audience. In this phase, the course is tested to ensure that the components function as designed within the current environment. The development team may employ tools such as an implementation checklist to validate a successful implementation and identify anomalies that must be addressed before full deployment of training content.
The following activities are addressed in this phase (if applicable):
- Running and testing of the system in the specified environment (system and user acceptance testing)
- Planning for coordination and management of the training program
The Evaluation phase serves as the quality management component for training development and ensures that the course functions as designed through evaluations with actual end users. Surveys and questionnaires address training issues such as:
- Did students like the instructional experience?
- Did students achieve the goals that were set out for the instruction?
- Were students able to transfer what they learned in class out in the real world?
- Was there any long-term return on the investment in the instructional experience?
The answers to these important questions allow the instructional designer to perform two critical functions: 1) Certify that learning has actually occurred as result of the instructional experience they created; and 2) identify gaps in student performance that need to be addressed with additional training or enhancements and modification to existing content.
The importance of this phase cannot be understated. It is through this phase that results and lessons learned from user feedback can be captured and distilled. By analyzing these results, action plans can be defined to inform future course development efforts - thus "closing the loop" of the ADDIE process. ODP is making significant strides in this area to build a systematic evaluations function to support continuous improvement across its programs.
2.0 Delivery Methods Analysis Tool (DMAT)
The primary elements of the Delivery Methods Analysis Tool (DMAT) include the following:
- Training Delivery Strategy Tool
- Media and Technology Selection Tool
The DMAT includes a methodology to:
- Capture course information that can be used to make decisions about how to approach, design, and deliver course content and materials.
- Identify course modules and their instructional considerations as an initial step in the process of determining an appropriate delivery strategy for each module.
- Provide media and technology considerations for each of the following: Face-to-Face, Interactive Video Teletraining (IVT) including both live and videotaped broadcasts, videoconferencing (VC), Web-based training (WBT), asynchronous self-paced and synchronous, computer-based training (CBT) via CD-ROM, audio conferencing, and print-based/self-paced learning modules.
- Provide a framework for selecting appropriate delivery media or technology for course modules.
2.3 Training Delivery Strategy Tool
This section discusses the need to assess course content (from the perspective of strategies, concepts, and tested practices) to define modules that are appropriate for Blended Learning delivery. Specific course elements that will be addressed here include: Audience composition, course goals and objectives, course modules, content allocation, course prerequisites, course participant preparation, course follow-up, and course assessment and evaluation.
2.3.1 Audience Composition
The number of participants, their job duties, knowledge, experiences, and learning styles constitute a profile by which better decisions can be made about course development and delivery. This profile is used to inform the development and delivery process to maximize learner understanding, retention, and application.
2.3.1.1 Considerations for analyzing audience composition
- Approximately, how many participants will attend each session? ______
- Who is your target audience? Check all that apply.
Directors
- Will the audience be mixed (see positions checked above)?
- What is the general skill or knowledge level of participants? For example:
- What is the participant skill or knowledge level (from training, on-the-job training, other relevant experience) in the specific area covered by the course topic?
- What is the participant level of technology "literacy"?
- What is their experience level with Distance Learning technology?
- What are characteristics of your learners? Check all that apply. For example:
incorporate a new
program or policies- Why are participants taking this course? Check all that apply.
- How will they apply what they learn through this course to their jobs? Check all that apply.
2.3.2 Course Goals and Objectives
To effectively present training information, increase knowledge and understanding, and promote skill-building, training developers must be able to clearly express what they expect participants to learn. This section of the tool assists users in identifying specific course goals and objectives.
2.3.2.1 Goals
Course goals describe broad, encompassing learning outcomes. For example, goal of this course is to:
- Increase knowledge and understanding about...
- Foster communication about and resolution of....
- Establish consensus on...
- Familiarize EES employees with...
- Introduce the concepts and theories related to...
- Expose participants to the principles of...
Question: What is the overall goal of this course?
_________________________________________________
_________________________________________________
_________________________________________________2.3.2.2 Objectives
Course objectives describe precise behaviors participants will be capable of at the end of a course or session. Objectives are stated as specific, "hard" or action verbs. For example: By the end of this Mid-Career Retirement Planning course, participants will be able to:
- Identify federal retirement benefits
- Access information about time-sensitive relevant benefits
- Devise a personal financial plan
Question: What should participants be able to do after they complete the course?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________2.3.3 Course Modules
Course modules represent information that can stand-alone as a lesson or session. This part of the tool helps users to segment training information, so that it is easier for learners to grasp and retain. Modularizing content also aids the training developer in identifying content for a distinct method of training.
2.3.3.1 Considerations for analyzing course modules
- What are the major subtopics of the course?
1. 5. 2. 6. 3. 7. 4. 8. - Is there enough information for each of the above subtopics to justify a module?
- If not, are there subtopics that naturally can be grouped together?
If yes, which ones?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________- What are the learning objectives of each module? What should participants be able to do after they complete Module 1?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________- What should participants be able to do after they complete Module 2?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________
(Continue on back, if needed.)- Do each module's objectives appear to be distinct from one another, and important and substantive enough to stand on their own as subject matter?
- Is there a logical start and end to each module?
- Is there a logical connection from one module to the next?
2.3.4 Content Allocation and Course Compression
This section of the tool aids the user in identifying essential, optional, and supplemental information.
Course compression often takes place when existing course materials that are delivered in a face-to-face environment (i.e., workshop or seminar) are converted to a distance learning format (i.e., IVT, Web, etc.). Compression occurs for two reasons: First, course content is streamlined and divided into modules containing only content that is specific to that module's learning objectives. Secondly, module content is restructured and reorganized for the specific format of the delivery technology. For example, content is compressed when converted to Web-based training because it is not delivered linearly as in a face-to-face format.
2.3.4.1 Considerations for content allocation and course compression
- For each module, what information is essential (list below, or highlight on content outline)?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________- What information is not essential (strike out on outline, or make mental note)?
- What information is optional, should there be time or the opportunity available to include it?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________- What information is likely to trigger participant questions and comments?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________- What information could be provided before or after the core course delivery through another format or as supplemental material?
_________________________________________________
_________________________________________________
_________________________________________________- Does the compression of content above reorganize or restructure the modules identified in the previous section?
2.3.5 Course Prerequisites
This section highlights considerations for identifying previous experience, training, or education needed before training begins.
Considerations for analyzing course prerequisites:
- What previous experience, training, or education should participants have to take this course? For example:
Specific knowledge and experience:
____________________________________________________
Completions of a previous training course:
____________________________________________________
____________________________________________________
____________________________________________________- What are the exceptions, if any, to this course's standard prerequisites? For example:
Educational background may replace prerequisites
Similar experiences may replace prerequisites
____________________________________________________
____________________________________________________2.3.6 Course Participant Preparation
Course preparation is pre-course materials (i.e., questionnaires, exercises, self-assessments, or video tapes) or a pre-course learning event (i.e., audioconference or online conference) that the participant is required or recommended to complete before attending a course. The content of the materials or learning event are directly relevant to and part of the course content to be delivered and increased participant learning readiness for course content.
2.3.6.1 Considerations for analyzing participant preparation
- How important is setting participant expectations before the course?
- Would participants benefit from seeing some portion of course content before the course?
If yes, what portion(s)?
_________________________________________________
_________________________________________________
_________________________________________________
_________________________________________________2.3.7 Course Follow-up
This section of the tool enables users to capture course follow-up elements including:
- Addressing question-and-answer or discussion overflow
- Providing clarification/increasing understanding
- Supporting the transfer of new knowledge or skills to the job
- Providing additional details and examples
2.3.7.1 Considerations for analyzing course follow-up:
- Is there course information that is likely to trigger more questions than time allowed during the scheduled session?
If yes, list:
_________________________________________________
_________________________________________________
_________________________________________________- Is there information that might require clarification or elaboration?
If yes, list:
_________________________________________________
_________________________________________________
_________________________________________________- Are there topics that might require more details and examples?
If yes, list:
_________________________________________________
_________________________________________________
_________________________________________________- Is this information provided in other supplemental training materials (i.e. workbook, information on the agency Intranet)?
_________________________________________________
_________________________________________________
_________________________________________________2.3.8 Course Assessment: Evaluation
Feedback and evaluation data from end users supports continuous program improvement. This section of the tool helps users to understand the process and structure for assessment and evaluation of course content for future improvement. Considerations for analyzing course assessment include the following:
- What do you expect to learn from course evaluation?
Level 1: Participant reaction/satisfaction
Level 2: Learning effectiveness
Level 3: Ability to apply learning/behavior
Level 4: Results of application- How will evaluation results be noted and passed along to course instructors or managers? For example:
Copies of evaluation forms made available
Synthesis and interpretation of evaluation forms made available
Verbal
Written
______________________________________
______________________________________- How will evaluation data be used to improve the course? For example:
Coaching
Presenter/Instructor integrate on own
Course team meets and integrates evaluation results into process before
the next course delivery
__________________________________________________
__________________________________________________2.4 Media and Technology Selection Tool
The selection of appropriate media and technology for a specific course module is an essential element of effective and efficient use of Distance Learning technologies. The Media and Technology Selection Tool provides criteria that should be evaluated for each delivery technology to identify those that are most appropriate for the specific module being considered. Additional references regarding development timeframes, instructional strategies, and delivery costs will also be provided. This tool should be implemented after completing the Training Delivery Strategy and identifying a specific module for media selection analysis.
2.4.1 Decision Tree Process of Media Selection
This section discusses how the Media Selection Tool uses a "decision tree" process to guide users to the most appropriate delivery technology for each course module. It identifies when and how the tool should be used, including "knock-out" questions that indicate the delivery technology being considered does not meet the delivery needs of the specific module.
2.4.1.1 When and How to Use the Media and Technology Selection Tool
The Media and Technology Selection Tool is one of a number of "how-to" Distance Learning materials and learning opportunities custom-designed for program managers, training coordinators, instructors, and other staff involved in DL related programs. Ideally, this selection tool should be read and implemented after
- Determining training topics by (a) needs assessment(s),
- Completing the Training Delivery Strategy Tool worksheets, and
- Identifying course modules.
The Selection Tool guides the user to the most appropriate delivery technology for each course module and is designed to be used in the follow manner:
- After course modules have been identified, the Media and Technology Considerations section of this tool should be completed for EACH module.
- Users should answer questions for each technology in the following order: Face-to-Face, Interactive Video Teletraining, Video and Web-Based Conferencing, Web-based Training, Computer-Based Training, Audio Conferencing, and Print-Based/Self-Paced (pp. 3-10).
- Questions should be answered YES or NO in the order presented.
For some questions, direction is given to go to another delivery technology if a negative answer is provided. These "knock-out" questions indicate that the delivery technology being considered does not meet the delivery needs of the specific module and a more appropriate delivery technology should be considered.
For example, if the question below is answered NO, then the user would stop answering questions for IVT - Videotape Considerations and go to the WBT Considerations. The user would continue answering questions in the WBT Considerations section to determine if it is a more appropriate delivery technology.
IVT - VIDEOTAPE CONSIDERATIONS Is the use of motion video necessary to achieve the module's learning objectives? YES NO If NO, go to WBT Considerations. Once finished answering all appropriate questions, the user should count the number of YES answers for each delivery technology and write the number in the appropriate place on the Media and Technology Summary Results (pp. 11-12).
2.4.1.2 Understanding Summary Results
The Media and Technology Summary Results provide a framework for recommending technologies that are appropriate for the delivery of a specific course module. This is why it is so important to break the course into modules as specified in the Training Strategy Tool. The more YES answers, the more likely the technology is appropriate for delivery of the module. Selecting appropriate delivery media/technology can immensely increase learning, retention, and implementation of necessary skills.
However, it should be noted that although a blended approach to delivery technologies is encouraged, it is important to resist the temptation to select too many different delivery media and technology, especially in the planning phase. The "bells and whistles" syndrome can come back to haunt you when planning becomes development and you are confronted with the time, energy, and complexities of integrating multiple technologies.
2.4.2 Media and Technology Considerations
This section provides a series of Yes/No questions for each of the following technologies:
FACE TO FACE CONSIDERATIONS
1. Is live audio interaction between the presenter and participants necessary to achieve the module's learning objectives? YES NO If NO, go to
WBT Considerations
2. Is it necessary to have both live audio and visual interaction between the presenter and participants? YES NO If NO, go to Audioconference Considerations.
3. Is there budget allocated to cover the costs associated with the travel required for presenters and participants to attend the training event? YES NO If NO, go to IVT Considerations.
4. Is it possible for the presenter(s) to be at the same location at the same time for the training event? YES NO
5. Will the module be offered to a small number of participants? YES NO
6. Are all of the participants at or near the same location? YES NO
7. Does the module's content require the presenter to observe and certify the participants' ability to perform a specific action or task (e.g. firearms training)? YES NO If NO, go to IVT Considerations.
8. Do the module's learning objectives require the participant to learn to use equipment or tools that are located in one region or at only one site? YES NO If NO, go to IVT Considerations. IVT CONSIDERATIONS
1. Is live interaction between the presenter and participants necessary to achieve the module's learning objectives? YES NO If NO, go to WBT Considerations.
2. Is it necessary to have live audio interaction between the presenter and participants to achieve the module's learning objectives? YES NO
3. Is it necessary to have both live audio and visual interaction between the presenter and participants? YES NO If NO, go to Web Conferencing Considerations.
4. Is it sufficient for participants to be able to see the presenter live without the presenter being able to see the participants? YES NO If NO, go to VC Considerations.
5. Will the module be offered to a large number of participants? YES NO If NO, go to VC Considerations.
6. Are the participants at many different locations? YES NO
7. Is it sufficient to have participants interact live with each other through audio only or in small groups at local sites? YES NO If NO, go to VC Considerations.
8. Are the module's learning objectives primarily cognitive (fact-based knowledge)? YES NO If NO, go to VC Considerations.
9. Will the module's learning objectives be achieved through group, interactive instructional strategies such as Question and Answer sessions? YES NO
10. Will instructional strategies such as demonstrating, modeling behaviors, and illustrating concepts enhance the module's content? YES NO
11. Will the module materials include explanations with a high level of visualization? YES NO If NO, go to VC Considerations.
12. Will the module materials include the use of any of the following media?
- Videotape/animation
- CD-ROM
- Graphics of pictures, images, or concepts
- PowerPoint or Presentations slides
- Computer applications
YES NO
13. Is it necessary to have live access to experts during the module delivery? YES NO
14. If experts are not at the local site, is it sufficient for the presenter and participants to interact live with them through audio only? YES NO If NO, go to VC Considerations.
15. Is the content of the module time-sensitive or JIT (Just-In-Time) information? YES NO
16. Is it desirable to videotape the delivery of the module for later viewing? YES NO VIDEO CONFERENCING CONSIDERATIONS
1. Is live interaction between the presenter and participants necessary to achieve the module's learning objectives? YES NO If NO, go to WBT Considerations.
2. Is it necessary to have live audio interaction between the presenter and participants to achieve the module's learning objectives? YES NO YES NO If NO, go to WBT Considerations.
3. Is it necessary to have both live audio and visual interaction between the presenter and participants? YES NO If NO, go to Web Conferencing Considerations.
4. Do participants have access to a VC system on site or within easy local travel distance? YES NO If NO, go to Web Conferencing Considerations
5. Will the module be offered to a small number of sites? YES NO
6. Will the module be offered to a limited number of participants? YES NO
7. Is it necessary to have live access to experts during the module delivery? YES NO
8. Is it necessary to have live interaction-both visual and audio-with experts who may be at different locations? YES NO
9. Is it necessary for participants to have extensive, live interaction-both visual and audio-with other participants not at their location? YES NO
10. Will the module materials include any of the following media?
- Graphics of pictures, images, or concepts
- PowerPoint or Presentations slides
- Whiteboard
- Videotape
- Audio
- Internet/World Wide Web
YES NO
11. Is the content of the module time-sensitive or JIT (Just-In-Time) information YES NO WEB CONFERENCING CONSIDERATIONS
1. Is live interaction between the presenter and participants necessary to achieve the module's learning objectives? YES NO If NO, go to WBT Considerations.
2. Do participants have access to Web conferencing software at their desktop or at a local site? YES NO If NO, go to Audioconference Considerations.
3. Is it necessary to have live audio interaction between the presenter and participants to achieve the module's learning objectives? YES NO
4. Will the module be offered to a limited number of participants per session (less than 20)? YES NO
5. Is it necessary to have live access to experts during the module delivery? YES NO
6. Is it necessary to have live interaction with experts who may be at different locations? YES NO
7. Is it necessary for participants to have extensive, live interaction with other participants not at their location? YES NO
8. Is it necessary for participants to discuss the training content in breakout sessions? YES NO
9. Will the communication dialogue need to be captured and distributed? YES NO
10. Will the module materials include any of the following media?
- Graphics of pictures, images, or concepts
- PowerPoint or presentations slides
- Whiteboard
- Videotape
- Audio
- Internet/World Wide Web
YES NO
11. Will participants need to share software applications or collaborate on a shared document? YES NO
12. Will participants need to post ideas to a shared whiteboard? YES NO
13. Will it be helpful to poll or survey learners during instruction? YES NO
14. Is the content of the module time-sensitive or JIT (Just-In-Time) information? YES NO
15. Will it be necessary to add or modify content just before instruction or "on the fly"? YES NO WBT CONSIDERATIONS
1. Can this module be delivered without live instructor facilitation? YES NO
2. Can this module be delivered primarily as text with some supporting media such as flash animation, graphic images and minimal audio files? YES NO If No, return to IVT Considerations
3. Will the module be offered to a large number of participants? YES NO
4. Do the participants have access to the Internet or Intranet? YES NO If NO, go to CBT Considerations
5. Would participants benefit from the opportunity to discuss and exchange information with the presenter and other participants? YES NO If NO, go to CBT Considerations.
6. Is it necessary for course managers or administrators to capture administrative information and data from each participant, such as recording and reporting on participants' progress? YES NO
7. Do participants require the flexibility to progress at their own pace, opt out of material they are already familiar with, and revisit any material as a reference or job aid as they determine necessary? YES NO
8. Does the content of the module need to be delivered uniformly and consistently? YES NO
9. Is the current module or training material primarily print-based? YES NO
10. Is it necessary to update and change the content of the module frequently? YES NO
11. Can the module's learning objectives be achieved primarily through independent study? YES NO
12. Is it sufficient for the presenter and the participants to primarily interact asynchronously? YES NO
13. Is it necessary for participants to have access to training material or module content prior to the delivery of other modules, such as a course syllabus, outline, or handouts? YES NO
14. Do participants require easy access to any of the material as a job aid or refresher? YES NO
15. Is it necessary to provide participants with on-line support resources and information in addition to the content of the module? YES NO CBT CONSIDERATIONS
1. Can this module be delivered without live instructor facilitation? YES NO
2. Is the content of the module static, with a relatively long life span? YES NO
3. Are the module's learning objectives performance-based? YES NO
4. Can the module's learning objectives be achieved through independent study? YES NO
5. Can the module's learning objectives be achieved through task repetition, mastery, and practice? YES NO
6. Are participant responses required to be immediately evaluated and non-instructor feedback given for the participants to continue with the instruction? YES NO
7. Do participants require the flexibility to progress at their own pace, opt out of material they are already familiar with? YES NO
8. Do participants require easy access to any of the material as a job aid or refresher? YES NO
9. Is it desirable to provide participants with resources and information in addition to training? YES NO AUDIOCONFERENCE CONSIDERATIONS
1. Is live interaction between the presenter and participants necessary to achieve the module's learning objectives? YES NO If No, return to WBT Considerations
2. Is it sufficient for the presenter and participants to interact through audio only? YES NO If No, return to IVT Considerations
3. Does the module delivery require short-lead time with low costs? YES NO
4. Is it desirable to deliver the module using familiar technology? YES NO
5. Is it necessary to have live access to experts through audio only during the module delivery? YES NO
6. May module content include the clarification and/or elaboration of previously delivered module content? YES NO
7. Is module content time-sensitive or JIT (Just-In-Time) information? YES NO
8. Are the module materials print-based? YES NO
9. Is it necessary to update and change the module materials? YES NO
10. Is it desirable to record the delivery of the module for later listening? YES NO SELF-PACED / PRINT-BASED CONSIDERATIONS
1. Is it sufficient for the participants to complete this module on their own without instructor direction? YES NO If NO, go to IVT Considerations.
2. Is this module's content outside of required curriculum? YES NO
3. Is it sufficient for participants to complete this module at their own pace? YES NO
4. Is there a varied number of participants at different locations who would benefit from taking this module? YES NO
5. Is the content of the module highly personalized and not appropriate for group instruction? YES NO
6. Does the module need to be available where computer connectivity may not be present? YES NO
7. Is the content of the module suitable for a desk reference or job aide? YES NO
8. Will the module include materials that are primarily paper-based or in an electronic format that can be emailed, down-loaded and printed? YES NO
9. Will the module materials include any of the following support media?
- Videotape
- Pre-recorded audio
- Graphics of pictures, images, or concepts
- Internet/World Wide Web
YES NO 2.4.3 Summary Results
The Media and Technology Analysis Summary highlights decisions made in the tool and provides a framework for recommending technologies that are appropriate for delivering a specific course module. The section also warns against the temptation to select too many different delivery media and technology, which can lead to complexities in integrating multiple technologies, as well as a learning environment that is overwhelming and confusing to the learner.
Face-to-Face Summary Results Total YES answers to Face-to-Face Considerations Recommendation A total of 5 or more YES answers to Face-to-Face considerations questions indicates that Face-to-Face is likely an appropriate delivery technology for the module considered.
IVT Summary Results Total YES answers to IVT Considerations Recommendation A total of 10 or more YES answers to IVT considerations questions indicates that IVT is likely an appropriate delivery technology for the module considered.
VTC Summary Results Total YES answers to VTC Considerations Recommendation A total of 8 or more YES answers to VTC considerations questions indicates that VTC is likely an appropriate delivery technology for the module considered.
Web Conferencing Summary Results Total YES answers to WC Considerations Recommendation A total of 9 or more YES answers to WC considerations questions indicates that Web Conferencing is likely an appropriate delivery technology for the module considered.
WBT Summary Results Total YES answers to WBT Consideration Recommendation A total of 7 or more YES answers to WBT considerations questions indicates that WBT is likely an appropriate delivery technology for the module considered.
CBT Summary Results Total YES answers to CBT Considerations Recommendation A total of 6 or more YES answers to CBT considerations questions indicates that CBT is likely an appropriate delivery technology for the module considered.
Audioconference Summary Results Total YES answers to Audio conference Considerations Recommendation A total of 5 or more YES answers to Audio conference considerations questions indicates that Audio conference is likely an appropriate delivery technology for the module considered.
Self-Paced Print-Based Summary Results Total YES answers to Self-Paced Print-Based Considerations Recommendation A total of 4 or more YES answers to Self-Paced Print-Based Considerations questions indicates that Self-Paced Print-Based is likely an appropriate delivery technology for the module considered. 2.4.4 Additional References
In addition to the Training Delivery Strategy and the Media and Technology Selection tools, two additional reference documents will support users as they determine an appropriate delivery technology. The Distance Learning Technologies Matrix examines the characteristics, advantages, and disadvantages of each delivery technology. The Distance Learning Glossary identifies select terminology used in both tools. Users will have access to both documents as they complete the tools.
2.4.5 Conclusion
Upon completion of the DMAT, course content will be properly segmented into modules and lessons and topics with associated learning objectives. Media and technology considerations will have been identified with suggested delivery strategies.
Designers should now have the material they need to create a detailed course outline. As stated in the Blended Learning Strategy Document, this outline serves as the foundation for the complete course and guides the designer to organize content considering knowledge of the target audience, the course goals and objectives, and any resources materials, including information obtained from subject matter experts (SMEs). Objectives and activities are grouped and sequenced according to learning level hierarchy.
The analysis phase of the ADDIE model is complete and the Design phase begins to create the "blueprints" for the instructional experience.
3.0 Web-based Training Style Guide
3.1.1 Purpose
The purpose of this Style Guide is to provide the ODP Training Partners with recommended Web-based Training (WBT) courseware design and development guidelines. The Style Guide provides generally accepted WBT strategies and best practices for navigation, user interfaces, design and development processes and tools, instructional strategies, and courseware development and delivery.
3.1.2 Audience
The intended audience for this Style Guide is the ODP Training Partners. Specifically, those users include institute directors/deputies, project managers, content designers and developers, graphic artists, and programmers.
3.1.3 Structure
This Style Guide will serve as a reference document for ODP Training Partners when designing WBT courses. A brief description of each document section is listed below.
- Section 3.1, Blended Learning Strategy, introduces the Style Guide.
- Section 3.2, Overview of the Web-based Course Development Process, provides a high-level description of WBT.
- Section 3.3, Design Strategies and Best Practices, presents strategies and best practices that support the instructional design process.
- Section 3.4, Development Strategies and Best Practices, provides an overview of WBT design, development, and production standards.
- Section 3.5, Delivery Environment Strategies and Best Practices, addresses technical considerations pertaining to the delivery of Web-based content.
- Section 3.6, Process and Best Practices, presents the recommended course development process and corresponding templates.
- Appendices, provides supplemental document and storyboard templates, a glossary, and process checklists.
3.2 Overview of the Web-based Course Development Process
3.2.1 What is Web-based Training?
Web-based Training (WBT) is a form of computer-based training in which the training material is accessed via the World Wide Web (WWW). WBT presents live content in a structure that allows self-directed, self-paced instruction for any topic. WBT is media-rich training fully capable of evaluation, adaptation, and remediation, across various computer platforms. Typical media elements used in this environment include text, graphics, photographs, animations, video, and audio.
WBT is often a viable training solution offering:
- Flexibility and convenience - users can access training when they want and at their own pace
- Cross platform functionality - training can be accessed from various computer platforms
- Web-browsers and Web accessibility - most computers include either Microsoft's Internet Explorer or Netscape Navigator and have a direct connection to an Intranet/Internet
- Lower cost for distribution - training is delivered via the Web
- Availability - users dispersed over a large geographic area can complete the training without incurring any travel expenses
- Simplicity - changes and updates to training can be implemented from one location
3.2.2 The Instructional Systems Design Process
Instructional Systems Design (ISD) is the systematic process of translating principles of learning and instruction into plans for effective, efficient, and relevant instructional materials, activities, and programs. This process consists of interrelated phases that work together toward a defined goal and ensures that training is developed to meet measurable outcomes. The suggested tool for use in Analysis, Design, Development, Implementation, and Evaluation phases of WBT course development is the ADDIE model.
The ADDIE model provides a blend of various documented ISD course development processes. This model supports five phases of WBT course development. These include:
- Analysis
- Design
- Development
- Implementation
- Evaluation
3.2.2.1 Analysis
The Analysis phase serves as the baseline for the course development process. The main purpose of analysis is to perform a systematic exploration of the way learners are currently performing a process and the way that process should be performed. This difference is called a performance gap. Analysis helps ensure that training is the appropriate solution. The product of this phase is the foundation for all subsequent development activities. The following is defined in the Analysis phase:
- Who and what must be trained
- When the training will occur
- Where the training will take place
The following tasks are completed during the Analysis phase:
- Needs Assessment
- Task Analysis
- Definition of Objectives
- Course Detailed Outline
3.2.2.2 Design
After the Analysis phase is complete, the instructional designer creates the "blueprints" of the instructional experience. This is known as the Design phase. In this phase, the instructional designer plans the elements of instruction. Supporting documentation includes:
- Course Design Plan
- Storyboards
- Evaluation Plan
3.2.2.3 Development
When the Design phase is complete, development begins. Development is the production phase of the ADDIE process, where ideas and content generated during the Design phase are given life. It is during this phase that graphics are created, video is captured, photographs are taken, animations are created, and the course is programmed and assembled.
3.2.2.4 Implementation
The Implementation phase may also be called roll out or execution. In this phase, the course is tested to ensure it functions correctly in the client environment. The implementation team will validate functionality and identify any issues that must be resolved prior to full deployment of the course. Specifically, The Implementation phase involves:
- Testing the course in the client environment
- Preparing a plan for the coordination and management of the course
3.2.2.5 Evaluation
The Evaluation phase serves as the quality management component for training development and ensures that the course functions as it was designed through evaluations with end-users. Surveys and questionnaires are used to collect feedback and answer training related questions, such as:
- Did the learners achieve the objectives that were defined for the instruction?
- Did the learners like the instructional experience?
- Were the learners able to transfer what they learned in class out in the real world?
An additional evaluation can occur 6-12 months following training. This evaluation determines if there is a long-term return on investment from the instructional experience.
The answers to these important questions enable instructional designers to certify that learning has or has not occurred as result of the course they created.
3.3 Design Strategies and Best Practices
3.3.1 Overview
This section will present strategies and best practices that support the instructional design process.
3.3.2 Needs Analysis
A thorough needs analysis is critical to guaranteeing that the course material addresses the learners' needs. There are two components of a needs analysis. They are:
- Needs Assessment
Needs assessment involves analyzing the target audience, the learning and work environment, and completing the five-step process for determining a training need.
- Task Analysis
Task analysis involves breaking down and organizing the proposed learning material and determining objectives to meet the training needsThe ODP Training Strategy provides brief descriptions of a training need and needs assessment. "A training need is a need for human performance improvement that can best be met by training..." Needs assessment is the process of determining if there is a need to improve performance and, if so, in what area and to what extent. A critical element of the needs assessment is the identification of "the differences between the ideal and the actual characteristics of the targeted learner group."
Conducting a thorough needs assessment is critical to the development of a successful training product. A needs assessment is conducted in order to:
- Establish entry behaviors and performance levels of potential learners
- Agree on the desired outcomes, or terminal behaviors, expected of the participants following training
3.3.2.1.1 Audience Analysis
Conducting an audience analysis guarantees the instructional designer develops appropriate learning material for the WBT. The instructional designer needs to determine the significant characteristics of the target audience. These characteristics may include but are not limited to:
- Reading and education level
- Computer and Web skills
- Experience with the proposed training material
- Job descriptions
3.3.2.1.2 Environmental Analysis
An environmental analysis provides the instructional designer with insight into the target audience's learning and working environments. Conducting this sort of analysis enables the instructional designer to create a WBT course that will function correctly and accommodate the target audience's needs in the intended learning environment as well as reflect their work environment.
3.3.2.1.3 Five-step Process for Determining a Need
The standard model for conducting a thorough needs assessment involves following a five-step process. Adhering to this process is critical to creating accurate, relevant, and appropriate content for ODP courses.
The process steps are listed below :
- List the course goals. During this step, the instructional designer determines what learners must be able to do or know when they complete the course. The instructional designer determines how well the identified goals are being achieved by interviewing and observing people who are experts in performing the skills or having managers describe the type of knowledge or skill they wish their staff to possess.
- Determine how well the identified goals are already being achieved. The instructional designer determines how well the identified goals are being achieved by observing people on the job, conducting assessments, or evaluating job performance. A combination of these techniques provides the most reliable estimate of the current level of performance of instructional goals.
- Determine the gaps between "what is" and "what should be." In order to determine if there are learning gaps, the instructional designer identifies the gap between what learners should be able to do and what they are currently able to do for each identified goal.
- Prioritize gaps according to agreed-upon criteria. The instructional designer uses the following criteria to assist with prioritizing learning gaps:
- Gap size (addressing the biggest gap first)
- Importance of the goal (working on the most critical goals first)
- Number of learners affected (choosing those gaps that affect the most learners first)
- Consequences of not meeting the goal (selecting those gaps having the most serious consequences if not closed first)
- Probability of reducing the learning gap (attending to the gaps that have the greatest probability of being closed with the available resources first)
- Determine which gaps are instructional needs and which are most appropriate for design and development of instruction. The instructional designer needs to be careful not to assume that a poor performance in learning tasks implies a need to correct or revise training. For example, a high rate of learner absenteeism or insufficient equipment for learners to perform their job may be the cause of poor work performance.
There are numerous tools that the instructional designer may use to accomplish the five-step process mentioned above and to ensure an accurate and complete needs assessment. Chapter 3 - Assessment of Training Needs to Determine Curricular Content, of the ODP Training Strategy identifies several tools and/or approaches for gathering requisite information from and about the target audience and proposed learning material for a given course. These additional tools and/or approaches, as adopted from Kern et al. and Finch & Crunkilton , include:
- Informal Discussions
- Formal Interviews
- Focus Groups
- Questionnaires
- Direct Observation
- Proficiency Tests
One of the most reliable and accurate tools used during a needs analysis is the task analysis. The task analysis breaks down and organizes the proposed learning material, resulting in a list of measurable objectives describing what learners should know or be able to do upon completion of a given course. The objectives determine the skills, knowledge, and situations that must be developed and provide a framework for the content. The task analysis also assists the instructional designer with identifying any prerequisite skills or knowledge learners must possess prior to beginning a course to achieve the stated objectives.
Typical steps of a task analysis include determining prerequisites, identifying job functions, identifying tasks within each function, and determining the task as procedural, process or principle based. The outcome of this process is the identification of the skills and knowledge that will structure the training program. Figure 6 (below) provides an illustration of a sample task analysis.
Figure 6. Sample Task Analysis
The same tools the instructional designer uses during the needs assessment may also be used during a task analysis. These tools enable the instructional designer to capture the information required to ensure the development of a detailed task analysis.
3.3.3 Web-based Training Objectives
Objectives are defined during needs analysis when the instructional designer conducts the task analysis. Specifically, objectives are detailed statements of what the learners will be able to achieve or be able to demonstrate as a result of completing a course. Thus, objectives are statements of learner behavior. They describe the result of the learning process rather than what or how the learner will be taught.
Well-defined objectives:
- Enable learners to know what is expected from them
- Enable learners to see whether or not the instruction is effective
- Provide guidelines for evaluation by specifying desired outcomes that are assessable
- Stress the behavioral changes expected rather than attitudes or insights that cannot be measured
The instructional designer must be careful to use objectives appropriately. Objectives are not a description of:
- Learning materials content
- What the instructor says or does
- A specific instructional experience
WBT objectives differ from standard objectives because they must be measurable by a course accessed via the Web. A course distributed over the Web cannot measure whether a learner "knows" or "understands" how to perform a specific task or "define" a term. WBT courses can enable learners to "identify" the proper steps to complete a task or a correct definition.
For a more thorough review of objectives, refer to Chapter 4 - Curriculum Development and Revision, of the ODP Training Strategy, this chapter also discusses the Taxonomy of Educational Objectives, i.e. Blooms' Taxonomy.
3.3.3.1 Terminal and Enabling Learning Objectives
Objectives are often categorized according to the hierarchical level of the skills, behaviors, or tasks identified during the needs analysis. There are two commonly used levels of objectives, Terminal Learning Objectives (TLOs) and Enabling Learning Objectives (ELOs).
- TLOs - TLOs are objectives that correspond to the overall instructional goals of the course. TLOs describe what learners will be able to do at the end of the overall instructional course.
- ELOs - ELOs, also known as subordinate objectives, correspond to the skills that are required to accomplish the TLO. Specifically, they define the skills, knowledge, or behaviors that learners must master to successfully achieve the TLO.
3.3.3.2 Writing Web-based Objectives
Effective Web-based objectives are critical to the success of any WBT course because they ensure learners are mastering the training material in a non-traditional learning environment. The objectives must be measurable via the Web. Effective objectives, regardless of training delivery, are comprised of the following three parts:
- Performance - Performance expresses what the learner will do, usually as an action
- Condition - Condition expresses what learners are given, such as tools or resources
- Criterion - Criterion expresses how well learners must perform
3.3.3.2.1 The Performance
The performance component of an objective is a statement of what the learner will be able to do. This component should be generated from the needs analysis and must include a clearly identifiable behavior. The objective must include a verb that is measurable over the Web and not open to a range of interpretations (e.g., it is impossible to measure or test whether a learner is aware of something or fully understands a concept via the Web). However, it is possible to test whether a learner is capable of selecting/identifying the correct concepts, list items, or labels. The objective must state specifically how learners will demonstrate what they know, understand, appreciate, etc.
Choose a measurable verb when describing what learners will be able to accomplish. For example, "The learner will be able to match each Emergency Level to its corresponding description."
3.3.3.2.2 The Condition
When the performance component is clearly defined, it may be necessary to state the important conditions under which the performance is to occur. The condition refers to the situation or scenario under which the learner will perform the behavior or performance. The condition can be defined by asking the following questions:
- What cues or conditions will learners need to accomplish the behavior?
- What resource material will learners need when performing the behavior?
- What will learners not be allowed to use?
- Will learners be given any special equipment or assistance?
For example, "Using the drop-down lists provided, the learner will be able to match each Emergency Level to its corresponding description."
3.3.3.2.3 The Criterion
After informing learners what they will be able to do (the performance) and what they can or cannot use to do it (the condition), they should be informed of how well to do it. The criterion is the degree or criteria component of the objective that describes to what degree learners will be evaluated. It is the standard learners should achieve. The criterion describes what is acceptable and also sets the criterion against which to assess the learners' performance (e.g., how to judge the outcome of the behavior or performance). It is often stated in terms of the limits within which a behavior must fall. When writing the criterion component, ask the following question: "What level of mastery is required for learners to demonstrate mastery of the task?"
For example, "Using the drop-down lists provided, the learner will be able to match each Emergency Level to its corresponding description with 75% accuracy."
3.3.3.2.4 Examples
Examples and non-examples of WBT objectives are provided in Table 3 (below). Each objective in the "Examples" column contains the three components of a good objective and most importantly is measurable over the Web. The objectives in the "Non-example" column are incomplete objectives and are not measurable over the Web.
Table 3. Examples and Non-examples of WBT Objectives
Examples Non-examples Given a list of potential crime scene protection procedures, first responders will be able to identify the proper steps required to protect a crime scene with 100% accuracy. First responders will know how to properly protect a crime scene. Given several different safety measures, public works employees will select those safety measures they need to employ in the event of a WMD incident with 100% accuracy. Public works employees will understand how to employ proper safety measures in the event of a WMD incident. Given a real life scenario, fire service personnel will be able to identify the proper steps involved with isolating and successfully terminating hazardous incidents with 100% accuracy. Fire service personnel will isolate hazardous incidents. 3.3.4 Instructional Strategies
Instructional strategies provide an approach to organizing and presenting information. This section of the ODP Style Guide discusses the WBT Learning Taxonomy, Course Structure, and recommended WBT Design Strategies.
3.3.4.1 Web-based Training Learning Taxonomy
The WBT learning taxonomy serves as the foundation for the instructional strategies. It establishes a relationship between different course components, which may include modules, lessons, units, or topics. These components correspond directly to the course learning objectives. The WBT learning taxonomy for ODP courses has three components:
- Course - associated with one or more terminal learning objectives or TLO
- Modules - associated with one or more enabling learning objectives or ELO
- Topics - associated with one ELO
Figure 7 (below) provides an illustration of this relationship.
Figure 7. WBT Learning Taxonomy
ODP courses will consist of modules, each containing a group of topics. Each topic is equivalent to one ELO. Module size is dependent upon the logical grouping of ELOs and the time required to complete a module.
3.3.4.2 Course Structure
Establishing a logical course structure is an important instructional strategy. Course content needs be organized to ensure learners are able to complete the course without frustration or confusion. The recommended structure for all ODP courses is to organize content into Modules and Topics.
3.3.4.2.1 Modules/Topics
The course should be organized into modules each of which are composed of separate topics. Each topic should have a stated objective and corresponding instructional content.
Each module of the WBT will include:
- Module Introduction and Objectives
These screens should include the module title, purpose and objectives.- Topics (content and practice)
Each topic should be based on one ELO with a Practice Exercise every 3-5 screens.- Module Summary
This screen should provide a summary of module topics and provide a transition to the next module.- Module Assessment
This is a tool used to measure learners' mastery level of the material presented and should consist of questions related to the ELOs with a minimum of one question per ELO.3.3.4.2.2 Suggested Course Progression
Described and illustrated below is a recommended course progression for ODP courses. This recommendation is based on industry best practices. When learners log on and begin an ODP course, they should be taken immediately to the Course Introduction. The course introduction should include the course purpose, overview, a description of the intended audience, and navigation instructions. Afterward, the learners can select modules sequentially to proceed through the course. After selecting a module, learners should complete all corresponding topics. Each topic should include content screens comprised of content specific and appropriate text, graphics, animations, pop-up information, Practice Exercises, and access to additional information (when necessary).
At the end of each module, a Module Summary screen should be presented. This screen should summarize topics contained within the module. After the Module Summary screen, learners should complete the Module Assessment. The Module Assessment is used to measure learners' mastery level of the material presented. When the learners complete all of the modules and corresponding assessments, they should return to the appropriate training portal.
Figure 8 (below) depicts a comprehensive flow through a standard ODP Web-based course:
Figure 8. Course Progression
3.3.4.3 Web-based Training Design Strategies
The requirements for effective WBT are different from the requirements for effective classroom training, Video Teletraining (VTT), or other types of instruction. The following strategies should be developed to ensure effective learner interaction within ODP courses:
- Course Design - Describes the overall instructional approach for the course. The strategy includes using extended metaphors, scenarios, and supplemental materials.
- Course Assessment - Describes the approach for Practice and Assessment items. Refer to section 3.3.5.3 (page 62) for more information concerning learning assessments.
- Module Design Strategies - Describes the strategy for developing individual modules, to include module descriptions, module objectives, instructional strategies, module assessment strategies, and content outline.
The instructional strategy, as part of the module design strategy, defines how learning is presented through text and graphics. Table 4 (below) describes the strategy examples for text and graphics presentation.
Table 4. Instructional Strategy Examples
Strategy Definition Complex Animation A complex animation is one that attempts to depict a process. Complex animations require much more resources than simple animations. Examples include Virtual Reality or 3-D animations. Graphic Graphic refers to a static visual element. In a WBT course, both photographs and original artwork are used as graphics. In some cases, text is treated as a graphic, (i.e., a visual image rather than words). This happens most often with calculations or lists. Graphic Build A graphic build is a series of graphics shown in sequence or as overlays. Label A label is text included as part of a graphic or animation. It does not require an action by the learner in order to be displayed. Pop-Up Text Pop-up text appears in the graphic area of the screen on a cue given by the learner (either clicking on an item or rolling the mouse over an item). Simple Animation A simple animation is similar to a complex animation, but generally shorter in length and easier to produce (e.g., 2-D rather than 3-D animation). Text Build A text build is a series of text items shown in sequence. Text builds add text until the instruction is complete. Text builds never replace one set of text with another on a single screen. 3.3.5 Interactivity
Interactivity is a powerful tool used to support content and engage learners. The level of interactivity in a course corresponds directly to the type of material that is presented and the degree of learner involvement required for a course. Table 5 (below) illustrates the four levels of interactivity strategies and their corresponding descriptions.
Table 5. Interactivity Strategy Levels
Level Description Level I Passive The learner acts solely as a receiver of information. The learner is required to read the text on the screen, view graphics, illustrations, charts, and use the navigational buttons to progress forward through the program or move back. An example of this type of WBT product may also contain pop-ups and hyperlinks to Websites, materials, and other information interspersed between the text and graphic presentations Level II Limited Interaction The learner makes simple responses to instructional cues. The WBT product includes learning activities listed in Level I as well as multiple choice and column matching related to the text and graphic presentation. A good example is a WBT product that includes these types of test items at the end of a unit of instruction to test the learner's grasp of the information. Level III Complex Participation The learner makes a variety of responses using varied techniques in response to instructional cues. The responses may include those listed for a Level II -Limited Interaction as well as text entry boxes and manipulation of graphic objects to test assessment of the information presented. A good example of this type of WBT product is desktop software training requiring the learner to perform as though using the program Level IV Real-time Participation The learner is directly involved in a life-like set of complex cues and responses. This involves engaging the learner in a simulation that mirrors the work situation with stimuli-and-response coordinated to the actual environment. An example of this type of WBT product is using artificial intelligence similar to computer games and flight simulators. 3.3.5.1 Objective and Interactivity Level
Table 6 (below) represents the six levels of objectives with their corresponding levels of interactivity. Interactivity strategies should be selected based on the level of objectives. Objective levels are determined by the following criteria:
- Type of learning (e.g., knowledge, skill, or attitude)
- Level of learning (e.g., fact, concept, process, procedure, or principle)
Table 6. Level of Objective and Recommended Interactivity Level
Level of Objective Level of Interactivity Knowledge Level I & Level II Comprehension Level II Application Level II Analysis Level III Synthesis Level III Evaluation Level IV 3.3.5.2 Guidelines for Interactivity
When an interactivity strategy level is selected, the following guidelines should be used to implement the requisite level of interactivity for all ODP courses.
Table 7. Guidelines for Interactivity
Guideline Description Explanation Provide opportunities for interaction at least every three or five screens. However, mandatory interaction with the computer should not be superficial. Without interaction, the course risks becoming an electronic page-turner. However, if an action required is unnecessary, the learner could be distracted by it and become frustrated. Learners prefer not to have unnecessary interactions. Group the content into small segments and build in questions (with feedback), periodic reviews, and summaries for each segment. Grouping content into smaller units and providing opportunities for interaction (e.g., questions) within each information segment allows learners to interact with the program more frequently. Ask as many relevant questions as possible without interrupting the continuity of the instructional flow. Questions should provide immediate feedback to learners regarding their own performance, and sustain learner attention by keeping them mentally engaged in the learning process. Ask questions at the application level rather than at the memory level. Application questions enhance attention and comprehension and facilitate transfer of learning. Use rhetorical questions during WBT to get learners to think about the content or to stimulate their curiosity. Also use them as a natural transition between frames. A rhetorical question is a question that does not require learners to overtly provide an answer. It invites learners to mentally interact with the content. Used as a transition aid, it can direct learners' attention to what is coming up next. Consider designs where the learner is not presented with information in a linear format, but rather discovers information through active exploration in the program. Developers should consider structuring a course for learner exploration without a predetermined "order" to the learner's selection of modules and topics. This approach adds variety, challenges learners, and maintains their interest. 3.3.5.3 Costs Associated with the Different Interactivity Levels
There are four levels of interactivity. Description of each level can be found in Table 5 (page 56). Factors that affect the cost of interactivity are listed below:
- Desired amount of interactions by learners
- Complexity of graphics
- Use of audio and video
- Number of frames per module
- Special features (e.g., exploratory branching)
Additionally, these factors affect each phase of the courseware development process (Analysis, Design, Development, Implementation, and Evaluation), which in turn affect the costs per interactivity level.
Development hours are directly proportional to the complexity of interactivity. Generally, the industry considers that for a Level I course, the number of development hours per one hour of seat time is in the range of two hundred (200) to three hundred (300) hours. This gradually increases such that Level IV is in the range of five hundred (500) to two thousand (2000) hours.
The table below represents the industry standards as documented by the American Society for Training and Development.
Table 8. Cost Per Interactivity Level
Level Cost Level I $10,000-$20,000 Level II $25,000-$35,000 Level III $40,000-$50,000 Level IV $55,000-$70,000 3.3.6 Learning Assessments
Learning assessments measure the competence or capability of learners when determining whether or not they have met the course objectives. This section discusses writing assessment questions, assessment strategies, and types of learning assessments.
3.3.6.1 Writing Assessment Questions
Learning assessments should enable learners to demonstrate what they can do or what they know. When determining each assessment item, the instructional designer should ask the following questions:
- Is it relevant to the stated objectives?
- Is the wording/structure clear?
- Is each objective tested twice (during and at the end of the module)?
To accurately measure learners' mastery of the training content, a minimum of one assessment question per objective - more complex content requires that multiple questions should be asked. The level of objective measured determines the type of question asked. For example, use scenarios for analysis level objectives.
3.3.6.2 Assessment Strategy
The ODP assessment strategy ensures that learners master the objectives, defined during Needs Assessment, for each course.
3.3.6.2.1 Pre- and Post-tests
Pre- and Post-tests are valuable assessment tools used to measure learners' knowledge pertaining to a specific course.
- Pre-tests - Pre-tests are used to measure learners' entry knowledge and provide them with a sense of what they will need to know upon completion of training. They also provide learners the opportunity to "test out" of specific modules within a course.
- Post-tests - Post-tests are completed upon completion of an entire course and are used to measure whether learning actually occurred.
3.3.6.2.2 Number of Tries to Answer a Question Correctly
There are several options available to instructional designers when determining the number of tries learners should be allotted to answer a given question correctly. The option chosen is often dependent upon the type of question/exercise (e.g., Practice or Module Assessment) presented. These options include, but are not limited to:
- Unlimited number of tries with immediate feedback
- Limited number of tries with immediate feedback
- Limited number of tries with remediation
3.3.6.2.3 Learner Mastery Level
The recommended level of mastery for all ODP courses is a score of 75% on the post-test.
3.3.6.3 Types of Learning Assessments
The following types of learning assessment interactions will be used in ODP courses:
- Practice Exercises (embedded questions)
- Module Assessments
3.3.6.3.1 Practices
Practices are a type of interaction used to ensure learners comprehend the training content. Practice Exercises are presented within a topic (typically once every three to five screens) depending on the requirements of the related learning objective. There will be at least one practice question for each ELO.
Learners are presented with one opportunity to complete a Practice Exercise. After learners complete the Practice Exercise, they are provided with immediate feedback informing them of the accuracy of their response. For correct responses, learners will be told that their response is correct, and given the reason why. For incorrect responses, learners will be told that their response is incorrect and given the correct response and a reason why.
3.3.6.3.2 Module Assessments
A Module Assessment will be provided at the end of each module. At least one assessment question will be provided for each ELO. In addition, assessment questions will be used to test learners at the TLO level. Assessment questions may use any of the four standard question types described in the next section.
There are two methods to structure a Module Assessment. The first will require learners to complete all questions for the module in a specific order. Learners will have one attempt to complete a question and will receive immediate feedback informing them of the accuracy of their response. For correct responses, learners will be told that their response is correct, and given the reason why. For incorrect responses, learners will be told that their response is incorrect and given the correct response and a reason why.
The second method to structure a Module Assessment is to incorporate remediation into the assessment sequence. In this structure, learners complete all assessment questions in a specific order and are notified of the accuracy of their responses on a Module Summary screen. The summary screen will contain the number of correct and incorrect answers. For every incorrect response listed, learners will be prompted to select each to receive remediation for the corresponding missed question. After they complete remediation, learners will be presented with the original question they missed and given the opportunity to answer it again. After they have responded to the question, they are returned to the Module Summary screen. Learners will continue with this process until they have responded correctly to at least 75% of the questions.
3.3.6.3.3 Guidelines for Standard Types of Practice Exercises/Module Assessments
This section provides guidelines for Practice Exercises/Module Assessment questions. The following types of questions are recommended for use in Practice Exercises and Assessment questions:
- Multiple choice
- Multiple-multiple choice
- Matching
- Case Study
3.3.6.3.4 Multiple Choice Questions
Multiple choice items are used for the recall of facts and the application of rules and procedures. Use multiple choice items to check discriminations and problem solving. A multiple choice item consists of:
- One stem
- One correct response
- Three distracters (incorrect responses)
Use the following guidelines for multiple choice items:
- For each item, develop one correct answer and at least three realistic distracters (a, b, c, d).
- Make the language of the correct answer and the distracters brief, similar in grammatical structure and word length, and worded in positive language.
- Avoid using negatives in any part of the multiple choice item (e.g., no, nor, not, none).
- Vary the correct answer position to avoid establishing an answer position pattern (e.g., "c" is generally correct).
- Do not use inclusive or exclusive answers or distracters (e.g., "All of the above", "None of the above", "Both a and b".)
- If responses are numerical, list them in ascending order.
- Capitalize the first word at the beginning of each answer and distracter when the stem is a question.
- Do not repeat initial words in the responses. Load repetitive initial words into the stem (e.g., if "the" is the first word in each response, move it to the stem).
3.3.6.3.5 Multiple-Multiple Choice Questions
Use multiple-multiple choice items when more than one correct response is possible. A broad scope of information can be covered within a single item of this type. This type of item has greater complexity and difficulty than simple multiple choice items. Use the following standards to develop multiple-multiple choice items:
- Group all items of this type together on a Practice Exercise/Module Assessment, so directions can be given only once for several items.
- Provide clear direction for choosing the correct answer(s).
- Use singular/plural verbs in the stem to prevent grammatical cues for the correct response.
- Use four or five responses.
3.3.6.3.6 Matching
Use matching items to measure a learner's ability to recall facts and label parts in pictures, schematics, diagrams, or drawings and for matching terms with their correct definition. Drop-down lists provide a solution for incorporating matching items while adhering to Section 508 Accessibility Requirements.
Use the following guidelines for labeling assessment items:
- Make all graphics and relevant components clear and sized appropriately to accommodate the available screen space.
- Provide sufficient information to indicate what the graphic item is and which part is to be labeled.
- Specify clearly the parts to be labeled or identified.
- Provide one definitive answer.
3.3.6.3.7 Case Studies
Use case studies to present real-life scenarios with follow-up question(s) pertaining to the scenario. The follow-up questions can be multiple choice or multiple-multiple choice. This presentation/question format allows learners to use their skills by analyzing a situation and applying knowledge that they have previously gained in the module.
Use the following guidelines for case study items:
- Present a real-life scenario that is applicable to the information previously presented and now being tested. Avoid uncommon or unrealistic scenarios, as they will present a distraction to the learners.
- Ensure all follow-up questions relate to the case study presented and are in accordance with all previously defined question standards.
3.4 Development Strategies and Best Practices
3.4.1 Overview
This section will present strategies and best practices that support the Web-based course development process. Specifically, this section will discuss course User Interface Design Specifications, General Screen Development, Media Types, Digital Audio, Digital Video, Programming, and Accessibility Compliance (Section 508).
3.4.2 Course User Interface Design Specifications
This section describes the recommendations for screen design, interface specifications, and course control elements, indicators, content layout, and content graphics.
3.4.2.1 Screen Design
The following interface templates illustrate the best practices and recommendations for screen design.
3.4.2.1.1 Course Main Menu
A course Main Menu is essentially a list of all the sections of a course. The Main Menu can always be returned to from inside the course and is a starting point from which the learner navigates through the course. Visual cues may be used to track progress through the course. The following sample provides a visual representation of the learner's progress. The main menu screen contains a legend to explain the visual cues.
Figure 9. Course Main Menu - SAMPLE
3.4.2.1.2 Topic Menu
From the Main Menu, learners can select a module. Learners are then presented with a Module Introduction screen from which they can select a topic. The following two figures provide examples of a closed (Figure 10, below) and an open topic drop-down menu (Figure 11, below).
Figure 10. Topic Menu (Closed) - SAMPLE
Figure 11. Topic Menu (Open) - SAMPLE
3.4.2.2 Interface Specifications
Table 9 (below) identifies the characteristics of the HTML and graphical elements of the interface (See Figure 12, page 73, for a diagram describing the location of the interface elements). All finished graphics should be saved as 16-bit compressed .jpg /.gif files. Table 9 also contains recommendations and best practices for interface elements.
Table 9. Interface Specifications
Interface Item Description Global Interface All screens will use a standard background and top bar banner layout, logo and course title. The global interface will display as:
- Page background color: bin hex code
Global Buttons These buttons will be placed in the top right portion of the course window layout.
The global buttons will display as three possible states, Normal, Selected (highlight) or disabled (grayed out).
Global buttons will be displayed as:
- .jpg/.gif graphics
- Button names may be overlaid as part of the graphic
Breadcrumb title The breadcrumb bar should display the specific learners' location within the course.
Breadcrumb bar displays as:
- HTML text
- Arial or Verdana font
- Left-justified
Screen Title The title bar contains the screen title. The screen title is displayed as:
- HTML text
- Arial or Verdana
- Left-justified
Content Body Text The body text consists of the main text area content for any given screen. This text displays as:
- HTML text
- Arial or Verdana 12-14 pt. usually black font
- Left-justified
Content Hyperlinks Hyperlinks should be used to link to pop-up content within the course, and should include indications of the following states:
- Normal
- Visited
- Rollover
Content Bullets Bullets are used to separate, organize and chunk information.
Bullet text is displayed as:Bullet Shape can be customized according to course needs.
- HTML text
- Arial or Verdana
Page Indicator
(i.e., page X of X)The page indicator is displayed as:
- HTML text
- Arial or Verdana
- Left-justified
Course Content Navigation Buttons These button controls "Back" and "Next" will have three states: The buttons will display as:
- Normal
- Selected
- Disabled
- .jpg /.gif graphics
- Futura Md Bold 8 pt. font overlaid as part of the graphic
Prompt Text Prompt text is instructional text informing learners of how to proceed. Prompt text displays as:
- HTML text
- Arial or Verdana font (bold font when referring to a button),
- Left-justified
Content Specific Buttons Content-specific buttons can have three to four states: Normal, Highlighted, Selected/Down, and Disabled. Content-specific buttons may include any or all of the following:
- More
- Scenarios
- Questions
- Print (Certification)
Text window pop-ups The storyboard design may include references to linked text within the course content area that will display in a window specific to that word or phrase, this is referred to as a "pop-up".
The pop-up link will display as:The pop-up text box will display as:
- Underlined HTML text (hyperlink)
- HTML text
- Window will be sized appropriate to amount of text content and will be color coordinated to match the course look and feel
System alert boxes System alert boxes will follow the standard MS Windows layout including a close window "X" control. An example of this window would be if the user selects a module from the main menu that they have not been through and is out of sequential order. This box may be color coordinated to be similar to pop-up windows.
