Development – Testing Online Learning Materials


Goal: Examination of some methods and the importance of testing online learning resources.

An important part of the development process is testing what has been created. These tests include the steps taken to ensure that all of the objectives in the class are addressed, ensuring that the materials and activities created to address objectives actually do address the intended objectives and have a reasonable chance of making sure that student learn, and the tests needed to ensure that the quality and functionality of the delivery mechanisms are sufficient to ensure that students can reliably access the learning materials as they are intended to be used. – Brian Newberry

1. List the two levels of testing as outlined in the presentation and discuss each one in turn. Then describe the types of testing activities for each level of testing.

In the Session Eight lecture, Professor Newberry (2015) notes that online learning materials must be tested for quality assurance. Testing must be done to ensure both the fidelity of the media and to ensure the instructional quality. Simply put, testing the media ensures that the materials are 1) well produced and user friendly (e.g. that they work properly) and that they 2) lead to the desired student outcomes.

  • Ensuring Fidelity of Learning Materials – Fidelity testing addresses the technical aspects of the media. The first level of fidelity testing is done by the media creator, who should make sure the media is of a high quality, complete, and generally functional pre-release. However, since the creator’s desk and computer aren’t the “real-world” setting in which the media will be used, it’s important that additional levels of fidelity testing take place before release. Newberry (2015) states that outside reviewers should be enlisted. These reviewers should approach the material as an end-user and looks at things such as links, navigation, coding, ADA compliance, bandwidth, and operating system/browser compatibility. Some of these things can be tested with web-based checkers and most CMS’s have built-in checkers as well. In ETEC 674 we examined and used several reliable online checkers such as The WAVE Accessibility Tool and W3C Markup Validation Service. These checkers are easy to use and often free. However, nothing beats live testers with different types of computers and devices, in different locations, using the media in a variety of ways, to ensure the fidelity of learning materials. And, as hard as one might try to avoid post-release errors, once the media is released the students themselves will serve as testers. It’s not uncommon for students to point out technical issues, problems, and errors as the course progresses.
  • Ensuring the Instructional Quality of Learning Materials – Content testing addresses the instructional quality of the media. As Newberry (2015) notes, this is a much more complicated aspect of testing, as it examines how well the materials result in the desired student outcomes. If the media creator is not the instructor/subject expert, the creator must, at the very least, understand the purpose of the lesson. This will involve working closely with the instructor/subject expert. Transcripts, outlines, ideas, and revisions will come from these experts. The media creator will package them into a usable learning object that accomplishes the objective, working with the experts all along the way. Instructors and students should be asked to actively test the learning materials prior to release to see if the outcomes are met. In our text, Horton (2012, pg.49) said that “Each objective leads us to create a learning object that completely accomplishes the learning objective and can prove it.” Testing the learning materials is a way to prove it.

2. What is ADA and how does it apply to the design and development of eLearning materials?

The Americans with Disabilities Act of 1990 “prohibits discrimination and guarantees that people with disabilities have the same opportunities as everyone else to participate in the mainstream of American life – to enjoy employment opportunities, to purchase goods and services, and to participate in State and local government programs and services.” In short, this important civil rights legislation makes it illegal for organizations to discriminate against individuals with disabilities. Section 504 of that act requires that school districts provide a free appropriate public education to each qualified student with a disability who is in the school district’s jurisdiction, regardless of the nature or severity of the disability. Section 508 of that act requires that all government agencies, public schools and universities, provide accessible technology and web content.

In ETEC 501 and ETEC 674 we discussed ADA and accessibility in detail. I learned there are four main categories of disability that may require special accommodation in an e-learning environment:

  • Visual disabilities such as blindness, low vison, color blindness
  • Hearing disabilities such as deafness and hard of hearing
  • Mobility disabilities such as the inability to use hands, hand tremors, slow muscular movement, immobility
  • Learning disabilities caused by reading disorders, attention deficit disorders, etc. (Waterhouse, 2005, p. 172)

However, in an online environment, it is not uncommon for instructors to be unaware of their students’ disabilities. That’s why it’s important to keep ADA in mind when creating course content. In the introduction to ETEC 674 Session Four, Professor Newberry noted that “The American’s with Disabilities Act (ADA) provides directives for how to properly accommodate the needs of individuals with various disabilities. This includes things as diverse as curb cuts and online materials. As developers of or instructors of online courses we need to be aware of these requirements, and how to achieve them.” To that end, the ADA website provides not only the law and regulations but also design standards and technical assistance materials. As instructional designers in eLearning we should all become very familiar with this resource.

Some examples of accommodations that can be made in eLearning materials include:

Blind students

  • Make sure LMS and other course content is compatible with the reading device and provides suitable alternatives to mouse clicks, long index columns, frames, etc.
  • Make time accommodations in the LMS for exams and other assignments to make up for the delay in the screen reader processing time.
  • Descriptive alt tags should be applied to all images.
  • Column and rows in tables should be labelled so they are machine readable.
  • Create audio files as alternates to text files.

Deaf students

  • Add captions to all videos and audio files.
  • Create text files as alternates to audio files.
  • Add signing to videos if appropriate.
  • Use highlights, underlines, and other textual indicators to provide emphasis on important words and phrases.

Note: While investigating ADA for ETEC 501, I learned that ADA, accessibility, Universal Design, and Universal Design for Learning are inextricably linked in e-learning. As educators we must keep in mind all abilities when creating courses and content. Today, accommodating the online needs of individuals with disabilities is part of larger movements called universal design and universal design for learning which emphasize the need for material to anticipate the diversity of all students, with or without disabilities. The goal of Universal Design in the academic setting is to design instruction, learning content, and learning activities that are usable by all. In this way learning objectives will be achievable by student with different abilities, language skills, or learning styles. There are many websites and guidelines to help instructors insure that their course is accessible to all and that it adheres to the principles of Universal Design and Universal Design for Learning. I’ll re-post this useful information here.

Universal Design
According to The Center for Universal Design, “The intent of universal design (UD) is to simplify life for everyone by making products, communications, and the built environment more usable by as many people as possible at little or no extra cost. Universal design benefits people of all ages and abilities. It’s the design of products and environments to be usable by all people, to the greatest extent possible, without the need for adaptation or specialized design.” (Center, 2008)

Universal Design for Learning
One aspect of UD that is of particular importance to educators is Universal Design for Learning (UDL). According to the educational research and development organization CAST, “Universal Design for Learning is a set of principles for curriculum development that give all individuals equal opportunities to learn. UDL provides a blueprint for creating instructional goals, methods, materials, and assessments that work for everyone – not a single, one-size-fits-all solution but rather flexible approaches that can be customized and adjusted for individual needs.” (CAST, 2013)

The following websites provide valuable information on UDL:

3. What is your institutions (You may use CSUSB’s) policy towards ADA and eLearning? Explain what this means in practical terms and what you think the strengths and weaknesses of the policy.

Disabled Student Services at Cerritos College
At Cerritos College, Disabled Students Programs and Services (DSPS), oversees disabled students and services. They assist all disabled students (on-campus and distance) in all courses (traditional, hybrid, online). I must admit I had never really examined their website before this class assignment, although I’ve referred students to their office on several occasions. I’m pleased to say I was very impressed with the depth and breadth of their program. They offer a wide variety of services and assistive technologies including:

  • Notetakers
  • Materials in alternate format
  • Sign language interpreters
  • Real time captionists
  • Assistive listening devices
  • Kurzweil reading machine
  • Priority enrollment
  • Assistive technology training

The DSPS webpage includes an Academic Accommodations for Disabled Students Policy as well as a Section 504/ADA Complaint Policy, both of which are informational and thorough.

They also have valuable information for faculty on their website, including information about individual consultation to support faculty efforts to accommodate a student with a disability and find solutions to specific issues. It seems clear to me that the Cerritos College DSPS understands that there are times when a faculty or staff member may be struggling with how to assist a student in the most effective manner. On their webpage they note that “having a better understanding of a disability and what the educational limitations are, as well as which accommodation are recommended will help all involved.” (Cerritos, 2015) They have a page called What are Accommodations? which helps faculty understand disabilities and identify appropriate accommodations.

DSPS Mission Statement
The mission of D.S.P.S. is to assess, address, and provide reasonable accommodative service and referrals to students with disabilities, enabling them to equally access and fully participate to the best of their abilities in the curricular and related activities of the Cerritos College community. (Cerritos, 2015)

Cerritos College Universal Access
Cerritos College is committed to establishing a barrier free learning community, or Universal Access, to all individuals. This includes a partnership of Information Technology Department, Disabled Student Programs & Services, Purchasing and Human Resources.  Below are the major categories of services we support:

  • PROCUREMENT – We assist in the assessment for purchasing software, hardware, telecommunications, video, and multimedia products and other electronic miscellaneous technologies across the campus.  As an entity of the State of California receiving Federal and State funding Cerritos College is bound by Federal and State laws.
  • ALTERNATIVE MEDIA – We provide assistance for the creation of textbooks, instructional materials, and other printed information converted to another format such as Braille, large print, or electronic text.
  • ASSISTIVE Technology – We purchase, install, and support the computer programs provided by Cerritos College for use on the campus.  This software includes screen readers, screen magnification programs, and text to voice programs. Software Locations:  Where assistive software and hardware are located.
  • CAPTIONING – We establish standards and oversee the captioning process (words superimposed on television or motion picture frames) that communicate or translate audio dialogue.
  • JAWS AIDS – Shortcuts and strategies for using JAWS for Windows with the MyCerritos and SchedulePlus websites. (Cerritos, 2015)

ELearning and ADA at Cerritos College
Cerritos College has a faculty committee that oversees online and distance education. The Senate Committee on Technology-Based Learning:

  • Recommends policies and procedures on distance learning
  • Develops and recommends the specifications for electronic classrooms
  • Develops and recommends the specifications for professional development for
    participation in technology-based learning
  • Develops and recommends the specifications for course/learning management
  • Monitors trends and practices on issues in the field and communicates them to the college

ADA requirements for eLearning fall under this committees’ mission and responsibilities. They work closely with DSPS to make sure eLearning students with special needs are accommodated.

Cerritos College instructors are required to include information on their syllabus and their course web/LMS page about DSPS and the services they provide. However, in regard to online content being used in classes, it’s my impression that instructors who use the campus CMS and LMS generally don’t think too much about accessibility due to the built-in accessibility checkers. For instance, when a page or other content is added to the campus CMS, the system will prompt the user to check for accessibility before publishing or updating the page. A report is returned that lists any accessibility problems. The page can’t be published until the problems are corrected. Instructors who don’t know how to correct accessibility errors, must contact he campus web master or his assistant for help. However, many faculty create webpages outside the CMS and they may not be aware of the accessibility requirements or may not know how to make appropriate accommodations.

Sakai is the Cerritos College LMS. According to the Sakai website, all core features of the Sakai project are accessible and usable by the greatest number of potential users, including people with disabilities. However, instructors have to be aware that content they add to the LMS, such as documents, videos, links, etc. must be accessible.

Last but not least, Cerritos College has recently adopted Quality Matters, a faculty-centered, peer review process that is designed to certify the quality of online and blended courses. The QM process includes ADA and accessibility in its checklist. In addition there are many online course checklists that accessibility and usability. It would be useful if Cerritos College had such a checklist for faculty to use.

In addition to the above, the Cerritos College Library CMS provider, LibGuides, has taken many steps to ensure that their code is accessible and that librarians understand how to add content that is also accessible. However, LibGuides staff recommend that librarians pass their guides through one of the web-based accessibility checkers to be certain they are in compliance. In fact, this is good practice for all web pages used in courses and in libraries.

Overall, Cerritos College has a very strong, inclusive policy and practice towards ADA issues. The college serves a large population of disabled students on campus who benefit from the DSPS services. And while eLearning does falls under the DSPS umbrella, I do wonder how well they connect with distance students who need their services. It seems to me that off-campus students are probably being served by services closer to their actual location even if they’re taking online course through Cerritos College.

4. Revisit the 11 instructional design steps presented in chapter 1 of the text (Design Quickly and Reliably).* Revise this 11 step system using what you now know about development and testing. Try to create your own instructional design process/template that you might actually use. Briefly explain your modifications.

I actually like this process a lot. I was at a loss as to how to start the design process for our assignment and these steps helped me organize my ideas and create a pretty thorough outline for my course. However, I would like to rearrange the last four steps. In my library research course I tend to come up with the idea for learning activities first, based on the objectives of the assignments. After that I would choose the media, and then create the object and tests.

In addition, given what we just discussed in this module, a major step is missing from this process. As Professor Newberry (2015) points out, testing online learning materials is crucial. Yes, identifying, recruiting, and acting on the feedback of beta testers may be time consuming, but it’s an important step in the process. As Benjamin Martin of Learning Solutions Magazine points out, “The purpose of the Beta Test for an on-line course is to simply check the effectiveness, usability, and functionality of the course from a typical user perspective.” (Martin, 2010)

I would also add a Final Checklist step. It may seem simplistic, but I like having an authoritative list to which I can compare my product. In ETEC 674 I came across the very thorough Online Course Readiness Checklist created for the San Diego Community College District. (Gustafson, n.d.) All the components listed are worth addressing for every online course. The major components of the Online Course Readiness Checklist are:

  • Instructional Design – the organization and architecture of the course
  • Navigation – how students access the course content and tools
  • Pedagogical Effectiveness – the instructional techniques
  • Accessibility and Usability – course meets ADA compliance and universal access standards
  • Copyright Compliance – adherence to the institutional copyright compliance policy.
  • Technology – use of technology tools and multimedia elements, hyperlinks
  • Schedule – updated time-sensitive items, schedules, calendars, announcements

Last but not least, I’d include a step that encourages follow-up with the instructor and the students who taught and took the course. In this way the course designer can ascertain whether the course and the learning objects were successful and make changes as needed. An online survey that addresses the functionality, ease of use, layout, and effectiveness should be sent at the end of the course to both instructor and students. Encourage user suggestions by including open ended response options, not just ratings scales or yes/no responses.

A 14 step instructional design process based on Horton’s 11 step process as presented in the text:

  1. Identify your underlying goal
  2. Analyze learners’ needs and abilities (Add an analysis of content and instructor needs/abilities/preferences.)
  3. Identify what to teach
  4. Set learning objectives
  5. Identify prerequisites
  6. Pick the approach to meet each objective
  7. Decide the teaching sequence of your objectives
  8. Decide which learning activities you’d like to include
  9. Choose media
  10. Create learning objects to accomplish objectives
  11. Create tests
  12. Test all media
  13. Checklist comparison
  14. Follow-up and revise for next time


CAST. (2013). What is universal design for learning? Retrieved from

Center for Universal Design. (2008). About UD. Retrieved from

Center for Universal Design. (1997). The principles of universal design. Retrieved from

Cerritos College. (2015). Disabled student programs and services. Retrieved from

Gustafson, K. (n.d.) Assuring that the online course is ready for prime time. Retrieved from

Horton, W. (2012). E-learning by design. San Francisco, CA: John Wiley and Sons.

Martin, B. (2010). Beta testing an online course. Learning Solutions Magazine. January 11. Retrieved from

Newberry, B. (2015). ETEC 541 Session Eight Lecture. [Word document]. Retrieved from CSUSB Blackboard ETEC 541.

Newberry, B. (2015). ETEC 674 Session Four Introduction. [Web page]. Retrieved from CSUSB Blackboard ETEC 674.

United States Department of Justice. Civil Rights Division. (n.d.) Information and technical assistance on the Americans with Disabilities Act. Retrieved from

Waterhouse, S.A. (2005). The power of eLearning: The essential guide for teaching in the digital age. Boston: Pearson.