Piloting your e-Learning Course – A Beginner’s Guide

By Hannah Hunter, Instructional Writer

Hannah(250).png Did you know that the pilot episode of Star Trek included only one member of the Enterprise crew: Mr. Spock? And in that version, Spock had dark red skin and absorbed food through a plate in his stomach? Viewers of the pilot were not shy in their opinion that the show was boring and Spock was creepy. Thankfully, Star Trek’s creators listened and completely reworked the cast and the script, creating the successful franchise we know today. Thank you, pilot testing!

No reputable company would launch a product without first conducting a pilot. Why should our e-learning courses be any different? Here are five guidelines to facilitating a rewarding pilot.

1. Set the stage

Conduct your pilot in a room with enough computers or tablets for learners to have their own devices. If you’d like to test the course in your LMS environment, ensure that the course is loaded into the LMS and assigned to participants. Schedule enough time for a thorough review – good feedback can mean lengthy discussions. For each half-hour of e-learning, allot at least an hour to discuss and gather insights from learners. Before setting learners loose on the course, explain the purpose of the pilot test. Encourage learners to take their time and provide honest feedback.

You may want to provide a checklist of broad concepts for learners to consider during their review, such as media, ease of navigation, relevance of content, grammar, spelling, clarity of instructions, etc.

2. Invite the right people

The right audience for a pilot test is the same audience who will be using the finished product: your learners! For best results, your pilot group should include the full spectrum of potential learners for your course: new hires, tenured employees, the technically savvy, the computer averse, learners who require 508 accessibility, low performers, high performers…you get it. I suggest having one person facilitate the pilot test and one to two other people from the project team present to observe and offer help if learners get stuck. If possible, the pilot facilitators should be people who the learners do not know. They should not be supervisors, corporate big wigs, or anyone whose prescience may intimidate learners or dissuade them from providing honest feedback.

And, I shouldn’t have to say this, but I do: don’t conduct a pilot test on project stakeholders. Stakeholders are going to have a very different perspective from learners and are likely too close to the project to provide new insights or catch mistakes at this stage.

3. Observe

While learners are taking the course, walk around the room and take note of what you observe. Are learners able to navigate easily through the course? How much time are they spending on each activity? Do they seem engaged or do they look bored? Are there any areas where people seem to be having difficulty? If learners ask you for help, you may provide it, but for the most part, do your best to avoid interfering with their review. You want learners to formulate their own opinions and experience of the course without outside input.

4. Ask questions and collect data

There are a number of ways to collect feedback from learners during a pilot test, including:

Discussion
Post-course surveys
Built-in QA tools, such as JIRA
Paper questionnaires

Ask questions that are open-ended and encourage thoughtful responses, such as:

What did you like about this course?
What impressions stick with you after completing this course?
What, if anything, was confusing or difficult about this course?
How did you feel about the timing of this course?
Would this course help you perform better on the job? Why or why not?
What suggestions do you have for improvement?
How does this course compare to other training you have taken?

5. Be open to change

Pilot tests can be ego crushers. It is disappointing to hear negative feedback about something on which you’ve worked so hard. But don’t conduct a pilot test if you are not willing to make changes to the course based on the feedback you receive. In practice, this means conducting your pilot at a stage of development when you still have the time and resources to make changes, usually at Alpha or before.

Remember, the purpose of a pilot is not to validate that the course is perfect, but to identify opportunities for improvement so you can adapt and ensure a successful launch. So, listen to and trust your learners. They are your most important, and honest, audience.