Chapter 4. Getting Feedback

Usability

For years, my library would create tutorials with the assumption that learners were completing the tutorials as designed, that they did not experience any problems, and that they were thoroughly understanding and grasping the content. We later implemented usability testing and found that our assumptions were entirely incorrect. We found that students were confused with most of the navigation, missed important directions and content, took twice the amount of time to complete a tutorial as we had expected, and did not fully learn what we thought they were learning. Most importantly, we learned that we should never create tutorials without conducting usability tests.

There are several definitions of usability, but the simplest and most straightforward is a “quality attribute that assesses how easy user interfaces are to use.”1 Usability testing, which is task-based in nature, should not be confused with testing for performance (Did the learner actually learn something?) or satisfaction (What did the learner think of the tutorial?). Yes, we are concerned with these aspects, but usability testing for e-learning helps to eliminate barriers that may impede learning. Along with irrelevant content and overly lengthy tutorials, “poor design and usability of e-learning applications” are two leading contributors to low tutorial completion rates.2 In addition to helping you discover interface and navigation issues with your tutorials, usability testing can also help you learn about your learners’ preferences and get suggestions from them on improvements. You can also learn a lot about design from usability testing and then later build these design elements into future tutorials. Because of the many benefits of usability testing, it should be built into your tutorial design and development process, and not treated as an afterthought.

To conduct effective usability testing, you don’t need sophisticated tools or expensive software. Nor do you need a dedicated space or a lot of participants. Guerrilla usability testing is a quick and low-cost way of getting feedback from users. It can be done in any public place, including heavily trafficked areas like the library lobby. It is defined as “a low-cost, lean, and agile method of collecting data for testing and validating a hypothesis in a short session focused on specific tasks.”3 Many UX practitioners believe you need only five users to discover the majority of interface problems, but others argue that more are needed. Spillers, for example, advises having from eight to twenty users depending on the number of iterations.4 In our usability testing, we found that five users did not suffice and that different user segments (second language learners, nontraditional students) had different difficulties with the interface and the wording than did traditional freshman students. Although it is not always possible, you should aim to recruit users who are representative of your student population. Although usability testing is often conducted only with a finished product, you should conduct it throughout the development process. Here wireframes can help. Wireframes are simplified versions of your tutorial, and they can be digital or analog. Analog versions, known as paper wireframes, are sketches of your tutorial with all the navigation elements. Users complete tasks by pointing to different elements on the screen. Wireframes should not include any colors, only black and white, or any images and styling elements but should include all content and all navigation. Using wireframes allows you to gain a better understanding of how the user will experience the tutorial because they are not focused on how the tutorial looks.

Here are some tips for conducting usability and testing on tutorials:

• If possible, recruit students from across campus and not just the library. This way you are not introducing bias into your testing by speaking only with library users.
• Don’t go it alone. Have at least two people conduct the testing: one person gives directions while the other observes and takes notes.
• Introduce yourself to the user, explain the tasks, and explain everyone’s roles. Make sure to emphasize that you are testing the tutorial and not judging the user.
• Encourage users to think aloud as they complete tasks. However, do not require it, as some users do not feel comfortable thinking aloud.
• Let the user know that they can quit at any time, and let them know how long the testing will take.
• Ask and record any demographic information that is relevant to the testing. Gender identity is likely not relevant, but major can be. Because you are not conducting human subjects research nor gathering any personally identifiable information, usability testing does not require institutional review board approval.
• Remain objective and do not offer help with any task. It is your job to observe and take notes.
• Keep it short. This type of testing should take minutes, not hours, to complete. Aim for less than ten minutes.
• Even though testing takes a short amount of time, it is free labor and users should be compensated in some way. Gift cards are best, but healthy snacks can help too.
• Test for tutorial tasks such as
  • Can the learner start the tutorial?
  • Do they know where to get help?
  • Can they exit the tutorial?
  • Can they easily move through the tutorial?
  • Do they understand what they will learn in the tutorial?
  • Do they know how to answer question types?
  • Do they understand what is being asked in a question?
  • Can they easily adjust the volume if there is audio?
  • Do they know how to use all buttons and other elements on the screen?
• Set time limits for tasks. If you notice a student struggling with a particular question, move on to the next task or question.
• After completing the testing, thank the student for their time. Ask learners for feedback about areas they struggled with and suggestions for improvement. Often our learners know what will work best. Also, make sure to answer any questions the user may have.
• It is rare to have such one-on-one time with our learners, so make the most of this time and ask them a question or two on the content, usefulness, or their satisfaction with the tutorial.
• Make sure you use the feedback from the testing to make changes to your tutorial.
• If needed and you have the time and resources, repeat with the new changes.

Satisfaction Surveys

In addition to usability, surveys are another way to get feedback about tutorials from learners. Surveys can provide you with a greater understanding of what your learners think of your tutorials and what they feel they learned from them. Most often, a survey is our only opportunity to get feedback from our learners. However, since surveys and feedback forms are not required, learners often skip them, but there are several ways to ensure greater motivation to complete them. The following tips can be applied to different types of surveys but are focused on satisfaction type surveys.

• First, keep surveys short. Nobody wants to complete a long survey. Aim for a survey that will take only a few minutes to complete. For many of our tutorials, we include only one to three questions.
• To keep surveys short, avoid asking demographic questions. It may be interesting to know how different learner segments feel about tutorials, but if it is not necessary, do not ask for demographic information. If you do ask for demographic information, make sure it is inclusive in terms of race, gender, and ethnicity.
• You should be able to articulate the aim of your survey in one succinct sentence: for example, “I want to know if learners found the tutorial helpful,” or “I want to know if learners enjoyed the tutorial.”
• Use simple question forms. You don’t want the learner to think too much about how to answer a question. Rather you want them to reflect on their experience and opinion.
• Don’t feel the need to use different question types in order to not bore the user or to mix things up. Use the best question format for each question.
• When using multiple-choice questions, include an Other option with a free response text box. It is likely that you have not exhausted all options.
• For Likert scale questions, use a five-point scale. A three-point scale will not provide you with sufficient information.
• If asking a question about frequency, use specific time periods and not terms that may be unclear, such as “regularly.” Instead use terms such as “weekly” and “monthly.”
• Avoid any library jargon, slang, and any other terms your learners may not be familiar with.
• If asking potentially sensitive questions like age and income, use a range and not a specific number.
• Pilot your survey with different user groups before publishing it. What is clear and straightforward to you may not be to others.
• If your survey is more lengthy, in order to increase motivation to complete it, compensate users with a monetary incentive such as a gift card.
• Use skip logic so that users can skip questions that are not relevant to them or that they don’t feel comfortable answering.
• Avoid leading questions. Leading questions are written so that users are forced to answer in a certain way. For example, the question “How much do you enjoy the tutorial?” implies the user enjoyed the tutorial. “Rate your experience using the tutorial” is more neutral.
• Use at least one open-ended question. Although more taxing on the user to answer, open-ended questions allow you to gain additional insights into the user’s experience.
• Use the responses from the survey to make changes in your tutorials.

Notes

1. Jakob Nielsen, “Usability 101: Introduction to Usability,” Nielsen Norman Group, January 3, 2012, https://www.nngroup.com/articles/usability-101-introduction-to-usability/.
2. Panagiotis Zaharias and Angeliki Poylymenakou, “Developing a Usability Evaluation Method for e-Learning Applications: Beyond Functional Usability,” International Journal of Human–Computer Interaction 25, no. 1 (2009): 75–98, https://doi.org/10.1080/10447310802546716.
3. Darpan Sunwar, “Guerilla Testing,” Open Practice Library, March 23, 2021, https://openpracticelibrary.com/practice/guerilla-testing/.
4. Frank Spillers, “The 5 User Sample Size Myth: How Many Users Should You Really Test Your UX With?,” Experience Dynamics, Medium, March 6, 2019, https://expdyn.medium.com/the-5-user-sample-size-myth-how-many-users-should-you-really-test-your-ux-with-675fa4bd8ac.