UX Research Projects

Usability Study: Prototype Feedback

I lead a usability test of a new product prototype of an AR book/app in order to provide feedback to the product, development and design team.

*Some parts of the image have been blurred for confidentiality reasons.

<My Role>

Senior Researcher (team of 3)

I was the product researcher that managed the whole product lifecycle from initial research, content design, prototyping, testing, software development, system management, and launch.

<Project Background and Goals>

A prototype for a new augmented reality (AR) workbook series was created for grade 2 English Language Arts. We needed to find out if the main user experience (completing workbook activities with AR features) was easy for users (grade 2 students) to complete, and fun.

<Challenges>

Due to COVID-19 restrictions, we had to be creative with the research design and test set-up because we (the product research team) were physically located in Seoul, Korea, but wanted to conduct a usability test with participants in the US. We proposed conducting a user test in a hybrid format, where some parts of the test were conducted remotely (via Zoom) and some in-person (facilitated by a teacher on-site).

<Research Questions>

Can users (grade 2 students) use the workbook and AR device independently?
How long does it take a user to complete 1 workbook? Can it be done within 15 minutes?
What can we learn from the user flow? What app features are working well and what's not? Are there under-served user needs?
Is it engaging and fun?

<Methodology>

Usability test (15 min session/ea.);
Post-test user survey (w/ SUS and CSAT questions);
Student observations (conducted by grade 4 students with prior ARpedia experience)

*The usability test was conducted in a school setting, where new students tested the prototype and experienced students observed.

<Participants>

* Divided into 2 large moderated sessions (each participant will have their own device to test)

Usability testers: 21 grade 2 students from a private school in Massachusetts without prior exposure to ARpedia
Observers: 19 grade 4 students with prior ARpedia experience.

<Key Performance Indicators (KPIs)>

(1) Task success rate (TSR): 78% or above

(2) System Usability Scale (SUS): average score above 68

(3) Customer satisfaction (CSAT): average score above 80

*Some parts have been blurred for confidentiality reasons.

*For additional findings and learnings, please contact lim.jsy@gmail.com

<Recommendations>

Provide more guidance on initial product on-boarding, either as app content, help features or via instructional video materials.
Simplify materials and decrease the number of supplemental learning items.
Improve recognition rate of marker images.

<Impact>

An initial product on-boarding game has been created for users to learn how to use all parts of the product (app, workbook, coloring markers and action markers).
Pop-up help bubbles were added in the app to assist first-time users with product on-boarding.
An additional "How to set up" on-boarding informative mini-video was created to explain how to use the product. In a classroom setting, teachers can use this video to introduce the product to students.
Revised and decreased the number of supplemental learning items for each workbook.
Improved marker recognition rates with image correction and design changes

<Reflection>

Augmented reality (AR) makes learning content more engaging and fun for learners. However, AR technology is relatively new to most of the general audience, and therefore more guidance is needed for new users to understand how to use the technology in the product. Diverse ways of refining the product design to be more intuitive is needed. For this particular project, once the "on-boarding process" is improved, another usability study should be conducted as a follow-up to ensure that the experience is more intuitive for users.

< Back