Companies are investing billions of dollars on implementing VR/AR solutions in their business. In 2020 alone, it’s forecasted that $18.8B USD will be spent, a 78.5% increase from 2019. VR training is the leading use case for VR/AR corporate spend because companies have started to realize the potential of immersive technologies. However, many of the companies we’ve talked to still don’t have a systematic approach to measuring success.
This is concerning because a 2011 survey by the CEB found that 75% of 1,500 managers surveyed from across 50 organizations were dissatisfied with their company’s Learning & Development function.
Source: Pixabay
How do you know if VR training you’re doing is effective?
Far too often, companies don’t have a way to prove efficacy from VR training. Or, they pay attention to vanity metrics that don’t provide value. The only way to translate virtual insights into real world results is by holistically evaluating human behavior within VR.
In this article, I will outline the 3-step process for getting actionable insights from VR.
3-Step Process For Getting Actionable Insights:
- Data collection: Put together a hypothesis for the learning outcomes you expect. Then decide what data you’ll need to collect to prove it.
- Analysis: Process the data and extract insights about user behavior. Was there anything unexpected? What did you learn?
- Decision making: Put the insights into action and direct your training based on your learnings.
Data collection:
The first thing to understand is that new capabilities are needed to collect Spatial Data, which refers to data from 3D environments.
Traditional metrics designed for 2D surfaces aren’t relevant.
Making use of hardware you can collect:
- Eye Tracking: VR/AR headsets increasingly have built-in eye trackers. This lets you see through the eyes of users.
- User Interactions: Instead of just looking at how users proceed down a funnel. VR/AR lets you see how users manipulate the space around them.
- Pathing – Users have the agency to move through simulations as they see fit. You can see where they spend the most time and the paths they take.
- Biometrics: Sensors can be added to measure the emotional state of users. Such as heart rate, EEG, ECG, etc.
Most importantly, these inputs should NOT be measured in isolation.
Instead, they should be understood in relation to each other to give you a holistic evaluation of behavior.
Once you’ve collected this data, you can qualify success based on the outcomes. Last year we worked with a medium sized electrical utility company in the United States to help them modernize their training methods. Employees at the company relied on individual trainers to deliver procedures which had drifted over time. This meant the lessons became inconsistent between employees. Our hypothesis was that through VR training, employees would be able to learn faster, retain more knowledge and make less mistakes.
Analysis
The next step is separating the signal from the noise.
Look for behavioural patterns that lead to or prevent successful outcomes.
In the past, the company didn’t have a systematic approach and this made it difficult for them to provide best practices to employees.
With VR, employees can get more mental repetitions under the exact same training conditions every time.
Managers can quantitatively assess employee performance based on their actions. For example, eye tracking provides an unbiased method of evaluation.
This makes the measurements from VR much more in-depth and we were able to systematically evaluate behavior for compliance and competency.
For example, we wanted to know:
- Did employees that hesitated before proceeding with tasks fail at higher rates?
- Does a stable heart rate during high pressure situations indicate competence?
- Does the time it takes the user to notice hazards affect success?
Employees were also given the chance to fail. So that we could determine how they got to doing it correctly.
The process is just as important as the outcome and it gives them the opportunity to improve over time.
Decision making
After analyzing the data from 3 days of training, we concluded that VR provided significant improvements.
- A 59% increase in knowledge retention.
- A 40% increase in compliance.
- A 20% decrease in task duration.
You’ll have a better understanding of what actions are conducive to success. VR training should be an iterative process, as you learn more about what’s working, you can apply them.
Eventually, the goal should be to be able to use VR training to qualify employees and find correlation between hours they spend in VR and real world proficiency.
This can only be done by holistically evaluating behavior and understanding whether learning outcomes are actually being improved.
