The Extended Reality Key Performance Metrics Roadmap

Extended reality (XR) is increasing competitive advantage in companies looking to reduce fixed expenses and product lifecycle costs, while increasing sales. The positive disruption of XR provides great opportunity for how businesses design, produce, and distribute. Setting clear Key Performance Metrics (KPMs) and measuring Return on Investment (ROI) becomes an essential  discussion point.

Testing and measurement of XR projects varies at the enterprise level, but the process  — and results — are proving more consistent and specific with each project. At AVATAR Partners, our continuous performance testing has resulted in several key KPM’s that strongly influence the development and progression of XR solution success. If you’re an organization with an augmented, mixed and virtual reality solution coming into effect, below are a few KPMs we’ve identified, and what else you can expect when testing and validating your next XR program.

Reduction In Training Time

One of the most important performance metrics to identify can be derived from test results based on the comparison of a company’s current methodology versus the use of XR. These tests include accuracy, time to train, and time to complete the task. Cost savings were derived based on an assumed reduction of errors based on increased accuracy, and would vary depending on the application and cost of parts. In a recent use case, we’ve seen a reduction in training time by 81% after implementation of XR, a highlight proven to enable technicians with little experience to be able to complete advanced tasks with little training. 

Elimination of Errors – Safety of Life – $72M Asset

As a result of the US Navy maintainer community requesting a maintenance training system for its high-value, high-risk assets that would result in Zero Errors, the KPM for the Navy was singular: Aircraft Readiness. Particularly, one additional strike fighter aircraft per squadron. Using internal research and development funds, AVATAR Partners designed and developed its first iteration of the Augmented Reality Maintenance Aid using Mixed Reality. Elimination of errors is associated with several collerary metrics, including the reduction of the cost of parts, reduction of labor rework, speed of readiness, and even higher stakes: the cost of a highly expensive asset and loss of life. In 2006, a V-22 Osprey aircraft went down, in part due to wiring error. Fortunately, the pilots survived. However, the $72 million asset was totaled

Reduction in Troubleshooting, Time on Task, and Improved Accuracy

Upon a recent review of an XR integration with a partner, we witnessed a significant reduction in troubleshooting time by 75%. When focused on the AR side for Quality Assurance, our team assessed a 90% reduction in inspection time – a decrease from 40 hours down to 4. Additionally, the AR-enabled QA inspection system found one error that was completely missed on the physical drawings.

Reduction in Development Time

AVATAR’s secret sauce? Quite simply, is SimplifyXR: a Unity plug-in that allows Unity  development, sustainment, configuration management and real-time device distribution in a codeless, plug-and-play standardized authoring environment. The product speeds development and maintenance time by tenfold by enabling rapid workflow-based updates and reuse of assets and contents. Originally built in-house to enable AVATAR Partners to be more competitive, the product was so compelling to our customers that it is now available for resale to any company using Unity or Avatar-developed solutions. 

Turning Novices into Experts, Upskilling, Job Satisfaction

Our testing has shown that by using hands-free AR to perform parts of a complex engine maintenance task, a novice (2 months on the job with no prior training or experience on the equipment) successfully completed the task at the level of a senior technician, the first time around. In this initial pass, there were 3 errors that were noted, and fixed, due to AR tracking, which took a total of 3 minutes. The novice technician took 27% longer complete this first-time task versus expert technicians. When tested by the expert technician using the AR application, the task completion time was reduced by 25%. 

Qualitative (or anecdotal) user feedback is an important metric to track, as it clarifies the effectiveness of the AR application design. As expected, our early user satisfaction surveys rated from scale of 1-10, started at 7, and are now generally between 9-10. The form factor of the wearable device or the handheld frequently impacts this metric. However, so does the procedural flow for the user, as well as the use of audio, gamification, interaction, and — most importantly — confidence of the user in the application, which always comes down to absolute precision of the AR overlay on the object.  

As a last comment, XR should not be more costly to sustain over time as a company implements enterprise adoption. The overall architecture and cost — including license fees, software updates, hardware costs, distribution methods, and overall openness and extensibility of the solution  – need to be understood up front to avoid getting blindsighted. 

If you’re looking to better target those metrics that matter to you, get in touch with us here.