Detailed UX metrics
This is part of a series discussing our practices and values in the context of unemployment insurance modernization with the US and NJ DOL.
Ok, so UX investment pays dividends over a continuous development software project. But how do we operationalize that information as something we can track, measure, and communicate?
Primary metrics
Multiple states (including New Jersey) and the Department of Labor shared two high-value metrics for UI modernization: reduced agent intervention (as a proportion of total claim volume) and overall lower support call volume (as a proportion of the unemployed population). That in turn leads to a secondary UX metric of lower stress and frustration for both claimants and agents.
As detailed above and in other detailed research studies to date, an improved claimant experience (e.g. simplifying language, providing helper text, conditionally showing/hiding questions, saving progress) leads to a more accurate and coherent claim. This in turn means fewer people need to call the DOL for support, and claims are less likely to have conflicting or missing information.
Unemployment disproportionately impacts marginalized people—in terms of class and wealth, language, age, disability, and others—making UX improvements and reduced stress a significant quality of life and health metric for people experiencing unemployment. Within the DOL, constant exposure to intense emotional and ethical situations is a stressor for agents, affecting their job satisfaction, performance, and retention. By reducing causes of stress, that in turn may improve agents’ experience and retention, saving costs for the DOL.
General usability metrics
In modern software design, we have tried-and-true methods for assessing progress in usability maturity. Digital.gov gives a couple of relevant examples: overall experience and completion rate of tasks. The former may be described as qualitative, which can give nuance to problems and indications of how to improve, but is harder to generalize to broader conclusions. The latter is a typical quantitative metric where an error rate can be derived from a large number of research sessions, however that comes at the cost of specificity. The goal of a good usability research plan is to do both, using mixed methods, where one may, for example, track the error rate from a large number of people to get a generalizable conclusion and supplement it with specific examples from qualitative feedback from each participant.
As a framework for future research, we recommend using the CASTLE framework. The authors point to its strength for “applications where users cannot choose the product,” which applies just as well to the single UI application for applicants in New Jersey. CASTLE is an acronym that refers to six dimensions of user experience: cognitive load, advanced feature usage, satisfaction, task efficiency, learnability, and errors. Each of those dimensions has one or more metrics, which we have used in various studies throughout our project . We recommend using this to plan large research projects that might span multiple studies, such as a general usability test for the form once it is available.
One example of a tool used to track metrics in the CASTLE and other frameworks is the System Usability Scale (SUS). The SUS is one of the simplest industry-standard ways to quantify three dimensions of basic usability: efficiency, effectiveness, and satisfaction. It is a set of ten questions (each mapping to a dimension) to ask participants in a study, usually summatively at the end of a session or task. Participants answer each question (e.g. “I found the application unnecessarily complex”) as a measure of agreement on a scale of 1-10, though we recommend adding adjectives to those numbers to make it easier to interpret and share results.
Image from MeasuringU comparing scales
Sustainability and cost improvements
Improvements to the user experience can also pay dividends in lower energy cost and progress towards carbon neutrality. We made design decisions that increased the efficiency of use to claimants:
Saving progress to avoid re-filling a timed-out form
Plainer language that reduces the need to ask questions
Reducing page weight ~90% to improve loading on slow and spotty connections
By increasing efficiency through UX, less energy may be needed to serve claimants via energy-consuming resources (phone lines, systems, people in offices). Once launched in a public location, New Jersey might consider engaging again with Truss or using tools like Ecograder to take further steps, including:
Using a greener web host
Lazy loading all images and serving them in AVIF/WebP format
Minifying served files and removing unused files
Other infra sustainability improvements
Let’s do it again!
This is the final post in our series about our latest UI modernization project. If you’d like to see a snapshot of our work, the USDOL published one on Github. It can be used as an example and reference to other states who want to modernize their UI infrastructure, or you can browse it as an example of the other topics covered in this series. And yes, we said ‘snapshot’, though we and NJDOL both suggest future work be done fully in the open from the start. That may sound daunting, but one of our core values is to “Act Without Fear”. Being transparent and fostering open-source collaboration allows the community to share lessons learned and decisions made to reduce rework and repeating mistakes. Just look at how our fully open source React and Figma libraries for the U.S. Web Design System (USWDS) have contributed to the improved state of web design in government contexts!
Are you looking for better ways to plan and study UX improvements? Or maybe you want to work more in the open? Truss can help. Please let us know what questions you have. We’d be happy to jump on a quick call and open up the conversation.
This is part of a series discussing our practices and values in the context of unemployment insurance modernization with the US and NJ DOL: