Usability testing DNS/TLS
Usability testing is vital if you’re going to make something for other humans. It should be done early and often (like voting!).
Typically, we think about usability testing for applications and products: “what does this button with a mystery icon mean?” or “this option I need all the time was nested 5 menus deep” are well known responses from end-users. It’s our responsibility as technologists to ensure our intentions of an optimal user experience actually match the user’s real experience.
When it comes to tools for managing infrastructure, we think about user experience less often. A lot of the time folks developing tools for technologists appear to assume they don’t need to be “nice”. A common refrain is “read the docs” or “google it”. There’s no reason for this dichotomy between “regular” user and technical user.
Recently, we developed a technical tool for one of our clients which required a significant educational lift. The service replaced a heavily manual process with a more automated system that relied on a lot of relatively modern technology (Terraform! Atlantis! GitHub?). Our customers are developers, but they have wildly varying comfort levels with many of our infrastructure practices, so we knew we'd have to focus on creating a service that felt accessible and intuitive.
We decided to approach developing our technical tool as though we were developing a web app. We recruited users, tested the product, and iterated on our solution based on user feedback. The only difference? We were testing a product where users wrote code instead of using a web app.
About the project
We faced a highly manual process to create public DNS records and TLS certificates that required PDFs, online forms, back and forth JIRA tickets, and manual creation resource changes. Every step of this process was vulnerable to human error and long timelines.
At Truss, we think anything predictable enough to have a computer do (after a certain threshold), should be automated. We use Terraform extensively in our work, and one of the tools we’ve been excited about is the Terraform pull request automation tool Atlantis. This project looked like a great opportunity to use Atlantis to create an automated system to run infrastructure via code. Leveraging Atlantis and our Terraform knowledge, we replaced the web UI and PDFs with a GitHub pull request where app developers could submit change requests for DNS records and TLS certificates as code. We defined the business owners (the folks responsible for verifying the requested resources match their team’s needs) in a Github CODEOWNERS file to enforce their approval of the request. They just had to approve the change, merge, and the changes would go into effect. Best of all, the need for a third party to make manual changes disappears.
Since we successfully restructured the information that defined DNS records and the creation of TLS certificates as code, we provided visibility into these systems by keeping a running log of code changes. We were also able to see when code had been manually changed because it would no longer align with our source of truth. This made it easier for developers to create changes directly and for those running the system to validate those changes in a centralized location.
If any of that was outside your wheelhouse, you might be someone who thinks engineers’ tools don't need usability testing. But stick with us!
Usability testing the project
We huddled with our Human Centered Design (HCD) team to better understand what we were looking to achieve, what we wanted to learn, who we should talk to, and what challenges we might face on this project. The HCD team designed a process for us that included an introduction email, a usability testing script that helped moderators facilitate the usability test, and recommendations on how to make sure the user feel comfortable throughout the process.
We scheduled hour-long sessions, recorded with consent, with 5 engineers identified by the client as likely users of this tool. We also scheduled time with 3 engineers who were less likely to use the tool to cast a wide net of feedback.
For the usability test, we would send the user the URL of a GitHub repo with read-write permissions and a scenario we made up (“You’ve just been asked by your team lead to get a new DNS record made for your website ilovedogs.com in a production environment. Your colleague says ‘hey, I hear there’s a new process for that! Here’s the GitHub link to make it happen!’”) and ask them to walk through the GitHub repo, talking through what they were thinking.
We got a lot of different feedback. When we heard something consistently, we iterated, tweaking the README and examples. Folks completing the usability tests had widely variable comfort levels with our proposed processes and tools. Some users weren’t able to go through the process at all whereas others set up a DNS record with no questions. The best part was getting some converts—people who were so appreciative of the new process that they went to bat for us, telling our client how great the new process was and suggesting other uses.
While we did iterate and apply some of the feedback received, one of our main challenges with this project was our lack of access to users. As referenced above, our customers had extremely varied experience with infrastructure as code, and our inability to talk to users meant that we had to design the system before verifying the concept, something we try our hardest to avoid. Once we made it through our first batch of usability tests, we again found ourselves stuck. Ideally, we wanted to apply changes and retest usability, collecting more feedback until we got to a point that our tool was truly serving our users.
Technologists also like to have happy environments and useful tools
Usability research isn’t just for the non-technical user. It can also be used to hone and shape infrastructure projects or other tools developed for engineers! With that in mind, in order to be successful in testing technical tools, make sure you have direct and consistent access to end-users. Usability testing will help you build user-centered tooling but you need to take the time to recruit the specific type of end-user who will be using your services.
So try it out! Let us know how it goes, and let’s all keep building (and maintaining) with each other.