Bazel, Kubernetes, and Truss: Building Robust Infrastructure for Gyroscope

Gyroscope, a burgeoning data science startup, came to us early hoping to lay a solid infrastructure foundation for their service. From day one, they wanted to be able to deploy a collection of microservices written in different languages and, later, to be able to add new services easily.  This was a sophisticated stack for a startup so young, driven by their need to be able to experiment with new Machine Learning models in production at will. We worked together to stand up a build system, continuous integration, deployment system, and monitoring, so that their small team could focus on the more pressing needs of their startup.

We started with Bazel, a new build system from Google, to turn all their services from code into Docker containers. Bazel’s ability to handle dependencies in any languages and its guarantee of reproducible, hermetic builds gave Gyroscope confidence that their software will be built correctly, whether it’s running on one of their own laptops, a new hire’s machine, or in CI. Anywhere their code gets built, Bazel builds it the same way with the same dependencies. How many hours have you lost debugging an issue that depended on a dev machines specific config? We wanted “But it works on my machine!” to be a thing of the past.

Bazel is still very young, and we certainly paid a price for that peace of mind. From outright limitations* in what it could build to the odd day here or there working out how to get it to work with Docker or Python scripts, there were a lot of ways that Bazel felt unfinished. But it was certainly worth it for Gyroscope, as the alternative of building and deploying a heterogeneous set of services with Make was a set of regular complications they couldn’t afford to deal with at this point in their business’s growth.

To further prevent developmental inconsistencies, we set up deployments with Kubernetes to reduce the possible production configuration woes, and did all local development in minikube, which runs a Kubernetes cluster on your dev machine. As much as possible, we wanted running things locally to match what happens in production. And, because of the way Bazel can both build and run scripts, building and deploying all their software was run by a single command.

Gyroscope’s desire for rapid experimentation and iteration with their ML backend made them want robust infrastructure for their services from the beginning, rather than getting speed today at the price of technical debt tomorrow. The best infrastructure gets out of your way and lets you focus on the problems you really want to solve. This modern stack designs away problems that have plagued software engineering for decades, giving Gyroscope more time to spend on their core product and growing their business. For early-stage startups, it can feel like an acceptable trade to aim for an MVP rather than something scalable and reliable from day one, but our clients have found that their later success is much better enabled when they invest in the right infrastructure as early as possible. (Their engineers fare better too.)

Head over and check out our joint blog post for more specifics about how our approach supported Gyroscope’s ambitious machine learning goals. If you’re curious about how a modern infrastructure stack could give you more time for solving the problems that matter to your business, contact us at why@truss.works.

*There are a few examples, but the one I reached for first has actually been fixed in the last month. Bazel development is certainly running at full steam.