Taming Dependencies at Opsee

On June 14th, 2016, Heavybit member companies Opsee, LightStep, and Iron.io hosted the GoSF Meetup at the Heavybit clubhouse. Below is a video of Greg Poirer’s talk on Taming Dependencies at Opsee, along with extended thoughts on the path they followed to get to their current solution.

About a year ago, Opsee began a project of retooling all of its internal communication around gRPC and protobuf v3. We chose these because we have microservices in a few different languages–as happens when you’re rapidly prototyping functionality and allowing developers the flexibility to choose their own language. As we coalesced as a team around Go, we ended up with a considerable amount of shared, generated code that we were reusing in a number of different projects.

We placed all of our protobuf definitions and the generated Go files in a single repository. Standard practice in Go is to check generated files into source control, because projects have to be able to build with a simple call to go build. This has some interesting implications when you aren’t developing in a monolith however. We inadvertently created a shared dependency around a specific version of gRPC and protobuf without realizing it.

After taking a very close look at our predicament, we decided that we would do things a little different. Instead of a single shared repository, we’ve chosen to break that up; the service responsible for a given protobuf definition houses that protobuf spec file. Instead of checking the generated code in, we allow services to duplicate the protobuf definition and keep its own copy and the generated code in version control.

This works for us, because we’ve chosen an internal contract that states

We will not remove or change fields in protobuf specifications.

This is similar to a versioned API contract. As we add new fields to those objects, the service that copied the protobuf file can choose to update or not–if it needs access to the new fields. This change allowed us to associate versions of protobuf and gRPC with each project instead of coordinating versions across all projects.

In this talk, we explored some of our options–including our process of considering a monolith. There were a number of, to us, unacceptable trade-offs for working with a monolith. Most of these centered around a lack of a good continuous integration story for projects in a monolith.

Learn more about Opsee at Opsee.com, and check out the Heavybit events page to RSVP to other developer focused events in our San Francisco clubhouse.