Charity Majors’ Mission to Monitor Microservices – InApps 2022

The following interview is part of a series, called Open Source Leaders, where we profile project leaders in the open source IT community, to learn more about how they developed their software as well as the challenges and benefits that come with running an open source project.  

Charity Majors knows first hand the pain every software developer feels upon releasing a carefully built and strenuously tested project into the wild — i.e., that moment where performance monitoring becomes virtually impossible. So she co-founded Honeycomb.io, to provide monitoring and debugging tools that create real-time windows into an app’s “real world” performance.

Majors honed her skills at Parse, the mobile application development company, where she worked as infrastructure tech lead on Parse’s mobile backend-as-a-service. When Facebook acquired Parse in 2013 Majors became manager of a team of production engineers responsible for the care and feeding — infrastructure, database adminsitration, automation, incident response, continuous deploy pipelines, Facebook integrations — of more than 500,000 mobile apps.

Charity Majors spoke to InApps to talk about leaving Facebook to found Honeycomb.io with partner/CEO (and also former Parse engineer) Christine Yen — along with a little help from the open source ecosystem.

What is your background?

I’m a music major dropout who turned techie. I’ve always gravitated towards the ops and DBA engineering roles — I have very low tolerance for abstract computer science, I have to do work that needs to be done. So I’ve been a systems engineer and an engineering manager, but somehow always seem to end up responsible for the databases too — I’ve been on call since I was 17! I’ve been flipping back and forth between engineering management and IC roles for a while now. I founded Honeycomb.io January 2016 with Christine Yen, another ex-Parse engineer. The company hasn’t failed yet.

How did you come to start Honeycomb.io?

I fell in love with Scuba while I was working at Facebook. Scuba is Facebook’s internal observability tool with this great, fast and flexible UI. When I was ready to move on from Facebook I was just so horrified at the thought that I’d be going back to a world of old-fashioned metrics and dashboards. I realized that I’m simply a far more powerful engineer with a rich, event-based, interactive tool like that, and I never again want to go stabbing around in the dark and bouncing from tool to tool like the bad old days. So we built Honeycomb, to give smaller companies that same sort of real-time analysis of application performance that huge enterprise orgs like Facebook get to use.

What business or technology problems does Honeycomb help solve?

One word: complexity. We are looking at a near-future scenario where everyone has to become a distributed systems engineer to manage the exploding complexity. The LAMP stack generation of tooling was built to answer known questions quickly and efficiently. It does that very well, with Graphite, New Relic, ELK/elastic, etc. But, in the future, we won’t have the luxury of known unknowns. By which I mean, in this system, if you knew what the question was, you could probably answer it. Increasingly now, though, you don’t have a clear question. Look at microservices and distributed systems: it can be incredibly challenging to figure out which component(s) is(are) where the symptom(s) — which may or may not exist, reported by unreliable narrators or equally flaky tooling — originate. Which are the causes and which are the effects?

“Most dashboards are artifacts of past failures — if you think that serves your current and future situations, you are blinding yourself.”  — Charity Majors

Honeycomb, which has lots of Scuba DNA, is a radical, fundamentally different way of approaching problem-solving, and it’s completely and utterly necessary. I’ve seen this over and over Facebook and elsewhere: we are lost if we stick to our old ways. Most dashboards are artifacts of past failures — if you think that serves your current and future situations, you are blinding yourself.

These days companies need something like Honeycomb whether they know it or not. Fortunately, people increasingly understand that running large multitenant platforms and developing deep data tooling isn’t free, and isn’t part of their core business proposition, and thus they should hire people like me to handle it for them. Honeycomb handles high cardinality data really well, and no other system does this. You can find every single needle in that fucking haystack with Honeycomb.

And where does open source come into this?

The open source component of Honeycomb thus far is a repository of integrations, the software that gets the data to our API. If I had to guess, at least 50 people so far have committed code to something Honeycomb related, and we will be ramping up substantially over the next two quarters.

Making Honeycomb’s front end open source is huge. All the code for all the connectors that transport the client’s data to us — we can’t possibly write all the connectors for all the types out there, that would be such an incredibly long tail and we simply don’t have those resources. But having open sourced Honeycomb’s integrations, now anyone can go read the code and write something in say Erlang or some other lang we are not ever going to support because there simply not enough people who use that. Open source serves everyone, in this case.

We haven’t open sourced Honeycomb’s backend yet. Mainly because it isn’t ready — we are focused on survival as a business and don’t have the resources to go there right now. We’d like to someday.

I’ve contributed to open source outside of my day job as well. The biggest I’ve made recently was my Terraform config files. Terraform is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned, and its GitHub repo has over a thousand contributors. I also wrote apiary, the original tool for capture/replay benchmarking MySQL/HTTP traffic. It’s now maintained by Lex Neva.

What about the potential for monetization?

We had discussed the idea of making the whole of Honeycomb open source and monetizing by selling enterprise licenses and customization and support, some model like that. But again we are a startup and focused on survival, getting ourselves established in the marketplace.

I am super pro open source, I have been using OS software my entire career and passionately support it. But I am also pragmatic — and in my experience, open source helps you expand faster, and brings people in. But you need to be growing out from a stable, established base. Open sourcing an entire startup is just more than almost anyone could pull off unless you have all the VC in the world. So we do what we can for now and are definitely looking to expand our involvement as we grow.

You are both a seasoned open source contributor and the founder of a software company. How, in your view, do open source and enterprise best fit together?

As a consumer and someone who now does enterprise on a relatively small scale, in that realm, I feel like we should all lean toward open source. If there’s a debate, lean to open. Unless you have a reason to not be transparent, do it!

Bigger picture, we should be shaming very large companies that use OS software for key parts of their core business, but who don’t return in kind by assigning an engineer or developer — or two, or three — to work on contributing back to that OS resource that is helping them profit, build back into making that resource better for everyone else using it too.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Honeycomb.io.