Weaveworks Hires SoundCloud Engineer Peter Bourgon, Releases Container Mapper – InApps Technology 2022

Last year, SoundCloud engineer Peter Bourgon was one of the software development world’s most outspoken skeptics of the benefits of distributed programming. In an interesting turn of events, Bourgon’s services, along with those of colleague David Kaltschmidt, were officially acquired Wednesday by Weaveworks, the makers of the Weave SDN framework for Docker.

As an apparent result, Weaveworks is now claiming stewardship of Weave Scope, a Docker container visualization system built by Bourgon, Kaltschmidt, and colleague Harmen Bus. Its aim is to plot containers in a Docker or microservices environment, in a way that illustrates their relationships to one another, as well as how they connect.

Change of Commitment

“We were interviewing a terrific developer called Peter Bourgon, who while at SoundCloud, had made his mark in the broader community,” wrote Weaveworks CEO Alexis Richardson in a company blog post Wednesday afternoon. “We were not at all surprised to find that Peter had very developed thoughts on monitoring and visualization. But things got interesting when it turned out that he had reached very similar conclusions to our own, but from a different starting point, i.e. operating large scale systems himself.  And then he showed us a demo that did exactly what we hoped for.”

Bourgon may have had this project brewing in his mind for some time. Only weeks old, there is very little documentation for this project. A readme file explains the concept simply: “Weave Scope automatically generates a map of your containers, enabling you to intuitively understand, monitor, and control your applications.”

While at SoundCloud, Bourgon contributed to the Go language client for Prometheus, the highly praised monitoring system for microservices. He is also a principal author of the gokit framework for using the Go language to build distributed microservices using service-oriented architecture and best practices.

In a now-famous, oft-repeated lecture called “Consistency Without Consensus” (a play on “consistency without concurrency”) Bourgon explains how failures in distributed systems may be tolerated through the introduction of partitioning. Once failures become not only tolerable but overcome through best practices, he believes the conventional constraints of the CAP Theorem — which states that of consistency, availability and partition tolerance, you can only have two out of three — can be overcome.

The Next Modern Enterprise

In the instructions that accompany gokit is a bit of personal philosophy concerning what he calls the “modern enterprise.”

“When we read the word “enterprise” we probably think of older, slow-moving bureaucracies, like IBM, HP, or even Red Hat,” he wrote. “But it’s been a long time since companies like those have been technical leaders. Decades, in some cases. Now, companies like Google, Amazon, Twitter, Netflix, Facebook, Spotify, or even SoundCloud set the tone for our industry.”

His ideal modern enterprise, as he reiterated during a talk he gave two months ago for a Go conference, is a tech-related, consumer-focused company with a critical mass of 100 or more engineers, focused around a service-oriented architecture. He counts SoundCloud among this mix.

Bourgon’s last day at SoundCloud was April 30, according to his Twitter feed. Just since that time, he has been pushing commitments to GitHub for Weave Scope.

In a version of his talk at the Berlin Buzzwords show a year ago, Bourgon praised the DevOps movement as awakening developers to the new realities of their industry, including taking responsibility for the maintenance of code as part of the evolution of that code.

“In the bad old days, we used to build these binary artifacts,” he told attendees, “and we’d all be like, ‘Works on my machine!’ and we’d, like, throw it to the sysadmins, right? And we’d say, ‘You guys, you should run this in production. And you should get paged when it goes down, and you should scale it out.’

“But we eventually recognized an invariant in software engineering,” he continued. “As the authors of a piece of software, we are uniquely qualified to run that piece of software in production. We’re going to be the best person to do it. So we had this DevOps revolution, right? And we changed our methodology, and we said, programmers are expected not just to type the code in, build a WAR file, and then throw it at somebody, but to deploy it to production ourselves, and to monitor it, and to get paged when it breaks, and to fix it, and then to scale it up when we need more capacity and scale it down when the next version of the software comes along.

“And we’re better for it as an industry, I think,” Bourgon commented. “All the best software shops work this way; I hope you work this way. If you don’t, it’s bad, and you should feel bad.”

HP and Red Hat are sponsors of InApps Technology.

Feature image via Flickr Creative Commons.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.