Jupyter Notebooks Could Help SREs Better Sleuth Incidents – InApps Technology 2022

Science!

“Every incident is a detective story. The whole point is to figure out who done it,” asserted Moshe Zadka, senior site reliability engineer at Twisted Matrix Laboratories, speaking at the USENIX SREcon20 America conference, held virtually last month.

Jupyter notebooks, an open source tool originally designed for the scientific research community, could be a valuable aid in helping site reliability engineers (SREs) and other operational folk investigate, document and even recreate fixes for site incidents, Zadka said. Both jobs, after all, require sleuthing and scrupulous notetaking to get at the truth.

“Incident resolution is by nature exploratory,” Zadka said, an activity well supported by the notebook format. Much like the data scientist, the SRE has to do a lot of exploring, testing of multiple hypotheses to see what went wrong. “When you have an incident, you have no idea what is going on,” he said.

As much of the data science community knows, Project Jupyter provides a web interface for users to create and share documents with live code, equations, visualizations and text.

The researcher documents into the notebook the entire process of fact-finding, with live code, sample data and narrative notes. It can then be exported to  HTML where others can review and even step through the same steps.

A New Case Arrives…

Zadka offered a basic example of how an engineer would use Jupyter to document an incident. The most notable part of this example is that each step of the incident process was conducted within the Jupyter notebook itself.

“If someone needs to rewind the same step, they can just get the notebook and run it again,” he said.

In this hypothetical incident, a cloud-based service is failing to meet its services level agreements (SLAs). Servers are dipping below their P90 latency limits, though each server is acting somewhat differently. It turns out one of the services crashed, which is resulting in occasional long response times to queries. Which service? And what instance of that service?

The attending engineer will open a new Jupyter notebook and document each step — with data, the code used and commentary — even the dead-ends and missteps. Documenting each step along the way, the SRE in effect creates a document of how the problem was fixed. “That record will be relevant later on,” Zadka said.

In this hypothetical, the engineer opens Prometheus to get some performance numbers on the servers, not knowing what to look for, yet:

Eyeballing this graph, Zadka advises not to look at the highest or lowest line first, but rather the one showing the greatest amplitude. You could, in theory, write a detailed Prometheus query to get at that number, but “that’s often annoying,” he said, explaining that PromQL (Prometheus Query Language) is geared more towards basic queries (“Dear God, it is not really designed like a programming language.”)

Instead, parse the performance data with Python, a much easier task. Although originally created to run Python, Jupyter is a language-agnostic tool. The code needed here would be to query five minutes worth of performance data in 10-second increments, seeking the greatest amplitude across all the servers:

The next trick is to find out which EC2 instance the malfunctioning service is running on. Fortunately, Python has a tool to connect directly to the Amazon Web Services’ APIs. So another set of code can be crafted to query the AWS API that can describe all the instances, along with the handy identifying tags that the SRE put in place earlier:

Once the EC2 instance is identified, a next step is to connect to the instance, via SSH, to confirm it has a core dump file, indicating the crashed service:

And, to finish the job, the core file is moved elsewhere.  Easy peasy.

At least the incident itself is resolved. Any organization, however, would want to run a post-mortem or retrospective to shed light on what the problem was, how it was resolved. “Luckily, you already have most of the notes” already within the notebook itself, Zadka said. Usually, some additional documentation is needed at this point, usually the thought process behind the decisions, which can be added into the notebook with markdown.

Once done, the engineer can export to notebook to HTML, where it can be viewed as a presentation by anyone, and attached to the incident ticket.
Someone can even run that same code again, to do further analysis.

View the entire presentation here:

Feature image by shell_ghostcage de Pixabay