DevOps Metrics That Matter – InApps Technology 2022

Tricentis sponsored this post.

Every other aspect of application delivery has been scrutinized and optimized for DevOps. Why not re-examine quality metrics as well?

Are “classic” metrics for automated tests, test case coverage and pass/fail rate important in the context of DevOps, where the goal is an immediate insight into whether a given release candidate has an acceptable level of risk? What other metrics can help us ensure that the steady stream of updates are truly fit for production?

To provide the DevOps community an objective perspective on what quality metrics are most important for DevOps success, Tricentis commissioned Forrester Research to research the topic. The survey results are published in the 55-page report, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics — Ranked by Industry Experts.” The report provides detailed rankings, heat maps, quadrant mappings and analysis.

In this article, we describe the heat maps and analysis for each of the four major metric categories: build, functional validation, integration/API testing and end-to-end testing.

Build Testing

When measuring builds, unit testing done well matters. Counting unit tests is a waste of time but understanding change impacts matters. Tracking “unit” tests prioritized by risk is the key. As the code base evolves, developers and testers need immediate feedback about change impact. This feedback is significantly more actionable if prioritized by level of risk.

Sixty-three percent of these firms consider the number of unit tests prioritized by risk as one of their top desired metrics. But far fewer can actually do so — while 34% of advanced DevOps firms track the number of unit tests run, only 27% prioritize by risk. And less-advanced DevOps firms use it even less — just 15% can track the metric today. Other important metrics tracked in builds focus on ensuring code quality, such as the number of successful code builds (61%), unit test pass/fail rate (60%) and the total number of defects identified (59%).

Functional Validation

When functional testing kicks in, user story/requirements coverage gets the focus. The top goal for testers in this stage is to minimize risk by ensuring that the functionality expressed in user stories works as expected. Extending this coverage concept to business risk coverage is an additional metric that leading DevOps firms execute to advance release automation.

Running functional tests and checking against covered requirements (69%), the density of bugs (62%) or number of functional defects found (66%) and the ratio of tests passed vs. failed (64%) are metrics that successful teams identify as important to manage risk and track quality during functional testing.

Integration Testing

To deal with modern distributed architectures, measuring integration testing and API tests win all around. With application architectures becoming more decoupled, decomposed in services and microservices,

API testing becomes more relevant. So much is going on beyond the user interface that it’s impossible to keep quality high and diminish risk without addressing the API layer. Advanced DevOps firms prioritize many of the same metrics for integration testing as they do for functional testing phase; this time, the focus is on APIs specifically.

These metrics include tests run against functional requirements (75%), total number of new API defects found (64%) and API bug density (63%), API test pass vs. fail rate (62%) and API code coverage (62%). Monitoring API risk coverage is an important metric as well, with 62% of advanced DevOps firms prioritizing it as a top metric for the category.

End-to-End Regression Testing

End-to-end regression testing also gets a first-class citizen role. In this category, leading organizations automate end-to-end tests at the process or transaction level. Automating these types of tests is not easy and advanced test automation tools are required because speed matters. The choice of the testing technology is important because achieving and, more importantly, maintaining high levels of automation is crucial. Therefore, the more tests that are automated, the better and so 70% of leading DevOps teams prioritize the percent of automated end-to-end test cases as a top metric.

Many of the other metrics that leading firms rank as important in this category are quantitative and measure coverage of functionality vs. requirements (70%), number of test cases executed (65%) and total number of defects identified during testing (66%).

For quadrant mappings, metrics details and additional analysis, see the complete 55-page report, “Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics — Ranked by Industry Experts.” The report is available for free on the Tricentis web site.

Feature image by Frank Winkler from Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Tricentis.