How DevOps Quality Trends Vary Around the World – InApps Technology 2022

Tricentis sponsored this post.

Delivering high-quality software fast is no longer a luxury — it is a prerequisite for digital success and mandatory for customer loyalty and acquisition. We are in an era where customers are empowered to choose and switch to the company that can provide them the best experience quickly and at minimal cost. Organizations that cannot keep up with their customers’ expectations and adapt to new business paradigms will find themselves losing customers and business to competitors that can.

How can firms deliver with quality at speed? Implementing Agile+DevOps best practices and building in quality is the first step — and this is not trivial. Automation is the lever that development teams pull to go faster. However, if not properly managed, automation can jeopardize quality and increase risk. To ensure that quality issues and unacceptable risk do not negate the velocity benefits of automation, firms must ensure they are tracking the right metrics to measure quality throughout development-testing-deployment.

Tricentis recently commissioned Forrester to research how 75 quality metrics are used and valued among 603 DevOps leaders and laggards. The goal: provide the software testing community an objective perspective on which quality metrics matter most for DevOps success.

In past columns, we’ve provided a high-level overview of the study’s findings, and shared the DevOps quality trends across build, functional validation, API testing, and end-to-end testing. For this article, let’s take a deep dive into how the quality metrics usage and valuation varied across regions.

Global Results

To recap the global findings that we covered in previous articles, the following metrics were ranked as the most valuable by the DevOps experts who measure them:

The three key takeaways from the global results:

• Understanding of business risk is a critical factor in DevOps success. Once organizations reframe the way they think about risk, they also alter their quality metrics to help them better understand the level of risk in their release cycle.
• DevOps experts focus more on contextual metrics (e.g., requirements coverage, risk coverage) while others focus on “counting” metrics (e.g., number of tests).
• DevOps experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.

Ultimately, this underscores the fact that DevOps success requires much more than increased automation and a shiny new toolset. A broader transformation is required to align on business risk and release with confidence. It’s not easy. But, the effort truly pays off in terms of enabling the team to deliver better software faster.

Europe DevOps Quality Metrics Trends

Looking specifically at Europe, the “Top 20” changes as follows:

Some interesting trends in this region include:

There Is a Greater Commitment to Measuring Quality Metrics

European respondents reported a higher level of DevOps quality metrics measurement across the board. For almost all metrics, the usage rate was at least 6% higher than the global average. For metrics related to time, coverage, risk, effectiveness, and efficiency, the usage rate was over 14% higher. This speaks to European organizations’ commitment to scrutinizing and continuously optimizing their quality processes — especially in terms of time and resource utilization.

Risk and Coverage Metrics Are Valued More than in the Global Average

European respondents also ranked risk and coverage quality metrics a surprising 21% higher than the global average. This could be related to the fact that the respondents from this region came primarily from the financial services and insurance sector, with health care and government close behind. In such highly-regulated industries, measuring and mitigating risk is certainly a core concern. This finding could also indicate that European organizations place a greater emphasis on protecting the corporate brand.

Test Data Preparation Time Seems to Be a Greater Concern

European respondents were more likely to measure (+18%) and highly-value (+23) time spent preparing test data than their global peers. Given the restrictions that GDPR placed on test data as of May 2018, it seems likely that European organizations have significantly changed their test data management processes (e.g., to masking and more synthetic test data generation), and are cautiously monitoring how the changes are impacting their overall efficiency.

Asia-Pacific DevOps Quality Metrics Trends

Now, let’s shift focus to Asia Pacific. The following 20 metrics were ranked as the most valuable by the Asia Pacific DevOps experts who measure them.

Notable trends in this region include:

End-to-End Testing Metrics Are Valued — and Measured — More than in the Global Average

Although Asia Pacific respondents measured fewer Build and Functional Validation metrics than the global average, they measured (and valued) End-to-End testing metrics much more than their peers around the world. For example, Percent of Automated End-to-End Tests was measured by 47% of the organizations (vs 36% globally) and highly-valued by 84% (vs 70% globally). Risk Coverage measurement was significantly higher; it was measured by 49% (vs 34%) and highly-valued by 71% (vs 59%). This speaks to the region’s focus on digital transformation and commitment to delivering exceptional user experiences.

API Testing Metrics Were also Valued — and Measured — More than in the Global Average

Asia Pacific respondents also measured and valued API testing quality metrics more than the global average. Overall, API quality metrics were measured by 16% more organizations in this region than globally. The highest valued API quality metrics were API Test Coverage (63% vs 39% globally)  and API risk coverage (79% vs 62% globally). This prioritization of API testing is likely a side effect of the regional trend towards API-driven open banking (the vast majority of respondents indicated they were in the Financial Services and Insurance sector).

There Is a Greater Leader/Laggard Quality Metrics Measurement Gap

Part of the study involved classifying the respondents as either DevOps leaders or DevOps laggards, based on their responses to various questions about the maturity of their processes. Although the percentage of DevOps leaders in the region was lower than the global average (18% vs 26%), the DevOps leaders from Asia Pacific generally measured quality metrics at a comparable rate to their global peers. However, the DevOps laggards in Asia Pacific generally measured quality metrics at a much lower rate than their global peers. This suggests that the select set of firms that have truly prioritized DevOps initiatives have made great strides — and the laggards have a lot of catching up to do in order to remain competitive.

For more details on the study’s methodology and findings, see the complete 55-page report: Forrester Research on DevOps Quality Metrics that Matter: 75 Common Metrics — Ranked by Industry Experts. The report is available for free on the Tricentis website.

Feature image from Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Tricentis.