The Necessary Cultural Evolution Driving Continuous Delivery – InApps 2022

CloudBees sponsored this post.

As continuous delivery — also known as continuous integration and continuous deployment or CI/CD — becomes the modus operandi for speeding software delivery in digital organizations, we’re seeing cases where companies will invest a whole lot of money to mount CI/CD tooling and processes only to see staff turnover rise and productivity stall. It’s because the people weren’t included in this change.

Today we’re going to discuss the kind of culture you need to embrace with your employees, your teams, and your full organization before, during, and after you launch continuous delivery.

CI/CD Has to Be a Communicated Cultural Change First and Foremost

Tom Petrocelli, research fellow at Amalgam Insights, told InApps: “It’s not really a matter of architecture that drives the need for the culture change. It’s business demand… to meet market needs more quickly.”

“You have to do the culture changes before you start doing the technology. I absolutely reject the idea that you adopt the technology and it makes the culture changes. The tech fails because the org wasn’t ready for it.” — Tom Petrocelli, Amalgam Insights

If DevOps is just a multifunctional team that cuts across silos, you first have to disrupt the culture.

“People like silos because it tells them what they are doing and who they are and it gives them comfort and it gives them power,” Petrocelli said. “A lot of power structures are there to keep people doing what they are already doing in place.”

He continued that new technology comes with an implied threat that someone could lose their job. You need to gently roll out any of these cultural changes in a way that describes the value. Particularly highlighting the value to their current position.

DevOps is a great idea in theory and a hard reality. Petrocelli said lot of companies put in a CI/CD pipeline and say they are doing DevOps.

“No you’re not. You’re doing automation,” he said. “And when the teams aren’t working and nothing is really coordinated it’s because you didn’t have DevOps in place, you didn’t have the culture in place, you just had automation in place.”

Part of the problem is that DevOps and CI/CD is too top-down mandated. And, as Petrocelli said, the stakeholders that are sold on the idea are usually the stockholders, not the employees.

And of course, each employee stakeholder has a different way of seeing it. Developers tend to embrace CI/CD and DevOps fairly quickly because they get to move more independently to serve customers faster.

Petrocelli says that if you can tell operations that having coordinated teams means they can solve problems faster — and that they won’t be up late figuring out what goes wrong — you are more likely to get them on board.

Most importantly, he says to make sure the different cross-functional teams sit together and learn from each other.

Technical Performance in Continuous Delivery

Dave Farley is co-author of the book with Jez Humble that made continuous delivery popular, “Continuous Delivery.” The theme of this book — and one that Farley still says holds strong ten years later — is that continuous delivery — the combination of CI/CD and the culture and processes needed — is a genuine engineering-driven approach to software.

“Continuous delivery is about establishing mechanisms and cultures that allow us to work more experimentally so that we can learn more effectively.” — Dave Farley, Continuous Delivery

He told InApps that for most of software development’s history we’ve been applying production-line thinking — in the form of Waterfall project management — when we should be applying scientific and mathematical principles.

“The problem that we’ve been trying to fix is that nearly everything else we do as a species is making a physical thing. The design is important, but also how do I production-nize that understandability in process and repeatability,” Farley said.

He contrasted that “software development is never like that. We never had a production problem. Our problem is always that of exploration and design.”

Farley said this misconception exists because most software development isn’t treated like an engineering discipline. He says it’s based on guesswork:

• We guess about the architecture
• We guess what scalable and resilient means
• We guess if our software has bugs in it before we go into production

He said that the objective of software development is constantly optimizing learning and exploration. He pointed to the scientific method as a way to do less guessing and more measurement:

1. Characterization — look at the problem and try to figure out what you’re observing
2. Hypothesis — exploration that will describe
3. Design an experiment — to verify or falsify the hypothesis
4. Experiment and look at results

Farley offers the example of automated tests. With these we measure the behavior of our software for performance and resilience.

He explained, “In order to be experimental, you control the variables — you automate all the configuration and deployment of our systems — so we eliminate whole classes of failures. And then we optimize to make it as fast and efficient to get very high-quality feedback.”

This gives CI/CD teams an experimental platform to try out on. Then the business can carry out those ideas and experiment with them in the market.

“Continuous delivery is a technical approach. We automate all the drudgery and we apply the human creativity to come up with clever ideas.” — Dave Farley, Continuous Delivery

This is what Farley calls technical performance. His definition of continuous delivery also includes cultural and organizational performance.

Cultural Performance in Continuous Delivery

Following the Scrum recommendations, CI/CD also needs small teams — five to nine people — so, as Farley says, you have the people working very close to the ideas. These small cross-functional teams need to have a holistic understanding of the piece of the code they are sharing, and need solid, visual communication that enables playing with code.

Farley said this is facilitated through test-driven development (TDD), pair programming, and pair rotations. He says the most successful organizations he’s worked at saw people arguing at whiteboards all the time.

He cautiously described these teams as “intellectually disciplined in the way they approach problem-solving,” with the constant refrain of:

• That’s an interesting idea, how would we test that?
• What sort of experiment would we try to verify or fail that?

This is in contrast with a lot of organizations where the highest paid person is in charge of guessing the solution. In CI/CD, you try to move away from guesswork and into exploration and discovery.

“In the teams where I felt like we’re doing best, we got to a meritocracy for ideas. There was less personal ownership about ideas. Whoever came up with the idea, anyone can test it,” Farley said. “To outsiders, it might seem like an argumentative culture, but the most junior member of my team could dispute my idea.”

Farley said cultural performance in continuous delivery comes from collective ownership and responsibility. No individual is responsible for a release, tracking, or the fixing of a feature. The whole team is responsible.

This falls well in-line with the self-organizing principles of agile software development.

Farley said this can also only be achieved with truly cross-functional DevOps teams, that retain the ability to design, build, test, deploy and manage their software in production.

He said that pair rotation was essential on all these teams. This is an offshoot of the agile practice of pair programming where two developers sit and work at one workstation on one computer, switching off on roles of writing and review code. Pair rotation sees these pairs constantly changing so empathy and collaboration is fostered among all team members — even ones that initially don’t get along. If a feature carries over to the next day, one developer would come off it, and another would join.

“Within a given sprint, everybody worked with everybody else and everybody worked on the features that would be released at the end of the sprint,” Farley said.

He continued that this process allows teammates to learn learn more quickly.

“Pair rotation builds the culture of the team. The team gets to know each other better and trust each other better. You get to understand strengths and weaknesses.” — Dave Farley, Continuous Delivery.

Organizational Performance in Continuous Delivery

Now both of these all sound well and good, but neither this cultural or technical change can happen without top-down buy-in. How you organize teams to give them autonomy?

Farley said, “In a perfect organization, any given requirement would have a natural team — decompose your org so people can do work without having to ask for stuff. No overhead, meetings and coordinating activities.”

He said this doesn’t necessarily mandate an adoption of microservices, but microservices certainly is the most popular approach that leads to autonomous, cross-functional teams.

Farley said you can still achieve continuous delivery with a shared codebase. You just do a functional decomposition of teams, rather than a technical one. Instead of technically aligned teams like a database team and a user interface team, you’d drive development around the user’s point of view, like a shopping cart team and a playlist team.

Farley warned that breaking up teams along technical boundaries means that you’ll always be in meetings. By breaking up teams around use cases, it frees you up to evaluate if things actually work within your team’s domain.

GuardRails not Anarchy

When we talk about autonomous teams in DevOps and CI/CD organizations, it should be clarified that the best success comes from teams that work independently, but within the boundaries of the organization and industry. In regulated industries that work with financial or medical compliance, that’s pretty straightforward. In fact, a new generation of tooling is being released to automate a lot of the security and compliance enforcement, detection and response.

But how does this work on regular organizations? CircleCI CEO Jim Rose told InApps that companies are finding they’ve gone too far with the freedom of tooling choice. After the relatively rapid transfer away from a big monolith with one standard framework and database, the process of coordinating how my service works with your service within modern microservices architectures is adding back the overhead decision making and slowing down processes — “because none of us knows what the other is doing.”

Rose said he is now witnessing companies enact what Netflix famously dubbed guardrails not gates.

“The way that teams are starting to handle that is through what we call the Team Back Plane — they build all the standards and tools and all the templates that everyone else uses,” he explained.

This is done on the Engineering Productivity Team at Google and on the Developer Acceleration Team at Shopify. They consolidate the tooling choice down to a package that is easy for developers to use and that eases the integration process with the whole system.

Rose says teams see a burst of productivity when they get started with CI/CD but then it slows down when you have seven different versions of 18 different kinds of software and databases working asynchronously.

He says this drives a secondary consolidation.

“What we see as best practice in some of our really advanced customers. What they actually create is instantiating the entire toolchain you need when you start,” Rose said.

For his customers, this includes a shared code repository, a CircleCI pipeline that dictates all the rules, and a place in the cloud where they can run their services with all the necessary keys and secrets.

Rose said that “Devs, as they are getting more cloud-native, are getting farther and farther from compute and infrastructure — they want to worry about their app, not the Kubernetes base.”

By setting up these guardrails early on, they can get their pieces of the CI/CD pipeline in minutes.