‘Jenkinstein’ and a Cloud Native Jenkins – InApps 2022

“What if Jenkins stopped being a snowflake?” asked its creator, Kohsuke Kawaguchi, arguably the architect of whatever snowflake his automation platform may have become.

“Meaning, you understand every part of how Jenkins’ configuration is put together,” he continued, “because there’s a record of who made what configuration change, when, and for what purposes. So every time you’re making changes, you know that you are not breaking some piece of the platform underneath, and you don’t have these pockets of things that you don’t understand.”

As the CloudBees CTO told attendees during the Day 1 keynote session at DevOps World | Jenkins World 2018 (historically also known as Jenkins World), there’s a place in certain organizations for a cloud native automation platform. But “cloud native” is not some type of “journey” that all companies are inevitably bound to take, just so they can use the platforms at the end of the cloud native vendor’s rainbow. Sure, no single platform suits everyone out-of-the-box in a one-size-fits-all scenario (if it could, there’s a good chance Jenkins would never have been necessary). Adaptations, both structural and cultural, will be necessary.

But as the success of containerization has demonstrated all too clearly, adaptability is rendered much simpler when the core — the part that must remain integral — is decoupled from the parts that require the adaptation. That may not have been the case for Jenkins in the beginning. Yet it certainly will need to be the case now.

It’s such a nuanced message from Kawaguchi that it may be easy for some to mistake it as a call to split Jenkins up.

Automation Without the Configuration

“Jenkins Evergreen is an effort we started at the beginning of this year,” said Kawaguchi, “because continuous delivery keeps getting more and more mainstream, and becoming more and more a daily part of the developer’s life. What that means is, the bar of simplicity, ease of use, ease of administration, is rising rapidly… With Jenkins Evergreen, what we are trying to do is to give a pre-assembled Jenkins distribution with batteries included, so to speak.”

The Evergreen distribution is an effort to make Jenkins self-updating. You might think a tool positioned in the market as a continuous delivery engine would have the least difficulty providing essentially the same services for itself.

The issue here is architectural: Plug-ins can change a platform into a unique different order of beast, a species unto itself. You could devise some kind of configuration platform (perhaps with a little butler for its logo) to specify how this beast may be taken apart and reassembled, which a self-updating platform would have to do. But automation is about devising repeatable processes, and if you can’t repeat the process of updating the platform across all the deployed instances of that platform, there really isn’t any point.

Ideally, Kawaguchi suggested, a Jenkins Evergreen user should never have to know how updates to itself are delivered, or even when, unless she looks into it herself. What’s more, now that organizations are settling on how they deploy Docker containers and orchestrate them using Kubernetes, it becomes more feasible to leverage this componentized system to rein in a broader base of core automation functionality — one that more clearly decouples the core from the adaptations.

“It’s great that Jenkins has 1,400 plugins, but you’re not supposed to install 1,400 plugins”–CloudBees CEO Sacha Labourey

Kubernetes may be giving rise to a new class of workload-centered user who has simpler, more concrete expectations of their automation platform. One such platform can then be geared toward those expectations, providing this class with what Kawaguchi described as a kind of guided, guardrail-equipped path to success, without so much time spent in configuration. The analogy Kawaguchi chose this time was comparing the act of booking a commercial airplane flight against learning to fly a plane.

“This idea of Jenkins upgrading itself might feel a little scary to many users,” said the CTO, “and to those of you who have been using Jenkins for a while. But I do believe that this is a better way to deliver a better Jenkins faster… When we roll out a new change, and we notice that something bad is going on, then we can detect that probably before most of you get to notice that. And then we can take corrective actions, and your instance can keep running.”

The Jenkins Guy

It’s becoming typical within organizations, said CloudBees CEO Sacha Labourey later in the keynote session, for all of their software deployments to depend on a narrow deployment channel. Acting as the gatekeeper for that channel is someone Labourey described as “the Jenkins guy… this superstar devoted to making Jenkins great on his team.”

Because this person is the authority on deployment within his organization, multiple teams come to rely on him to meet their scheduling goals. Yet this leads to a technical issue that few folks outside of IT operations take time to consider: The Jenkins Master (not to be confused with the “Jenkins Guy” himself, but rather the server managing a distributed scheme with multiple agents) becomes bloated, like an old telephone directory or the Windows XP System Registry.

And because that organization’s Jenkins deployment is not only dependent upon the Guy, but somewhat bound to his choices of plug-ins, the result is what the CEO called “Frankenstein Jenkins,” and what other developers and engineers at the conference Tuesday had dubbed “Jenkinstein.”

“It’s great that Jenkins has 1,400 plugins, but you’re not supposed to install 1,400 plugins,” the CEO said, wearing his trademark, glossy teal sneakers. “It’s a big no-no, and yet we see it everywhere… This Jenkins becomes bloated, slow to start. When it crashes, it takes forever to start. Hundreds of developers are pissed. And nobody wants to fix it, because if you touch it, you own it, right?”

Kawaguchi presented some snapshots of a concept for Jenkins where launching a configuration file effectively spawns the container, places it in operation and establishes a best fit for the desired configuration of that container.

For such a version to evolve into a “cloud native Jenkins” deployment — one that could conceivably advance the goals of what’s been called “cloud native DevOps” — it would actually need to advance several steps further.

“What if, when you start running Jenkins, you don’t really have to do any up-front provisioning of the compute, storage, or anything?” asked the CTO.  As different teams added to the automation workload, it could simply scale itself out?

Ephemeral Jenkins

In a separate session Tuesday, three CloudBees engineers who contribute to the Jenkins Cloud Native SIG made astonishingly clear what should have been obvious to anyone pondering the proper definition of “cloud-native DevOps:” The components of a modern cloud-native application are microservices, which means they’re ephemeral in nature. Sustaining and maintaining their existence would run contrary to the whole point of their design. DevOps isn’t really about the lifecycles of these micro-organisms. They’re all part of a broader goal, a bigger application — and that’s the actual product of DevOps processes.

“What we want to achieve,” explained CloudBees principal software engineer Carlos Sanchez to attendees Tuesday, “are things that today seem, for somebody who is totally new towards, doesn’t know, Jenkins at all, but comes from Kubernetes, new technologies, new stacks: things like infinite scalability. It’s something that’s possible today, if you architect your things right. Multitenancy, multiple Jenkins builds happening at the same time, without interruption from another plug-in killing your master and screwing everybody’s builds.”

Sanchez and CloudBees colleagues Jesse Glick and Oleg Nenashev spoke of a vision they called “ephemeral Jenkins” — not short-term in itself, but comprised of ephemeral components like the microservices-based applications they would serve. Historically, Jenkins agents have been tied to a single master. In a distributed system of hundreds of instances, a traditional deployment would soon become untenable. But in this model, the master would become virtualized, in a sense. Agents would continue to perceive a single master, while on the other side of the abstraction, Amazon S3 storage is being automatically provisioned and scaled to store images that collectively form the image, if you will, of the master.

[left to right: Carlos Sanchez, Principal Software Engineer; Jesse Glick, Developer; Oleg Nenashev, Principal Software Engineer, CloudBees]

“Essentially, the problem is that currently, Jenkins by default is going to store practically everything related to what it does inside this one gigantic directory called ‘Jenkins Home,’” explained Glick. “That includes all kinds of configuration, various kinds of runtime state, all of your build records, including metadata about files that they produced, logs from all those builds, test results, fingerprints of files — a whole long list of things… If you’re unlucky to have this on a network share, you’re going to definitely have some performance problems in terms of transferring that.  So there are a lot of scalability issues with it.”

Architecture First

As it turns out, implementing what some have called “cloud-native DevOps” does require another architectural shift for software, perhaps before we even begin to consider a cultural shift for organizations and their people. We asked Kohsuke Kawaguchi, is there a concept of an organization that, because of its structure, is better suited to a cloud-native implementation of automation than any other kind.

“If I think about this like a broader push toward more automation, like a century-long [benefit] to mankind,” the CTO responded, “the context at that level would be, what cloud-native is allowing software development to do is just more sophisticated automation at the lower level, that can be harnessed to greatly advance things like scalability. But in order to take advantage of that, it dictates a certain kind of software architecture to really deliver that underlying capability. So if you have a 20-year-old app that’s running on AWS, that’s not really leveraging the underlying platform capabilities.”

Younger companies, Kawaguchi told InApps, may have an easier time devising processes that are better suited to cloud-native automation. For one thing, there’s a greater chance that the engineers originally hired to design those processes, are still employed there. But older organizations face an evolutionary challenge: not to become replicas of these newer ones, but instead to devise processes that keep their processes relevant.

Kawaguchi’s complete interview can be heard in an upcoming edition of InApps: Makers podcast.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.