Even More Problems with GitOps – InApps Technology 2022

Codefresh sponsored this post.

GitOps is an emerging way to manage the actual state of systems. But while GitOps as an idea is great, we are not even close to having that idea be useful in a practical sense. There is still a lot of work to be done.

In two previous posts, I explored a number of initial issues around the current practice of GitOps — such as it being too often thought of as only a way to manage Kubernetes deployments, and that GitOps principles often can not even be applied to GitOps tools.

In this post we will discuss the problem of no well established patterns, and the connection between continuous delivery (CD) and GitOps.

There Are No Well Established Patterns

The general principles of GitOps are well known, so I will not repeat them here. They are well established, even though many vendors are trying to “bend” them to fit their products. What is not well defined, however, are the practical implementations:

• When is it more appropriate to use the push mechanism and when to rely on pulling?
• How should we define permanent environments, and should they differ from temporary preview environments — like those related to pull requests?
• How should environments be organized?
• Should we use separate repositories for each environment, or rely on monorepos, or use branches, or something completely different?
• How do deployments based on changes in Git repositories fit into broader continuous delivery pipelines?
• When should we let the system synchronize itself, and when should we employ manual confirmations?
• Where should we perform those confirmations?

And so on and so forth. There are many unanswered questions and quite a few “good practices” that are yet to be defined.

I cannot say that no one has defined any “good practices.” But most of the time, those practices are tied to specific implementations that are not applicable anywhere else. Flux, for example, is very opinionated. That can and should be considered a set of good practices. But they do not work anywhere outside of Flux. Heck, those practices are challenging to implement to anyone but those starting from scratch. Similarly, Jenkins X made quite a few strides in defining an opinionated workflow that can also be considered a set of best practices. Yet, just like Flux, it failed to define a “standard” that can be followed anywhere but inside of Jenkins X and by no one but those who are starting from scratch and do not have any existing definitions. In other words, it was so inflexible that teams with existing systems could not use it.

We need GitOps patterns that can be universal and that can be applied no matter the tooling; or, to be more precise, that the tools can adopt and create their own implementations on top.

The Connection Between Continuous Delivery And GitOps Is Not Yet Well Established

I dare you to create a continuous delivery pipeline that follows GitOps principles.

Now, you might say that it is not a big deal. You might even create one relatively quickly. But, if you do, chances are that it will either not follow GitOps principles and is a half-baked solution that will need to be redone a couple of times, or that it will fail miserably and make you feel silly.

The problem is that most CD pipelines are not based on GitOps principles, because most of the tools in the market were not based on them. Most of those currently in use are going in the opposite direction. Most assume that pipelines should interact directly with the clusters, or rely on other tools that do that. That is wrong. It is in direct opposition to GitOps, which assumes that every change to the actual state is preceded by a change to the desired state stored in Git. What that means is that it is OK for CD pipelines to build artifacts, to create releases, to run tests, and do quite a few other tasks. But, when it comes to changing the actual state, CD pipelines should be pushing changes to Git and not changing our systems directly or indirectly by invoking other more specialized tools. Heck, pipelines should not be interacting with clusters, except those dedicated to building and running tests.

The problem is that most CD pipelines are not based on GitOps principles, because most of the tools in the market were not based on them.

More often than not, you are left on your own when trying to apply GitOps principles to CD pipelines. You’ll need to clone environment repos; add, modify, or delete files; change some parameters and arguments; create pull requests (PRs) or push changes directly to the mainline; and so on and so forth. Can all that be scripted? Sure, it can. Should CD tools provide a means for users to avoid reinventing the wheel? Definitely.

Unfortunately, apart from a few exceptions, that is mostly missing in CD tools. They are not designed around GitOps principles, and so they are having difficulties adapting to them.

… And The List Goes On and On

Do not think that we are finished with GitOps issues and misunderstandings. The list could go on for a while, so I’ll cut it short with only a few more mentions — like the difficulties to use it at scale, problems to fit secrets into GitOps principles, and so on.

There is a long road ahead of us. We’ve come a long way since the inception of the idea, but we are still far from the final destination; and we are not even sure whether we are going in the right direction. This might be the time to hit the reset button and start over and build something much better based on the experience we have.

GitOps is Bad?

Does all this mean that GitOps is a bad idea or that we do not yet have the tools that will allow us to implement it? Not at all. GitOps is a great idea. It existed long before the term GitOps was even invented.

We have been using the principles behind GitOps for a long time now. In a way, any tool that can react to Git webhooks or monitor Git for changes can be a GitOps tool, as long as it uses the information from Git to change the actual state. There are so many tools that fit that description that I will not even bother trying to list them.

That being said, there is a lot of room for improvement. Even though the existing tools can be used to apply GitOps principles, we can and we should work on improving them. Even though GitOps principles have existed for a while, there is still a lot that can be improved.

If you are interested in GitOps and continuous delivery, please try Codefresh. You’ll get free unlimited builds for life.

Feature image via Pixabay.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.