The Git Vulnerability and its Aftermath – InApps 2022

To explore the response to the recently-disclosed Git security vulnerability (which we wrote about here) and to provide some context for it in a world of imperfect code, InApps Founder Alex Williams called upon Tal Klein of Adallom and Bryan Helmkamp, CEO and founder of Code Climate, for this episode of InApps Analysts.

For more episodes, check out the podcast section of InApps.

#27: The Git Vulnerability and its Aftermath

Listen to all TNS podcasts on Simplecast.

Bryan refreshes us on the nature of the Git vulnerability: “It allows an attacker who has control of a Git repository to execute arbitrary code on the client machine of anybody connecting to that Git repository with a vulnerable version of the Git client.”

Tal is not at all surprised by this news: “Vulnerabilities are going to happen; there’s no such thing as perfect code,” he says. “Git was another popular attack vector for the Shellshock vulnerability,” says Tal, describing Git as the perfect candidate through which to attempt to obtain privileges to escalation. “It’s actually the second scenario in which Git itself becomes an attack vector,” he says.

Most importantly, this vulnerability appears to have been discovered before it could be exploited in the wild. “I’m always relieved when the CVE is published by researchers rather than after a breach,” says Tal, meaning the good guys got to it first.

Bryan agrees on the importance of this up-side. “GitHub actually ran some analysis of all of the source code repositories that they host, and my understanding of their conclusion was that they have no evidence of any malicious Git repositories on GitHub at this time, and they’ve also added protections so that nobody could plant a malicious repository on GitHub,” says Bryan, calling this a good indication that the problem was identified and that fixes could be implemented before exploits resulted.

Alex asks, “What are the sources of these vulnerabilities? Are these just gaps that no one has ever seen before, or are they gaps that are created?”

“In this case, it was effectively an interaction between two behaviors: the behavior of the Git program, and the behavior of case-insensitive file systems,” says Bryan. “Most Linux developers,” Bryan continues, “use case-sensitive file systems. However, most Macs and many Windows machines do use case-insensitive file systems.”

“The vulnerability may be much less obvious to somebody who’s developing on Linux, and that is the primary operating system used by the Git core team. But there’s subtle behavioral interplay between the underlying file system and the program itself.”

Bryan expects to see a string of follow-on, ripple disclosures and patches. “I usually recommend people keep their ear pretty close to the ground for a few weeks after something like this for any additional patches.”

Alex then asks Tal to provide Adallom’s perspective on the science of vulnerabilities.

“It’s important work,” says Tal. “I think GitHub should reward their researcher who discovered this. I think they’ve done a great service to the community. We’re very supportive of it and we think that this needs to be a standardized process where people should not only feel supported by the community to reveal these kind of things, but are actually financially rewarded for doing so.”

“It’s better when a zero-day stops being a zero-day as a result of research rather than as a result of threat actors,” says Tal. “This is the system at work. From our perspective, the more people pay attention to the underlying behavior of code, especially code that’s so vastly popular and utilized, the healthier everybody is.”

“The security of open source is entirely dependent on the community participation and the development of the code. If you have open source code that is fairly obscure or doesn’t have enough eyeballs on it, then the likelihood of somebody malicious being able to find something is much greater, because the ratio of eyeballs to code is very relevant,” says Tal. “There’s a magic number to lines of code and participants in a project.”

If this were a closed source code management system, says Bryan, and with no evidence of a breach in the wild, it’s likely that the product would see a point release and that no one would be much the wiser for it. He suggests that good processes for patches and for notifying the community are the best pro-active steps for any system.

Again citing the Shellshock vulnerability in Bash, he says, “That was around for a long time, and there’s no mailing list of everybody who uses Bash. We got lucky here with GitHub, because they have such a visible position in the community.”

The GitHub vulnerability is more limited in scope in comparison to Shellshock, says Bryan. “It’s not as exploitable in a passive way. You have to make a connection to a service.”

Alex asks what are the critical things that users need to do right now to close the window for good on this vulnerability?

“Upgrade to a safe version of Git,” Bryan answers. “The Git release team did what looks to be a very good job of back-porting the fix across at least four different versions of Git, so even people who are using an outdated version of Git can get the security of the patch without needing to jump to an entirely new version.”

“I posted some instructions on Twitter for Mac users that are using the Home Brew package management system,” says Bryan. “It’ll vary a little bit depending on your installation mechanism.”

“In cases like this, I strongly recommend people patch hosts that they don’t believe are affected, just for safety’s sake,” says Bryan, referring to perhaps forgotten or unused servers on which Git is installed.

Tal wonders whether there is an easy way to scan networks or infrastructure to determine what versions of Git are installed?

“With servers, I typically just use the package manager as much as possible,” says Bryan, adding that automation tools do exist that can assist with quicker reaction, which is key. What is Bryan’s advice if you don’t have automation going right now? “It’s not the time to wait to get that going before patching.”

However, says Tal, “the most dangerous time in a vulnerability’s life is not before, but after it’s been disclosed.”

“The attack surface has moved from the sophisticated attacker to an unsophisticated attacker,” Tal says. “There’s lots of people who can, without having to do a lot of research or a lot of work, try to exploit this wherever they can.”

“The system we have right now could be improved greatly, and that’s one of the reasons I’m very excited about Docker and CoreOS,” says Bryan. “They’ve done a lot of work to shrink down the amount of software on a base CoreOS system as much as possible to reduce that attack surface.”

“The safest code is no code,” Bryan continues. “The safest package is the one that you don’t have installed.”

Bryan is excited about the way CoreOS has created a minimal base system with simple automatic upgrades. “Then,” Bryan says, “when you’ve got a case like this, where suddenly everything needs to be upgraded, hopefully your base operating system can just automatically roll forward in a very graceful way, or hopefully it wasn’t vulnerable in the first place.”

In a framework like Rails or Node, some vulnerabilities might require the developer to actually think through how they’re going to patch and cut a new version of their application, says Bryan, “and that process can take a good amount of time.”

“Some of these new deployment systems being pushed forward are going to help with these cases a lot,” he says.

Tal agrees, but with some reservations. “A lot of times, Docker shies away from accountability,” says Tal, observing that when Alex asked Docker about two recently-announced Docker vulnerabilities, “their response was, ‘yeah, but ultimately it’s not Docker’s fault if there’s a vulnerability in the application or the operating system.’”

“Obviously containers are the way forward, without a doubt,” says Tal. “I completely agree that the reduction of attack surfaces is a huge benefit, but somebody needs to take ownership of at least insuring that these new mechanisms don’t introduce net new attack vectors.”

Alex agrees that Docker should be responsible for driving fixes, but he concedes that such heroics are not always in Docker’s best interests, considering that they are occupied with improving and exporting their own enabling technology.

“I always try to ascribe good intent,” says Bryan, “but the (Docker) project is moving very fast.”

“More code, and more code-changing, means more implicit behaviors that are created in the interactions between components,” says Bryan. “In this case, the Git vulnerability was the boundary between the file system and Git.”

“Docker deals with a lot of different components, and a lot of different technologies, and there are a lot of boundaries kicking around there,” Bryan says, adding that he expects significant vulnerabilities to emerge as these newer, container-based technology projects receive more scrutiny. Bryan believes that the maintainers for those projects should take a strong position for eradicating vulnerability, by working with the community constructively and prioritizing the extra labor required for proper releases and fixes, “even though they have a commercial interest in driving forward new features.”

Alex wonders what actions can be anticipated on the part of potential attackers in the wake of this latest news?

“If I’m an attacker and I’m looking at this,” says Bryan, “I’m adding this to my bag of tricks to springboard other vulnerabilities into more privilege,” adding that this is where attackers excel. “Once somebody’s on your network who’s not supposed to be on your network, or running code on a host,” says Bryan, “it’s pretty much game over.”

“Threat actors essentially create Rube Goldberg devices,” says Tal. “Most attacks are usually looking for a pinhole. As long as you have a way in, then there’s a whole bunch of other tools at your disposal to cascade the attack; they call it the ‘long tail’ of the attack.”

Tal explains that an attacker is basically trying to get in, and the easiest way is to compromise a privileged user, either by trying to phish them, or to get them to download a malware package.

“The biggest attack surface of any infrastructure is the end user,” says Tal. “Once you get in, there’s very few defenses that exist that are designed to stop somebody who should have access to something from accessing it.”

Image via Wikimedia Commons.

Adallom is a sponsor of InApps.

InApps is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Docker.