The Challenges of Securing the Open Source Supply Chain – InApps Technology 2022

As organizations shift more resources to the cloud, some security risks increase: misconfigurations and cloud supply chain insecurity.

Breaches caused by infected IT management software from SolarWinds and Kaseya are among the most notorious recent examples of supply chain insecurity. But it’s the chain of code used to build cloud infrastructure that’s increasingly at risk, driving the move to shift security left.

Open source software can be a weak security link in that chain. According to a report published in September by Sonatype, as threat actors move upstream along the chain to infiltrate open source software, “next-generation” software supply chain attacks have risen by 650% within the past year.

These attacks are more easily scaled, letting cybercriminals distribute malware throughout the supply chain to wreak maximum havoc.

What’s often overlooked in discussions of supply chain risk is that “attackers don’t necessarily modify source code repositories to facilitate these breaches,” as Matthew Chiodi, vice president and chief security officer, cloud for Prisma Cloud by Palo Alto Networks, wrote in a September report from the company’s Unit 42 threat research team.

“They don’t have to,” he added. “They find weaknesses in the software development pipeline and attack those.”

Chiodi, who led the team that created the report, makes it clear that open source is not the problem.

“In fact, using open source is, in my view, a requirement for innovation,” he said. “Anyone who says their company can’t use open source should likely be looking for another job, as they’re sealing the fate of their company from an innovation standpoint.”

Almost all modern cloud native applications are developed using open source components, including almost any software as a service (SaaS) platform. The problem is, no one is tasked with maintaining or securing that code.

“Many new cloud features and services, including SaaS applications, are launched every day,” Chiodi told InApps Technology. “Yet some popular open source modules used in these applications have not been updated for years. And that’s the crux of the problem.”

Vulnerabilities in Open Source

The Unit 42 report was sparked by a Cloud Native Computing Foundation (CNCF) white paper on software supply chain best practices. This paper found that 98% of all code bases include open source software, and 92% of those code bases contain outdated or vulnerable code ripe to be exploited, said co-author Andres Vega, CNCF tech lead for the Security SIG and a member of the Security Technical Advisory Group.

In its own research, Palo Alto Networks Unit 42 team found that 63% of third-party code templates used in building cloud infrastructure contained insecure configurations, and 96% of third-party container applications deployed in cloud infrastructure contain known vulnerabilities.

“Proactively addressing these threats is of the utmost importance,” wrote Chiodi.

Most modern cloud native applications are developed using infrastructure as code (IaC), and multiple different components are deployed. This process, Chiodi noted, depends on multiple IaC packages and typically Helm charts, a package manager for Kubernetes applications.

Each Helm chart, in turn, depends on multiple container images, and each of those images depends on multiple application packages.

“In other words, each of those modules that a developer might be bringing in depends on other modules, so there’s this chain of dependencies,” said Chiodi, which is responsible for most security issues.

There’s a direct correlation between the number of dependencies and a higher chance of misconfigurations, Chiodi told InApps Technology. For example, with one to 20 dependent packages for container images, you’ll have on average 22 vulnerabilities, and with over 100 dependent packages, the number jumps to an average of 515 vulnerabilities.

The Unit 42 team used Bridgecrew‘s static analysis tool, Checkov, which helps identify misconfigurations in IaC frameworks, to analyze Terraform modules in popular, public container registries. The team found that almost half of those modules contain at least one critical or high-severity misconfiguration.

“If you factor in the number of times each module has been downloaded, 64% resulted in at least one high or critical insecure configuration,” said Chiodi. “So most of what’s being downloaded from public registries has very poor configurations.”

What Is the Impact of Vulnerable Software?

Another damning report from Veracode, an application security company, found that, although more than 80% of developers said they consider security when choosing a library, most never update the third-party libraries used in their software.

While developers quickly patch some vulnerabilities in third-party software, the report found, they take up to three months to patch half of vulnerable libraries and one year to address 75% of them.

The Veracode report examined 13 million scans of more than 86,000 repositories containing over 300,000 unique libraries, as well as conversations with more than 1,700 developers.

Vulnerabilities found in widely used components can have huge potential consequences. For example, vulnerabilities and malware have been found recently in open source npm packages, one of which counts more than 14 million weekly downloads.

Many security problems traceable to open source software are related to trust, said Cole Kennedy, CEO and co-founder of TestifySec, a cybersecurity company.

A developer might want to incorporate a repository with “only one contributor who resides in mainland China, and who’s also employed by the Chinese military, which could be a problem,” he said. “And that’s really the problem we see in open source: a lot of this stuff is not vetted.”

Then there are possible legal consequences. For example, a survey by the cybersecurity company Venafi of more than 1,000 IT and development executives found that 94% of them want consequences such as fines and legal liability for software suppliers that fail to protect the security of their software build pipeline.

There’s also increasing pressure from existing and proposed legislation. President Biden’s executive order in May established that software bills of materials (SBOMs) detailing commercial and open source components are now required from suppliers to the federal government.

They’ve also been proposed for critical infrastructure by the Cybersecurity and Infrastructure Security Agency (CISA).

How to Prevent Risks to Software

The CNCF best practices white paper breaks down the problem of securing the software supply chain into five key areas, using the analogy of a physical manufacturing supply chain and adopting the practices used to secure it. These five areas are securing source code, materials, build pipelines, artefacts and deployments.

The organization also provides recommendations for security assessments and the security review process, and an adoption framework for assessing your own architecture.

The CNCF white paper looked at practices across multiple industries. “If we’re going to standardize on recommendations for what is state-of-the-art security for organizations that are security conscious and industries that are highly regulated, how do we piece that together?” said Vega, the paper’s co-author.

Another distinction to consider is the different security risks of small, one-off projects and libraries, and larger, commercially supported projects like Linux and Kubernetes, with lots of momentum behind them. “These large, open source projects are probably more secure than most closed-source enterprise projects because there are more eyes on them,” said Kennedy.

Third-party assessments already exist for some open source projects, usually corporate-sponsored ones. For example, Cure 53, a cybersecurity company, has provided some assessments for the CNCF, and the Open Source Security Foundation’s scorecards can be used to assess project security.

“If you have in-house expertise you can look at the actual source code, who committed it and where they’re from,” said Kennedy.

Sigstore is a new service and standard for signing, verifying and protecting software components, “for open source maintainers, by open source maintainers,” as the website says. “It keeps a public system of record that can be audited by anyone,” said Vega. The project, begun as an open source initiative at Red Hat, is now under the auspices of The Linux Foundation.

The companies CloudBees, Sophos, Venafi and Veracode are working to define a vendor-neutral map of standard controls for evaluating software that development teams need for purchasing, according to a Venafi blog post. These controls, along with details on potential exposure, are listed on GitHub.

Protecting the Development Pipeline

Traditionally, developers have relied on vulnerability scanning tools. But after compiling, any vulnerability inserted into your code is often hidden from scanning tools.

Language can also affect the ability to find weak spots in software. In C, a vulnerability can be hidden more easily, yet some aspects of memory in C that can make code more secure, said Kennedy. Vulnerabilities are easier to find in Python, but code is subject to more types of attacks.

Providing attestations — verifiable proof of software quality — definitely helps. TestifySec, for example, is putting together a services offering for helping organizations implement software supply chain security.

“We’re also developing a product, both open source and commercial SaaS,” said Kennedy. “We have an open source release coming out of our new product Witness, a command line tool that will provide these attestations.”

For its part, Palo Alto Networks has just launched Prisma Cloud 3.0, a cloud native application protection platform, which includes many new capabilities including Cloud Code Security.

“This allows organizations to integrate security for IaC and cloud native applications as part of their developer and DevOps workflows, and it lets you integrate security for open source directly into those flows,” said Chiodi.

“It also gives organizations the ability to monitor and manage their cloud security posture across all major cloud providers. We offer all this in one platform instead of having to buy 15 different tools.”

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Unit, Kaseya.

Featured image by Miltiadis Fragkidis via Unsplash.