Bring in the Bug Hunters or Maybe Not for Some Open Source Projects – InApps Technology 2022

Facebook, GitHub and the Ford Foundation have recently donated $300,000 to the Internet Bug Bounty, a program that rewards security researchers who find vulnerabilities in software deemed critical to the internet infrastructure. The plan is to use the money to expand the program’s coverage to data processing libraries and other widely used open-source software.

While this initiative is great for bug hunters and security in general, it doesn’t create the same incentives for developers who have to fix the reported issues. As a result, some open-source projects do not want to get involved because they lack the resources to deal with a sudden influx of vulnerability reports.

The Internet Bug Bounty (IBB) was set up in 2013 as a partnership between Facebook, Microsoft and HackerOne, a vulnerability coordination platform used by around 200 organizations, including the U.S. Department of Defense.

The projects currently covered by the IBB include programming languages like Ruby, PHP, Perl and Python; web servers like Apache and Nginx; web development frameworks like Ruby on Rails, Django and Phabricator; and libraries such as OpenSSL. Reporting critical vulnerabilities with a broad impact such as Shellshock, ImageTragick or TLS 3Shake, is also being rewarded through the program.

Since its launch, the Internet Bug Bounty has paid researchers more than $600,000 for 625 confirmed vulnerabilities, 250 of which were reported last year. Some bug hunters have chosen to donate their bounties to non-profit organizations like the Electronic Frontier Foundation, Hackers for Charity and Freedom of the Press Foundation.

The program’s planned expansion will target primarily widely used open-source libraries, like those used for image processing and hopefully compression too, said Shawn Davenport, vice-president of security at GitHub.

Davenport couldn’t name specific libraries that will be covered because the IBB organizers are still in the process of contacting project maintainers to ask if they want to come on board. However, he mentioned that there has already been some pushback from developers and some projects have declined to join the program.

And that’s to be expected because unlike PHP, Perl, Python or Apache, which have tens or hundreds of contributors, data processing libraries are typically small projects maintained by only a handful of individuals who often do the work in their spare time.

What Bug Bounties Mean for Developers

Projects that decide to participate in the IBB will likely have to deal with a large number of vulnerability reports, especially in the beginning, and they will have to solve them in a timely manner. Crowdsourced security testing is also known to generate a high amount of false reports due to a large number of people with different backgrounds and technical skill levels attempting to find flaws. Many bug hunters are not traditional developers, and while they can spot and understand code flaws, they’re not necessarily capable of developing and submitting patches for the issues they find.

Historically, data handling libraries have been a constant source of serious vulnerabilities because it’s not easy to parse complex input safely. Just over the past week, the Full Disclosure security mailing list has received reports of vulnerabilities in libmad, libao, libvorbis, libjpeg-turbo, libid3tag, OpenExif and some other media processing libraries.

All of these issues, combined with the general lack of funding and human resources that most open-source projects suffer from, makes it unlikely that many of them will agree to participate in a bug bounty program. And that’s not good news for end users because the vulnerabilities that could be weeded out through the IBB will remain in the code and might be exploited by malicious hackers in the future.

“I think that’s the next challenge that needs to be addressed because right now there’s a big imbalance: You might have 100 people who can find flaws and only one developer who can fix them, and that’s not a healthy situation,” Davenport said.

A Lack of Resources

The sad reality is that many open-source projects are poorly funded and understaffed, including some that maintain components used in thousands of products with millions of deployments.

In 2014, when the highly critical Heartbleed vulnerability was discovered in OpenSSL, people were shocked to learn that the core development team behind the world’s most popular cryptographic library had only four members and just one of them was working full-time. The project itself was getting around $2,000 per year in donations, despite tens of thousands of commercial products using OpenSSL.

Later in 2014, several critical vulnerabilities that came to collectively be known as Shellshock were discovered in the Unix Bash shell. Bash is the default login shell in most Linux distributions and Apple’s macOS, but for the past 25 years, it has been maintained by a single developer named Chet Ramey, who works on it as a hobby.

Even entire operating systems are underfunded. A few years back the OpenBSD project came close to shutting down because it didn’t even have the money to cover its electricity bill and keep the build infrastructure running.

All of these incidents motivated the Linux Foundation to launch the Core Infrastructure Initiative (CII), a program that provides open-source projects viewed as critical to the internet with the financial resources they need to hire additional developers and improve “their responsiveness to patch requests” and overall security. Twenty large companies including Facebook, Google, Microsoft, Amazon, Adobe, Cisco, Qualcomm, HP, Huawei, Intel and IBM sponsor the program.

Through the CII, the Linux Foundation engaged the Institute for Defense Analyses (IDA), a non-profit organization, to perform a census with the goal of finding critical open-source projects in need of assistance. Among the troubled projects identified by IDA were several widely used image processing and compression libraries. For example, bzip2, gzip, libexpat, zlib, libjpeg, libpng and unzip were found to be “relatively unmaintained.”

“While some software projects have many participants, perform in-depth security analyses, and produce high-quality software with strong security, other projects rely on small teams and limited budget and time to do the tasks necessary to ensure security,” the Linux Foundation said in the Census Project summary.

While the CII is a step in the right direction, the number of projects that have received funding through the program so far is relatively small compared to how many are actually in need of assistance.

The Hybrid Approach

Even well-funded projects that are used to dealing with vulnerabilities on a regular basis could find it hard to participate in a bug-bounty program because of all the logistics involved, but also because there’s generally a disconnect between security researchers and developers.

“I think this is a problem that private companies face as well,” Davenport said. “I’ve seen reports coming into GitHub’s bug bounty program for which it was really challenging to recreate the problem and verify where it exists. I would like to see bug bounty programs move towards a closer relationship between security researchers and developers, whether that means researchers actually providing patches or being helped to see and understand the code where the problem might have originated — a white-box approach instead of a black-box one.”

Some companies that run bug bounty programs already encourage this. For example, both Google and GitHub routinely pay more for higher quality reports that also propose a way to fix the identified problems.

Google has gone even further and launched a Patch Reward program in 2013 for third-party open-source projects that it views critical to its products and the internet. The program covers software like the Linux kernel, Apache httpd, lighttpd, nginx, OpenSSL, OpenSSH, OpenVPN, BIND, but also data parsing and compression libraries like libjpeg, libjpeg-turbo, libpng, giflib, zlib, libxml2, bzip2, tar, gzip, info-zip and lzo.

Google’s Patch Reward program doesn’t focus on fixes for individual bugs, but on patches that significantly improve the security of the in-scope applications and harden them against entire classes of exploits. Such contributions include improvements to privilege separation or sandboxing, memory allocator hardening, cleanups of integer arithmetics, systematic fixes for various types of race conditions, elimination of error-prone design patterns or library calls, contextual auto escaping in templates and more.

Google’s rewards range from $500 to $20,000 depending on the patch type and the company can pay even more for “unusually clever or complex submissions.” Patches submitted by a project’s own developers qualify for rewards and in some cases of external contributions the reward might be split between the third-party contributor and the project maintainers.

“Quite a few vulnerabilities trace back to preventable coding mistakes, or are made easier to exploit due to the absence of simple mitigation techniques,” Google says in its program’s description. “We are hoping to address this to some extent.”

And while Google doesn’t reward patches for individual bugs, maybe that’s something that sponsored bug-bounty programs like the IBB should consider doing in the future. Splitting the bounties between security researchers and developers might provide an incentive for more projects to join. And that would benefit everyone: bug hunters, developers and end-users, whether they’re consumers or companies who use those open-source libraries in their commercial products.

“Open-source software is free to use, not free to develop; it takes somebody’s time and effort,” Davenport said. “I feel that corporations have gotten a bit of a free lunch, which is one of the reasons why I’m happy that GitHub was able to contribute to the IBB. We depend on open-source software extensively. What we’ve done solves one part of the equation — rewarding researchers for finding vulnerabilities — but I do hope that collectively we get to the point where we also reward developers for patching them.”

Feature Image: “Head(less) Hunter” by JD Hancock, licensed under CC BY 2.0.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Census.