Everyone remembers when the Heartbleed vulnerability in the OpenSSL cryptographic library sent waves of panic ripping through the software industry and companies around the world. Software developers did not know enough about the open source components used in their own products to understand whether their software was vulnerable – and their customers using that software did not know either.
The questions remains, have software companies learned a lesson?
The answer is a resounding no. To prevent future Heartbleeds, companies need to put in place processes and automation to scan, track and manage the open source components they use in their products. Doing so will enable them to responsibly understand where the open source components reside within their software products, which of these components are vulnerable and what customers are exposed. Moreover, these producers should also have systems and automation in place to patch these vulnerabilities and ensure the software they are building, and the software their customers are running, is secure – minimising the risk of undocumented open source components and vulnerabilities from inadvertently compromising their supply chain and putting their customers at risk.
Today’s Reality
Imagine yourself reading your news feed right now and a new cutely named vulnerability in an open source component started appearing in posting after posting. There is a standard flow to these types of events – a slow burn, then more and more mentions and retweets, questions about how real the issue is, is it even exploitable, maybe some assignment of blame or the start of a code review and post mortem. While this is all going on, you start getting the first pings from your boss or the Product Managers at your company. “How does this news affect us?”, “What is our response, action plan or update to our customers?”, “Are we already affected by people taking advantage of this vulnerability to attack our users?”
Ten to 20 years ago, it was much easier for a VP of Engineering to know what they were shipping. They could turn around, look at the bookshelf and see the product boxes of the libraries they have licensed. Maybe there were a few digital downloads, or some code from a famous book. Slowly at first, then almost overnight, the development model changed from mostly homegrown software with a few commercial libraries to projects that are at least 50 percent open source, all digitally delivered, and not always in well-defined locations in the source tree. To answer the question “Are we affected by this latest vulnerability?” requires a current listing of dependencies or bill of materials (BOM).
While most companies believe that listing is currently being managed, the truth is that the vast majority of software companies would be hard pressed to produce such a list – even post-Heartbleed. Research shows that the typical software company is not able to create such a listing, and if they are, the average percentage of known components is only four percent! What this means is that for every known component, there are 24 other unknown and unmanaged components that are being used and delivered to customers.
A Quick Review
A simple way to check, is to see if you could produce a current BOM and see when it was last updated. If this list was created only using self-reporting by developers, it is almost certainly only a small percentage of reality. Perform some sampling of the codebase to check if the version numbers reported are still current and do a quick review of library names, a quick search on copyright strings, license and COPYING files and a review of file extensions that are likely third party (.JAR or .DLL, for instance).
Even a quick review will uncover large amounts of previously unknown third party software.
Even more eye opening can be a search for the string “OpenSSL”. It is pretty much guaranteed you will find multiple copies of previously unknown instances of OpenSSL in open source and embedded in commercial components. Both source and binary inclusions will be found.
These self-tests can quickly show that either your BOM is incomplete, or out-of-date. While this is not a happy thing to find out, you will find yourself with lots of company in the software space.
Use Cases
There are few main use cases where you will find open source components that may be affected by published vulnerabilities. The first case are things brought in as top-level components, often in clearly named files or directories. These are the most commonly disclosed due to their visibility, but are not the entire picture. The next place components found are as subcomponents of these top-level components. This type of open source software inclusion is harder to discover and is often overlooked, even in the case of a well-known vulnerability like Heartbleed. Versions of these components that have been compiled and linked into other larger packages are almost invisible to the typical developer when trying to get a handle on their complete list of dependences. Lastly, codebase owners will want to review the remaining source files to find cut and pastes, re-factorings or rearrangements of a larger open source package. This last case can be almost impossible to do by eyeballing the codebase alone.
Once a current BOM is created, it should be compared against components with known vulnerabilities. Expect multiple vulnerable components to be found, especially on the first review cycle. Some of these many be easily exploitable, while others may not have such a clear path to exploit. It is common to triage these results. Remediations include upgrading to latest version, patching, blocking access – and in some cases – removing the component and product features affected.
You may find that a vulnerability was introduced due to a subcomponent of a larger component that was delivered via your software supply chain. If you are lucky, you may find that your open source and commercial suppliers have already patched the affected component and it is ready for download. It is also common to find out that they are not aware, or not ready to remediate the issue.
Defect Logging Process
Familiarising yourself with the defect logging process for your open source and commercial components is an important part of participating in the vulnerability life cycle.
Once a new version of your product is delivered, the process of keeping a valid current BOM, and checking this list against known vulnerabilities, should continue as long as the product is both developed and used. This should be done for all versions used by your user community. Even though your development team may have moved to the latest version, you may have significant numbers of users happily using much older versions of your software.
Until the software industry puts into place processes similar to the ones detailed above, it will continue to be unprepared to quickly respond to new “Heartbleed” style vulnerabilities.
Jeff Luszcz, VP of Product Management at Flexera Software
Image Credit: Leolintang / Shutterstock