Open source Linux is a popular choice for developers of embedded systems and devices, but with ever-increasing numbers of interconnected IoT devices being deployed, Linux software vulnerabilities have become more widespread than ever. The proven four-step process for resolving Linux vulnerabilities involves monitoring, assessment, notification, and remediation. But taking responsibility for identifying vulnerabilities and making the necessary updates to mitigate threats is often beyond the capacity of device developers and manufacturers.

In the embedded industry, Linux is the most popular development platform. Ninety percent of public cloud workloads run on Linux (Ubuntu, 2021). This number will continue to grow as open source solutions push the boundaries of the intelligent edge. Securing Linux-based systems and devices has become one of the most pressing and perplexing challenges facing developers and device manufacturers. Gone are the days of “fire and forget” device deployment. Virtually every device made today is designed for interconnectivity with something, which makes them all susceptible to security vulnerabilities. The reality is that connected devices are likely becoming more vulnerable with every reported exploit.

Interconnected devices are proliferating at exponential rates. This massive increase in connections, data volume, network traffic, and users has brought a proportional increase in cyberthreats across a wider attack surface. In response, device manufacturers and developers of IoT applications are employing sophisticated methods to build in powerful security functionality at the earliest stages of design. And that’s a good thing. In fact, it’s essential. But it’s not enough. Threats are constantly evolving, and operators of open source systems need a mechanism to maintain security in devices over their entire useful lives.

Consider the example of your own laptop computer. There was a time when all you needed to secure it was a password, and the biggest external threat was an infected floppy disk. Once you connect it to the internet, however, it becomes a target for attackers, typically via the applications that reside on it. Chances are you receive weekly or monthly update advisories or auto-updates from app providers, intended to protect your computer from newly discovered software vulnerabilities.

Every IoT device running Linux needs that same level of ongoing protection. The question is how to accomplish it in a systematic, scalable, and cost-effective manner.

Below are some of the main challenges project teams are facing when using Linux in development:

• Quality: 21,957 security vulnerabilities were listed in 2021 (NIST NVD database), and 40% of a developer’s time is spent debugging (Infopulse, 2019).
• Cost: Fixing a bug in a deployed device is widely believed to cost 100x more than fixing it during development (The Register, 2021). The average time an embedded solution is supported after deployment is 10–15 years.
• CVEs: Over the last five years, the number of security vulnerabilities, also known as CVEs, has been rising at a rapid pace. In 2021, there were over 20,000 new CVEs reported. Midway through 2022 the number had already reached 16,000 (cvedetails.com). Staying on top of all the new security vulnerabilities, especially when you are trying to integrate new features and meet tight deadlines, is challenging but critical.
• Support: Long deployment lifecycles are the norm for embedded projects. Embedded devices can be in the field anywhere from 5–20 years. Until that device is properly decommissioned, it must be serviced and maintained.
• Skill and retention: All companies are competing for the same resources with the right skill set. Good, experienced software engineers are hard to find and even harder to keep.
• Lifecycle domain expertise: Most likely, the team that built and deployed an embedded project five years ago is no longer around to maintain the device that is currently deployed. The domain knowledge is gone or archived on someone’s retired laptop.
• Managing rate of change: Industries are becoming more complex at a faster pace. We are rapidly moving to a machine economy that leverages new technologies, tools, and processes. In addition, end customers are demanding new capabilities sooner. Release schedules are shortening.
• Managing export and compliance requirements: In a global economy, there are new requirements imposed on devices for export, compliance, and standards.

Before you can fix vulnerabilities in a system, you must know what and where they are. That’s becoming increasingly challenging, as security vulnerabilities are multiplying in parallel with the expansion of the IoT.

The Common Vulnerabilities and Exposures (CVEs) database is the widely accepted, de facto industry standard for identifying, repairing, and reporting vulnerabilities. Using CVE identifiers, the information about a vulnerability can be correlated to appropriate security patches or protection technologies, which is especially vital in the open source software world. Disclosure of vulnerabilities can come from a variety of sources, including the software vendor, security vendors, independent researchers, community mailing lists, and government agencies such as the U.S. Computer Emergency Readiness Team (US-CERT). However, the CVE database has been challenged to keep pace with the volume and scale of vulnerabilities resulting from the IoT world.

The last decade saw an explosive growth in the volume of CVEs over the previous decade, with thousands reported each year. Moreover, they have tended to increase in severity. According to the National Institute of Standards and Technology (NIST), 80% of all external attacks take advantage of known vulnerabilities in unpatched or misconfigured systems. Meanwhile, in its 2016 Threats Predictions report, McAfee Labs reported that many recent zero-day attacks — those that exploit vulnerabilities before they become known to the vendor — specifically targeted vulnerabilities in open source software.

Potential Pain Points

Here are some questions to help you identify potential pain points:

• Is your Linux platform architected to adapt to new requirements in the future?
• Does your team have the right technical skillset to address security, innovation, and ongoing maintenance needs?
• Do you have a security policy in place for the full product lifecycle?
• Can your team quickly move to resolve security issues as they emerge?
• Does your team have the skillset to mitigate and remediate CVEs?
• Do you know how to find and resolve compliance issues in your code?
• Are you able to manage and maintain multiple platform deployments?
• Are you able to keep pace with innovation while also managing existing platforms?
• Are resource staff able to maintain and manage CVEs and defects in the deployed Linux platform?

Ongoing threat mitigation in deployed systems requires a four-step approach: monitoring, assessment, notification, and remediation.

Monitoring

Think of monitoring as the “surveillance camera” in your security strategy. Assuming two houses have strong locks, the one with the surveillance camera is going to be better prepared against an intrusion. In this case, the cameras are operated by organizations that issue vulnerability alerts and advisories, such as US-CERT, NIST, the CVE database, various security vendors, private mailing lists, and communities focused on finding Linux vulnerabilities.

The challenge here is that, with dozens of organizations issuing advisories, there is bound to be a certain amount of speculation, making it critical to know which organizations can be relied upon for accurate and actionable information.

Assessment

Once an advisory or security report is received, the system operator or its software partners must decide whether their devices are vulnerable, and to what extent. Vulnerability is typically ranked as high, medium, low, or not present, and it is prioritized based on likely severity, difficulty of attack, and likelihood of avoidance.

Assessment requires knowing exactly which packages and versions are vulnerable, as well as the exact configurations of your systems. For vulnerable products, the clock for finding a patch starts the minute the vulnerability is exposed.

Notification

Once the vulnerability has been assessed, affected users must be notified of the issue, the determination of vulnerability, and the action plan for remediation. This step requires the right tools and methodologies, so that notifications are sent to all affected parties in a timely and efficient manner.

Remediation

The timing and method of remediation is usually based on priority. Vulnerabilities deemed to be of high severity may require an immediate “hot fix,” while others of lower severity may be covered in periodic software updates.

The challenge is having the ability to quickly deliver effective patches and distribute them to end users via a secure channel.

It is not uncommon for a software development team to start a platform project using a Linux OS provided by a semiconductor company. Platform teams realize benefits from using semi Linux that can accelerate a project through the prototype phase, and in some cases to full production. However, there is also the potential to run up against unrecognized or unaccounted-for technical debt and design challenges. This is where Wind River^® can help.

Wind River Studio Linux Services

With Wind River Studio Linux Services, you won’t have to make drastic trade-offs between resources, quality, and time-to-market. Studio Linux Services addresses the three biggest challenge areas for Linux developers:

1. We can help alleviate your technical debt from security issues, ongoing maintenance, and operations.
2. You can offload the resource burden of time-consuming tasks that inhibit your ability to innovate.
3. You can utilize our deep industry and technical experience to build a design that thrives throughout the product lifecycle.

» Learn More

How Studio Linux Services Helps

SECURITY SCANNING SERVICE

Security vulnerabilities are simply a fact of life in today’s interconnected world. Managing and mitigating them is essential for the protection of end users, but the process requires a level of engagement that is beyond the scope of most IoT solution developers, device manufacturers, and system operators. Fortunately, the open source community is vigilant in finding vulnerabilities that affect Linux software. By working with a software partner active in that community, with a proven process for monitoring, assessing, notifying customers, and fixing vulnerabilities, manufacturers and developers can help protect their customers against cyberthreats.

At Wind River, that process begins with a Security Scanning Service. Here's how it works:

» Learn More