Software Containment in VxWorks: VxPOD – A Separation Alternative to Virtualization

The challenge in hardware consolidation for real-time embedded systems is ensuring that sub-systems don’t interfere with each other.   To take advantage of the space and hardware cost reduction that comes with hardware consolidation, some systems use virtualization technology to separate sub-systems from each other.  With Virtualization, a hypervisor is run either on the hardware (Type 1 Virtualization) or an operating system (Type 2 Virtualization). The Hypervisor instantiates virtual machines (VMs) to run software loads in and is responsible for partitioning or virtualizing all hardware resources between the VMs. Entire operating systems would run inside these VMs as if they run natively on the hardware.

Virtualization systems are elegant to put together, but virtualization requires 1) hardware that supports virtualization, which is typically more expensive; 2) more resources as each VM runs its own full-fledged operating system; and 3) adds more complexity to the design because of the additional hypervisor to guest operating system interaction.

Wind River® VxWorks® has a built-in type 1 virtualization solution, and it also offers an alternative single-OS separation mechanism:  VxWorks Portable Deterministic Containers (VxPOD).  VxPOD is a set of technologies for defining time, space, and resource constraints around a group of VxWorks processes within a single real-time operating system (RTOS). VxPOD can be used on any processor with an MMU.  It does not require hardware virtualization extensions. This solution requires less resources and technology compared to virtualization.

This blog post uses the example of a manufacturing cell to show the problems that VxPOD addresses.  The solution can be applied to many other hardware consolidation scenarios: reducing space and weight in an industrial control system or combining several medical devices into one unit.

Consolidation Example:  Manufacturing Cell

A manufacturing cell is a self-contained system responsible for creating and assembling a manufactured product.  The sub-systems in this example include:

• Robot control (high criticality sub-system)
• Environment control (medium criticality sub-system)
• Quality control (low criticality sub-system)
• Supply inventory with cloud connectivity (low criticality sub-system)

Figure 1 shows the four sub-systems and the qualities of each system.

Figure 1.  Sub-systems of a manufacturing cell to be consolidated into one controller hardware

If one manufacturing cell requires four separate controllers to operate each sub-system, the hardware cost for the controllers is quadrupled, and replicated in 50 to 100 of these cells across the factory floor and across several sites. To reduce hardware costs and conserve space, the manufacturer is consolidating the controllers of all four systems into one piece of hardware.First Challenge:  Multi-team DevelopmentThe robot control sub-system is developed by a sub-contractor, the environment control is developed by a team on the other side of the world, and the quality control and supply inventory sub-system controllers are developed locally by two separate teams.   Each development team runs sub-system tests in isolation. The controllers of each development team come together in a single piece of hardware equipment.  The first challenge, then, is to make sure each sub-system still runs as expected when it shares a physical box with other sub-systems.Second Challenge: TrustAs a result of the multi-team development, different teams could be using different approaches to development. Typically embedded development is done in C, but a team could take a lot of content from open source, or even use Java for some of the partitions. This means that there could be untrusted code running in the system that needs to be separated from the critical content.Third Challenge:  ReusabilityThe manufacturing plant decides to modify 20% of the cells to take on a different configuration.  The new cells use the same environment control and supply inventory control, but the robot control and the quality control are switched out and replaced by a laser cutting machine and a conveyor belt sorting system developed by another vendor. In fact, later on, another variation of the manufacturing cell combines various sub-systems of pre-existing systems to create a new cell.  The third challenge is to create sub-systems that can be integrated in different combinations to create multiple system configurations.How VxPOD Addresses These ChallengesAll VxPOD sub-systems run in a single OS together under the control of a single scheduler.  By allotting memory, time window, system objects, file systems, and system calls to specific process groups, VxWorks isolates each sub-system within the OS. In addition to isolation, VxPOD also provides a common operating environment for sub-systems.  By using the same separation mechanism among all sub-systems, each sub-system can then be recombined to form a new system. To illustrate VxPOD in the manufacturing cell, the software components of each sub-system are shown in Figure 2.

Figure 2 – Sub-system software components

Each sub-system consists of two to four processes in this example.  No hard-defined limits on resources have been defined between the sub-systems.   All of the sub-systems save files into the file system.  Some sub-systems use multi-process memory regions, multi-process semaphores and multi-process message queues.  Both the high criticality robot control system and the supply inventory system connect to networks – one to an isolated private sensor network and the other to the public cloud.

The final system could be a single core processor, with all the sub-systems sharing that single core, or it could be a multi-core system with the subsystems separated out over the cores. A single subsystem could be using multiple cores in this scenario.

The final system would likely consist of a C-programmed robot control, the user-interface could be in C++ using accelerated graphics and a high level Human Machine Interface, the supply inventory could be done in Java and the quality control could use some open source image analysis content. When the system integrator brings all of these components together into one microcontroller, the potential for interference exists.  The following table describes how VxPOD technology confines shared resources to a sub-system to prevent interference.

Together, all of these technologies serve to contain the sub-system into a coherent unit, with VxWorks as its common operating environment. In the case of multi-site and multi-vendor development, each development team creates a contained sub-system based on VxWorks and its VxPOD technologies.   Multiple teams can contribute to a library of contained VxWorks sub-systems.  An integrator picks and chooses multiple sub-system combinations to create variations of a whole system.VxPOD versus VirtualizationThere are times when consolidation of multiple workloads, each in their own OS onto the same MCU is preferred and worth the complexity and cost of a hypervisor solution.   VxPOD provides a simpler alternative.  Here are cases when a VxPOD would be chosen as the design over virtualization:

• A separate OS for the workload is required. In VxPOD all workloads run in the same OS instance, with virtualization each virtual machine gets its own OS instance.
• The silicon does not have hardware support for virtualization. Though virtualization is still possible in this case, it requires invasive changes into the workloads.
• Virtualization overhead cannot be tolerated.  Though small, virtualization does add some additional overhead to certain OS operations either in the hypervisor, or in the silicon. VxPOD avoids this overhead.
• Partial containment is lighter weight.  In some cases, several non-critical sub-systems may need to share a network stack, or to share some time and resources.  With VxPOD, a system integrator can be selective about system resources that cross sub-system boundaries.

You can learn more about VxPOD in the VxWorks Product Note http://www.windriver.com/products/product-notes/pn_vxworks/.  For those of you who have access to Wind River Online Help, the VxPOD technologies are documented under Safety Profile Facilities for RTP Application Control in the VxWorks Programmer’s Guide available from the Wind River Knowledge Library https://knowledge.windriver.com.