Enabling Automatic Testing of Anything with Simics

Automating testing is a key part of modern software development practice like continuous integration, continuous deployment, and agile development. Using only manual testing is an excellent way to reduce velocity, leave bugs in the product, and make the test department quit in frustration. Doing fully automated testing is sometimes harder than it sounds, in particular for embedded systems. It is quite common to hear people discussing continuous integration and testing, and admit that while it works nicely for general computing workloads, it is far more difficult to make it work well for embedded systems. The hardware needed to run particular tests can often be complicated to set up, expensive to obtain, scarce, or tricky to communicate with. As a result, testing will suffer, eventually impacting software quality and production schedules. Using virtual platforms to extend and augment the set of available test platforms makes automatic testing much more feasible for even the thorniest systems. Here is how.

What does automated testing with Simics really mean? Simics is not a test generation system or a test management system, but rather an excellent system for executing tests and collecting test outputs. Simics is typically integrated into an existing test automation framework alongside hardware, using the same tests as are run on the hardware. This reuses existing assets in terms of test designs and test scripts, and ensures that all tests can run anywhere, increasing flexibility.

Still, Simics adds a number of unique abilities that make testing more effective and efficient.

Compared to hardware, managing a Simics test system is much easier. As Simics is just software, it will not run out of control, hang, or become unresponsive due to a bad hardware configuration or total target software failure (see this blog post for more on the difference between virtual and physical hardware). Thanks to the scriptable setup system in Simics, recreating a particular configuration is instant, where it might take hours or even days to configure in hardware by finding boards, connecting cables, and uploading software. There is no need for users to physically go to the lab and physically reconfigure the system, it can all be done as part of a software flow.

Simics runs on standard servers or PCs, which allows for a much more agile testing system. The same physical hardware box can be used to run software for a wide variety of systems. With hardware boards and systems, the number of each type of system is limited, limiting the ability to ramp up testing for a particular type of system when needed. With Simics, it is also possible to temporarily increase the testing pool by borrowing servers or other computer resources from other groups within the same company, or even renting time on a cloud computing service.

The increased availability of virtual hardware and the ability to run in parallel typically has two beneficial effects.

One is that more testing can be done faster, and this makes it possible to run tighter test loops than with hardware. We have seen customers who were limited by hardware greatly increase test coverage and frequency – if you can run your regression test suite daily rather than weekly, errors will get found earlier and fewer errors will make it out in the field, reducing development costs and increases product quality. More available testing resources also reduce waiting time from the point in time where a test is submitted to when it gets executed, which is particularly important if some test case has to be rerun for analysis by the developers. If the analysis run has to wait for days to be run, the context will long have been lost and the developer will have to make an effort to remember what it was all about. If it can be performed within hours, there is much better continuity and productivity.

The other effect of virtually unlimited hardware availability is that tests do not have to be scaled down or modified to match available hardware; instead, the hardware can be set up to match the tests that need to be performed. Rather than use a development board, a real board can be used, reducing the number of build variants and increasing the fidelity of the testing.

Looking beyond convenience, configuration, and availability, using Simics for automated testing offers many opportunities to work smarter and save time in tests.

Using Simics checkpoints, it is possible to fan out tests with incremental value in a way very hard to do on hardware. Interesting intermediate points can be saved and reused for tests, providing both stable starting points for multiple tests and saving time in the setup phase (one interesting example of using checkpoints to fire off many test cases from the same initial state was covered in the blog interview with Hyungmin Cho we ran earlier this year). The picture below shows some common ways to use checkpoints. Tests A and B are run directly from the booted checkpoint, since their setup phases are short. For Setup Q, however, the work of doing the setup is much larger, and we have multiple tests that will run from this setup. Therefore, after the set up is done, a checkpoint Q is saved, and all the tests depending on setup Q is run from this checkpoint.

It is worth pointing out that saving checkpoints after a system has been configured for a test, but before the tests have been run, has the advantage that resetting and cleaning up the system between tests is very easy. In a traditional hardware-based test setup, you typically need to apply somewhat tricky cleanup code as well as a hardware reboot to get the system back to a state that is sufficiently pristine to not affect the next test to run. With Simics, you just pick up the checkpoint of the clean initial state and run from it, avoiding the risk of contamination between tests and often saving a lot of reboot time. It does mean that the cleanup phase and start phase managed from the test system needs to have special cases for Simics, but that is part of the testing framework and can be done once. Such use of checkpoints need not affect how test cases are expressed.

Checkpoints offer an excellent way to pick up the state after a test and pass the failed tests back to development. As discussed in my previous blog post on continuous integration with Simics, this feedback loop can save days of development time, as well as avoiding the need to grab and hold a test system to allow for the diagnosis of an issue.

When it comes to feeding test cases into the system running on Simics, there are many different options available. The simplest solution is to connect the network or serial port of the virtual platform to the outside world, and drive them just like a physical board would be driven. This is illustrated by the two external test data arrows in the picture below.

However, that is not the only option, as we can see in the picture above. For example, the target software stack itself can be the test data. A fresh build of an operating system would be loaded onto Simics to test that it boots and initializes hardware correctly. A test application with its own built-in test data would be loaded on the target system and executed with no external interaction. Simics can perform fault injection (see my previous blog posts about fault injection in general  and a specific example), and a script that injects a particular fault or sequence of faults would be considered the test input for a test of robustness, fault tolerance, and RAS.

The hardware configuration itself can be a relevant test stimuli. For example, creating a certain configuration of boards, a particular set of different boards, a certain network size,  or network topology. Indeed, way back in the early days of Simics, we did some interesting tests of this kind. In one instance, a target machine was configured to report itself as having 2^{^64} bytes of RAM. This amount RAM was (and is) impossible to acquire in physical form (there was not that much RAM in the entire world, and a quick back-of-the-envelope calculation indicated that the cost would be somewhat bigger than the entire US defense budget*). With Simics, on the other hand, it was simple, and we could check if the OS in question considered that size of memory as a negative value somewhere.  In another case, we maxed out a self-configuring network system to hold the maximum number of nodes the self-configuration was supposed to handle, to test that it would indeed work. In the lab, that test would have been impossible to do, since the developers had nowhere near the 200+ nodes needed available. It was more like ten.

For more on Simics, see the Simics product pages at Windriver.com.

* Note that today, in October 2014, that much RAM can be had for just 1% of the GDP of the US (or pretty exactly the GDP of New Zealand).