Simics helps running 60 000 GNAT Pro tests in 24 hours

A key aspect of AdaCore’s GNAT Pro offering is the quality of the product we’re delivering and our proactive approach to resolving issues when they appear. To do so, we need both intensive testing before delivering anything to our customers and to produce “wavefront” versions every day for each product we offer. Doing so each and every day is a real challenge, considering the number of supported configurations, the number of tests to run, and the limit of a 24-hour timeframe. At AdaCore, we rely heavily on virtualization as part of our testing strategy. In this article, we will describe the extent of our testing of GNAT Pro on VxWorks, and how Wind River Simics helped us meet these challenges.

Broad Support for VxWorks Products

The number of tests to run is proportional to the number of configurations that we support. We have an impressive matrix of configurations to validate:

• versions: VxWorks, VxWorks Cert, VxWorks 653… on the full range of versions that we support (e.g. 5.5, 6.4 to 6.9, 653 2.1 to 2.5…)
• CPUs: arm, ppc, e500v2, x86…
• program type: Real Time Process, Downloadable Kernel Module, Static Kernel Module, vThreads, ARINC 653 processes, with the Cert subsets…
• Ada configuration variants: zero cost exceptions versus setjmp/longjmp exceptions, Ravenscar tasking profile versus full Ada tasking…

Naturally, there are some combinations in this matrix of possibility that are not supported by GNAT, but the reality is we cover most of it. So the variety of configurations is very high.

The matrix is growing fast. Between 2013 and 2015, we have widened our offer to support the new VxWorks ports (VxWorks 7, VxWorks 653 3.0.x) on a large range of CPUs (arm, e500v2, ppc..), including new configurations that were needed by GNAT Pro users (x86_64). This represents 32 new supported configurations to be added to the 168 existing ones. It was obviously a challenge to qualify all these new versions against our existing testsuites, with the goal of supporting the new VxWorks versions as soon as possible.

Simics supports a wide range of VxWorks configurations, and has a good integration with the core OS itself. This made Simics a natural solution for our testing strategy of GNAT Pro. On VxWorks 7 and 653 3.0.x, it allowed us to quickly set up an appropriate testing environment, as predefined material exists to make it work smoothly with most Wind River OSes; we could focus on our own technology right away, instead of spending time on developing, stabilizing and maintaining a new testing infrastructure from scratch.

Representative Testing on Virtual Hardware

Another benefit of Simics is that it not only supports all versions of VxWorks, but also emulates a wide range of hardware platforms. This allows us to test on platforms which are representative of the hardware that will be used in production by GNAT Pro users.

A stable framework for QA led to productivity increases

Stability is an important property of the testing framework; otherwise, “glitches” caused by the testing framework start causing spurious failures, often randomly, that the QA team then needs to analyze each time. Multiplied by the very large number of tests we run each day, and the large number of platforms we test on, lack of stability can quickly lead to an unmanageable situation. Trust of the test framework is a key factor for efficiency.

In that respect, and despite the heavy parallelism required in order to complete the validation of all our products in time, Simics proved to be a robust solution and behaved well under pressure, thus helping us focus our energy on differences caused by our tools, rather than on spurious differences caused by the testing framework.

Test execution speed allowed extensive testing

Some additional info about how broad our testing is. On VxWorks, we have mostly three sets of testsuites:

• the standard Ada validation testsuite (ACATS): around 3,600 tests;
• the regression testing of GNAT: around 15,000 tests;
• tool-specific testsuites: around 1,800 tests.

All in all, just counting the VxWorks platforms, we are running around 350,000 tests each day. 60,000 of them are run on Simics, mostly on the more recent ports (VxWorks 7 and 653 3.x).

In order to run all these tests, an efficient testing platform is needed. With Simics, we were able to optimize the execution by:

• a proper tuning of the simulation target;
• stopping and restarting the platform at an execution point where tests can be run right away, using checkpoints;
• developing additional plugins to have efficient access to the host filesystem from the simulator.

We will provide more technical details about these optimizations in future articles.

Guest authors:

Joel Brobecker, Senior Software Engineer, AdaCore

Jérôme Guitton, Senior Software Engineer, AdaCore

Jérôme Lambourg, Senior Software Engineer, AdaCore