How to measure performance

Performance is generally considered one of the more discriminating aspects when choosing between different RTOS products. Many vendors provide figures about their products where every product looks fast and attractive. The question one should always ask is: “How is it measured?”. Saying a product is fast actually provides no information at all. The only objective way to say something about a products performance is to base the figures on results from an independent test suite. Such a test suite exists in the form of Thread-Metric. Thread-Metric is an open source test suite produced by Express Logic Inc. More information about this test suite can be found here...

How is the test performed

Initial tests are performed by Mr. David Thedens, a US based embedded systems developer and RTOS expert. Test results are published by Mr. Thedens on the Microchip web site in the Online Discussion Groups. You can find these results here. After conducting these initial tests both Express Logic (ThreadX) and AVIX-RT (AVIX) have published a new and performance optimized version. For both products the latest performance figures have replaced the original ones. Results for ThreadX compared to the previous version of AVIX you can find here. Below you find a graphical and a tabular representation of the test results. These are just different representations of the mentioned figures. Upon request, Mr. Thedens is willing to share the Thread-Metric source files used for a specific RTOS so you can validate the results. If you want to receive the Thread-Metric sources for a specific RTOS, please send a message through the contact page and AVIX-RT will take care of forwarding the request to Mr. Thedens. Thread-Metric sources for AVIX accompanied by a limited AVIX distribution is available from the AVIX-RT download page.

All products were tested with full error checking enabled. AVIX uses an error checking mechanism with centralized error handling, no function returns a status value. The fact that a function returns means it succeeded. Although this might seem AVIX does not check function results, this is not the case and just one of the advanced features offered by AVIX.

Which RTOSes are tested?

Even using the Thread-Metric test suite on a standalone product does not provide much information. Given a standard test, the real interesting question is “How do different products compare each other?”. For the performance test, AVIX is compared against ThreadX, uc/OS-II. TNKernel and AVA. These are all products for the 16-bit controllers. For PIC32 no representative test has been performed.

What are the results?

The results of running the mentioned RTOS’es against the Thread-Metric testsuite are shown in the graph below:

As shown in this graph, AVIX is the best performer on four out of the seven tests the Thread-Metric suite is composed of. Detailed figures obtained for each product can be found at the bottom of this page where all results are presented in a table. Also a description is provided of the way how the tests were conducted.

Details

All tests are conducted for a 24HJ256GP610 controller running 40MIPS in the MPLAB development environment. Code is compiled using the Microchip C30 compiler version 3.01. For each test the highest possible optimization that could be used for a certain product was used.

Detailed test results are shown in the table below where a green cell marks the best performer for a certain test. Where the table does not contain a value but a number between brackets, a special situation occurred which is described below with a reference to the number in the table.

(1) For a reason why this specific test is not conducted, please refer the Microchip Online Discussion Group where the original results are published. You can find this information here.