2012 Summer Project Week:SelfTesting
- Isomics: Steve Pieper
- Kitware: Jean-Christophe Fillion-Robin, Julien Finet, Benjamin Long
- BWH: Sonia Pujol
BIST (built-in self-test) is an important concept in delivery of high-reliability engineered systems. While Slicer and the packages in the NA-MIC Kit are fairly well tested by open source software standards, the focus has been almost exclusively on build-time testing. But this leaves out many important variables that can lead to user-level issues. A BIST should be runnable from the installed version of Slicer on the user's machine. Desirable features include:
- Ensure that packaging and installation have succeeded correctly, taking into account the variability of the user's platform (OS version, memory, graphics card, drivers, etc)
- Exactly recreate testing steps so that any user-level issues can be narrowed down (e.g. if a tool works on test data, but not on the user's data).
- Support testing in a continuous loop to look for memory corruption or leaks.
The goal at project week is to discuss these issues and formulate a standard plan that can be used by all slicer developers.
Some desirable features of a solution:
- easy for developers to write tests that capture the full workflow
- distribution of tests as extensions as part of the debugging and support process
- integration with QtTesting framework to define tests 'by doing'
- same tests should be available in CTest and BIST (ideally this will happen automatically)
- interface features:
- should allow selection of all tests or a subset of tests
- should allow running in a loop
- should present a summary of success/failure
- (possible feature) dialog when slicer first installs that suggests running the tests
- (possible feature) integration of performance testing (e.g. render speed)
- (possible feature) integration with CDash so users can submit test results
- a prototype implementation as a scripted module is being developed using a factory-style mechanism where modules can register self-tests with slicer that will then be optionally run as part of the self-testing process. Prototype scripted module
- The issue is being tracked as a feature request in mantis
- Example of a test
- Discussion with some developers revealed no preferred alternative built in self test architecture
- This approach will be further elaborated and integrated to the slicer trunk.