Difference between revisions of "2012 Summer Project Week:SelfTesting"

From NAMIC Wiki
Jump to: navigation, search
(Created page with '__NOTOC__ <gallery> Image:PW-MIT2012.png|Projects List </gallery> ==Key Investigators== * Isomics: Steve Pieper * Kitware: Jean-Christophe…')
 
 
(6 intermediate revisions by the same user not shown)
Line 18: Line 18:
 
* Exactly recreate testing steps so that any user-level issues can be narrowed down (e.g. if a tool works on test data, but not on the user's data).
 
* Exactly recreate testing steps so that any user-level issues can be narrowed down (e.g. if a tool works on test data, but not on the user's data).
 
* Support testing in a continuous loop to look for memory corruption or leaks.
 
* Support testing in a continuous loop to look for memory corruption or leaks.
 
 
 
 
 
 
  
 
</div>
 
</div>
Line 30: Line 24:
  
 
<h3>Approach, Plan</h3>
 
<h3>Approach, Plan</h3>
 +
 +
The goal at project week is to discuss these issues and formulate a standard plan that can be used by all slicer developers.
  
 
Some desirable features of a solution:
 
Some desirable features of a solution:
Line 48: Line 44:
  
 
<div style="width: 40%; float: left;">
 
<div style="width: 40%; float: left;">
 
  
 
<h3>Progress</h3>
 
<h3>Progress</h3>
  
* [https://github.com/pieper/Slicer/tree/1922-bist a prototype implementation as a scripted module] is being developed using a factory-style mechanism where modules can register self-tests with slicer that will then be optionally run as part of the self-testing process.
+
* [https://github.com/pieper/Slicer/tree/1922-bist a prototype implementation as a scripted module] is being developed using a factory-style mechanism where modules can register self-tests with slicer that will then be optionally run as part of the self-testing process. [https://github.com/pieper/Slicer/blob/75f79b165ac1f45571ae41c76e2b7d632d27853a/Modules/Scripted/Scripts/SelfTests.py Prototype scripted module]
 
* The issue is being tracked as a [http://na-mic.org/Bug/view.php?id=1922 feature request in mantis]
 
* The issue is being tracked as a [http://na-mic.org/Bug/view.php?id=1922 feature request in mantis]
 
+
* [https://github.com/pieper/Slicer/blob/7f43f0388a80735be5346b603b009f8d9a741e62/Modules/Scripted/EditorLib/Testing/ThresholdThreadingTest.py Example of a test]
 +
* Discussion with some developers revealed no preferred alternative built in self test architecture
 +
* This approach will be further elaborated and integrated to the slicer trunk.
  
 
</div>
 
</div>

Latest revision as of 11:45, 22 June 2012

Home < 2012 Summer Project Week:SelfTesting


Key Investigators

  • Isomics: Steve Pieper
  • Kitware: Jean-Christophe Fillion-Robin, Julien Finet, Benjamin Long
  • BWH: Sonia Pujol

Objective

BIST (built-in self-test) is an important concept in delivery of high-reliability engineered systems. While Slicer and the packages in the NA-MIC Kit are fairly well tested by open source software standards, the focus has been almost exclusively on build-time testing. But this leaves out many important variables that can lead to user-level issues. A BIST should be runnable from the installed version of Slicer on the user's machine. Desirable features include:

  • Ensure that packaging and installation have succeeded correctly, taking into account the variability of the user's platform (OS version, memory, graphics card, drivers, etc)
  • Exactly recreate testing steps so that any user-level issues can be narrowed down (e.g. if a tool works on test data, but not on the user's data).
  • Support testing in a continuous loop to look for memory corruption or leaks.

Approach, Plan

The goal at project week is to discuss these issues and formulate a standard plan that can be used by all slicer developers.

Some desirable features of a solution:

  • easy for developers to write tests that capture the full workflow
    • distribution of tests as extensions as part of the debugging and support process
    • integration with QtTesting framework to define tests 'by doing'
  • same tests should be available in CTest and BIST (ideally this will happen automatically)
  • interface features:
    • should allow selection of all tests or a subset of tests
    • should allow running in a loop
    • should present a summary of success/failure
  • (possible feature) dialog when slicer first installs that suggests running the tests
  • (possible feature) integration of performance testing (e.g. render speed)
  • (possible feature) integration with CDash so users can submit test results

Progress

Delivery Mechanism

This work will be delivered to the NA-MIC Kit as a (please select the appropriate options by noting YES against them below)

  1. ITK Module
  2. Slicer Module - YES
    1. Built-in - YES
    2. Extension -- commandline
    3. Extension -- loadable
  3. Other (Please specify)

References