Difference between revisions of "Slicer3:Execution Model Testing"

From NAMIC Wiki
Jump to: navigation, search
m (Text replacement - "http://www.slicer.org/slicerWiki/index.php/" to "https://www.slicer.org/wiki/")
 
(16 intermediate revisions by 4 users not shown)
Line 1: Line 1:
The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.
+
The design of [https://www.slicer.org/wiki/Slicer3:Execution_Model Slicer3's Execution Model] facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.
  
 
==Overview==
 
==Overview==
The Execution Model testing uses ITK's test driver. This test driver is located in [http://public.kitware.com/cgi-bin/viewcvs.cgi/Code/Common/itkTestMain.h?root=Insight&view=markup Insight/Code/Common/itkTestMain.h]. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to a CDash dashboard. The test driver can also compare the test results with baseline images for the tests.
+
The Execution Model testing uses ITK's test driver. This test driver is located in [http://public.kitware.com/cgi-bin/viewcvs.cgi/Code/Common/itkTestMain.h?root=Insight&view=markup Insight/Code/Common/itkTestMain.h]. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to the [http://www.cdash.org/CDash/index.php?project=Slicer3 Slicer3 CDash dashboard]. The test driver can also compare the test results with baseline images for the tests.
  
 
In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.
 
In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.
  
 
The CLP testing follows a number of conventions.
 
The CLP testing follows a number of conventions.
# Tests reside in Slicer3/Applications/CLI/Testing
+
# Tests reside in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/CLI/Testing/ Slicer3/Applications/CLI/Testing]
# Most test data resides in Slicer3/Testing/Data/Input
+
# Most test data resides in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Input/ Slicer3/Testing/Data/Input]
# Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
+
# Baselines for regression tests reside in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Baseline/CLI/ Slicer3/Testing/Data/Baseline/CLI]
 
# Naming conventions:
 
# Naming conventions:
 
## The test driver for a CLP program called Foo is in FooTest.cxx
 
## The test driver for a CLP program called Foo is in FooTest.cxx
Line 16: Line 16:
 
==Creating a Test==
 
==Creating a Test==
 
There are 5 steps to create and run a test for a CLP program:
 
There are 5 steps to create and run a test for a CLP program:
# Create a test driver file in Applications/CLI/Testing
+
# Create a test driver file
 
# Design a test and optionally, create baseline images for the test
 
# Design a test and optionally, create baseline images for the test
 
# Configure the test
 
# Configure the test
Line 25: Line 25:
  
 
===Creating a Test Driver===
 
===Creating a Test Driver===
In Applications/CLI/Testing create the file ''CurvatureAnisotropicDiffusionTest.cxx''
+
In [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/CLI/Testing/ Applications/CLI/Testing] create the file ''CurvatureAnisotropicDiffusionTest.cxx''
{| border="1"
+
{|class="wikitable" border="1"
! CMake Code
+
! C++ Code
 
! Explanation
 
! Explanation
 
|-
 
|-
Line 47: Line 47:
 
===Designing a Test===
 
===Designing a Test===
 
Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:
 
Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Input/?rev=8264 Testing/Data/Input]
+
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Input/ Testing/Data/Input]
 
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData/ Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData] and its sub directories.
 
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData/ Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData] and its sub directories.
 
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Libs/MRML/Testing/TestData/ Libs/MRML/Testing/TestData]
 
# [http://viewvc.slicer.org/viewcvs.cgi/trunk/Libs/MRML/Testing/TestData/ Libs/MRML/Testing/TestData]
Line 56: Line 56:
 
# Define appropriate parameters.
 
# Define appropriate parameters.
 
# Run the program.
 
# Run the program.
If the program produces an image, use File->Save to save the resulting image as a Baseline in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Baseline/CLI/?rev=8264 Testing/Data/Baseline/CLI]. The name of the baseline image should follow the naming conventions described above.
+
If the program produces an image, use File->Save to save the resulting image as a Baseline in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Baseline/CLI/ Testing/Data/Baseline/CLI]. The name of the baseline image should follow the naming conventions described above.
  
 
===Configure Build and Add the test===
 
===Configure Build and Add the test===
Edit the CMakeLists.txt file in Application/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:
+
Edit the [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/CLI/Testing/CMakeLists.txt?view=markup CMakeLists.txt] file in [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/CLI/Testing/ Applications/CLI/Testing]. For the CurvatureAnisotropicDiffusion program the additional lines are:
{| border="1"
+
{| class="wikitable" border="1"
 
! CMake Code
 
! CMake Code
 
! Explanation
 
! Explanation
Line 86: Line 86:
 
   )
 
   )
 
|
 
|
This cmake code creates the test and specifies the command line arguments. Tests are run using the ''--launch'' option of the Slicer3 program. This assures that all required environment variables are set properly. The ''--compare'' flag and its two arguments ''${BASELINE}/${CLP}Test.nhdr'' and ''${TEMP}/${CLP}Test.nhdr'' are only present for regression tests. Notice that the output of the test is stored in the ''${TEMP}'' directory. For regression tests, this output is compared with the image in ''${BASELINE}''.
+
This cmake code creates the test and specifies the command line arguments. Tests are run using the ''--launch'' option of the Slicer3 program. This assures that all required environment variables are set properly. The ''--compare'' flag and its two arguments ''${BASELINE}/${CLP}Test.nhdr'' and ''${TEMP}/${CLP}Test.nhdr'' are only present for regression tests. Notice that the output of the test is stored in the ''${TEMP}'' directory. For regression tests, this output is compared with the image in ''${BASELINE}''.[1]
 +
|-
 +
|}
 +
 
 +
[1] If the output of your command line module is not a volume, but is a different type of file on disk, you can use the cmake compare command to test the output. See an example with the MergeModels test checked in [http://viewvc.slicer.org/viewcvs.cgi?rev=13123&view=rev here].
 +
 
 +
ITK testing framework is defined in $ITK_HOME/Code/Common/itkTestMain.h, which utilizes [http://www.itk.org/Doxygen/html/classitk_1_1DifferenceImageFilter.html itkDifferenceImageFilter] for baseline comparison. Additional regression testing functionality allows to specify the following comparison parameters (NOTE: the order is important: these parameters must follow in this order after ''--compare image1 image2''):
 +
 
 +
{|class="wikitable" border="1"
 +
! parameter
 +
! explanation
 +
|-
 +
|
 +
--compareNumberOfPixelsTolerance <integer> (default 0)
 +
|if this number (or less) of pixels is different from the baseline, the test will succeed
 +
|-
 +
|
 +
--compareRadiusTolerance <integer> (default 0)
 +
|search in the neighborhood of this radius for the matching pixel (be careful with this, since setting this to non-0 will increase drastically the execution time of your test!)
 +
|-
 +
|
 +
--compareIntensityTolerance <float> (default 2.0)
 +
|pixels with the intensities that differ by the specified value are considered same
 
|-
 
|-
 
|}
 
|}
Line 99: Line 121:
  
 
=== Check in the Test Files===
 
=== Check in the Test Files===
In Applications/CLI/Testing,
+
In [http://viewvc.slicer.org/viewcvs.cgi/trunk/Applications/CLI/Testing/ Applications/CLI/Testing],
  
 
  svn add CurvatureAnisotropicDiffusionTest.cxx
 
  svn add CurvatureAnisotropicDiffusionTest.cxx
Line 105: Line 127:
 
   CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx
 
   CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx
  
In Testing/Data/Input
+
In [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Input/ Testing/Data/Input] add and commit any new input data.
  
Add and commit any new input data.
+
In [http://viewvc.slicer.org/viewcvs.cgi/trunk/Testing/Data/Baseline/CLI/ Testing/Data/Baseline/CLI]
 
 
In Testing/data/Baseline/CLI
 
  
 
  svn add CurvatureAnisotropicDiffusionTest.nhdr
 
  svn add CurvatureAnisotropicDiffusionTest.nhdr
Line 116: Line 136:
 
         CurvatureAnisotropicDiffusionTest.nhdr
 
         CurvatureAnisotropicDiffusionTest.nhdr
 
         CurvatureAnisotropicDiffusionTest.raw.gz
 
         CurvatureAnisotropicDiffusionTest.raw.gz
 +
 +
== References ==
 +
* [http://www.itk.org/Wiki/ITK_Procedure_for_Adding_a_Test ITK Procedure for adding a test (ITK wiki)]
 +
* [http://en.wikipedia.org/wiki/Regression_testing Regression testing (Wikipedia)]

Latest revision as of 17:44, 10 July 2017

Home < Slicer3:Execution Model Testing

The design of Slicer3's Execution Model facilitates testing, since the command line programs (CLP's) can be run as stand-alone programs. CMake, CTest and CDash provide mechanism to build, run and report the results of tests.

Overview

The Execution Model testing uses ITK's test driver. This test driver is located in Insight/Code/Common/itkTestMain.h. The ITK test driver runs each test as a subprocess, collects the output of the test and wraps the test output into an XML description of the test. Optionally, after all tests are run, CTest submits the results to the Slicer3 CDash dashboard. The test driver can also compare the test results with baseline images for the tests.

In ITK, many test are run from a test executable. Currently, in Slicer3, each test executable runs one test.

The CLP testing follows a number of conventions.

  1. Tests reside in Slicer3/Applications/CLI/Testing
  2. Most test data resides in Slicer3/Testing/Data/Input
  3. Baselines for regression tests reside in Slicer3/Testing/Data/Baseline/CLI
  4. Naming conventions:
    1. The test driver for a CLP program called Foo is in FooTest.cxx
    2. The baseline image for a CLP program called Foo is in FooTest.nhdr or some other image format.

Creating a Test

There are 5 steps to create and run a test for a CLP program:

  1. Create a test driver file
  2. Design a test and optionally, create baseline images for the test
  3. Configure the test
  4. Run the test
  5. Check in the test files

The following sections create a test for the CLP program CurvatureAnisotropicDiffusion.

Creating a Test Driver

In Applications/CLI/Testing create the file CurvatureAnisotropicDiffusionTest.cxx

C++ Code Explanation
#include <iostream>
#include "itkTestMain.h" 
void RegisterTests()
{
  REGISTER_TEST(CurvatureAnisotropicDiffusionTest);
}
#undef main
#define main CurvatureAnisotropicDiffusionTest
#include "CurvatureAnisotropicDiffusion.cxx"

This is the ITK test driver. ITK tests usually REGISTER several tests. However, because of the nature of Slicer3 command line programs, only one test can exist in each test driver.

Designing a Test

Find a set of data that you can use for the test. There are several sources of data in Slicer3 including:

  1. Testing/Data/Input
  2. Modules/EMSegment/Testing/TestData/TutorialTest2/VolumeData and its sub directories.
  3. Libs/MRML/Testing/TestData

The dataset should be fairly small, but be large enough to produce a result that you can verify.

  1. Now run Slicer3 and load the data.
  2. Select the program that you will test.
  3. Define appropriate parameters.
  4. Run the program.

If the program produces an image, use File->Save to save the resulting image as a Baseline in Testing/Data/Baseline/CLI. The name of the baseline image should follow the naming conventions described above.

Configure Build and Add the test

Edit the CMakeLists.txt file in Applications/CLI/Testing. For the CurvatureAnisotropicDiffusion program the additional lines are:

CMake Code Explanation
##################################
# CurvatureAnisotropicDiffusion tests
set (CLP CurvatureAnisotropicDiffusion)
add_executable(${CLP}Test ${CLP}Test.cxx)
add_dependencies(${CLP}Test ${CLP})
target_link_libraries(${CLP}Test ITKIO)

This cmake code creates the test driver executable for the test. The target_link_libraries should be the same ones specified when the command line program is build in Applications/CLI. This ITK-based command line program only needs the ITKIO library.

add_test(${CLP}Test ${Slicer3_EXE} --launch ${CLP}Test
  --compare ${BASELINE}/${CLP}Test.nhdr
            ${TEMP}/${CLP}Test.nhdr
  ${CLP}Test
    --conductance 2
    --timeStep 0.0625
    --iterations 2
    ${TEST_DATA}/MRHeadResampled.nhdr
    ${TEMP}/${CLP}Test.nhdr
  )

This cmake code creates the test and specifies the command line arguments. Tests are run using the --launch option of the Slicer3 program. This assures that all required environment variables are set properly. The --compare flag and its two arguments ${BASELINE}/${CLP}Test.nhdr and ${TEMP}/${CLP}Test.nhdr are only present for regression tests. Notice that the output of the test is stored in the ${TEMP} directory. For regression tests, this output is compared with the image in ${BASELINE}.[1]

[1] If the output of your command line module is not a volume, but is a different type of file on disk, you can use the cmake compare command to test the output. See an example with the MergeModels test checked in here.

ITK testing framework is defined in $ITK_HOME/Code/Common/itkTestMain.h, which utilizes itkDifferenceImageFilter for baseline comparison. Additional regression testing functionality allows to specify the following comparison parameters (NOTE: the order is important: these parameters must follow in this order after --compare image1 image2):

parameter explanation

--compareNumberOfPixelsTolerance <integer> (default 0)

if this number (or less) of pixels is different from the baseline, the test will succeed

--compareRadiusTolerance <integer> (default 0)

search in the neighborhood of this radius for the matching pixel (be careful with this, since setting this to non-0 will increase drastically the execution time of your test!)

--compareIntensityTolerance <float> (default 2.0)

pixels with the intensities that differ by the specified value are considered same

Configure the Test

Run CMake followed by make (for linux) or the Visual Studio IDE.

Run the Test

ctest -V -R CurvatureAnisotropicDiffusionTest

If the test passes, proceed.

Check in the Test Files

In Applications/CLI/Testing,

svn add CurvatureAnisotropicDiffusionTest.cxx
svn commit -m"ENH: New test for CurvatureAnisotropicDiffusion."
 CMakeList.txt CurvatureAnisotropicDiffusiontest.cxx

In Testing/Data/Input add and commit any new input data.

In Testing/Data/Baseline/CLI

svn add CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz
svn commit -m"ENH: Baselines for CurvatureAnisotropicDiffusionTest"
        CurvatureAnisotropicDiffusionTest.nhdr
        CurvatureAnisotropicDiffusionTest.raw.gz

References