2009 Winter Project Week Command Line Program Testing

From NAMIC Wiki
Jump to: navigation, search
Home < 2009 Winter Project Week Command Line Program Testing



Key Investigators

  • Bill Lorensen (His Basement)

Objective

There is NA-MIC funded infrastructure in place to regression test the command line programs found in Applications/CLI. Surprisingly, of the over 50 command line programs in the current Slicer3 distribution, only 11 have tests. This project will illustrate the importance of testing and show how easy it is to create tests for command line programs using the existing infrastructure. The project will document the process, create tests and help others to create tests.

Approach, Plan

Using the existing infrastructure, the project will create regression tests for many of the existing command line programs in Applications/CLI. The project will provide expert assistance to those who feel that creating a test is difficult. A Wiki page will provide step-by-step procedures to test the command line programs.

Progress

There is really no excuse for the lack of testing in Applications/CLI. The infrastructure that has been in place for a few years makes it almost painless to test command line programs. Better documentation and peer pressure will help improve the situation.

Tests have been created for all of the command line programs in Applications/CLI (not including sub directories). Three types of tests exist:

  1. --help tests just run the command line program with the --help flag. These tests are as basic as a test can be. All they really test is whether or not the command line program and build, link and report help.
  2. Smoke tests run the program put do not check the results of the program. These are one step above the --help tests and should be the minimum test for each program.
  3. Regression tests run the program and compare the resulting output with a baseline output. Currently only image regression testing is available. Each command line program that produces an image should have a regression test.

Documentation: TODO



Working Notes

Program Test Type Command Line

Add

Regression

${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/AddTest.nhdr

AffineRegistration

None (--help)

BSplineDeformableRegistration

Smoke

--resampledmovingfilename  ${TEMP}/BSplineDeformableRegistrationTest00.nhdr
--outputtransform ${TEMP}/BSplineDeformableRegistrationTest00Transform.txt 
--default 0
--maximumDeformation 1.0
--constrain
--spatialsamples 10000
--histogrambins 32
--gridSize 5
--iterations 50
${TEST_DATA}/CTHeadAxial.nhdr 
${TEST_DATA}/CTHeadAxial.nhdr 

CheckerBoard

Regression

--checkerPattern 5,3,2
${TEST_DATA}/CTHeadAxial.nhdr
${TEST_DATA}/CTHeadAxialDoubled.nhdr
${TEMP}/CheckerBoardTest.nhdr

ConfidenceConnected

None (Regression)

--smoothingIterations 1
--timestep 0.0625
--iterations 5
--multiplier 2.5
--neighborhood 1
--labelvalue 2
--seed 9.71622,-8.1,14.2189 --seed -12.9459,-8.1,18.577
${TEST_DATA}/MRHeadResampled.nhdr
${TEMP}/ConfidenceConnectedTest.nhdr

CurvatureAnisotropicDiffusion

None (Regression)

--conductance 2
--timeStep 0.0625
--iterations 2
${TEST_DATA}/MRHeadResampled.nhdr
${TEMP}/CurvatureAnisotropicDiffusionTest.nhdr

DicomToNrrdConverter

None (--help)

DiffusionTensorEstimation

None (--help)

DiffusionTensorMathematics

None (--help)

DiffusionTensorTest

None (--help)

DiffusionWeightedTest

None (--help)

DWIDicomLoad

None (--help)

ExecutionModelTour

None (Smoke)

--integer 30
--double 30
-f 1.3,2,-14
--string_vector foo,bar,foobar
--enumeration Bill
--boolean1
--transform1 ${TEST_DATA}/ExecutionModelTourTest.mrml\#vtkMRMLLinearTransformNode1
--transform2 ${TEST_DATA}/ExecutionModelTourTest.mrml\#vtkMRMLLinearTransformNode2
${TEST_DATA}/MRHeadResampled.nhdr
${TEST_DATA}/CTHeadAxial.nhdr

ExtractSkeleton/ExtractSkeleton

None (--help)

FreesurferSurfaceSectionExtraction

None (--help)

GaussianBlurImageFilter

Regression

--sigma 2.5 ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/GaussianBlurImageFilterTest.nhdr

GradientAnisotropicDiffusion

None (--help)

GrayscaleFillHoleImageFilter

None (--help)

GrayscaleGrindPeakImageFilter

None (--help)

GrayscaleModelMaker

None (--help)

HistogramMatching

None (--help)

ImageLabelCombine

None (--help)

ImageReadDicomWrite

None (--help)

LabelMapSmoothing

None (--help)

LinearRegistration

Smoke

--resampledmovingfilename  ${TEMP}/LinearRegistrationTest00.nhdr
--outputtransform ${TEMP}/LinearRegistrationTest00Transform.txt 
--initialtransform ${TEST_DATA}/itkAffineTransform00.txt
--spatialsamples 10000
--histogrambins 32
--iterations 50
--learningrate 0.01
--translationscale 100
${TEST_DATA}/CTHeadAxial.nhdr 
${TEST_DATA}/CTHeadAxial.nhdr 

MedianImageFilter

Regression

--neighborhood 1,2,3 ${TEST_DATA}/CTHeadAxial.nhdr ${TEMP}/MedianImageFilterTest.nhdr

ModelMaker

None (Smoke)

--labels 2
--modelSceneFile ${TEMP}/ModelMakerTest.mrml\#vtkMRMLModelHierarchyNode1
${MRML_TEST_DATA}/helix-roi-lable2.nrrd

MultipleModelsExample

None (--help)

OrientImage

Regression

--orientation Axial ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestAxial.nrrd


--orientation Coronal ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestCoronal.nrrd


--orientation Sagittal ${MRML_DATA}/fixed.nrrd ${TEMP}/OrientImageTestSagittal.nrrd

OtsuThresholdImageFilter

None

OtsuThresholdSegmentation

None (Regression)

--brightObjects
--numberOfBins 128
--faceConnected
--minimumObjectSize 1000
${TEST_DATA}/CTHeadResampled.nhdr
${TEMP}/OtsuThresholdImageFilterTest.nhdr 

ProbeVolumeWithModel

None (--help)

Realign/Realign

None (--help)

ResampleVolume2

None (--help)

ResampleVolume

None (--help)

RigidRegistration

None (--help)

Subtract

None (--help)

TestGridTransformRegistration

Smoke

--gridSize 5
--outputtransform ${TEMP}/TestGridTransformRegistrationTest00Transform.txt
${TEST_DATA}/CTHeadAxial.nhdr ${TEST_DATA}/CTHeadAxial.nhdr

VotingBinaryHoleFillingImageFilter

None (Regression)

--radius 3,3,3 
--majorityThreshold 1
--background 0
--foreground 1
${TEST_DATA}/CTHeadResampledOtsuSegmented.nhdr
${TEMP}/VotingBinaryHoleFillingImageFilterTest.nhdr

ZeroCrossingBasedEdgeDetectionImageFilter.xml

None (Regression)

${TEST_DATA}/CTHeadResampled.nhdr
${TEMP}/ZeroCrossingBasedEdgeDetectionImageFilterTest.nhdr