Difference between revisions of "Summer2009:Registration reproducibility in Slicer"
(19 intermediate revisions by 3 users not shown) | |||
Line 1: | Line 1: | ||
__NOTOC__ | __NOTOC__ | ||
<gallery> | <gallery> | ||
− | Image:PW2009-v3.png|[[2009_Summer_Project_Week| | + | Image:PW2009-v3.png|[[2009_Summer_Project_Week#Projects|Projects List]] |
Image:transform-gui.jpg|Transform for test data/GUI registration | Image:transform-gui.jpg|Transform for test data/GUI registration | ||
Image:transform-cli.jpg|Transform for test data/CLI registration | Image:transform-cli.jpg|Transform for test data/CLI registration | ||
Line 21: | Line 21: | ||
# There is also significant difference between command line invocations of the module on different platforms. | # There is also significant difference between command line invocations of the module on different platforms. | ||
− | The issue has been confirmed independently by Andriy, Bill, Jim and Steve. It was also confirmed by Kilian. The problem is thoroughly documented in [http://www.na-mic.org/Bug/view.php?id=416 Slicer3 bug 416]. | + | The issue has been confirmed independently by Andriy, Bill, Jim and Steve. It was also confirmed by Kilian. The problem is thoroughly documented in [http://www.na-mic.org/Bug/view.php?id=416 Slicer3 bug 416]. The discrepancy is present with seed (and, for RegisterImages, seed and number of threads) set to the same value between runs. |
We would like to understand the source of this inconsistency, together with the understanding what is reasonable to expect expect in terms of reproducibility from such complex numerical codes like rigid registration. | We would like to understand the source of this inconsistency, together with the understanding what is reasonable to expect expect in terms of reproducibility from such complex numerical codes like rigid registration. | ||
Line 43: | Line 43: | ||
* added testing mode to RigidRegistration to measure the maximum difference | * added testing mode to RigidRegistration to measure the maximum difference | ||
* abused Slicer3 dashboard to collect the magnitude of difference on various platforms | * abused Slicer3 dashboard to collect the magnitude of difference on various platforms | ||
+ | * number of threads used apparently has impact on registration reproducibility, should be specified | ||
+ | * Number of threads should not affect the result of registration; Luis reproduced the problem outside Slicer, confirmed as an ITK issue: NAMICSandbox/PNL/RegistrationTestBench/Testing | ||
+ | |||
+ | Reference for the initiated [http://www.itk.org/Bug/view.php?id=9222 ITK bug #9222] (open as of Dec 20, 2009) | ||
</div> | </div> | ||
Line 66: | Line 70: | ||
</pre> | </pre> | ||
− | 4. To get the actual difference in the corner of your image using one transform vs another, you can run RigidRegistration in the testing mode. '''Testing feature is not available in stable release -- you need to download nightly, or | + | 4. To get the actual difference in the corner of your image using one transform vs another, you can run RigidRegistration in the testing mode. '''Testing feature is not available in stable release -- you need to download nightly, or build from sources!'''. In testing mode, initial transform is NOT used as the initial tranform, but is simply a "ground truth" transform to test reproducibility. |
<pre> | <pre> | ||
Line 77: | Line 81: | ||
|[[Image:rigid_registration_reproducibility_setup1.jpg|thumb|250px|RigidRegistration initialization in GUI]] | |[[Image:rigid_registration_reproducibility_setup1.jpg|thumb|250px|RigidRegistration initialization in GUI]] | ||
|} | |} | ||
+ | |||
+ | ==Participate!== | ||
+ | |||
+ | Here are the results of running RigidRegistration module with the parameters set up as above, on the provided data. Please feel free to run the test on your platform using this command line (you need build from source or nightly build to do this): | ||
+ | <pre> | ||
+ | Slicer3 --launch RigidRegistration --testingmode --initialtransform MRMeningioma01.tfm | ||
+ | --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 | ||
+ | --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 | ||
+ | --outputtransform ./RigidRegistrationTransform-CLI.txt MRMeningioma0.nrrd MRMeningioma1.nrrd | ||
+ | </pre> | ||
+ | |||
+ | Or use this version, which you can cut and paste into a Slicer3-build directory: | ||
+ | * mac/linux: | ||
+ | <pre> | ||
+ | ./Slicer3 --launch ./lib/Slicer3/Plugins/RigidRegistration --testingmode --initialtransform ../Slicer3/Testing/Data/Input/MRMeningioma01.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt ../Slicer3/Testing/Data/Input/MRMeningioma0.nrrd ../Slicer3/Testing/Data/Input/MRMeningioma1.nrrd | ||
+ | </pre> | ||
+ | * windows: | ||
+ | <pre> | ||
+ | ./Slicer3 --launch ./lib/Slicer3/Plugins/Debug/RigidRegistration.exe --testingmode --initialtransform ../Slicer3/Testing/Data/Input/MRMeningioma01.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt ../Slicer3/Testing/Data/Input/MRMeningioma0.nrrd ../Slicer3/Testing/Data/Input/MRMeningioma1.nrrd | ||
+ | </pre> | ||
+ | |||
+ | Once you do this, you can contribute the transform, details of your system, and the error value here: | ||
+ | |||
+ | * Baseline: george.bwh.harvard.edu (linux) -- the e-06 discrepancy is the same for all fat nodes at SPL! Does it have to do with multi-threading? | ||
+ | <pre> | ||
+ | MRMeningioma01.tfm: | ||
+ | |||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: QuaternionRigidTransform_double_3_3 | ||
+ | Parameters: 0.00993255 0.00724952 0.00477429 0.999913 7.40702 4.83973 -0.0941685 | ||
+ | FixedParameters: 0.00025 0.00025 -1.42109e-14 | ||
+ | --- | ||
+ | Magnitude of error vector: 4.7826e-06 | ||
+ | </pre> | ||
+ | |||
+ | * cluster.bwh.harvard.edu (linux) | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: QuaternionRigidTransform_double_3_3 | ||
+ | Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 | ||
+ | FixedParameters: 0.00025 0.00025 -1.42109e-14 | ||
+ | --- | ||
+ | Magnitude of error vector: 2.72849 | ||
+ | </pre> | ||
+ | |||
+ | * Andrey Fedorov's laptop / Windows XP, dual core | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: QuaternionRigidTransform_double_3_3 | ||
+ | Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 | ||
+ | FixedParameters: 0.00025 0.00025 -1.42109e-014 | ||
+ | --- | ||
+ | Magnitude of error vector: 2.72849 | ||
+ | </pre> | ||
+ | |||
+ | * transplant (Steve Pieper's Mac Book Pro - dual core) | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: QuaternionRigidTransform_double_3_3 | ||
+ | Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 | ||
+ | FixedParameters: 0.00025 0.00025 -1.42109e-14 | ||
+ | --- | ||
+ | Magnitude of error vector: 2.72849 | ||
+ | </pre> | ||
+ | |||
+ | * Joe.bwh.harvard.edu - Windows XP64 8 core | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: QuaternionRigidTransform_double_3_3 | ||
+ | Parameters: 0.00825374 0.00297963 0.0129941 0.999877 7.56843 5.34521 -0.0495314 | ||
+ | FixedParameters: 0.00025 0.00025 -1.42109e-014 | ||
+ | --- | ||
+ | Magnitude of error vector: 3.3349 | ||
+ | </pre> | ||
+ | |||
+ | == Reproducibility for RegisterImages module== | ||
+ | |||
+ | {| | ||
+ | |align="left"|RegisterImages also produces results that have discrepancies. Here's the command line we used: | ||
+ | <pre> | ||
+ | ./Slicer3 --launch RegisterImages --saveTransform register_images_transform.txt | ||
+ | --initialization CentersOfMass --registration Rigid | ||
+ | --expectedOffset 10 --expectedRotation 0.1 --expectedScale 0.05 --expectedSkew 0.01 | ||
+ | --verbosityLevel Standard --sampleIntensityPortion 0 | ||
+ | --randomNumberSeed 12345 --numberOfThreads 1 | ||
+ | --baselineNumberOfFailedPixelsTolerance 1000 --baselineIntensityTolerance 10 --baselineRadiusTolerance 0 | ||
+ | --rigidMaxIterations 100 --rigidSamplingRatio 0.01 --affineMaxIterations 50 | ||
+ | --affineSamplingRatio 0.02 --bsplineMaxIterations 20 --bsplineSamplingRatio 0.1 --controlPointSpacing 40 | ||
+ | MRMeningioma0.nrrd MRMeningioma1.nrrd | ||
+ | </pre> | ||
+ | |||
+ | Note, that the parameters passed to the executable during the command line invocation are identical to those passed to shared library call, with the exceptions in invocation and data passing (this is available from the Slicer log window, and corresponds to the GUI parameter configuration on the right)): | ||
+ | <pre> | ||
+ | slicer:0x7f74cc6752bd --processinformationaddress 0x7f74c8137e70 --saveTransform register_images_transform.txt | ||
+ | --initialization CentersOfMass --registration Rigid | ||
+ | --expectedOffset 10 --expectedRotation 0.1 --expectedScale 0.05 --expectedSkew 0.01 | ||
+ | --verbosityLevel Standard --sampleIntensityPortion 0 | ||
+ | --randomNumberSeed 12345 --numberOfThreads 1 | ||
+ | --baselineNumberOfFailedPixelsTolerance 1000 --baselineIntensityTolerance 10 --baselineRadiusTolerance 0 | ||
+ | --rigidMaxIterations 100 --rigidSamplingRatio 0.01 --affineMaxIterations 50 | ||
+ | --affineSamplingRatio 0.02 --bsplineMaxIterations 20 --bsplineSamplingRatio 0.1 --controlPointSpacing 40 | ||
+ | slicer:0x10479e0#vtkMRMLScalarVolumeNode2 slicer:0x10479e0#vtkMRMLScalarVolumeNode3 | ||
+ | </pre> | ||
+ | |||
+ | |[[Image:register_images_reproducibility_setup.jpg|thumb|250px|RegisterImages initialization in GUI, which corresponds to the command line invocation]] | ||
+ | |} | ||
+ | |||
+ | Note: You *must* use the same number of threads to have consistent results between runs, otherwise the result is not reproducible on the same platform with the same method of invocation. | ||
+ | |||
+ | * b3_d4_5.bwh.harvard.edu transform when using GUI invocation: | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: AffineTransform_double_3_3 | ||
+ | Parameters: 0.999879 -0.0101701 0.0118087 0.0104324 0.999695 -0.0223654 -0.0115776 0.0224858 0.99968 7.27052 4.66624 -0.100641 | ||
+ | FixedParameters: 0.469 0.469 0.6 | ||
+ | </pre> | ||
+ | |||
+ | ...and CLI invocation: | ||
+ | <pre> | ||
+ | #Insight Transform File V1.0 | ||
+ | # Transform 0 | ||
+ | Transform: AffineTransform_double_3_3 | ||
+ | Parameters: 0.99993 -0.00522759 0.0106436 0.00544963 0.999766 -0.0209396 -0.0105316 0.0209961 0.999724 7.05399 4.50586 -0.00851314 | ||
+ | FixedParameters: 0.469 0.469 0.6 | ||
+ | </pre> | ||
==References== | ==References== | ||
* [http://www.na-mic.org/Bug/view.php?id=416 Mantis entry for bug 416] | * [http://www.na-mic.org/Bug/view.php?id=416 Mantis entry for bug 416] |
Latest revision as of 23:37, 20 December 2009
Home < Summer2009:Registration reproducibility in SlicerKey Investigators
- BWH: Andriy Fedorov, Steve Pieper, Tina Kapur
- GE: Jim Miller
- Kitware: Luis Ibanez
- EAB: Bill Lorensen
Objective
Rigid registration in general, and RigidRegistration module of Slicer3 in particular, are important workflow components for a number of applications. Two issues are of our interest:
- We found gross inconsistency between the result of using rigid registration module on volumetric brain MRI in GUI vs command line invocation.
- There is also significant difference between command line invocations of the module on different platforms.
The issue has been confirmed independently by Andriy, Bill, Jim and Steve. It was also confirmed by Kilian. The problem is thoroughly documented in Slicer3 bug 416. The discrepancy is present with seed (and, for RegisterImages, seed and number of threads) set to the same value between runs.
We would like to understand the source of this inconsistency, together with the understanding what is reasonable to expect expect in terms of reproducibility from such complex numerical codes like rigid registration.
Approach, Plan
- use clinical data to demonstrate the problem, once again
- collect feedback from registration experts
- create a test that demonstrates the problem, and add it to the ITK dashboard
- Questions to consider:
- is reproducibility a function of the architecture? is it dependent on multi-threading? can it be quantified and estimated?
- what is the correct way to test reproducibility for this kind of applications?
Progress
- added testing mode to RigidRegistration to measure the maximum difference
- abused Slicer3 dashboard to collect the magnitude of difference on various platforms
- number of threads used apparently has impact on registration reproducibility, should be specified
- Number of threads should not affect the result of registration; Luis reproduced the problem outside Slicer, confirmed as an ITK issue: NAMICSandbox/PNL/RegistrationTestBench/Testing
Reference for the initiated ITK bug #9222 (open as of Dec 20, 2009)
Test setup
1. Get images and transform from Slicer3 trunk:
2. To get the transform in GUI, go to RigidRegistration module, use the default parameters to initialize (see figure), EXCEPT reduce iteration cout from the default to "100,100,50,20" 3. To get the transform with CLI, here's the command line you need to run: Slicer3 --launch RigidRegistration --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt MRMeningioma0.nrrd MRMeningioma1.nrrd 4. To get the actual difference in the corner of your image using one transform vs another, you can run RigidRegistration in the testing mode. Testing feature is not available in stable release -- you need to download nightly, or build from sources!. In testing mode, initial transform is NOT used as the initial tranform, but is simply a "ground truth" transform to test reproducibility. Slicer3 --launch RigidRegistration --testingmode --initialtransform RigidRegistrationTransform-GUI.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt MRMeningioma0.nrrd MRMeningioma1.nrrd |
Participate!
Here are the results of running RigidRegistration module with the parameters set up as above, on the provided data. Please feel free to run the test on your platform using this command line (you need build from source or nightly build to do this):
Slicer3 --launch RigidRegistration --testingmode --initialtransform MRMeningioma01.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt MRMeningioma0.nrrd MRMeningioma1.nrrd
Or use this version, which you can cut and paste into a Slicer3-build directory:
- mac/linux:
./Slicer3 --launch ./lib/Slicer3/Plugins/RigidRegistration --testingmode --initialtransform ../Slicer3/Testing/Data/Input/MRMeningioma01.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt ../Slicer3/Testing/Data/Input/MRMeningioma0.nrrd ../Slicer3/Testing/Data/Input/MRMeningioma1.nrrd
- windows:
./Slicer3 --launch ./lib/Slicer3/Plugins/Debug/RigidRegistration.exe --testingmode --initialtransform ../Slicer3/Testing/Data/Input/MRMeningioma01.tfm --fixedsmoothingfactor 0 --movingsmoothingfactor 0 --histogrambins 30 --spatialsamples 10000 --iterations 100,100,50,20 --learningrate 0.01,0.005,0.0005,0.0002 --translationscale 100 --outputtransform ./RigidRegistrationTransform-CLI.txt ../Slicer3/Testing/Data/Input/MRMeningioma0.nrrd ../Slicer3/Testing/Data/Input/MRMeningioma1.nrrd
Once you do this, you can contribute the transform, details of your system, and the error value here:
- Baseline: george.bwh.harvard.edu (linux) -- the e-06 discrepancy is the same for all fat nodes at SPL! Does it have to do with multi-threading?
MRMeningioma01.tfm: #Insight Transform File V1.0 # Transform 0 Transform: QuaternionRigidTransform_double_3_3 Parameters: 0.00993255 0.00724952 0.00477429 0.999913 7.40702 4.83973 -0.0941685 FixedParameters: 0.00025 0.00025 -1.42109e-14 --- Magnitude of error vector: 4.7826e-06
- cluster.bwh.harvard.edu (linux)
#Insight Transform File V1.0 # Transform 0 Transform: QuaternionRigidTransform_double_3_3 Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 FixedParameters: 0.00025 0.00025 -1.42109e-14 --- Magnitude of error vector: 2.72849
- Andrey Fedorov's laptop / Windows XP, dual core
#Insight Transform File V1.0 # Transform 0 Transform: QuaternionRigidTransform_double_3_3 Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 FixedParameters: 0.00025 0.00025 -1.42109e-014 --- Magnitude of error vector: 2.72849
- transplant (Steve Pieper's Mac Book Pro - dual core)
#Insight Transform File V1.0 # Transform 0 Transform: QuaternionRigidTransform_double_3_3 Parameters: 0.00687841 0.00580804 0.0113849 0.999895 7.65038 5.16471 0.0120382 FixedParameters: 0.00025 0.00025 -1.42109e-14 --- Magnitude of error vector: 2.72849
- Joe.bwh.harvard.edu - Windows XP64 8 core
#Insight Transform File V1.0 # Transform 0 Transform: QuaternionRigidTransform_double_3_3 Parameters: 0.00825374 0.00297963 0.0129941 0.999877 7.56843 5.34521 -0.0495314 FixedParameters: 0.00025 0.00025 -1.42109e-014 --- Magnitude of error vector: 3.3349
Reproducibility for RegisterImages module
RegisterImages also produces results that have discrepancies. Here's the command line we used:
./Slicer3 --launch RegisterImages --saveTransform register_images_transform.txt --initialization CentersOfMass --registration Rigid --expectedOffset 10 --expectedRotation 0.1 --expectedScale 0.05 --expectedSkew 0.01 --verbosityLevel Standard --sampleIntensityPortion 0 --randomNumberSeed 12345 --numberOfThreads 1 --baselineNumberOfFailedPixelsTolerance 1000 --baselineIntensityTolerance 10 --baselineRadiusTolerance 0 --rigidMaxIterations 100 --rigidSamplingRatio 0.01 --affineMaxIterations 50 --affineSamplingRatio 0.02 --bsplineMaxIterations 20 --bsplineSamplingRatio 0.1 --controlPointSpacing 40 MRMeningioma0.nrrd MRMeningioma1.nrrd Note, that the parameters passed to the executable during the command line invocation are identical to those passed to shared library call, with the exceptions in invocation and data passing (this is available from the Slicer log window, and corresponds to the GUI parameter configuration on the right)): slicer:0x7f74cc6752bd --processinformationaddress 0x7f74c8137e70 --saveTransform register_images_transform.txt --initialization CentersOfMass --registration Rigid --expectedOffset 10 --expectedRotation 0.1 --expectedScale 0.05 --expectedSkew 0.01 --verbosityLevel Standard --sampleIntensityPortion 0 --randomNumberSeed 12345 --numberOfThreads 1 --baselineNumberOfFailedPixelsTolerance 1000 --baselineIntensityTolerance 10 --baselineRadiusTolerance 0 --rigidMaxIterations 100 --rigidSamplingRatio 0.01 --affineMaxIterations 50 --affineSamplingRatio 0.02 --bsplineMaxIterations 20 --bsplineSamplingRatio 0.1 --controlPointSpacing 40 slicer:0x10479e0#vtkMRMLScalarVolumeNode2 slicer:0x10479e0#vtkMRMLScalarVolumeNode3 |
Note: You *must* use the same number of threads to have consistent results between runs, otherwise the result is not reproducible on the same platform with the same method of invocation.
- b3_d4_5.bwh.harvard.edu transform when using GUI invocation:
#Insight Transform File V1.0 # Transform 0 Transform: AffineTransform_double_3_3 Parameters: 0.999879 -0.0101701 0.0118087 0.0104324 0.999695 -0.0223654 -0.0115776 0.0224858 0.99968 7.27052 4.66624 -0.100641 FixedParameters: 0.469 0.469 0.6
...and CLI invocation:
#Insight Transform File V1.0 # Transform 0 Transform: AffineTransform_double_3_3 Parameters: 0.99993 -0.00522759 0.0106436 0.00544963 0.999766 -0.0209396 -0.0105316 0.0209961 0.999724 7.05399 4.50586 -0.00851314 FixedParameters: 0.469 0.469 0.6