NAMIC Wiki - User contributions [en]

Project Week 25/Next Generation GPU Volume Rendering

2017-06-28T17:16:34Z

Sankhesh: /* Addendum */

__NOTOC__

Back to [[Project_Week_25#Projects|Projects List]]

==Key Investigators==
* [http://nist.mni.mcgill.ca/?page_id=369 Simon Drouin] (Montreal Neurological Institute, Canada)
* [http://www.spl.harvard.edu/pages/People/pieper Steve Pieper ] (Isomics Inc., USA)
* [http://perk.cs.queensu.ca/users/lasso Andras Lasso] (Queen's University, Canada)
* [http://www.sintef.no/en/all-employees/employee/?empId=541 Ole Vegard Solberg] (SINTEF, Norway)
* [https://www.kitware.com/alvaro-sanchez/ Alvaro Sanchez] (Kitware Inc., USA <i>Remote attendance</i>)
* [https://www.kitware.com/sankhesh-jhaveri/ Sankhesh Jhaveri] (Kitware Inc., USA <i>Remote attendance</i>)

==Project Description==
{| class="wikitable"
! style="text-align: left; width:27%" | Objective
! style="text-align: left; width:27%" | Approach and Plan
! style="text-align: left; width:27%" | Progress and Next Steps
|- style="vertical-align:top;"
|

Develop a specification for the next generation of GPU volume processing and rendering in VTK and Slicer

The specification should support

* Multiple input volumes for ray-casting and slice-based volume rendering
* GPU-based volume slicing and compositing
* Programmable volume rendering shader
* Smart management of volumes in GPU memory
* Possibility to run simple processing operations on the volume (e.g. Non-linear transforms)
* Consider possible complications
** GPU features not available / software only fallback
** Volumes too big to for GPU memory
|

Compare the architectures of different existing projects where parts of the required functionality has been implemented:

* [http://ibisneuronav.org Ibis Neuronav]/PRISM (Multi-volume ray-casting with programmable shaders in VTK)
* Steve Pieper's volume processing on the GPU with [https://github.com/pieper/CommonGL OpenGL in VTK and GLSL] and with [http://afruehstueck.github.io/TF.html WebGL (with Anna Fruehstueck)] and [https://www.youtube.com/watch?v=wMkuwzA7RDo WebGL2] (see videos below)
* [https://www.custusx.org/ CustusX ] (GPU volume slicing and compositing with multi-window context sharing)

Determine a sensible way to integrate all those contribution in VTK.
|

|}

==Illustrations==
https://www.dropbox.com/s/bqg1fyd42z6lawn/prism-demo-simon-drouin-imic-2016.mp4?dl=0

3D Image Filters in WebGL2
<embedvideo service="youtube">https://www.youtube.com/watch?v=wMkuwzA7RDo&modestbranding=1&rel=0</embedvideo>
Multivolume rendering and nonlinear transforms in WebGL2
<embedvideo service="youtube">https://www.youtube.com/watch?v=ML9_JWAz1kY&modestbranding=1&rel=0</embedvideo>

==Background and References==

[https://docs.google.com/document/d/1-4Up_Shq6oFTGhwXIF5DuiXUYsdIMlAC1oK7eNHWP_o/edit#heading=h.ut6owd65r1fu Some notes about sharing GLSL code between desktop OpenGL and WebGL]

Schott M, Pascal Grosset AV, Martin T, Pegoraro V, Smith ST, Hansen CD. Depth of Field Effects for Interactive Direct Volume Rendering. Comput Graph Forum. 2011 Jun;30(3):941-50.

Volumetric shadows and light scattering:

Kniss J, Premoze S, Hansen C, Shirley P, McPherson A. A model for volume lighting and modeling. IEEE Trans Vis Comput Graph. 2003;9(2):150–62.

Ambiant occlusion (this one should be possible to do with ray casting, but not as efficient):

Hernell F, Ljung P, Ynnerman A. Local Ambient Occlusion in Direct Volume Rendering. IEEE Trans Vis Comput Graph. 2010 Jul;16(4):548–59.

=== Planning hangout June 13 ===
(Simon, Jc, Alvaro, Sankhesh, Steve, Hina)

Recent features in VTK Volume Rendering (in master, blog post coming)
* Volume peeling - translucent geometry with volumes in GPU ray cast mapper
* Render to texture
* 2D lookup tables (value and gradient magnitude)
Work in progress
* Overlapping volumes - multiple inputs to mapper
* vtk charts to work with 2D transfer functions

Slicer to migrate to latest version once cmake hierarchy is sorted out (Jc)

Questions / discussion points:
* Ray cast vs view aligned plane-based algorithms
** depth of focus, shadows, diffuse lighting...
** How to integrate multiple features
* Nonlinear transformation
* Custom shaders
* Dynamic shader generation in python
* multiple components
** RGBA
** Independent components?
* 2D lookup tables
* Volumes that live on the GPU
** sharing across contexts
** use as input textures and render targets
** tiling?
** streaming?
* Large volumes?
* Mesa backend?
** Offscreen support for OpenGL2 backend should work in VTK now

=== Hangout at Project Week ===

* Attendees: Alvaro, Sankhesh, Aashish, Andras, Steve, Jc, Simon, Ole

* Open Questions:
** What is best to accomplish at Project Week?
** What architecture is best for the future and how can we get there?

* Alvero announced that volume peeling supports the OpenGL render pass, which could support custom shaders
** two calls: Prereplace and Postreplace (as in vtkPolyDataMapper)
** newer OpenGL2 version is much more flexible than the older version in OpenGL1
** allows user to replace parts of the rending process

* Need to be able to change
** per-sample ray accumulating value
** compositing function
** initialization of ray path parameters
** control over when ray terminates
** uniforms
** textures

* RenderPath vs RenderTags example?
** Pattern is used in the vtk volume rendering mapper
** existing implementation serves as a working example for customization
** may need to define new spots to inject code (TBD)

Debugging
** Pattern to add unique identifiers to shader variables
** There are vtk hooks to help debug
** Access to compiler errors, and full shader scene
** Can use an apitrace program (linux and mac) to track opengl state

How to make it easier to customize?
** e.g. coloring samples by depth for 4D US
** high level API?
** set of worked out examples for common use cases

Next steps
* At project week:
** collect examples of things that should be simple
** provide feedback and suggestions to vtk developer
* At Kitware:
** propose a high level API for feedback
** multiple overlapping volumes
** look at sharing volumes across mappers (class to represent a volume in memory)
** shared GL context (see CustusX for worked out example of a "hacked in" way to support this)

Long-Term
* Be able to create volume processing pipelines that connect CPU and GPU processing seemlessly
* Investigate the capabilities of vtk-m

===== Addendum =====
* [https://blog.kitware.com/vtk-8-0-0/ VTK 8.0.0 release notes]
* [http://m.vtk.org/ VTK-M]

Project Week 25/Next Generation GPU Volume Rendering

2017-06-28T17:15:54Z

Sankhesh: /* Hangout at Project Week */

Project Week 25/Next Generation GPU Volume Rendering

2017-06-28T17:07:04Z

Sankhesh: