Difference between revisions of "Project Week 2008 Special topic breakout: GWE"

From NAMIC Wiki
Jump to: navigation, search
 
Line 1: Line 1:
 
  [[2008_Summer_Project_Week#Agenda|Back to 2008 Summer Project Week Agenda]]
 
  [[2008_Summer_Project_Week#Agenda|Back to 2008 Summer Project Week Agenda]]
 
= Agenda =
 
  
 
== GWE Overview ==
 
== GWE Overview ==
Line 7: Line 5:
 
* GWE Client (End User - You) <=== Secure RPC Network ===> GWE Daemon(s) (Cluster Head Node)  
 
* GWE Client (End User - You) <=== Secure RPC Network ===> GWE Daemon(s) (Cluster Head Node)  
 
* Detailed information can be found on the [http://www.gridwizardenterprise.org GWE Project Site].  
 
* Detailed information can be found on the [http://www.gridwizardenterprise.org GWE Project Site].  
* [[Media:GWE-Core.ppt|Overview Presentation]]
 
  
 
== GWE Setup ==
 
== GWE Setup ==
Line 13: Line 10:
 
=== Installation ===
 
=== Installation ===
 
* Download bundle, unbundle & set appropriate environment variables.
 
* Download bundle, unbundle & set appropriate environment variables.
* Detailed information can be found on the '[http://www.gridwizardenterprise.org/guides/install.html GWE Installation Guide]'
+
* Detailed information can be found on the [http://www.gridwizardenterprise.org/guides/install.html GWE Installation Guide]
  
 
=== Configuration ===
 
=== Configuration ===
 
* Define your grid and provide your authentication information to access "grid resources".
 
* Define your grid and provide your authentication information to access "grid resources".
* Detailed information can be found on the '[http://www.gridwizardenterprise.org/guides/config.html GWE Configuration Guide]'
+
* Detailed information can be found on the [http://www.gridwizardenterprise.org/guides/config.html GWE Configuration Guide]
  
 
=== Daemon Setup ===
 
=== Daemon Setup ===
* ONLY IF YOU ARE GOING TO ENHANCED A NEW CLUSTER WITH GWE CAPABILITIES (GWE Enabled).  
+
* ONLY IF YOU ARE GOING TO ENHANCE A NEW CLUSTER WITH GWE CAPABILITIES (GWE Enabled).  
* "Standalone GWE" will be available in the next release. GWE will run on your machine. Parallelization at the OS level, not distributed across cluster(s).  
+
* "Standalone GWE" will be available in the next release (To run GWE on your machine. Parallelization at the OS level, not distributed across cluster(s)).  
 
* From the machine where you installed GWE, run the utility "gwe-util-daemon-fullsetup.sh". That's it!
 
* From the machine where you installed GWE, run the utility "gwe-util-daemon-fullsetup.sh". That's it!
 
* After that a "GWE Daemon" will be running in your cluster having your cluster user as this "GWE Daemon" administrator (different than roles in the cluster OS!).
 
* After that a "GWE Daemon" will be running in your cluster having your cluster user as this "GWE Daemon" administrator (different than roles in the cluster OS!).
 
* A "GWE Daemon" is just another application in a machine, therefore "GWE Daemon"s can run in a single machine, although it is not recommended.  
 
* A "GWE Daemon" is just another application in a machine, therefore "GWE Daemon"s can run in a single machine, although it is not recommended.  
* Detailed information can be found on the '[http://www.gridwizardenterprise.org/guides/admin.html GWE Administrator Guide]'
+
* Detailed information can be found on the [http://www.gridwizardenterprise.org/guides/admin.html GWE Administrator Guide]
  
 
== GWE Clients ==
 
== GWE Clients ==
Line 37: Line 34:
 
* Uses P2EL proprietary language to describe set of commands as permutation of templated commands with an associated parameter space.
 
* Uses P2EL proprietary language to describe set of commands as permutation of templated commands with an associated parameter space.
 
* Many other monitoring/control commands available.
 
* Many other monitoring/control commands available.
* Detailed information can be found on the '[http://www.gridwizardenterprise.org/guides/user.html End Users Guide]'
+
* Detailed information can be found on the [http://www.gridwizardenterprise.org/guides/user.html End Users Guide]
* [[Media:GWECore-Demo.mov|Video Demo]]
 
  
 
=== GSlicer3 ===
 
=== GSlicer3 ===
Line 45: Line 41:
 
* Setup: Download GSlicer3 bundle, unbundle it, set appropriate environmental variables.
 
* Setup: Download GSlicer3 bundle, unbundle it, set appropriate environmental variables.
 
* Config: Same as regular GWE.
 
* Config: Same as regular GWE.
* Detailed information can be found on the '[http://www.gridwizardenterprise.org/integration/slicer.html GSlicer3 Guide]'
+
* Detailed information can be found on the [http://www.gridwizardenterprise.org/integration/slicer.html GSlicer3 Guide]
* [[Media:GWE-GSlicer3.ppt|Overview Presentation]]
 
* [[Media:GSlicer3-Demo.mov|Video Demo]]
 
  
 
== Future Work ==
 
== Future Work ==
Line 54: Line 48:
 
** "Standalone GWE" (requests parallelized as processes running in your machine - no cluster(s) needed).
 
** "Standalone GWE" (requests parallelized as processes running in your machine - no cluster(s) needed).
 
** Resolve "Windows" issues (cygwin does not work as expected).
 
** Resolve "Windows" issues (cygwin does not work as expected).
** Remote files directory staging (currently only single files).
+
** Support for array type of iteration variables.
 
** Users application registry database and auto-deployment module to clusters.
 
** Users application registry database and auto-deployment module to clusters.
** Support for array type of iteration variables.
+
** Daemon admin utilities (shutdown, restart, download DB, etc.).
** Daemon admin utilities (shutdown, restart, download DB, etc.)
+
** Order behavioral parameters (max concurrent jobs running, job execution timeout, launch mode, vfs clean up policy, pre-install applications, etc).
** Order behavioral parameters (max concurrent jobs running, job execution timeout, launch mode, virtual file system size, etc).
 
 
** Alert notification module (email, pager, etc).
 
** Alert notification module (email, pager, etc).
 
** Scheduling module.
 
** Scheduling module.
Line 65: Line 58:
 
** Multi-cluster functionality. Currently only single cluster mode (not true grid computing yet). Pending: module to daisy chain "GWE Daemon"s.
 
** Multi-cluster functionality. Currently only single cluster mode (not true grid computing yet). Pending: module to daisy chain "GWE Daemon"s.
 
** Single daemon services registry (Single IP Port/RMI registry) for multiple daemons.
 
** Single daemon services registry (Single IP Port/RMI registry) for multiple daemons.
** More file system drivers: GridFTP, SRB, etc.
+
** More file system drivers: GridFTP, SRB, XNAT/XCAT?, etc.
 +
** More resource manager drivers: Torque, GWE, Cloud, etc.
 
** X.509 certificates support for authentication.
 
** X.509 certificates support for authentication.
 +
** Order chaining (revamping prototyped framework for chaining Slicer3 modules).
 +
** MPI support.
 
** "GWE Powered" Portlets, FreeSurfer, FIPS, etc.
 
** "GWE Powered" Portlets, FreeSurfer, FIPS, etc.
** MPI support.
 
** Order chaining (revamping prototyped framework for chaining Slicer3 modules).
 
** More resource manager drivers: Torque, GWE, Cloud, etc.
 
 
** JDSL support (standard submission protocol).
 
** JDSL support (standard submission protocol).
 
** JMX support (runtime diagnostics).
 
** JMX support (runtime diagnostics).
Line 81: Line 74:
 
** Auto-deployment of Slicer3 in running clusters (as an extension of the users application auto-deployment module).
 
** Auto-deployment of Slicer3 in running clusters (as an extension of the users application auto-deployment module).
 
** GWE requests monitoring module to replace progress report in the status bar. UI with a realtime status of all GWE requests submitted using GSlicer3.
 
** GWE requests monitoring module to replace progress report in the status bar. UI with a realtime status of all GWE requests submitted using GSlicer3.
 +
 +
== Links ==
 +
* GWE Core - [[Media:GWE-Core.ppt|Overview Presentation]] / [[Media:GWECore-Demo.mov|Video Demo]]
 +
* GSlicer  - [[Media:GWE-GSlicer3.ppt|Overview Presentation]] / [[Media:GSlicer3-Demo.mov|Video Demo]]

Latest revision as of 14:52, 26 June 2008

Home < Project Week 2008 Special topic breakout: GWE
Back to 2008 Summer Project Week Agenda

GWE Overview

  • Distributed enterprise system, which leverages on clusters of computers, in order to provide a grid computing platform for end users to easily and effectively parallelize the execution of inter-independent processes.
  • GWE Client (End User - You) <=== Secure RPC Network ===> GWE Daemon(s) (Cluster Head Node)
  • Detailed information can be found on the GWE Project Site.

GWE Setup

Installation

  • Download bundle, unbundle & set appropriate environment variables.
  • Detailed information can be found on the GWE Installation Guide

Configuration

  • Define your grid and provide your authentication information to access "grid resources".
  • Detailed information can be found on the GWE Configuration Guide

Daemon Setup

  • ONLY IF YOU ARE GOING TO ENHANCE A NEW CLUSTER WITH GWE CAPABILITIES (GWE Enabled).
  • "Standalone GWE" will be available in the next release (To run GWE on your machine. Parallelization at the OS level, not distributed across cluster(s)).
  • From the machine where you installed GWE, run the utility "gwe-util-daemon-fullsetup.sh". That's it!
  • After that a "GWE Daemon" will be running in your cluster having your cluster user as this "GWE Daemon" administrator (different than roles in the cluster OS!).
  • A "GWE Daemon" is just another application in a machine, therefore "GWE Daemon"s can run in a single machine, although it is not recommended.
  • Detailed information can be found on the GWE Administrator Guide

GWE Clients

  • GWE architecture provides multiple extensible modules, one of which is the "GWE Client API".
  • Using the "GWE Client API", developers can create their own "GWE Client" applications.
  • "GWE Client" applications are those; which allow end users to submit their "grid computing" requests to the GWE backend system.
  • Currently, the GWE project team has created two "GWE Client" applications:

GWE Terminal

  • Console based GWE client.
  • Uses P2EL proprietary language to describe set of commands as permutation of templated commands with an associated parameter space.
  • Many other monitoring/control commands available.
  • Detailed information can be found on the End Users Guide

GSlicer3

  • Slicer3 based GWE client ("GWE Powered" Slicer3).
  • Basically a Slicer3 product enhanced with "GWE Client API" and custom development to provide "grid computing" capabilities to Slicer3.
  • Setup: Download GSlicer3 bundle, unbundle it, set appropriate environmental variables.
  • Config: Same as regular GWE.
  • Detailed information can be found on the GSlicer3 Guide

Future Work

  • Usual fixes, improvements, user support, documentation and project management.
  • GWE Core
    • "Standalone GWE" (requests parallelized as processes running in your machine - no cluster(s) needed).
    • Resolve "Windows" issues (cygwin does not work as expected).
    • Support for array type of iteration variables.
    • Users application registry database and auto-deployment module to clusters.
    • Daemon admin utilities (shutdown, restart, download DB, etc.).
    • Order behavioral parameters (max concurrent jobs running, job execution timeout, launch mode, vfs clean up policy, pre-install applications, etc).
    • Alert notification module (email, pager, etc).
    • Scheduling module.
    • Requests control: abort, pause, resume (under construction).
    • Descriptive request result parsing. P2EL semantics + parser framework.
    • Multi-cluster functionality. Currently only single cluster mode (not true grid computing yet). Pending: module to daisy chain "GWE Daemon"s.
    • Single daemon services registry (Single IP Port/RMI registry) for multiple daemons.
    • More file system drivers: GridFTP, SRB, XNAT/XCAT?, etc.
    • More resource manager drivers: Torque, GWE, Cloud, etc.
    • X.509 certificates support for authentication.
    • Order chaining (revamping prototyped framework for chaining Slicer3 modules).
    • MPI support.
    • "GWE Powered" Portlets, FreeSurfer, FIPS, etc.
    • JDSL support (standard submission protocol).
    • JMX support (runtime diagnostics).
    • Full feature statistical module.
    • etc...
  • GSlicer3:
    • Centralized configuration.
    • Usage of currently available specialized widgets (and maybe new GWE specialized widgets?).
    • API to build specialized "GWE Powered" Slicer3 modules.
    • Auto-deployment of Slicer3 in running clusters (as an extension of the users application auto-deployment module).
    • GWE requests monitoring module to replace progress report in the status bar. UI with a realtime status of all GWE requests submitted using GSlicer3.

Links