You are here: Foswiki>SBIR Web>WebHome (07 Jun 2007, RobGardner)Edit Attach

SBIR Project

Meeting notes and action items

Description

The figure below shows a conceptual architecture for a validation process engine (VPE) which:
  • processes validation dialogs (eg. written in BPEL).
  • manages extended validation sessions using the execution engine.
  • process diaglogs attach to machine level validation scripts (eg. written in bash, perl) that invoke specific grid component functions (eg., gt primitives).
  • A composer component allows editing of validation dialogs.
  • A publisher component stores these in a database for archival use.
  • A console component initializes and provides online monitoring of the valiation process engine (which may run as a service).
  • The VPE has a state database that provides backend robustness
  • A metrics database (mysql) collects information about validation sessions (runs of a given validation dialog)
  • A dashboard component provides a display of these sessions in various ways (web-based queries against the mysql metrics database)
  • Execution of validation scripts (provided from various sources, executed as components of a dialog composition) are managed by the VPE. They interact directly with the OSG grid services (could be any service, including application services).
  • Logging and other data from these executions are collected, organized, compressed and made available for analysis (a la splunk).
vpe-v1.jpg

References

  • SoW - Statement of Work
  • Process framework project with Intalio
  • Original SBIR project abstract located here

Use Cases

  • UC1: Develop a testing framework that can programmatically exercise the atomic operations of the OSG pre-web services gatekeeper tests. (See scripts below.)
  • UC2: Extend to SRM (StorageResourceManager) functionality on a storage server.
  • UC3: Extend to web-services GRAM functional tests.
  • UC4: Extend to grid-wide testing on the OSG validation testbed.

Software Components

VPE

VPConsole

  • Would something like HypericHQ be useful here?
  • There are also similar management approaches like Nagios/Cacti (based around RRDs). There is now a Nagios "appender" for Log4J, making it easy to dump output from ODE and other Java tools into a Nagios system.

A traditional commerical monitoring/management framework may be useful in outfitting otherwise unmanaged components within the grid infrastructure with instrumentation. This is orthogonal to applying orchestration to validate gross functions of the grid, but the results of verifications could be merged with other management/monitoring data

VPComposer

  • Perhaps something along the lines of http://www.eclipse.org/mylar/zest.php.
  • One of the discoveries of working on visual metaphors for BPEL and other orchestration environments was that it is difficult to capture the details of an interaction visually and uniformly, and it proved to be useful to have both a bubbled-up "view" suitable for interpretation at a high level and a drilled-down no-details-barred "view" of a system available. Depending on the design decisions made in the project, a "view" could be as simple as an icon representing status (green checkmark or red "x"), a flowchart, or a non-graphical representation like a textual program or script.

If we elect to do it in anything other than BPEL directly, it will be more expedient to create a non-graphical approach to defining processes. There are any number of tools that could be used to turn a single language (DSL) into BPEL for execution in ODE. Any approach would generate both BPEL and WSDL for primitive test/verification actions. Wrapping test/verification actions as web services would be a prerequisite. -- PaulBrown

VPPublisher

VPmetricsDB

VPdashboard

This component provides various views of current and archived validation sessions, and may be organized in various ways, eg., according to service. dashboard.jpg

See, for example, the concept of "sparklines" introduced by Tufte. There are a variety of libraries available to implement sparklines.

ValScripts

Globus

Condor

CEMon

Squid

Testing Infrastructure

Results

SBIR Web Utilities

ISorted ascending Attachment Action Size Date Who Comment
dashboard.jpgjpg dashboard.jpg manage 70 K 18 Jan 2007 - 18:03 RobGardner dashboard
vpe-v1.jpgjpg vpe-v1.jpg manage 242 K 18 Jan 2007 - 17:49 RobGardner VPE v1
gt-prews-tests.shsh gt-prews-tests.sh manage 1 K 18 Jan 2007 - 18:07 RobGardner gt2 test script
gt-ws-tests.pl.txttxt gt-ws-tests.pl.txt manage 2 K 18 Jan 2007 - 18:08 RobGardner gt4 perl script
robots.txttxt robots.txt manage 26 bytes 31 May 2007 - 19:23 UnknownUser  
Topic revision: r53 - 07 Jun 2007, RobGardner
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki? Send feedback