SBIR Project

Meeting notes and action items

• NotesJun8
• NotesJun1
• NotesMay18
• NotesMay4
• NotesApr27
• NotesApr13
• NotesApr6
• NotesJan18
• NotesNov17

Description

The figure below shows a conceptual architecture for a validation process engine (VPE) which:

• processes validation dialogs (eg. written in BPEL).
• manages extended validation sessions using the execution engine.
• process diaglogs attach to machine level validation scripts (eg. written in bash, perl) that invoke specific grid component functions (eg., gt primitives).
• A composer component allows editing of validation dialogs.
• A publisher component stores these in a database for archival use.
• A console component initializes and provides online monitoring of the valiation process engine (which may run as a service).
• The VPE has a state database that provides backend robustness
• A metrics database (mysql) collects information about validation sessions (runs of a given validation dialog)
• A dashboard component provides a display of these sessions in various ways (web-based queries against the mysql metrics database)
• Execution of validation scripts (provided from various sources, executed as components of a dialog composition) are managed by the VPE. They interact directly with the OSG grid services (could be any service, including application services).
• Logging and other data from these executions are collected, organized, compressed and made available for analysis (a la splunk).

References

• SoW - Statement of Work
• Process framework project with Intalio
• Original SBIR project abstract located here

Use Cases

• UC1: Develop a testing framework that can programmatically exercise the atomic operations of the OSG pre-web services gatekeeper tests. (See scripts below.)
• UC2: Extend to SRM (StorageResourceManager) functionality on a storage server.
• UC3: Extend to web-services GRAM functional tests.
• UC4: Extend to grid-wide testing on the OSG validation testbed.

Software Components

VPE

• ODE (PXE offspring), http://incubator.apache.org/ode/index.html

VPConsole

• Would something like HypericHQ be useful here?
• There are also similar management approaches like Nagios/Cacti (based around RRDs). There is now a Nagios "appender" for Log4J, making it easy to dump output from ODE and other Java tools into a Nagios system.

A traditional commerical monitoring/management framework may be useful in outfitting otherwise unmanaged components within the grid infrastructure with instrumentation. This is orthogonal to applying orchestration to validate gross functions of the grid, but the results of verifications could be merged with other management/monitoring data

VPComposer

• Perhaps something along the lines of http://www.eclipse.org/mylar/zest.php.
• One of the discoveries of working on visual metaphors for BPEL and other orchestration environments was that it is difficult to capture the details of an interaction visually and uniformly, and it proved to be useful to have both a bubbled-up "view" suitable for interpretation at a high level and a drilled-down no-details-barred "view" of a system available. Depending on the design decisions made in the project, a "view" could be as simple as an icon representing status (green checkmark or red "x"), a flowchart, or a non-graphical representation like a textual program or script.

If we elect to do it in anything other than BPEL directly, it will be more expedient to create a non-graphical approach to defining processes. There are any number of tools that could be used to turn a single language (DSL) into BPEL for execution in ODE. Any approach would generate both BPEL and WSDL for primitive test/verification actions. Wrapping test/verification actions as web services would be a prerequisite. -- PaulBrown

VPPublisher

VPmetricsDB

VPdashboard

This component provides various views of current and archived validation sessions, and may be organized in various ways, eg., according to service.

See, for example, the concept of "sparklines" introduced by Tufte. There are a variety of libraries available to implement sparklines.

ValScripts

Globus

• gt2: https://twiki.mwt2.org/bin/../pub/SBIR/WebHome/gt-prews-tests.sh
• gt4: https://twiki.mwt2.org/bin/../pub/SBIR/WebHome/gt-ws-tests.pl.txt

Condor

CEMon

Squid

Testing Infrastructure

• See the VTB software and pacman cache, http://osg-vtb.uchicago.edu/
• UCTier compute hosts and OSG installation ITBInstallProcedure

Results

Related Links

SBIR Web Utilities

•   - advanced search
• WebTopicList - all topics in alphabetical order
• WebChanges - recent topic changes in this web
• WebNotify - subscribe to an e-mail alert sent when topics change
• WebRss, WebAtom - RSS and ATOM news feeds of topic changes
• WebStatistics - listing popular topics and top contributors
• WebPreferences - preferences of this web