COMOS Data Visualization Design Document

1.0 Overview

The COSMOS Data Visualization framework aims to provide value by building a standards based integration infrastructure that provides access to two dimensions of the COSMOS management architecture, COSMOS Data Collection and COSMOS Resource Modeling. COSMOS Data Visualization provides a well defined set of service interface types that facilitate the exploration of the COSMOS Resource Modeling's repository of models and the retrieval of the COSMOS Data Collection's repository of state, event, and performance information. The model repository is semantically linked to the information in the monitor data repository. This relationship, which is managed in SML through the Resource Modeling component, is used to connect the exploration of the model with data that has been collected.

1.1 Use Cases

1.1.1 Use Case A - Determine the health of a network of computers

A company has a network of computers that is managed by a system administrator. The system administrator wants to determine if the network of computers are running under nominal conditions. It is expected that the system administrator uses a web based tool that shows the health of the system. At a glance the tool would visually show (via an icon) whether a computer is down or running. The system administrator will navigate through a visual structure representing the network of computers to drill down on a particular machine to look at a more detailed view. The detail view will present properties on resources in the system. For example the properties view may show the IP information of the machine. The view may also show deployment information, monitoring information or any other form of system management information. If data collection information is available for a particular machine the user may create a report to on the collected data to find trends or isolate problems. It is also expected that this tool be very useable in the sense that the data may be filtered based on the user. For example the system administrator may want a more detail view than a business analyst. As a result the user interface will be composed of sub views that can be arranged within a page. The views will interact with each other. For example the navigation view will trigger the properties view to present reports based on what was selected in the navigation view. Similarly the properties view may trigger a view that presents a report.

1.1.2 Use Case B - Vendor wants to visualize a vendor specific resource

A storage vendor has a set of storage servers that they provide. They define the necessary resource modeling and data collection components to collect and model the SAN units they provide. They want to be able to visualize their data in a system administration tool as discussed in the use case A. They want to create views which they can seamlessly contribute to the system administration console. The end result would provide system administrator with a more informed views on their particular SAN units.

2.0 Requirement

2.1 Summary

The initial goal of the resource model visualization UI is to provide a user friendly interface to navigate the complex graph-like resource model. Another goal is to visually expose properties to the end user. It should be noted that there can be many "flavors" of the visualizations of the resource model. The visualization may be driven by the domain (i.e. visualizing a network topology or a visualizing a computer system), user type (presenting the information to a system admin as opposed to a business analyst), or company driven (i.e. branding). As a result the ui framework should be flexible to accommodate many flavors of the visualization.

To simplify the design for the June release consider the following requirements:

• Visualize a read-only model. The ui is not expected to modify the resource model or data collection information
• Provide a simple navigation system based on a resource graph
• Show properties of a resource
• Provide visualization of collected resource information

The following are stretch goals:

• Provide operation support. This entails invoking a web service via a wsdm interface.

2.2 Views

Lets step back a bit and not consider the underlying technology before discussing the design. We first need to define a set of "views" that are needed.

• Navigator - the role of the navigator is to present the resource model in some form of visual structure such as a tree or a topology. The idea is that end users can at a glance visually grasp the relationships between the resources. An action to view the properties on a resources should be possible.
• Properties View - the role of the properties view is to present name value pairs in a table structure. Some common operations may be filtering, sorting, and highlighting. Note this is a simple visualization of the properties. This view will drive more in depth visualizations of the properties. For example, if a resource has statistical data an action would be placed on the property to launch a report to visually show the statistical data.
• Detailed Views - These are the views that organize the properties in more detail so that end users can grasp the information easily. For example, statistical data can be views in a chart instead of a flatten table structure.

The following shows an illustration how the views interact with one another.

Note that the communication between the different views is a one way communication which simplifies the design.

2.3 Layout

Another aspect to the user interface is the layout of the views. The UI should be flexible to accommodate different layouts of the views.

2.4 Environment

The target users for the resource model ui is mainly administrators that want to manage the system. In the industry these tools are typically web based tools. Hence the user interface should be web based.

2.5 Rapid Development and Extensibility

It should also be the goal to allow exploiters to easily contribute and extend the user interface to create different flavors of the visualization.

3.0 Design summary

The initial focus of the COSMOS Data Visualization framework is on three sets of service interface types: Structured Data Visualization Service - The lowest-level service interface types deliver a structured view of the underlying data. Graphical Visualization Service - The second set of service interface types is focused on integrating higher-level visualizations of the retrieved information. Presentation Service - The third set of service interface types is focused on serving a variety of user interface mechanisms. The COSMOS Data Visualization framework will provide a comprehensive set of service interface types at each of these three levels of abstraction. Each level offers an adoption point for implementers of management applications. The icon is intended to designate these adoption points. The COSMOS Data Visualization framework will also provide examples of implementations of management application components that adopt the service interface types. Some of these examples are: Eclipse BIRT ODAs, Libraries, and Reports that consume the structured data views. Eclipse RCP components and JSR-168 Portlets that consume the visualization APIs. A set of Web2.0, AJAX-enabled, mashup-ready user interface gadgets that allow for light-weight, browser-based integration of visualization components.

3.1 Structured Data Visualization Service

This set of service interface types is organized for adoption by management application components that consume structured data, for example, tabular displays of information about modeled resources, status information, events, and performance data.

Lets step back and consider the following basic requirement on the data visualization component.

"Organize a complex data structure into simpler views to allow users to visualize trends in the data."

The following diagram illustrates the above requirement.

The service will leverage a data layer provided by the resource model project to access the resource model. Similarly the data collection project will provide a data layer to the data collection information. These data layers should provide apis to transform a complex data structure to a normalized structure. The data layer apis should also provide the following data manipulations to help organize the data in the visualization.

• data filtering - ability to show certain data that meet a specific criteria (e.g. create a report to show hosts available in a specific domain)
• data grouping - organize the data based on a grouping criteria (e.g. create a report that shows available application grouped by host)
• data sorting - provide apis to sort the data (e.g. create a report that shows the top 10 cpu usage computers)
• data counts - provide apis to count a set of data that meets a certain criteria(e.g. display a bar chart that show total # of applications running per host)
• apis should be locale dependant. Time formats or strings may need to go through some locale specific transformation before presenting the information in a report.

Stateless facades APIs will be created that will use the data layer APIs to construct UI object models that are needed to visualize the resource and data collection information. Note for BIRT reports these facades are the BIRT data sets. These facade APIs will have intimate knowledge of the structure of the resource model and data collection persistent schema. For example, the following describes a method signature for a facade API.

3.2 Graphical Visualization Service

From the requirement summary it is clear that the user interface is "visualization service component" based (i.e. pluggable user interface components that are managed and displayed by some kind of user interface container). At a high level we can define a set of components required to create the resource model user interface.

• visualization service framework
• a set of user interface components

3.2.1 Web Visualization Service Framework

A framework is required to render pluggable user interfaces in an overall view. Consider the following diagram:

The user interface will be composed from a library of ui components that will register with the visualization service framework. Some form of registration mechanism is required to determine where to place the component within a page. Note the above diagram suggests that a "tag" is assigned to a specific section of the page.

Some form of registration file that binds the component to a section is needed. Consider the following XML file that can be packaged with the component:

<component id="Simple_Properties_View">
<section tag="properties">
</component>

Each component would contain a similar registration file that will tell the visualization service framework where to place the component on the page.

3.2.2 Event Model

An event model is required to communicate between the different visualization service components. One view needs to "drive" the rendering of other views. For example the navigator could present a tree. A user can select a resource in the tree to show properties of the resource in the properties view. Design consideration: Can we make some assumptions to simply this model for resource models?

3.2.3 Adding Ajax to the Web Visualization Service Framework

The visualization service framework should provide a mechanism to trigger the rendering of one component from another component using AJAX technologies. Consider a case where a user uses a navigation tree to select a resource. At this point the navigation tree sends an event to the properties component to render itself. A request is sent to the server to render the properties component and a response is sent back with a html fragment that will be intercepted by java script to replace the the properties section with the new html fragment. Refer to the following diagram that illustrates this flow.

Note this is the extent that the framework should provided ajax support. Each ui component would be responsible for providing their own ajax support to manipulate ui widgets that it contains. For example for a table wants to sort its column the ui component would be responsible for adding the necessary ajax support.

3.2.4 User Interface Components

The responsibility of the user interface component is to render fragments that can be used to compose a content page. Obviously we need to implore a model-view-controller architecture so that we can support various types of rendering technologies such as dhtml, xul, svg, xml, csv, etc.

We should define a simple contract between the component and the underlying visualization service framework so that these components can be reused in various admin consoles or content management system. The components should not have any knowledge of the visualization service framework they are rendering in. With that said lets consider the following design requirements for a component.

• Components are self contained modules.
• Components can be aware of event triggers created from themselves
• Components are not aware of event triggers that are created from other components. The visualization service framework intercepts event triggers from other components and notifies the target component to render itself.

Consider the following diagram that shows how this may be possible:

1. The end-user clicks on a resource in the navigator.
2. The navigator adds the context object to a communication bus.
3. An event is triggered to render the properties page
4. The context object is read from the communication bus.
5. A request is sent to the properties page to render itself.
6. A response with the updated output fragment is sent to the client to replace its content.

Note that the bus allows components to communicate with each other in a standard way without knowing the event model. With this architecture the components have a tighter contract with the model instead of the visualization service framework interface.

3.2.4.1 MVC and User Interface Components

As mentioned in the previous section components should be made up of a model, view and controller.

• Model - provides the object structure that the view will use to render the component.
• Controller - provides the logic to manipulate the model
• View - provides the rendering logic to visualize the model. When creating views for the web environment we want to implore AJAX technologies to improve the usability of the tool.

At each level of the MVC architecture exploiters can add their own customizations. Notice that cascading style sheets should be part of the view to reinforce reuse and extensibility. A preference mechanism should also be in place that stores user define configurations for a view. For example a preference to highlight certain properties in the properties view may be required.

3.2.4.2 Basic Well-Defined Components

A set of well defined components are needed to visualize the resource model. These components are as follows:

• Navigation view - the navigation view is the entry point view that the user uses to navigate through the model. The initial rendering of this view should be a tree model that shows the relationships between resources. When a user selects a node the properties view is rendered with the properties associated with the resource. Note that it is possible in the future to represent the navigation view as a graph. Additionally, there may be multiple actions that can occur on a specific node other than triggering the rendering of the properties view. There may be requirements on this view to filter out certain nodes or highlight certain nodes based on a criteria.
• Properties view - the properties view provides a table view of the properties associated with a resource. The table should provide the ability to sort, filter and search on certain properties. Certain properties in the table may require special rendering (i.e. operation status may require an icon). There are also cases where the properties may be a complex structure other than a simple value. The table should allow for the rendering of complex properties.
• Reporting view - A wrapper user interface component should be created to wrap the rendering of BIRT reports. This component will delegate rendering to the BIRT report viewer.

3.2.4.3 Reporting Component

Reporting is just another type of visualization on the data. The data visualization user interface should not treat reporting as a special case. Lets assume that we have a BIRT reporting system on the network that services reports. This system can provide html reports, xml, csv, or pdf. Consider the following high level illustration.

Note that the BIRT reporting system is a standalone application in the sense that end-users can directly make request to the reporting system to see a rendering of the report. The request by the end-user would have information telling the reporting system which report to render and any other report parameter necessary to render the report.

The data visualization UI would delegate request to the reporting system and render the reports within the UI. Consider the following diagram that shows a mockup of the COSMOS data visualization admin console showing two reports. The rendering of the report are generated by the report system.

Note that the data visualization is composed of a navigator, a report view and a properties view. The report view will make requests to the report system to get a http response containing html fragments that the data visualization component will arrange in the ui.

3.3 Presentation Service

The presentation service will provide a set of ui component libraries that will be adopted by management application components. The idea is that these components can be consumed by user interface visualizations to compose UI mashups. The library will be separated into two categories

• Widget Library
• Report Component Library

3.3.1 Widget Library

There are several widget libraries within the open source communities. The widget library should implore Web 2.0 technologies such as AJAX to create dynamic user interfaces. Obviously the library should provide a rich set of widgets. COSMOS will extend the basic widget set to render specific system management notions. For example an operation status value can be rendered as an image with a specific color to indicate the status value.

A meta data schema is required to associate widgets with specific properties. These widgets can be read only in the sense that the widget just presents the data. Otherwise, these widgets can be write only and send information back to the server to trigger an action on a WSDM API.

3.3.2 Report Component Library

A set of report components will be utilized to construct reports. This library will contain pre configured datasources and pre configured report items such as charts so that basic reports can be constructed without the need to know any binding logic to the data source. Note: BIRT provides a framework to create these types of report components.

3.4 Utility Modules

With any kind of application there are basic modules that the application is built on top of. This promotes reuse and best practices. The following are a list of utilities modules that will be required by the resource model user interface.

• database functions
• security functions
• validation functions
• http processing functions
• url decoding/encoding functions
• xml parsing
• xpath processing
• service invocation
• etc.

Ideally COSMOS should leverage existing open source frameworks to provide the necessary function. This will improve reliability and increase development.

4.0 Technology Consideration

4.1 PHP

PHP is a scripting language that allows one to create very lightweight applications. Furthermore, PHP's scripting capabilities makes it relatively easy to create applications rapidly compared to other lanagues such as java. Within the open source community PHP has become one the leading technologies. There are existing frameworks that we can leverage one of these frameworks is the "Zend Framework" that supports many utility modules.

One of the concerns with PHP is the ability to integrate with a java backend. The data layer for COSMOS will be java based. A bridge is needed to communicate between PHP written code and java written code. There may be interoperability issues. The following excerpt is from the article on PHP/Java integration (http://ca.php.net/java).

The Java Servlet SAPI builds upon the mechanism defined by the Java extension to enable the entire PHP processor to be run as a servlet. The primary advantage of this from a PHP perspective is that web servers which support servlets typically take great care in pooling and reusing JVMs. Build instructions for the Servlet SAPI module can be found in php4/sapi/README. Notes:

• While this code is intended to be able to run on any servlet engine, it has only been tested on Apache's Jakarta/tomcat to date. Bug reports, success stories and/or patches required to get this code to run on other engines would be appreciated.
• PHP has a habit of changing the working directory. sapi/servlet will eventually change it back, but while PHP is running the servlet engine may not be able to load any classes from the CLASSPATH which are specified using a relative directory syntax, or find the work directory used for administration and JSP compilation tasks.

4.2 JSF - JSR-127

Java Server Faces is a user interface framework for java web applications. This framework is backed by the JSR-127 specification from several vendors such as IBM, Borland, Oracle, and SUN. As a result there is tooling surrounding JSF to support and adopt the specification. JSF provides a UI component framework to create reuseable components.

4.3 JSF or PHP Conclusion

In COSMOS we define 3 components. We have a navigator, properties view and a birt viewer. We want to make these components reusable in the sense that we can extend their capabilties or add these components to existing pages. For example, lets say I want to create a tab view that has two tabs. The first tab will show the properties table while the second tab may show an operation status table. As a developer I should have the ability to reuse the properties table in my tab view and add the operation status table.

Some of the advantages of using JSF are as follows::

1. JSF is an insdustry standard that has tooling around this standard. This specification was created for componitization. With PHP we have to develop our own componitization framework that is not a standard.
2. There will be no interoperability issues with a java backend.
3. There is a good knowledge based how to use JSF components in a JSR 168 portlet. This is based on the fact that both JSF and Portal are based on a specification and there are articles on the web how to use JSF components with a JSR 168 portlet.

4.4 JSR 168

Pure servlet based frameworks like Struts or PHP 4 (with java binding ) are very difficult to use in a Portlet environement because they only support and understand a single request event. May need a Portal Bridge to allow the mapping of an action event to the underlying servlet handler implmentation. Refer to Portals Bridges.

4.5 Ajax Toolkits

There are several opensource ajax toolkits available. The following is a list of ajax toolkits that should be considered. This list was composed from the ajaxian.com survey. http://ajaxian.com/archives/ajaxiancom-2006-survey-results

• Prototype
• Scriptaculo.us
• Dojo
• DWR
• Moo.fx
• jQuery
• Yahoo! UI
• Rico
• Atlas
• MochiKit
• XAJAX
• GWT - works with java. does not work too well with PHP

"An Open Source AJAX Comparison Matrix" is an article that makes a comparison between the popluar AJAX toolkits.

"Exploring Ajax Runtime Offerings" discusses several AJAX toolkits and compares their architecture.

We should also consider whether other eclipse projects are currently using any existing AJAX toolkits. DOJO is a toolkit that the Aperi project is considering. BIRT ustilizes prototype. However prototype does not have a rich set of widgets.