Re: [stem-dev] Project Structure Change Proposals: Features, Update Sit

[Stefan Edlund <edlund@xxxxxxxxxxxxxxx>]

Hi,

here's my comments:

1. The update site URL for data plugins works for me.

2. For the feature naming scheme option number one looks cleanest and simplest to me, and I'm all for simplicity.

3. Your proposed list of features is good. A few suggestions. All the features that contain static data starts with "org.eclipse.stem.data...". How about we use the prefix "org.eclipse.stem.models..." for computational stuff, for instance disease models and population models. It would make it cleaner extending STEM with other types of models in the future which has always been the plan. Also, I think perhaps documentation is a feature, and I think this is a good time to add a separate feature for distributed STEM. Also, the analysis component is STEM is very much epidemiology oriented right now, so maybe a feature like org.eclipse.stem.analysis.epidemiology. Finally, data fitting in STEM is right now fairly generic and not tied to epidemiology (it all depends on the error function you plug in). Here's how that would look:

org.eclipse.stem.feature
org.eclipse.stem.prereq.feature
org.eclipse.stem.core.feature (the solvers would also belong in this one I think)
org.eclipse.stem.models.epidemiology.feature (disease models)
org.eclipse.stem.models.populations.feature (population models)
org.eclipse.stem.data.geography.feature
org.eclipse.stem.data.population.human.feature
org.eclipse.stem.data.transportation.feature
org.eclipse.stem.data.earthscience.feature (this one would include the single elevation plugin I presume)
org.eclipse.stem.data.earthscience.[2000-2009].feature
org.eclipse.stem.analysis.epidemiology.feature (contains the org.eclipse.stem.util.analysis plugin)
org.eclipse.stem.optimiziation.feature (contains org.eclipse.stem.analysis, org.eclipse.stem.analysis.automaticexperiment)
org.eclipse.stem.distributed.feature (contains the single plugin org.eclipse.stem.graphsynchronizer)
org.eclipse.stem.doc.feature
org.eclipse.stem.tests.feature

4. As for the repository reorg. (Indigo time-span sounds reasonable), how about:

/trunk/
/trunk/features - Contains the 'feature' projects
/trunk/core - Contains the 'core' projects, that make up the 'core'
feature
/trunk/models/epidemiology - Contains the epidemiology-centric projects (make up
the 'epidemiology' feature)
/trunk/models/population - Contains the population-centric projects (population models).
/trunk/analysis/epidemiology - Contains the analysis plugin and associated plugins
/trunk/optimization - Contains the Nelder Mead code and Automatic Experiment perspective.
/trunk/distributed - For the graph synchronizer
/trunk/data/skeletons - Contains the data plug-in skeletons
/trunk/data/internal - Contains plug-ins that are intended for internal
use only (internal.data plugins)
/trunk/tests - Contains the test case projects
/trunk/releng - Contains projects related to release engineering
/trunk/doc - Contains the documentation project
/tags/*
/branches/*

I can't decide whether subdividing the /data/skeletons and /data/internal paths is needed. On one hand, it would make it easier for users to only check out the stuff they need, but on the other hand the names of the plugins should make that fairly clear. I can go either way.

Thanks for starting this discussion!
/ Stefan

Stefan Edlund
Public Health and Computer Science Research
IBM Almaden Research Center
(408) 927-1766 edlund@xxxxxxxxxxxxxxx


Matthew Davis---05/11/2011 02:08:34 PM---Hi All, As has been discussed on our weekly phone call, upcoming additions to STEM

From:	Matthew Davis/Almaden/IBM@IBMUS
To:	stem-dev@xxxxxxxxxxx
Date:	05/11/2011 02:08 PM
Subject:	[stem-dev] Project Structure Change Proposals: Features, Update Site, Build Process, Repository restructure, etc
Sent by:	stem-dev-bounces@xxxxxxxxxxx

---

Hi All,

As has been discussed on our weekly phone call, upcoming additions to STEM 
are driving the need for changes in how we distribute project components. 
Notably, due to the addition of new Earth Science data, our binary 
distribution would more than double in size.  The following is a list of 
actions already taken and proposals for discussion on how to augment, 
extend, and support these changes.

The following proposals and changes will be applied as of STEM version 
1.2.1.  I appreciate any comments, feedback, or other discussion regarding 
these proposals. 

My apologies for the long email.

Thanks,
-Matt




P2 Repositories and Update Sites
---------------------------
To make retrieving and using this new data in STEM easy, we will begin 
distributing optional data using a P2 Repository/Update Site.  This will 
allow STEM users to retrieve/install automatically using the "Install New 
Software" menu in STEM.  For now, the only things we will distribute 
through the update site are static (generated) data features/plugins.  We 
will not distribute STEM application updates at this time, although this 
initial groundwork can be used to move to that in the future.

By enabling P2 in STEM, we also get additional functionality of the 
provisioning architecture, such as the "dropins" folder.  This may be 
useful in the future.

* Proposal For Update Site URL for Additional Data plug-ins:

http://download.eclipse.org/stem/update-site/addons/

Because the data is historical and should not change, the update site will 
be statically built and distributed for now.  We will not integrate it 
into the build process.




Feature-oriented Builds
---------------------------
As a prerequisite to using P2 Repositories and Update Sites, we have made 
initial changes to the STEM Product and build process to become 
"feature-oriented" (versus "plug-in-oriented").  The initial changes 
extend existing work that someone else began (Dan?).  The following is a 
proposal with discussion on how to proceed.

* Initial/Current Features

Before I began looking at this, the following features existed, but were 
unused, in the STEM repository:

org.eclipse.stem.feature - Top-level container that represents the STEM 
application.
org.eclipse.stem.feature.prereq - Third-party (non-STEM) dependencies, 
including Eclipse Platform, EMF, BIRT, Zest.
org.eclipse.stem.feature.core - The plug-ins that make up the "core" 
functionality of STEM

In addition to tweaking these features, I added a new feature using the 
existing naming scheme:
org.eclipse.stem.feature.internal.data - The plug-ins that contain raw 
data and generator code to create data distributed with STEM

* Naming Schemes

Dan pointed out a couple weeks ago that this naming scheme is inconsistent 
with the naming scheme used by other Eclipse projects.  I agree, and I 
propose that we change the naming scheme to conform.  I think there are 
two options:

1. Append ".feature" to plug-in grouping id
This appears to be the most widely used approach (and was suggested by 
Dan).  The features would be renamed to look like this:
org.eclipse.stem.feature
org.eclipse.stem.core.feature
org.eclipse.stem.internal.data.feature
etc

OR

2.  Remove "feature" altogether from the name
This is currently in use by the EMF project.  As plug-ins and features can 
share names, you can simply drop the .feature from the feature name (for 
workspace compatibility, however, you do need to append -feature to the 
directory).  In this case, the features would be renamed as such:
org.eclipse.stem (directory name org.eclipse.stem-feature)
org.eclipse.stem.core (directory name org.eclipse.stem.core-feature)
org.eclipse.stem.internal.data (directory name 
org.eclipse.stem.internal.data-feature)
etc

* How to split STEM into Features

This is the most important question - how exactly do we split STEM up into 
"features".  Remember that an Eclipse "feature" represent discrete but 
integrate-able groupings of Eclipse plug-ins and other features. 
Basically, a feature groups together plug-ins/features that provide 
specific new functionality and/or extends existing functionality in an 
Eclipse-based application.

With that in mind, we re-open an old, mostly philosophical question --- 
"What is STEM?"

By design, STEM is an event modeler that can model and simulate anything 
that can be represented on a graph.  The core of STEM was designed 
specifically with this in mind.
In practice, STEM is an epidemiological modeler that models and simulates 
infectious diseases.

After looking at the relationships between components, there is no 
absolute separation between generic modeling components and 
epidemiological-oriented components.  As a result, it's currently not 
possible to create a clean split between generic and disease-specific 
functionality.  What I mean by this is:  there's no way to create isolated 
"core" and "epidemiology" features.  That said, there are "epidemiology" 
components that can be split off into their own groupings.

I propose the following features be created in STEM.  Features with a (*) 
will be part of the binary distribution on the Web site.  Other features 
will be distributed via Update SIte.  The actual breakdown of plug-ins for 
each feature will be determined after discussion.

* Proposed Features

(*) org.eclipse.stem.feature - Top level feature, target of the builder. 
Contains references to other (*) features.
 + (*) org.eclipse.stem.prereq.feature - Third-party dependencies
 + (*) org.eclipse.stem.core.feature - Core functionality, including core, 
definitions, basic geography, and core UI.  Effectively the minimum set of 
plug-ins required to run STEM
 + (*) org.eclipse.stem.epidemiology.feature - Contains plug-ins 
pertaining to epidemiology that are  NOT required by default to run STEM. 
Mostly the disease models and their UI plug-ins.
 + (*) org.eclipse.stem.data.geography.feature - Political geography data, 
including country-level maps, names, and associated graphs and models
 + (*) org.eclipse.stem.data.population.human.feature - Human population 
data plug-ins, including associated graphs and models
 + (*) org.eclipse.stem.data.transportation.feature - Transportation data 
plug-ins, including associated graphs and models

org.eclipse.stem.data.earthscience.feature - Earth Science Data plug-ins
 + org.eclipse.stem.data.earthscience.[2000-2009].feature - Individual 
years of earth science data.

org.eclipse.stem.tests.feature


Source Code Repository Restructure
---------------------------

1. Do we want to restructure the repository?

As of this morning, there are 147 plugins, features, and other projects in 
the STEM source root.  That is a lot.  And, as is the case with our 
changes to enable extra functionality to be distributed outside of the 
main STEM application, there's really no need to check out all these 
projects.

Other Eclipse projects have more structure in their repositories, 
separating various components.  Adding more structure, of course, makes 
the checkout process more complicated ("What do I need, and where does it 
live?").

2. If yes, then when?

A repository restructure is a complicated affair, where the easiest way to 
approach it from a developers point of view is to commit all diffs, freeze 
changes, clean you workspace, and re-checkout.  We have discussed the most 
logical time would be when we switch from Helios to Indigo, probably in 
July.

3. What does the new structure look like?

This is a tough question.  There's not a lot of consistency across Eclipse 
projects for this, so we're really open.  I'm going to take a stab at a 
proposal that incorporates ideas from other Eclipse projects.

/trunk/
/trunk/features - Contains the 'feature' projects
/trunk/core - Contains the 'core' projects, that make up the 'core' 
feature
/trunk/epidemiology - Contains the epidemiology-centric projects (make up 
the 'epidemiology' feature)
/trunk/data/skeletons - Contains the data plug-in skeletons
/trunk/data/internal - Contains plug-ins that are intended for internal 
use only (internal.data plugins)
/trunk/tests - Contains the test case projects
/trunk/releng - Contains projects related to release engineering
/tags/*
/branches/*


An alternate suggestion is as follows:

/stem/trunk/features
/stem/trunk/core
/stem/trunk/epidemiology
/stem/trunk/tests
/stem/trunk/releng
/stem/tags/*
/stem/branches/*

/data/trunk/features
/data/trunk/internal
/data/trunk/skeletons
/data/trunk/tests
/data/tags/*
/data/branches/*


We can then distribute a project set file (PSF) that contains the 
appropriate references to automatically checkout and build a developer's 
workspace depending on what they want.  The Eclipse Metadata contains 
provisions for this.

4. Subversion vs. Git

A lot of Eclipse projects are moving to Git for their source repository 
over CVS/SVN.  If we want to do this in the next year or two, this will be 
the time to do it.



Changes to Build Process
---------------------------
* Java 6 Base Requirement
As was voted on several weeks ago, we are now using a Java 6 JDK with 1.6 
source and class compatibility requirements in our builder.  This 
effectively deprecates Java 5.

* Feature-oriented PDE Build, P2 enablement 
Additional changes to the build process to support feature-oriented builds 
and enable P2 repositories have been made.  These changes are committed 
into the org.eclipse.stem.releng.feature project, but will be rolled into 
the org.eclipse.stem.releng project after the final changes are in place

* Use of Hudson/Jenkins for automated builds
We've set up a Jenkins instance on an internal server to handle running 
and automating STEM builds.  I realize that Eclipse provides projects with 
infrastructure to do this in the open, but we have significant hardware 
assets and there's really no need to place additional burdens on their 
infrastructure.  I will publish the Jenkins/Hudson configuration files for 
anyone that wants to set up their own automated STEM builds.

* Our plan right now is to run nightly builds to test code compliance but 
we will not externally publish these builds (as we're talking 2 GB total 
per build).  Weekly, probably on Sunday morning, we will run a build that 
will be published automatically to the STEM Web site.

(Side question of whether we should use Hudson or Jenkins:  We will 
standardize on whatever Eclipse standardizes on.  If the Eclipse Hudson 
project is approved, and they resolve all IP concerns, we will use Hudson 
upon the first Eclipse/EPL release.  Until then, we will use Jenkins).

* Automatic updating of download site
I've made a handful of changes to the PHP script that controls downloads 
to enable automatic updating of build versions.  These changes will be 
documented in the wiki.

* Use of Buckminster and/or Target Platforms
Currently out of scope for this discussion.  If someone wants to start a 
side conversation - and implement - the build process using Buckminster, I 
encourage it.  However, this was not a part of this project.
_______________________________________________
stem-dev mailing list
stem-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/stem-dev