platform-core-home/documents/update.html

Parent Directory Parent Directory | Revision Log Revision Log

Revision 1.1 - (download) (as text) (annotate)
Tue Feb 17 16:34:54 2004 UTC (5 years, 9 months ago) by jeff
Branch: MAIN 
add the plugin versioning and update site proposals

<html>
<head>
<title>Eclipse Update Site: Eating our own dog food</title>
	<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   	<link rel="stylesheet" href="http://eclipse.org/default_style.css" type="text/css">
</head>

<body bgcolor="#FFFFFF" text="#000000">

<h1>Eclipse Update Site Proposal</h1>
<blockquote> 
  <p><b>Summary</b> <br>
    The current packaging of the Eclipse SDK as a set of large zips (one per platform) 
    is a convenient one stop shop for getting up and running with the SDK. Unfortunately, 
    this approach does not scale well. As more and more Eclipse projects (e.g., 
    EMF, GEF, ...) come into common use, there is increasing interest in including 
    these in pre-packaged zips. Similarly, as Eclipse is used in more varied scenarios, 
    so increases the demand for different packagings (smaller and larger). It 
    is not feasible for eclipse.org to create, manage and distribute pre-packaged 
    configurations to meet all the demands. This proposal details a structure 
    which enables Eclipse component providers to contribute their features and 
    plugins and for Eclipse consumers to efficiently select, download and install 
    their components of interest.<br>
    Last Modified: 1100 February 17, 2004</p>
</blockquote>

<h2>Problem Definition</h2>
<p>The base Eclipse function is currently distributed as roughly the following 
  zip files</p>
<ul>
  <li>binary Platform (one per target OS/WS) (7 x ~24MB)</li>
  <li>binary JDT (one OS/WS independent) (~12MB)</li>
  <li>SDK zips includind source and all doc (one per target OS/WS) (7 x ~80MB)</li>
  <li>assorted other zips (e.g., examples, webdav, ...)</li>
</ul>
<p>Eclipse tools and technology project downloads are available separately and 
  come in a variety of flavours, sizes and organizations. People wanting the base 
  SDK are quite comfortable with the current situation. They need only download 
  the single zip and go. If they need to run on two different platforms they download 
  the zip for that platform and ignore the fact that 95% of the second download 
  is the same as the first.</p>
<p>Users looking to combine the base Eclipse with additional plugins/features 
  are somewhat less happy. They have to search over several web sites, identifying 
  downloads (zips) of interest and correlating versions manually. In general, 
  the suppliers of these additional zips do not make it easy for users to understand 
  their offerings.</p>
<p>The main issues with the current setup are:</p>
<ul>
  <li>Does not promote the easy provision, acquisition and use of plugins from 
    a variety of souces. The basic premise of Eclipse is that plugin consumers 
    can gather a set of interesting plugins, put them together in an Eclipse platform 
    configuration and enjoy an integrated seemless environment. Currently plugin 
    producers provide their plugins in various forms and the lack of any managed 
    acquisition mechanism forces potential users to grovel around for plugins 
    and then hope that they got the right thing.</li>
  <li>Does not scale as the scope of Eclipse increases. The current approach addresses 
    the provision/acquisition problem by packaging collections of interesting 
    plugins together in a single zip. This works for Eclipse since it is at the 
    bottom of the plugin stack. However, others providers are not so lucky and 
    are challenged with questions like &quot;how many of my prerequisites should 
    I include in my download?&quot;. Further, as more and more plugins are added 
    to the Eclipse world, the size of the complete zip increases. A zip of a reasonable 
    set of SDK drops from projects on eclipse.org could easily reach 150MB (i.e., 
    double the Eclipse SDK size). Following the current model, there would be 
    one of these uber-zips per target OS/WS.</li>
  <li>Does not scale as the Eclipse use scenarios increase. As the range of people 
    using Eclipse broadens, there will be increasing demand for alternative collections 
    of plugins. These will contain either more or less than the Eclipse SDK. While 
    we can take a stab and produce some common packagings, we will not</li>
  <li>Makes poor use of existing disk and bandwidth resources. As mentioned earlier, 
    something like 95% of each OS/WS specific SDK zip is identical content. Further, 
    from build to build, not all plugins change. The user doc for example tends 
    to change slowly. Near the end of a milestone or release cycle we frequently 
    rebuild to fix isolated problems. Rather than rebuilding just the plugins 
    in question and providing these for download, we rebuild the whole stack and 
    repackage an entire set of full zips. Subsequently, interested developers 
    are forced to redownload the bulk of what they already have. This wastes diskspace 
    and bandwidth for both eclipse.org and our users.</li>
  <li>Finally, it is somewhat ironic that the update/install mechanism has been 
    designed in part to address exactly these kinds of concerns yet it is one 
    of the few components that does not see regular use by the Eclipse team itself. 
    As a result, the update/install team misses out on the regular feedback enjoyed 
    by the other Eclipse teams. It is seemingly hard to justify putting update/install 
    forward as a useful technology but not using it ourselves. </li>
</ul>
<h2>Scenarios</h2>
<h3>Scenario 1: minimal install</h3>
<p>User starts with a minimal Eclipse base install. This contains just the runtime, 
  some UI elements and an Update manager UI. Based on this they want to build 
  an install containing additional elements from various sources.</p>
<ol>
  <li>download and unzip the minimal install form eclipse.org or a mirror</li>
  <li>start the new install </li>
  <li>the user is presented with an update manager UI </li>
  <li>user discovers or is presented with update sites </li>
  <li>user selects features to install (the candidates presented are appropriate 
    to the the current environment e.g., OS/WS/etc as well as already installed 
    components)</li>
  <li>the selected features are downloaded</li>
  <li>the features are installed into the current configuration</li>
  <li>the appropriate product/application (one that was just downloaded) is started 
    and the update manager GUI goes away</li>
</ol>
<h4>Variations</h4>
<p>After following the above scenario, the user can manage their configuration 
  either using the steps outlined in Scenario 2 or by restarting their configuration 
  with an indication that the update manager UI should be started. This puts them 
  back at step 3 but with all their previously installed plugins still installed.</p>
<h3>Scenario 2: building on a product install</h3>
<p>In this scenario, the user has an Eclipse product installed (possibly just 
  the Eclipse SDK from a downloaded zip file or the configuration from the steps 
  in scenario 1) and wants to add more function. This pretty much follows the 
  normal update/install scenarios.</p>
<ol>
  <li>user installs an eclipse based product (e.g., Eclipse SDK)</li>
  <li>at some point, while running the product, the user invokes the update manager 
    UI </li>
  <li>user discovers or is presented with update sites </li>
  <li>user selects features to install (the candidates presented are appropriate 
    to the the current environment e.g., OS/WS/etc as well as already installed 
    components)</li>
  <li>the selected features are downloaded</li>
  <li>the features are installed into the current configuration</li>
  <li>the installed function is now available to the user</li>
</ol>
<h3>Scenario 3: Web install</h3>
<p>Note: This is an advanced scenario that may not be supported in the near term.</p>
<p>This scenario is very much like scenario 1. The main difference is in how the 
  initial install is obtained. Rather than manually downloading and unzipping 
  the minimal install, JNLP (Webstart) is used to fetch and run the minimal install. 
</p>
<ol>
  <li>user browses some website and clicks on a JNLP link</li>
  <li>the appropriate jars are downloaded and &quot;installed&quot; using the 
    JNLP mechanism</li>
  <li>the minimal Eclipse install is started </li>
  <li>go to step 3 of scenario 1</li>
</ol>
<h2>Problem Solution</h2>
We propose to use the Eclipse update technology to address the described problems 
and implement the listed scenarios. Note that this new structure is a complement 
to the existing large zip packagings of Eclipse. We are not proposing to stop 
creating the familiar SDK etc zips. 
<h3>Update site</h3>
<p>eclipse.org (and others) will provide update sites sporting all the relevant 
  features and plugins. Collections of projects can cooperate and share an update 
  site or each supply/support their own update site. Managing your own eliminates 
  the need for some sort of shared site update mechanism. However, the user must 
  be able to navigate these individual sites seamlessly (see the section on Update 
  manager). </p>
<h3>Versioning</h3>
<p>Update sites are not possible unless we use the Eclipse version numbering scheme 
  effectively. The main challenge is for plugins to have version numbers that 
  indicate their content. Updating versions so they correctly reflect semantic 
  change would be good but not essential. What is required however is that if 
  two plugins have the same id and version, their content is identical. The easiest 
  way to do this is to use the version qualifier (i.e., the forth version element). 
  The qualifier should be the version tag from the map file used for the build 
  which generated them (e.g., 3.0.0.v20040212). The map file defines the content 
  going into the build. If that does not change, then we can assume that the content 
  of the output plugin will not change. In this way, identical plugins will not 
  be duplicated on the update site and consumers will not have to download them 
  if they already have them.</p>
<p>Features on the other hand are packagings of plugins. They are also relatively 
  small. Since the content of a feature is indirectly defined by the content of 
  the plugins included in the feature, it is hard to detect change. The safe bet 
  here is to use the build stamp as the qualifier for feature version (e.g., 3.0.0.I200402131200). 
  This has the benificial effect of clearly identifying the features for a particular 
  build. Further, since the entire feature set is regenerated, the collection 
  of features with the same version number completely defines the entire build 
  output.</p>
<p>Nightly builds would not go into the update site. There is no easy way to correlate 
  the content in a nightly build (out of HEAD) with that of an integration build 
  (using versioned resources). As a result, all plugins would be new and the economies 
  of the proposed solution would not be realized. Further, nightly builds are 
  primarily there for the individual teams to ensure they have not adversely affected 
  other components etc. That is, they do not figure prominently in the scenarios 
  driving this proposal.</p>
<p>Plugin directories currently reflect the id and version of the plugin. With 
  the addition of the map stamp, the directory names will grow by typically ~10 
  chars). It is unlikely but this may push some path length limits. The use of 
  the map stamp in a file/dir name may also place additional constraints the set 
  of chars that can be used in a map stamp (i.e, CVS tag).</p>
<p>Sidenote on version qualifiers: The qualifier (fourth segment) is
  not interpreted. Qualifiers are compared using lexographical sorting. As such,
  care should be
  taken to ensure that the qualifier for subsequent builds monotonically increase.
  For example, rather than prepending the build type identifier (e.g., I20040217...
  or M7), the build type should be appended and the build date used as the main
  body (e.g., 20040217-I and 20040213-M7).</p>
<p></p>
<p></p>
<h3>Update manager</h3>
<p>The main challenges for the update manager are in making all of this scale 
  and usable. </p>
<ul>
  <li>the UI must support a structured (hierarchical?) view of the available features. 
    This view (and in fact the underlying feature structure) must be able to span 
    update sites/servers. </li>
  <li>users must be able to select some set of root features and versions and 
    then click &quot;all required features&quot;. Again, this should span update 
    sites/servers</li>
  <li>using these capabilities, user can then create their own uber-features/sites 
    which simply reference features/sites. This becomes in effect a favourites 
    list. They select their favourite root, click &quot;all required&quot; and 
    get everything they need.</li>
  <li>the update manager mechanism may cache downloaded elements
     (features and plugins) for use in future install operations. That is, if
    you  already have the plugin/feature, you don't need to download it again.
    The 
    cache location should be (optionally) set by the user.</li>
  <li>ideally the update/install scenarios described will be possible when running
    offline providing that all required elements have been previously cached.
    That is,
    
    if the user has already installed the features for a build, she should be
     able to create a new install without touching a server.</li>
  <li>as an advanced (i.e., longer term) item, update manager needs a way of
    managing its cache/store -- users need a way of purging old elements. Given
    a set
    of 1000
    plugin jars
    of various
    versions, 
    determining which to keep and which to delete is not possible without working
     from higher level feature groupings. In the near term, the user can delete
    the whole cache.</li>
</ul>
<p>Interesting side note: using the runtime's new ability to run directly out 
  of jars, it is possible to have many different eclipse configurations sharing 
  the same actual plugin files from the update manager's cache/store. This will 
  break some assumptions about plugins being together in a plugins directory etc 
  (the update reconciler may get upset) and it introduces some challenges for 
  cache clean up but it is an interesting possibility.<br>
</p>
<h3>PDE Build</h3>
<p>PDE Build already updates feature.xml files with the version numbers of the 
  plugins involved in the build. To support the version numbering scheme outlined 
  above, PDE Build will be updated to </p>
<ul>
  <li>modify the plugin version number in each plugin.xml (i.e., add the map stamp). 
    The version number of a plugin.xml is updated only if it contains &lt;_map 
    stamp_&gt; as the version qualifier. This allows plugins to opt out of this 
    update mechanism.</li>
  <li>modify the version number in the feature.xmls (i.e., add the build stamp). 
    The version number of a feature.xml is updated only if it contains &lt;_map 
    stamp_&gt; as the version qualifier. This allows features to opt out of this 
    update mechanism. </li>
  <li>(advanced) modify the version number on &lt;import&gt; statements. Some 
    plugins and features may specify a perfect match rule in when listing their 
    dependencies. While this is not recommended, it is certainly possible. Unfortunately, 
    there is a conflict if the intention was that plugin A depend perfectly on 
    the version of plugin B with which it was built. During the build, B's version 
    number will be updated according to its map stamp. Plugin A's &lt;import&gt; 
    statement for B needs to be updated to reflect this new version. While this 
    modification is technically possible, it requires more complex analysis of 
    the plugin.xml file. Updating the version of the plugin itself can be done 
    using simple string replacement. In this case the builder has to 
    <ul>
      <li>look at each &lt;import&gt; using perfect match and wanting build-time 
        map stamp substitutoin, </li>
      <li>identify the pluign being imported, </li>
      <li>discover the new version number</li>
      <li>replace the &quot;substitutution desired&quot; flag in the version number 
        with the correct version number without loosing any other formatting, 
        comments etc.</li>
    </ul>
  </li>
</ul>
<p>Note: the plugin.xml updating outlined above must be carried out on any manifest.mf 
  files present in the build.</p>
<p></p>
<p>PDE Build will also be updated to output directly to update site format. This 
  is already largely supported but there maybe some modest work required.</p>
<p>A possible longer term advance is to change PDE build to only build the plugins 
  which actually changed. Since we know the map stamp of the plugins we want in 
  the output, we can first look for those in the update site (or some cache). 
  If present, there is no need to rebuild them. This will have a huge (positive) 
  impact on rebuilds during release cycles.</p>
<h2>Summary</h2>
<p>The problem outlined above is making it hard to scale Eclipse. It is hard for 
  people to use large numbers of Eclipse components (e.g., EMF, GEF, ...) and 
  we are failing to put forward a good model of how others would build/stock component 
  systems. The ability to do this easily is key to the overall Eclipse component 
  model. The steps proposed here are a good first try at solving the problem. 
  The result will not be perfect but will allow us to understand what is wrong 
  and fix it. Fortunately, the bulk of the technology required exists in Eclipse 
  today. Also, since we are proposing a complementary approach to the current 
  download strategy, we can stage the implementation of this new mechanism both 
  as we have time and as we learn more.</p>
<p>The key steps that should be taken for 3.0 are </p>
<ul>
  <li>create/populate an update site</li>
  <li>update PDE build to automate the stamping of plugins and features</li>
  <li>a modest amount of work on update manager to enable reasonable workflows 
    (i.e., set a stake in the ground)</li>
</ul>
</body>
</html>