[cdt-dev] FYI Re: [eclipse.org-committers] Infra news: CVS slow, Nightly

[James Blackburn <jamesblackburn@xxxxxxxxx>]

Hi All,

Denis is making progress creating git mirrors of various platform
projects.  I've had some success using git with cdt and have made some
suggestions on how the CDT git mirror might be laid out.

Problem:
Unlike the platform plugins, all the CDT plugins live (mostly) in one
org.eclipse.cdt* namespace. This means there isn't an easy CVSROOT ->
Eclipse runtime project mapping.  In git it's the entire repository
that's versioned, and it's at this granularity that users perform
clones / checkouts.  You can't just checkout a subdirectory, you
*must* pull the full repository.

With the above said there's a choice as to how finely or coarsely
grained you want your git repositories to be.  There could be one
repository for all of org.eclipse.cdt, which would be >1Gb, so would
clearly be far to coarse. There could be one repo per component:
cdt-core, cdt-debug, dsf, p2, etc.  Or there can be one repository per
Eclipse Project.

I prefer the last option, mostly because I've found it works best for
what I do.  It makes the following use cases easy:
 * As it is today:
     - A user wants to fix a bug in managedbuild.core, they can
checkout just the managedbuild.core plugin.
       In CVS today, users are used to checking out individual
projects at a time, and not checking out a component followed by
importing the projects... Plugins are the quantum of PDE.
  * Feature patches:
     - If ISVs need to patch a plugin and distribute the patch as a p2
featurepatch, they don't need to build / package the rest of the
component
  * PDE Build:
     - The plugin is the unit of build in Eclipse. It makes sense that
users can checkout cdt.core from their local clone, but can pull a
tagged cdt.ui from 'net.  Creating an Eclipse IFetchFactory for PDE
based builds is straightforward, see the contribution on:
   https://bugs.eclipse.org/bugs/show_bug.cgi?id=289838
  * Disk space / Bandwidth
    - Big components mean big (and slower) repositories -- each git
repository has all the history, and cloning necessarily requires
transferring all of that (albeit compressed).
  * submodules / team shared sets
    - We can recreate the component structure using git submodules
without forcing all the plugins to live in one repository. This is
akin to the symlinking we've got going on in the CVS tree today :).
Alternative we could use team shared sets.

Clearly I'm all for project level granularity: it's more flexible than
component level granularity, and the features of component level can
be replicated using existing git / eclipse functionality. However this
is just my opinion.  I've posted a couple scripts in
https://bugs.eclipse.org/280583 for recreating the current
org.eclispe.cdt/ tree in git using a git repository per CDT Project as
I do here.  If you have any thoughts on this, do pipe up before the
repository layouts are set in stone!

Using git my product currently tracks CDT HEAD (crazy, I know :P), and
I can easily manage / merge my local feature which aren't ready to go
upstream yet.  I can build using the cdt.releng PDE based build
substituting cvs: for git: in the map.

I think CDT is unique in the number and variety of downstream
consumers.  Denis seems keen to do what's easiest, and consistent
across all the projects. I think it's important we get something
that's both convenient and flexible...

Cheers,
James

2009/10/26 Denis Roy <denis.roy@xxxxxxxxxxx>:
> Greetings,
>
> 1. CVS/SVN slow
>
> You may have noticed that CVS and SVN have been slowing down lately. The
> disk arrays hosting these files are very busy, partly because the entire
> world is doing all kinds of stuff with anonymous (pserver) CVS.  Because
> many teams use pserver for builds, I cannot offer a better class of service
> for committers.
>
> I am proposing a solution on this bug: http://bugs.eclipse.org/293355
> Essentially, I want to use a shadow (delayed) copy of CVS for pserver
> access. Please take a moment to read it and/or comment, because something
> will have to change to fix this.
>
>
> 2. Nightly/Integration builds on Amazon AWS
>
> As you may know, we try to exclude Nightly and I-builds from being mirrored,
> since they make our mirror size very large and require lots of bandwidth to
> mirror (but get very few downloads).  However, downloading a Nighly build
> from download.eclipse.org can be challenging.
>
> I have since altered the way we sync to our Amazon AWS service.  We now sync
> every 2 hours to AWS, and AWS contains everything on download.eclipse.org.
> If you need a Nightly build, getting it from AWS is your best bet.
>
>
> 3. git mirror of Eclipse CVS
>
> In bug 280583 I am working on providing a read-only git mirror of Eclipse
> projects.  Part of the discussion revolves around structuring the git
> repositories, and I will likely be using the project meta-data to map
> project<->cvs directories.
>
> Please take a moment to ensure your 'source repository' meta-data is current
> in the Portal.
>
>
> As usual, thanks for your help!
>
> Denis
>
>
>
> _______________________________________________
> eclipse.org-committers mailing list
> eclipse.org-committers@xxxxxxxxxxx
> https://dev.eclipse.org/mailman/listinfo/eclipse.org-committers
>
> IMPORTANT: Membership in this list is generated by processes internal to the
> Eclipse Foundation.  To be permanently removed from this list, you must
> contact emo@xxxxxxxxxxx to request removal.
>