Bug 374428 - source build is broken and will be removed from primary builds
Summary: source build is broken and will be removed from primary builds
Status: RESOLVED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: Releng (show other bugs)
Version: 3.8   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Platform-Releng-Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 271953 371163 (view as bug list)
Depends on:
Blocks: 376451
  Show dependency tree
 
Reported: 2012-03-15 14:47 EDT by Kim Moir CLA
Modified: 2012-05-06 21:01 EDT (History)
10 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Kim Moir CLA 2012-03-15 14:47:52 EDT
The current source build and source fetch build that are available on the build page are broken. They don't take into account our git migration, among other things. Should they be fixed or should we just delete them given the current CBI initiative?
Comment 1 John Arthorne CLA 2012-03-15 16:32:00 EDT
I say remove it, unless somebody who needs this is willing to make it work again. The current state is that the source build does contain the source, but the scripts to actually build that source are out of date and broken.

Everyone who builds Eclipse themselves seems to fetch their own source so it doesn't seem to be adding much value. For reference there were 800 downloads of the 3.7.0 source build, and about 1000 for the 3.6.0 source build.

CCing PMC members for more input. I think our viable options are:

1) Leave the srcIncluded zip in its current broken form. You couldn't build Eclipse with it, but maybe people are using it for other purposes - source scans, archiving, etc.

2) Remove it completely.

The srcFetch build I think we can safely remove. This one is completely broken because it is structured around fetching from CVS. CBI and Linux Tools already provide scripts for doing this if you want to go the "fetch it yourself" route.
Comment 2 Dani Megert CLA 2012-03-15 17:36:38 EDT
+1 for 2). I never used that build in the past 10 years.
Comment 3 Paul Webster CLA 2012-03-15 19:11:36 EDT
I'm happy to go with #1 as I've heard of people downloading the srcIncluded archive to see the source that went into a build.  But it's not something I've ever used.

+1 also for removing srcFetch ... that's totally broken.

PW
Comment 4 David Williams CLA 2012-03-15 22:41:02 EDT
To cross reference, a very old request for all projects to provide "source builds" was recently closed as "won't fix" with no complaints ... see bug 185146.
Comment 5 Alexander Kurtakov CLA 2012-03-16 07:39:03 EDT
We (Linux Tools) project would be willing to investigate a possibility to get source tarballs generated by the official builder that would be usable to build from source. Currently we use the following script http://git.eclipse.org/c/linuxtools/org.eclipse.linuxtools.eclipse-build.git/tree/eclipse-build/buildSDKSource.sh to produce tarballs like http://www.eclipse.org/downloads/download.php?file=/technology/linuxtools/eclipse-build/3.8.x/eclipse-3.8.0-M5-src.tar.bz2 which we use with additional scripts.
It would be very good if a similar tarball or at least a complete one with all the sources(no binaries) is generated by the regular buildsystem for every I-build. This will save time for us, keep on the same base and we are willing to spend time on this to happen if people are interested and will accept such thing.
Comment 6 David Williams CLA 2012-03-16 10:26:50 EDT
(In reply to comment #5)
> ... This will save time for us, keep on the same base and we are willing
> to spend time on this to happen if people are interested and will accept such
> thing.

I'm not the one to say "no", or "yes" ... but ... one question that occurs to me is you say it "saves you time" but doesn't it "cost us time (and storage)" every build then, whether used or not? I'm only asking since I'm probably missing your point I so wanted to clarify. 

If that is the case, then it might be nice to "have a separate job" and if/when needed then it could be generated (refetched from build input/output), and once generated it could be stored and linked in pages where everyone could find it easily ... but, offhand, we're trying to simplify the build and/or break it into "different pieces". 

To emphasize, I'm not trying to argue against the idea ... I'm not fundamentally opposed to it (and haven't looked at your scripts :(  ... but I am just trying to help clarify things. 

Thanks,
Comment 7 Alexander Kurtakov CLA 2012-03-16 11:38:13 EDT
(In reply to comment #6)
> (In reply to comment #5)
> > ... This will save time for us, keep on the same base and we are willing
> > to spend time on this to happen if people are interested and will accept such
> > thing.
> 
> I'm not the one to say "no", or "yes" ... but ... one question that occurs to
> me is you say it "saves you time" but doesn't it "cost us time (and storage)"
> every build then, whether used or not? I'm only asking since I'm probably
> missing your point I so wanted to clarify. 

well, if you look at platform builds only it will cost more storage but if you look from the point of Eclipse.org infrastructure it won't cost more storage as for Juno we follow the platform pretty close and sooner we will have the source tarballs for pretty much every I-build. About time it shouldn't cost development time as we would do the development and maintenance. About build time I'm fine with an approach that it doesn't get enabled by default unless it takes less than certain amount of time(10-20 mins?) and see whether we can meet such expectations.

> 
> If that is the case, then it might be nice to "have a separate job" and if/when
> needed then it could be generated (refetched from build input/output), and once
> generated it could be stored and linked in pages where everyone could find it
> easily ... but, offhand, we're trying to simplify the build and/or break it
> into "different pieces". 

Well, the idea behind the proposal is to make it easier for people to try building random I-build instead of having to generate the source tarball themselves first as for this task one have to pretty familiar with eclipse platform build system.

> 
> To emphasize, I'm not trying to argue against the idea ... I'm not
> fundamentally opposed to it (and haven't looked at your scripts :(  ... but I
> am just trying to help clarify things. 

Sure, I know that it's pretty complicated now and having two separate build projects is not helping to simplify things as people often mismatch instructions for the two builds and we duplicate so much work when both of our groups would have spent better. With the current amount of resources available I think it's more than mandatory to not duplicate work. But if one prefers simplicity than that I can understand him too.
> 
> Thanks,
Comment 8 Krzysztof Daniel CLA 2012-03-16 12:49:12 EDT
A really important aspect of source builds is the fact that they are crucial for Eclipse adoption in the Linux World. Difficulties with building Eclipse from source cause that various distros stick with old versions of Eclipse. Of course it is always possible to download Eclipse for Linux from eclipse.org, but you cannot then manage that Eclipse in those modern virtualization environments, where all users get the same version of Eclipse because of single admin's click.

Having Eclipse packaged by various linux distros is a must-have for me, and this will not happen until Eclipse is easy to build entirely from source. The low download stats does not necessarily come from the lack of purposeability, but may rather from the lack of usability?

As Alex said - we already build source tarballs on weekly basis, sometimes even more frequently. The problem is that fetching sources on any other server than build.eclipse.org takes about 4hours, is extremely fragile, and cannot be restarted. Building the same sources @ build.eclipse.org takes around 20 minutes.

We are definitely interested in automatization of those builds on a weekly basis, and accompanying each build with eclipse-build snapshot, so any package maintainer is able to download and build Eclipse for his Linux distro.
Comment 9 David Williams CLA 2012-03-16 13:04:38 EDT
(In reply to comment #8)

Ok, so this sounds like an important function, for someone to provide, in some manner. 

I can see the merit of "fetching and taring up" the source on build.eclipse.org (since 20 minutes, instead of 4 hours). 

If I'm hearing this right, though, still doesn't sound like it needs to be done by the Eclipse builder itself, for each and every build. 

As another example of merit of it being a "separate task" is that then it could likely be re-used by those using CBI as the platform hopes to eventually do. 

Am I making sense? Or am I misunderstanding? :)
Comment 10 Krzysztof Daniel CLA 2012-03-18 01:43:51 EDT
(In reply to comment #9)
> (In reply to comment #8)
> If I'm hearing this right, though, still doesn't sound like it needs to be done
> by the Eclipse builder itself, for each and every build. 
> 
> As another example of merit of it being a "separate task" is that then it could
> likely be re-used by those using CBI as the platform hopes to eventually do. 
> 
> Am I making sense? Or am I misunderstanding? :)

I do not insist on getting source as a part of a regular build. I'm fine for creating a separate task that would create the source for IBuilds + milestones and releases.
Comment 11 David Williams CLA 2012-04-01 20:47:27 EDT
*** Bug 371163 has been marked as a duplicate of this bug. ***
Comment 12 John Arthorne CLA 2012-04-02 09:49:59 EDT
*** Bug 271953 has been marked as a duplicate of this bug. ***
Comment 13 Hasan Ceylan CLA 2012-04-02 14:46:23 EDT
I have been included in the trail by chance.

I would like to stop by add my 2c.

srcFetch was used by downstream providers like Redhat to build rpms. At least that was true as of the last time I used Fedora. (Now I use ubuntu although not checked it personally I am guessing they use similar ways to produce deb packages.) 

If that is still true, IMHO it is worth publicly announcing the removal of the feature to let downstream providers adjust their position.

Regards,
Hasan Ceylan
Comment 14 Andrew Overholt CLA 2012-04-02 16:10:43 EDT
(In reply to comment #13)
> srcFetch was used by downstream providers like Redhat to build rpms.

We don't do that anymore as the Linux Tools eclipse-build effort produces its own source drops that are usable.

Thanks :)
Comment 15 David Williams CLA 2012-04-11 00:54:16 EDT
(In reply to comment #13)
> I have been included in the trail by chance.
> 
> I would like to stop by add my 2c.
> 
> srcFetch was used by downstream providers like Redhat to build rpms. At least
> that was true as of the last time I used Fedora. (Now I use ubuntu although not
> checked it personally I am guessing they use similar ways to produce deb
> packages.) 
> 
> If that is still true, IMHO it is worth publicly announcing the removal of the
> feature to let downstream providers adjust their position.
> 
> Regards,
> Hasan Ceylan

Thanks for your 2c, but I think those that are interested in this sort of thing follow releng list, bugzilla, etc., pretty close, so I do consider it "announced publically". If you think it should be announced somewhere else in some other manner, feel free. (Just, please, don't do a press release :) 

I'd like to use this bug to document the removal of the "source build packages" from the primary build and download pages, and have opened a clone, bug 374428, where others can volunteer to restore source packages if they'd like. While I'm pretty new to this build (and to Git) I've heard Git makes it much easier? So ... please comment in bug 374428 if anyone wants to take on the new task. 

Thanks.
Comment 16 David Williams CLA 2012-04-26 10:04:35 EDT
The "source builds" have been disabled/removed (I think) from all the visible places in the builds. 

There is still old, now dead code laying around in eclipse builder that should be removed at some point, but ... I think that can wait. So will count this bug as fixed, and track dead code removal in bug 377764.
Comment 17 David Williams CLA 2012-05-06 21:01:34 EDT
(In reply to comment #15)
> (In reply to comment #13)

> 
> I'd like to use this bug to document the removal of the "source build packages"
> from the primary build and download pages, and have opened a clone, bug 374428,
> where others can volunteer to restore source packages if they'd like. While I'm
> pretty new to this build (and to Git) I've heard Git makes it much easier? So
> ... please comment in bug 374428 if anyone wants to take on the new task. 
> 

I see I pasted in the wrong bug number for the new enhancement request, it is bug  bug 376451.