Bug 237588 - Get nightly Nebula builds up and running
Summary: Get nightly Nebula builds up and running
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Nebula (show other bugs)
Version: unspecified   Edit
Hardware: PC Mac OS X - Carbon (unsup.)
: P1 normal (vote)
Target Milestone: ---   Edit
Assignee: Thomas Schindl CLA
QA Contact:
URL:
Whiteboard:
Keywords: plan
Depends on: 271416 272403 285127 285594
Blocks:
  Show dependency tree
 
Reported: 2008-06-18 03:33 EDT by Thomas Schindl CLA
Modified: 2021-07-05 11:39 EDT (History)
12 users (show)

See Also:


Attachments
The diff for Nick (31.73 KB, image/tiff)
2008-11-25 04:04 EST, Thomas Schindl CLA
no flags Details
Am I blind??? (13.55 KB, image/png)
2008-11-25 07:48 EST, Thomas Schindl CLA
no flags Details
The build log (46.53 KB, text/plain)
2008-12-09 13:36 EST, Thomas Schindl CLA
no flags Details
Gallery build log (compilation failure) (157.44 KB, application/octet-stream)
2009-03-25 13:53 EDT, Nicolas Richeton CLA
no flags Details
Gallery build errors (linux) (13.49 KB, text/plain)
2009-03-31 06:37 EDT, Nicolas Richeton CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Schindl CLA 2008-06-18 03:33:48 EDT
This is a bug to ensure that we get up the nightly builds of our widgets. Chris. I'll go to take care of this bug once the vote is done.
Comment 1 Thomas Schindl CLA 2008-06-18 03:39:03 EDT
Nick, I know you have enough to do to make the EMF builds happen but it would be great if you could help us here over at nebula to get a build process up and running. The current one has stopped working some months ago. 

You came to my mind because nebula has like EMF many independent sub-components and is structured fairly loosely.
Comment 2 Nick Boldt CLA 2008-06-18 09:14:54 EDT
Loosely? Bite your tongue, man! We run a tight ship here. :)

As to helping w/ your build, by all means -- once Ganymede's in the can I've already offered to help w/ PDT, so you can join the queue too.

Does your build involve platform-specific building? If so, then I can prolly advise you, but won't be able to convert you over to my system entirely as we do 100% java builds w/ no .so or .dll issues to content with.

In the meantime, you can have a gander at http://wiki.eclipse.org/Modeling_Project_Releng for tips on everything from CVS structure to bugzilla integration.
Comment 3 Chris Gross CLA 2008-06-18 09:56:57 EDT
Hey guys,

We already do have a build for Nebula.  Its not working for some reason and I haven't had time to look into it, but its likely a simple problem (bad password, process was shutdown, etc).  You can find all the build scripts for Nebula in CVS.  

I think the biggest problem is that Eclipse.org doesn't provide infrastructure to run builds (at least not for us).  So its currently running (well not running ;)) on a machine in my group at IBM.  The machine is running tons of other stuff as well and generally I can't ensure that the Nebula build will continue to run unaffected.

-Chris
Comment 4 Nick Boldt CLA 2008-06-18 10:10:40 EDT
Need hardware? Open a bug in Community > Servers and ask for a vserver for your builds. You'll get root access to your own box.

Or, run your build on build.eclipse.org, to which you already have access if you have cvs access to dev.eclipse.org. On build.eclipse, you get shared access to the monster build server.

(To use build.eclipse, you might need to use the portal.eclipse.org site to request a full login shell or open a bug if the portal doesn't grant you that option.)

Or... you could borrow some space on emf.torolab.ibm.com from me, if you have a reason to run @ IBM rather than @ eclipse.org. :)
Comment 5 Chris Gross CLA 2008-06-18 10:55:01 EDT
Well that sounds particularly easy.  When I originally went about setting up the Nebula build I was told this kinda stuff wasn't available.

Tom do you want to go through the steps Nick offered (there's no reason we need to continuing running builds inside IBM)?  The build scripts should be pretty straight forward once you take a look at them in CVS.  If not, feel free to contact me as always.
Comment 6 Nick Boldt CLA 2008-11-21 00:22:15 EST
BTW, you might want to take a look at the Common Builder:

http://wiki.eclipse.org/Common_Build_Infrastructure

Comment 7 Paul Webster CLA 2008-11-21 09:25:50 EST
Tom,

./start.sh -projectid nebula.widgets.grid \
  -version 1.0.0 \
  -antTarget runWithoutTest \
  -noclean \
  -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology' \
  -projRelengPath 'org.eclipse.swt.nebula/releng/org.eclipse.nebula.widgets.grid.releng' \
  -basebuilderBranch R35_M2 \
  -javaHome /opt/public/common/ibm-java2-142 \
  -URL http://www.eclipse.org/external/eclipse/downloads/drops/S-3.5M3-200810301917/eclipse-SDK-3.5M3-win32.zip \
  2>&1 | tee ~/buildlog_`date +%Y%m%d%H%M%S`.txt

Run this script from /opt/public/cbi/build/org.eclipse.dash.common.releng/tools/scripts

By default you will get N builds in /opt/public/cbi/build/widgets/grid/downloads/drops/1.0.0

Comment 8 Paul Webster CLA 2008-11-21 09:30:10 EST
The nebula-widgets-SDK-N200811210902.zip will be in the N200811210902/eclipse/N200811210902 directory.  You should also be able to generate a p2 repository as well with something like:

p2.repo=${buildDirectory}/${buildLabel}/repository
generate.p2.metadata=true
p2.metadata.repo=file:${p2.repo}
p2.artifact.repo=file:${p2.repo}
p2.flavor=tooling
p2.publish.artifacts=true
p2.root.name=org.eclipse.swt.nebula.grid
p2.root.version=1.0.0.v${buildId} 

PW
Comment 9 Nick Boldt CLA 2008-11-21 17:38:30 EST
(In reply to comment #8)
> The nebula-widgets-SDK-N200811210902.zip will be in the
> N200811210902/eclipse/N200811210902 directory. 

If you're ending up with failures in the build system due to trying to compile (but not run) your tests, you can skip over that step entirely with -andTarget runWithoutTestBuild.

If all goes well, your resulting zips should be in N200811210902/, rather than N200811210902/eclipse/N200811210902/, and those temporary subfolders will be scrubbed from the disk.
Comment 10 Nick Boldt CLA 2008-11-21 17:39:13 EST
Sorry, that should be -antTarget, not -andTarget. I've updated the comments in start.sh to mention this additional target too.
Comment 11 Thomas Schindl CLA 2008-11-21 17:55:29 EST
Nick, what do I have to do to the build. My problem is that when I log in dev.eclipse.org or build.eclipse.org I get dropped of immediately.

tom-schindls-computer:~/Documents/eclipse-ws/uface-dev/proper/org.ufacekit tomson$ !ss
ssh tschindl@build.eclipse.org
Password: 
Last login: Fri Nov 21 08:22:09 2008 from 81.16.98.98
r$ 
Not allowed to do '' at /usr/bin/cvssh line 106, <STDIN> line 1.
Connection to build.eclipse.org closed.
tom-schindls-computer:~/Documents/eclipse-ws/uface-dev/proper/org.ufacekit tomson$ 

Tom
Comment 12 Paul Webster CLA 2008-11-21 18:50:44 EST
(In reply to comment #9)
> If you're ending up with failures in the build system due to trying to compile
> (but not run) your tests, you can skip over that step entirely with -andTarget
> runWithoutTestBuild.
> 

I was using -antTarget runWithoutTest  ... is that not enough?

PW
Comment 13 Nick Boldt CLA 2008-11-21 21:32:21 EST
(In reply to comment #11)
> Not allowed to do '' at /usr/bin/cvssh line 106, <STDIN> line 1.

0. By default all committers get a cvssh shell account.
1. log into portal.eclipse.org.
2. request a "full bash shell account"
3. enjoy full shell access to {dev|util|build|download1}.eclipse.org

Can you do me a favour? I've told people about this for ages as I have always just had access, I've never actually seen the UI in the portal to request this access. Can you grab a screenshot and attach it here? I would like to stick this into a FAQ in the wiki, but without the eye-candy it's kinda lacking. 

(In reply to comment #12)
> > If you're ending up with failures in the build system due to trying to compile
> > (but not run) your tests, you can skip over that step entirely with -antTarget
> > runWithoutTestBuild.
> I was using -antTarget runWithoutTest  ... is that not enough?

runWithoutTest == do SDK/runtime/examples build, then do tests build. DO NOT RUN the tests.

runWithoutTestBuild == do SDK/runtime/examples build. DO NOT BUILD OR RUN the tests.

Ultimately you should be using "run", but these extra targets are there to let you get up to speed more gradually. Testing should be a mandatory part of the build.
Comment 14 Nick Boldt CLA 2008-11-21 22:45:48 EST
Running as dashBuild@build.eclipse.org, I got this:

Build folder:

/opt/public/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158
http://build.eclipse.org/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158/ (links don't work yet -- bug 256192)

Log:

/opt/public/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158/buildlog.txt

Execution:

cd /opt/public/cbi/build/org.eclipse.dash.common.releng/tools/scripts; ./start.sh -projectid nebula.widgets.grid   -version 1.0.0   -antTarget runWithoutTestBuild   -noclean   -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology'   -projRelengPath 'org.eclipse.swt.nebula/releng/org.eclipse.nebula.widgets.grid.releng'   -basebuilderBranch R35_M2   -javaHome /opt/public/common/ibm-java2-142   -URL  http://www.eclipse.org/external/eclipse/downloads/drops/S-3.5M3-200810301917/eclipse-SDK-3.5M3-win32.zip  2>&1 | tee ~/buildlog_`date +%Y%m%d%H%M%S`.txt
Comment 15 Thomas Schindl CLA 2008-11-25 04:04:24 EST
Created attachment 118642 [details]
The diff for Nick
Comment 16 Thomas Schindl CLA 2008-11-25 04:06:42 EST
So what does this mean now - is there something wrong on our side?

Tom

(In reply to comment #14)
> Running as dashBuild@build.eclipse.org, I got this:
> 
> Build folder:
> 
> /opt/public/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158
> http://build.eclipse.org/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158/
> (links don't work yet -- bug 256192)
> 
> Log:
> 
> /opt/public/cbi/build/widgets/grid/downloads/drops/1.0.0/N200811212158/buildlog.txt
> 
> Execution:
> 
> cd /opt/public/cbi/build/org.eclipse.dash.common.releng/tools/scripts;
> ./start.sh -projectid nebula.widgets.grid   -version 1.0.0   -antTarget
> runWithoutTestBuild   -noclean   -projRelengRoot
> ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology'   -projRelengPath
> 'org.eclipse.swt.nebula/releng/org.eclipse.nebula.widgets.grid.releng'  
> -basebuilderBranch R35_M2   -javaHome /opt/public/common/ibm-java2-142   -URL 
> http://www.eclipse.org/external/eclipse/downloads/drops/S-3.5M3-200810301917/eclipse-SDK-3.5M3-win32.zip
>  2>&1 | tee ~/buildlog_`date +%Y%m%d%H%M%S`.txt
> 

Comment 17 Thomas Schindl CLA 2008-11-25 04:18:28 EST
Sorry Nick can't find the UI in portal myself to request such a thing but http://wiki.eclipse.org/IT_Infrastructure_Doc states that one has to file a bug to make this happen.
Comment 18 Paul Webster CLA 2008-11-25 07:40:42 EST
(In reply to comment #17)
> Sorry Nick can't find the UI in portal myself to request such a thing but
> http://wiki.eclipse.org/IT_Infrastructure_Doc states that one has to file a bug
> to make this happen.

It really did used to be there :-)  under All Project: [committer tools]

PW
Comment 19 Thomas Schindl CLA 2008-11-25 07:48:48 EST
Created attachment 118655 [details]
Am I blind???

Maybe I'm blind but this how my portal looks like when open :-(
Comment 20 Nick Boldt CLA 2008-11-25 12:49:20 EST
(In reply to comment #16)
> So what does this mean now - is there something wrong on our side?

It means I ran a build and you can verify the contents of the zips contain what you'd expect them to contain, and can be thus installed into your local Eclipse install. It means everything appears to be working, at least on the surface.

(In reply to comment #17)
> Sorry Nick can't find the UI in portal myself to request such a thing but
> http://wiki.eclipse.org/IT_Infrastructure_Doc states that one has to file a bug
> to make this happen.

Sorry, I don't maintain the portal code (and despite requests to make it open source, the Foundation won't) so I can't even poke at it to see when/why this link vanished.

Open a bug, point at your screenshot and ask "where did the link go?"

Comment 21 Thomas Schindl CLA 2008-12-08 06:38:35 EST
Nick everything worked [1]! What's the next step?

[1] http://build.eclipse.org/cbi/build/widgets/grid/downloads/drops/1.0.0/N200812080555/eclipse/N200812080555/
Comment 22 Nick Boldt CLA 2008-12-08 12:19:59 EST
(In reply to comment #21)
> Nick everything worked [1]! What's the next step?
> 
> [1]
> http://build.eclipse.org/cbi/build/widgets/grid/downloads/drops/1.0.0/N200812080555/eclipse/N200812080555/

Well, if everything was 100% working (please review your log) you'd have zips, md5 sums and test results in http://build.eclipse.org/cbi/build/widgets/grid/downloads/drops/1.0.0/N200812080555/. I suspect something is amiss because the zips are still in their temporary build folder, not their final destination. Can you attach a log, or re-run the build using start_logger.sh (instead of start.sh) so that a buildlog.txt appears in the build's target folder?



Comment 23 Thomas Schindl CLA 2008-12-09 13:36:46 EST
Created attachment 119945 [details]
The build log
Comment 24 Thomas Schindl CLA 2008-12-09 13:38:27 EST
Looks like that the problem is that we have no tests :-(
Comment 25 Nick Boldt CLA 2008-12-09 16:38:06 EST
(In reply to comment #24)
> Looks like that the problem is that we have no tests :-(

That's certainly a problem. :)

Workaround is noted in comment 13:

> runWithoutTest == do SDK/runtime/examples build, then do tests build. DO NOT
RUN the tests.

> runWithoutTestBuild == do SDK/runtime/examples build. DO NOT BUILD OR RUN the
tests.
Comment 26 Nick Boldt CLA 2008-12-09 16:46:37 EST
(In reply to comment #21)
> Nick everything worked [1]! What's the next step?

The next steps are:

* generate update site / p2 metadata repo
* publish build to download.eclipse.org so it's publicly accessible and mirrored.
* send emails / newsgroup announcements

I have a script for both but they haven't been ported for use on build.eclipse.org. See bug 252035, bug 253280, bug 253285.

You'll also want some web UI in the build folder (or that references the build folder) like what's in eclipse.org/modeling/emf/downloads/. 

And, if you're in Galileo, you'll want to do this:

* update Galileo contribution file (bug 257004)
Comment 27 Thomas Schindl CLA 2008-12-10 04:12:18 EST
(In reply to comment #26)
> (In reply to comment #21)
> > Nick everything worked [1]! What's the next step?
> 
> The next steps are:
> 
> * generate update site / p2 metadata repo
> * publish build to download.eclipse.org so it's publicly accessible and
> mirrored.
> * send emails / newsgroup announcements
> 
> I have a script for both but they haven't been ported for use on
> build.eclipse.org. See bug 252035, bug 253280, bug 253285.

So I need to wait, right?

> 
> You'll also want some web UI in the build folder (or that references the build
> folder) like what's in eclipse.org/modeling/emf/downloads/. 
> 

Can you help me to get something there. I have no clue at all how to administrate such a site.

> And, if you're in Galileo, you'll want to do this:
> 
> * update Galileo contribution file (bug 257004)
> 

We are not in Galileo so this doesn't apply to us.
Comment 28 Nick Boldt CLA 2008-12-10 10:41:07 EST
(In reply to comment #27)
> > I have a script for both but they haven't been ported for use on
> > build.eclipse.org. See bug 252035, bug 253280, bug 253285.
> So I need to wait, right?
Yes, sorry. 
 
> > You'll also want some web UI in the build folder (or that references the build
> > folder) like what's in eclipse.org/modeling/emf/downloads/. 
> Can you help me to get something there. I have no clue at all how to
> administrate such a site.

You can borrow from what's in /cvsroot/org.eclipse/www/modeling/, as implemented by GEF and PDT (/cvsroot/org.eclipse/www/{gef,pdt}), but if you wait we'll have something more generic as part of CBI one day.
 
Comment 29 Jeremy Dowdall CLA 2009-02-07 16:49:22 EST
> You can borrow from what's in /cvsroot/org.eclipse/www/modeling/, as
> implemented by GEF and PDT (/cvsroot/org.eclipse/www/{gef,pdt}), but if you
> wait we'll have something more generic as part of CBI one day.

Tom,
 I'll take care of putting this together.

Nick,
 Is there a bug or something to track when CBI implements this?
Comment 30 Nick Boldt CLA 2009-02-21 14:08:34 EST
>  Is there a bug or something to track when CBI implements this?

Not really. There's this bug, but that's not quite the same issue.

https://bugs.eclipse.org/bugs/show_bug.cgi?id=251945

You could open one, then attach patches and ideas to it. :)
Comment 31 Nicolas Richeton CLA 2009-03-17 13:45:33 EDT
Adding this item to the project plan.

Are we waiting for the 3 bugs Tom mentionned ? 
If the build process builds the plugins correctly, can we just do a simple scp to download.eclipse.org ? We don't need a p2 repo or mails right from the start.
Comment 32 Jeremy Dowdall CLA 2009-03-18 23:19:35 EDT
(In reply to comment #31)
> Adding this item to the project plan.
> 
> Are we waiting for the 3 bugs Tom mentionned ? 
> If the build process builds the plugins correctly, can we just do a simple scp
> to download.eclipse.org ? We don't need a p2 repo or mails right from the
> start.
> 

The only thing I'm waiting for is some time to get back to this... :(
I'm going to try to get to the build tutorial on monday morning, so hopefully that'll help me get this moving again.
In the mean-time, I don't think there's anything wrong with building locally and then uploading the jars directly to download.eclipse.org - like you said, we don't need all the extras right away.  Let me know if you don't have access and I can put them up there for you.
Comment 33 Nicolas Richeton CLA 2009-03-24 10:28:02 EDT
Jeremy, 

When you say "building locally", do you mean using a personnal computer/server to do the build? If we can't use build.eclipse.org, we should ask for a vserver to run all nebula builds (see Nick's comment).

Are you working on the build scripts from org.eclipse.nebula.build or the ones for build.eclipse.org ? (not the same, I guess). 
Comment 34 Nick Boldt CLA 2009-03-24 12:12:17 EDT
(In reply to comment #33)
> Jeremy, 
> 
> When you say "building locally", do you mean using a personnal computer/server
> to do the build? If we can't use build.eclipse.org, we should ask for a vserver
> to run all nebula builds (see Nick's comment).

The most recent improvements in the Athena builder involve being able to run it locally in your own Eclipse workspace, on a vserver, or on build.eclipe.org via Hudson (or via commandline). 

So... check out our slides & exercises, and try it for yourself locally, then on build.eclipse... and if you can get yourself added to the callisto-dev group, then you can use Hudson to manage your builds too.

Comment 35 Nicolas Richeton CLA 2009-03-25 13:52:17 EDT
Nick, 
I'm trying to build Gallery on OSX from terminal : the compilation begins but I get compile errors on each class (all dependencies are missing). See attached log.
 I tried to add -Djava.rt=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/classes.jar to start.sh 
 and bootclasspath=${java.home}/../Classes/*.jar to build.properties with no result. 
 
 Any idea ? 
 Where is the right place to ask questions on using athena ? 
 
 Full command is : ./start.sh -projectid org.eclipse.nebula.widgets.gallery -version 0.0.1   -projRelengRoot ':pserver:anonymous@dev.eclipse.org:/cvsroot/technology'   -projRelengPath 'org.eclipse.swt.nebula/org.eclipse.nebula.widgets.gallery.releng'   -writableBuildRoot $HOME/Desktop/Nebula/build/ -Djava.rt=/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Classes/classes.jar &
Comment 36 Nicolas Richeton CLA 2009-03-25 13:53:16 EDT
Created attachment 129872 [details]
Gallery build log (compilation failure)
Comment 37 Nicolas Richeton CLA 2009-03-27 03:16:08 EDT
When building on Linux , I get : 

/home/nricheton/nebula_builds/build/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.5.0.N20081008-2000/scripts/genericTargets.xml:92: Bundle org.eclipse.nebula.widgets.gallery_0.0.0 failed to resolve.:
	Missing required plug-in org.eclipse.swt_0.0.0.
	Optional plug-in org.eclipse.jface_0.0.0 is not available.

and a ton of missing plugins warning from [eclipse.buildScript]

(I changed dependencyUrl with a linux-gtk release of Eclipse.)
Comment 38 Nicolas Richeton CLA 2009-03-27 06:26:21 EDT
In addition of building the widget, we should also build the example project.
Comment 39 Jeremy Dowdall CLA 2009-03-27 12:50:40 EDT
(In reply to comment #33)
> When you say "building locally", do you mean using a personnal computer/server
> to do the build?

Hi Nicolas,

I meant just building on our personal computers using a plain 'ole PDE type build just like we've done to upload a release to Sourceforge - build the jars and then scp them to the down load server - doesn't get us any automation or signing, but does get the files up there (with mirroring) so that people can play with them...

At this point though, it sounds like several of us are pretty close to solving this, so finishing a "real" build is probably the way to go.

Tom mentioned that we'll _not_ be able to join callisto-dev and use Hudson, because we are not on the official release train... oh well, we have other issues to deal with before that anyway.

Getting the example built is a good plan too - I'd forgotten about that.
Comment 40 Nick Boldt CLA 2009-03-30 18:24:58 EDT
(In reply to comment #37)
> When building on Linux , I get : 
> 
> /home/nricheton/nebula_builds/build/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.5.0.N20081008-2000/scripts/genericTargets.xml:92:
> Bundle org.eclipse.nebula.widgets.gallery_0.0.0 failed to resolve.:
>         Missing required plug-in org.eclipse.swt_0.0.0.
>         Optional plug-in org.eclipse.jface_0.0.0 is not available.
> 
> and a ton of missing plugins warning from [eclipse.buildScript]
> 
> (I changed dependencyUrl with a linux-gtk release of Eclipse.)

Unless your deps change frequently, you're better to use build.properties to set the Eclipse SDK (or any other deps):

dependencyURLs=http://download.eclipse.org/eclipse/downloads/drops/S-3.5M6-200903130100/eclipse-SDK-3.5M6-linux-gtk.tar.gz

or

dependencyURLs=http://download.eclipse.org/eclipse/downloads/drops/S-3.5M6-200903130100/eclipse-SDK-3.5M6-macosx-carbon.tar.gz 

As to the compilation issues on mac, try this:

JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home
JAVA60_HOME=/System/Library/Frameworks/JavaVM.framework/Home

See also slides and exercises for building GEF or Linux Tools:

http://www.eclipsecon.org/2009/sessions?id=302
Comment 41 Nicolas Richeton CLA 2009-03-31 06:36:48 EDT
Thanks Nick, it works on OSX. I was trying to make build.properties independent of the OS and to set paths from the command line, but I'll use one file per OS as in gef.

BUT, it still does not work on linux, I'm using exactly the same command and properties (except for JAVA_HOME)
(see log attached)

Now I'm trying to generate javadoc and publishing downloads. 
Java doc : is this supported ? Are javadoc supposed to be packages in one archive ? 

Publishing : I suppose I have to be careful here. Currently, I get all files in drops/version/nighltyid.
I suppose I need a promote.properties file at the root of my releng plugin with a few options to upload to eclipse downloads ?
How can I list existing downloads in /technology/nebula or delete my tests files in necessary ?
Comment 42 Nicolas Richeton CLA 2009-03-31 06:37:35 EDT
Created attachment 130379 [details]
Gallery build errors (linux)
Comment 43 Nicolas Richeton CLA 2009-03-31 06:59:15 EDT
> How can I list existing downloads in /technology/nebula or delete my tests files in necessary ? 
forget this, I just asked for a full shell access too...
Comment 44 Nick Boldt CLA 2009-03-31 14:46:56 EDT
(In reply to comment #41)
> Thanks Nick, it works on OSX. I was trying to make build.properties independent
> of the OS and to set paths from the command line, but I'll use one file per OS
> as in gef.

You could set properties in build.xml based on OS:

http://ant.apache.org/manual/CoreTasks/condition.html
 
> BUT, it still does not work on linux, I'm using exactly the same command and
> properties (except for JAVA_HOME)
> (see log attached)

Which SDK are you feeding in? Does it match your linux box (gtk, 32- or 64-bit) ?
 
> Now I'm trying to generate javadoc and publishing downloads. 
> Java doc : is this supported ? Are javadoc supposed to be packages in one
> archive ? 

Yes, but that's non-functional at the moment. See bug 256211 and 269290.

> Publishing : I suppose I have to be careful here. Currently, I get all files in
> drops/version/nighltyid.
> I suppose I need a promote.properties file at the root of my releng plugin with
> a few options to upload to eclipse downloads ?

Yeah, but the promote.properties stuff isn't hooked up yet. So, I'd recommend simply using rsync to push from one box to another, eg:

rsync -aP /path/to/drops/version/nighltyid you@dev.eclipse.org:/home/data/httpd/download.eclipse.org/technology/nebula/downloads/drops/version/

(note the trailing slash on target dir and lack of trailing slash on source dir so that the source will be copied INTO the target.

> How can I list existing downloads in /technology/nebula or delete my tests
> files in necessary ?

There's web UI in /cvsroot/org.eclipse/www/modeling/includes/ which handles this, but it's not yet been genericized for use w/ Hudson builds done on build.eclipse.org. You could have a look at how GEF or PDT reuse it in /cvsroot/org.eclipse/www/{gef,pdt}/downloads/

As to deleting files, you need a bash shell, not a cvssh shell, which I assume you've asked for in another bug.


Comment 45 Nicolas Richeton CLA 2009-04-01 04:14:59 EDT
Thanks Nick, now I can build on OSX then rsync results to downloads :-) : http://download.eclipse.org/technology/nebula/downloads/drops/N200904010945/nebula-gallery-SDK-incubation-N200904010945.zip

I still have a few questions :
 - Chris, Tom, should we do one build that includes everything (widgets, example, ...) or one build per project. We already talked about creating subprojects for each widget so one build per project seems the way to go. 


 - Nick, I'm about to move the build process to build.eclipse. 
       - Should I use cron to schedule it or is there already a global build schedule ? 
       - I made a shell script which runs start.sh and then does a rsync. What is the best place for it ?
       - Should I use /opt/public/cbi/build/ as writableBuildRoot  ?


 - We can organise technology/nebula this way :
      downloads/drops/
                   buildId1   (both nightly and releases)
                   buildId2  
      update/                 (update sites)
             release/         (latest release)
             nightly/         (latest nighlty)

  Does this sounds good to you ?
    
 Note: if we use on build per widget, there will be one update site for every widget.
        

I hope my build problem on my linux box, will not block the move to build.eclipse.org. 

> Which SDK are you feeding in? Does it match your linux box (gtk, 32- or 64-bit)?
I'm using the right SDK : linux-gtk (32bit). I have another eclipse build process (not CBI) on it which works with that version. Strange.  
I'm trying with motif just to be sure.     


         
Comment 46 Nick Boldt CLA 2009-04-01 13:59:26 EDT
(In reply to comment #45)
>  - Nick, I'm about to move the build process to build.eclipse. 
>        - Should I use cron to schedule it or is there already a global build
> schedule ? 

You can use Hudson to manage/schedule builds if you'd like, but you need to get added to group "callisto-dev" first - open a bug assigned to webmaster and ask for it. Once there, log in using your committerid and pwd:

https://build.eclipse.org/hudson/view/Athena%20CBI/

>        - I made a shell script which runs start.sh and then does a rsync. What
> is the best place for it ?
>        - Should I use /opt/public/cbi/build/ as writableBuildRoot  ?

Have a look at the way we run the GEF or Linux Tools in Hudson; it'll give you details on how to hook up start.sh + rsync.
 
>  - We can organise technology/nebula this way :
>       downloads/drops/
>                    buildId1   (both nightly and releases)
>                    buildId2  
>       update/                 (update sites)
>              release/         (latest release)
>              nightly/         (latest nighlty)
>   Does this sounds good to you ?

+1, though I might include an extra level of nesting:

>       downloads/drops/
>                    version/ (eg., 0.7.0, 0.7.1)
>                       buildId1   (both nightly and releases)
>                       buildId2  

>  Note: if we use on build per widget, there will be one update site for every
> widget.

You can then have a build to aggregate the update site zips into a single unpacked update site. 
 
> I hope my build problem on my linux box, will not block the move to
> build.eclipse.org. 

Just make sure you switch to the linux ppc version when building there. :) Again, see the GEF and LT builds for examples of how to run in Hudson.
 
Comment 47 Paul Webster CLA 2009-04-01 14:05:39 EDT
(In reply to comment #46)
> You can use Hudson to manage/schedule builds if you'd like, but you need to get
> added to group "callisto-dev" first - open a bug assigned to webmaster and ask
> for it. Once there, log in using your committerid and pwd:

Just as an aside, they will only add us if we're part of the Galileo Release train.

PW
Comment 48 Nick Boldt CLA 2009-04-01 14:37:43 EDT
(In reply to comment #47)
> > You can use Hudson to manage/schedule builds if you'd like, but you need to get
> > added to group "callisto-dev" first - open a bug assigned to webmaster and ask
> > for it. Once there, log in using your committerid and pwd:
> Just as an aside, they will only add us if we're part of the Galileo Release
> train.

Add your votes to bug 270633 re: read-only access, and to bug 257265 for per-committer access (rather than "callisto-dev"-only access).


Comment 49 Nicolas Richeton CLA 2009-04-02 04:27:05 EDT
I tried to start the build on build.eclipse.org, but I can't write anything to /opt/public/cbi/build

the process needs to create the following folders:

org.eclipse.nebula.gallery.releng  (will need one for each widget)  
technology

Is there a wait to schedule the build with the dashBuild user ? (Other than Hudson, because seems we won't be allowed to use it right now)

Or can someone create the folders mentionned above and give me some write access on it ? 
 
Comment 50 Nick Boldt CLA 2009-04-02 18:00:43 EDT
To run via Hudson, you can use the config I've created for you:

https://build.eclipse.org/hudson/view/Athena%20CBI/job/cbi-nebula.widgets.gallery-0.5.x-nightly/

However, you'll have to tweak the build_build.eclipse.org.sh such that it sets writableBuildRoot="${WORKSPACE}/build"

If you prefer to run as your own user on build.eclipse.org, then set writableBuildRoot="${HOME}/some/folder"; in this case you'll need to set up a crontab (`crontab -e`) to set a schedule for builds. In Hudson I have you set to poll CVS every 6 hrs and do an N build if anything's changed.

I don't think the rsync steps will work if run by Hudson (permission problems).

You could also run commandline as the dashBuild user but that won't get you any benefit beyond either running in Hudson or running as yourself via commandline.

So, if you want builds and promotes to run as one step, build as your user in your $HOME dir. If you want the Hudson UI for monitoring status, sending emails on failure, and tracking changes between builds, you'll need to get into the callisto-dev group.
Comment 51 Nick Boldt CLA 2009-04-02 18:01:42 EDT
(In reply to comment #50)
> You could also run commandline as the dashBuild user but that won't get you any
> benefit beyond either running in Hudson or running as yourself via commandline.

I should also mention that to run as dashBuild you need to be a committer on the Athena project itself. If you'd like to contribute, we'd be happy to have you. :)
Comment 52 Nicolas Richeton CLA 2009-04-03 03:40:01 EDT
Nick, I got the same problem on build.eclipse.org than on my linux box : plugins in the eclipse install get deleted right after untaring. I commented out <stripversion> in getdepencies.xml and it works. I don't know why, but there is an issue there.
Comment 53 Nicolas Richeton CLA 2009-04-03 07:49:15 EDT
Gallery is new built daily from my HOME folder, 
published as drops/buildId   and drops/latest
The update site is also available. 

See http://www.eclipse.org/nebula/downloads.php

I'll do some clean-up and start to work on the other widgets.

A few problems : 
- It would be better to build from another directory, I can give access to nebula_builds in my HOME, but somewhere in /opt/public/ would be better
- my feature version ends with .qualifier. this is replaced by HEAD.  I was expecting the date time. This will prevent updating with the nighlty update site.
- The build is different from the standard one : StripVersionNumbers is removed.
Comment 54 Nicolas Richeton CLA 2009-04-03 15:25:59 EDT
Grid is building but using tag v20081203 as defined in the map of grid.releng.   

Can we build products or jnlp with athena ? 
Comment 55 Nick Boldt CLA 2009-04-03 16:33:12 EDT
(In reply to comment #54)
> Can we build products or jnlp with athena ? 

Products: not yet, but it's in plan and waiting for someone to have time to start working on it. Patches welcome. Hell, if you want to be the person to make that work, I'll submit you for a committer election today. (We're a lot easier to join than, say, WTP or the platform.)

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=271186

JNLP: no idea what's involved in that type of packaging. If you want that, feel free to open a new bug in Dash > Common Builder and explain what's needed. 

Comment 56 Nicolas Richeton CLA 2009-04-05 06:51:17 EDT
Nick,

Sounds a good idea : there is this problem with stripeversions to investigate and we'll need product and jnlp export for Nebula, so I can start to work on it. I've got a working product nighlty build at home using plain PDE  this'll be a good place to start. 

Unfortunately, I can only work on Eclipse project during my spare time, so I can't garantee this will be right away. 
Comment 57 Jeremy Dowdall CLA 2009-04-05 12:47:30 EDT
Thanks for all your work on this Nicolas and Nick!

Nicolas, I've created the projects and updated the files as you mentioned on the mailing list, but am getting the following error when trying to build CWT:

BUILD FAILED
/home/data/users/jdowdall/nebula_builds/org.eclipse.dash.common.releng/tools/scripts/buildAllHelper.xml:1356: The following error occurred while executing this line:
/home/data/users/jdowdall/nebula_builds/org.eclipse.dash.common.releng/tools/scripts/buildAllHelper.xml:1513: The following error occurred while executing this line:
/home/data/users/jdowdall/nebula_builds/org.eclipse.dash.common.releng/tools/scripts/buildAllHelper.xml:1600: java.net.MalformedURLException: no protocol: ${dependencyURLs}

does it work for you?


I haven't tried running CDateTime yet because it depends on CWT... do I just need to include it in the dependencyURLs?
Comment 58 Nicolas Richeton CLA 2009-04-05 14:04:08 EDT
Jeremy,

rename build.properties.build.eclipse.org to build.properties and restart the build.

You can build directly datetime by including cwt in the map file.
Comment 59 Nicolas Richeton CLA 2009-04-05 14:53:13 EDT
Jeremy, 

I fixed CWT build. You where checking out with a tag on which cwt didn't exist yet. (v200811...). I switched to HEAD. 
 
I added cwt to the build schedule. You can go on with datetime.

Comment 60 Jeremy Dowdall CLA 2009-04-05 16:43:00 EDT
(In reply to comment #59)
> Jeremy, 
> 
> I fixed CWT build. You where checking out with a tag on which cwt didn't exist
> yet. (v200811...). I switched to HEAD. 
> 
> I added cwt to the build schedule. You can go on with datetime.
> 

Thanks Nicolas,

Unfortunately though, the plugins directory of the zip file is empty... I looking into it, but let me know if you find something.
Comment 61 Jeremy Dowdall CLA 2009-04-05 17:30:01 EDT
CWT is Java 1.5, does JAVA_HOME need to point to a 1.5 jdk, or do I need to specify a JAVA15_HOME, or... ?
Comment 62 Nicolas Richeton CLA 2009-04-06 01:51:34 EDT
Jeremy,
Yes, it's better to point to a 1.5 jdk. JAVA_HOME will probably work. Nick will be able to tell us if JAVA15_HOME can be used.

I fixed the missing plug-in problem : in the feature's plugin list, you should use 0.0.0 as version. You were using 0.9.0.qualifier and since qualifier is replaced during the build, it was not selecting any plugin. 

I committed the fix for you.
Comment 63 Nick Boldt CLA 2009-04-06 02:10:22 EDT
(In reply to comment #62)
> Jeremy,
> Yes, it's better to point to a 1.5 jdk. JAVA_HOME will probably work. Nick will
> be able to tell us if JAVA15_HOME can be used.

I use both JAVA_HOME and JAVA50_HOME (not JAVA15_HOME), since one is added to the path and the other is used to calculate bootclasspath (see o.e.dash.common.releng/server.properties).

You should also be setting BREEs (Bundle-Require-Execution-Environment) in your MANIFEST.MF to explicitly tell the compiler if any plugins should be compiled to less than a 5.0 VM requirement. 

Comment 64 Nick Boldt CLA 2009-04-06 02:17:05 EDT
If you want to be able to use the same script for Hudson and non-Hudson use, consider changing this:

writableBuildRoot="$HOME/nebula_builds";

to this:

if [[ -w ${WORKSPACE} ]]; then #run as Hudson in https://build.eclipse.org/hudson/
  writableBuildRoot="${WORKSPACE}/build"
else #run as normal user via commandline
  writableBuildRoot="$HOME/nebula_builds"
fi

Comment 65 Nick Boldt CLA 2009-04-06 02:19:18 EDT
Better, check if ${WORKSPACE} is defined, is a directory, and is writable:

if [[ ${WORKSPACE} ]] && [[ -d ${WORKSPACE} ]] && [[ -w ${WORKSPACE} ]]; then 
   # run as Hudson in https://build.eclipse.org/hudson/
   writableBuildRoot="${WORKSPACE}/build"
 else 
   # run as normal user via commandline
   writableBuildRoot="$HOME/nebula_builds"
fi
 
Comment 66 Jeremy Dowdall CLA 2009-04-06 05:55:07 EDT
(In reply to comment #62)
> I fixed the missing plug-in problem : in the feature's plugin list, you should
> use 0.0.0 as version. You were using 0.9.0.qualifier and since qualifier is
> replaced during the build, it was not selecting any plugin. 

makes sense, but doesn't work for me... I probably have something setup wrong in my nebula_builds directory.
I'll try it some more locally, but as long as it works for you the builds can go out.
Comment 67 Jeremy Dowdall CLA 2009-04-06 05:57:26 EDT
(In reply to comment #63)
> I use both JAVA_HOME and JAVA50_HOME (not JAVA15_HOME), since one is added to
> the path and the other is used to calculate bootclasspath (see
> o.e.dash.common.releng/server.properties).

what's the path you use to the jdk?

> You should also be setting BREEs (Bundle-Require-Execution-Environment) in your
> MANIFEST.MF to explicitly tell the compiler if any plugins should be compiled
> to less than a 5.0 VM requirement. 

this is what's in the plugin MANIFEST:
 Bundle-RequiredExecutionEnvironment: J2SE-1.5
Comment 68 Nicolas Richeton CLA 2009-04-06 07:01:02 EDT
Jeremy, 

Please attach your build log so I can take a look at it.
Comment 69 Nicolas Richeton CLA 2009-04-08 03:36:13 EDT
for those with a failing build, I've made a temporary fix for bug 271416. WritableWorkDir have been updated to remove the underscore which causes the error. Look at gallery releng plugin or simply change the work directory.

Nick, thanks for this code, I'll add it ASAP.
Comment 70 Thomas Schindl CLA 2009-04-10 09:01:00 EDT
Nicolas,

I update the Grid stuff today - how can I now start a build on build.eclipse.org. Your wiki document doesn't explain where from I get
* the dash-releng stuff
* what writableDirectoryPath is I only see writableBuildRoot which points to $HOME/nebula_builds

Tom
Comment 71 Nicolas Richeton CLA 2009-04-10 09:07:23 EDT
Tom, 

Currently, I start everything from my build.eclipse.org account, I'll share this folder with the nebula admin_group. We can ask for a public folder some where (we aren't allowed to write on /opt/public/cbi/build


You can just update your releng plugin (be sure to have the map file targeting HEAD), I'll take care of the rest. 

To build locally, you just have to get start.sh from the dash releng project
in technology cvs / org.eclipse.dash/o.e.d.commonbuilder.releng

For the build dir, you're right it's writableBuildRoot.

Comment 72 Thomas Schindl CLA 2009-04-10 09:11:15 EDT
I've updated the grid.map so I think everything is ok. If you could launch the build that be great!
Comment 73 Nick Boldt CLA 2009-04-10 12:55:50 EDT
(In reply to comment #70)
> I update the Grid stuff today - how can I now start a build on
> build.eclipse.org. Your wiki document doesn't explain where from I get
> * the dash-releng stuff

I recommend creating and publishing a .psf file for your developers. Here's a sample:

http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.commonbuilder.releng/psf/gef.psf?root=Technology_Project&view=co

From that file, you can Import > Team > Team project set file and you'll get the three projects needed to build GEF (releng.basebuilder, dash.common.releng, gef.releng); change the file to point to your project's releng and you'll be set.

The latest version of the gef.releng now includes sample build.properties for different scenarios (linux, macosx, win32, hudson on build.eclipse) so you can more easily see what needs to change for each platform configuration.

You may also want to create a Nebula-specific version of this document to help out your developers/contributors:

http://wiki.eclipse.org/Common_Build_Infrastructure/Getting_Started/Build_In_Eclipse

You may also want to create .psfs for other uses, such as shown here:

http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ve/org.eclipse.ve.releng/psf/?root=Tools_Project
(-releng is for the three releng projects; -tree is for the whole CVS tree in one folder in workspace, -projects is to check out all the projects into individual projects in workspace).

> * what writableDirectoryPath is I only see writableBuildRoot which points to
> $HOME/nebula_builds

writableDirectoryPath is not a variable I created; maybe you've done something else?

writableBuildRoot is the one I use.
Comment 74 Nicolas Richeton CLA 2009-04-10 15:38:20 EDT
Tom, 
Ok, I've made some changes to your releng plugin, it's building now. Can you check that the zip contains everything needed ? 
http://www.eclipse.org/nebula/downloads.php

Just forget writableDirectoryPath, this never existed, that's my fault.

To start a build, you only need one file : start.sh. I'll add a link to the file in view cvs.
Comment 75 Thomas Schindl CLA 2009-04-10 17:58:55 EDT
Looks perfect.
Comment 76 Thomas Schindl CLA 2009-04-10 18:19:37 EDT
Nicolas,

I added 2 new releng plugins:
* org.eclipse.nebula.widgets.pgroup.releng
* org.eclipse.nebula.widgets.pshelf.releng

Maybe you could add them to the build.

Tom
Comment 77 Nicolas Richeton CLA 2009-04-11 09:13:38 EDT
Added and scheduled both pgroup and pshelf.
Moved gallery releng
shared the build directory with nebulaadmin
Comment 78 Nick Boldt CLA 2009-04-11 16:35:07 EDT
I'm curious why you have so many .releng projects.

You could build everything into a single update site using ONE .releng project which builds an .all feature. That .all feature would simply contain the individual widgets.pgroup, widgets.pshelf features.

If you don't need SDK and Runtime zips, then set your build.steps to include buildUpdate (not buildZips) in build.properties, and you'll get a single archived update site, which you can publish to download.eclipse.org as a zip, or unpack it and publish it as a regular site.

If you *DO* need those SDKs/Runtimes, you can still produce them, but there's a few extra packaging steps required to split off per-feature zips, should you want them. For example, GEF produces an All-features-in-one-zip, but also 6 other zips for GEF SDK/RT, Draw2D SDK/RT, Zest SDK/RT. Personally, I think it's a waste of time/space because the update mechanism is way more efficient, but that's just me. :)

Comment 79 Nicolas Richeton CLA 2009-04-12 07:38:36 EDT
Nick, 
I asked the question of using one or several releng projects on comment #45 :)  The thing is : all nebula widgets are really separate projects, and will probably not have the same release planning. In the future, there will probably be widgets that leave the incubation status while other won't (some of them doesn't even build right now :) ), so I choose to have separate releng projects. But we can change that if other nebula developers want to.

Thanks for the buildUpdate tip, there are still a few things that need to be improved in our usage of cbi (packaging, error detection, send mails on error, build location, ...) I'll work on these once every project is building. 
Comment 80 Nicolas Richeton CLA 2009-04-12 10:56:46 EDT
Examples are building :)
Comment 81 Nick Boldt CLA 2009-04-14 17:06:12 EDT
If anyone wants to use Hudson on build.eclipse.org to run builds, I can now grant access. Hudson is still default-limited to callisto-dev people, but other committers can be added on a per-job basis (bug 257265, bug 270633).

So, everyone in Nebula should now be able to READ this page ...

https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly

... and once logged in using their dev.eclipse.org CVS committer id, can be granted more control, such as ability to start/stop builds, explore the workspace, configure jobs, and comment on builds (eg., to explain a breakage or configuration change).

Please list any committer ids who'd like read-write access, and I'll add you.

Comment 82 Nicolas Richeton CLA 2009-04-15 04:52:12 EDT
Nick, 

The read only access works. 
Could you add 'rnicolas', so I can work on nebula build ? 

Thanks.
Comment 83 Nick Boldt CLA 2009-04-15 12:18:01 EDT
(In reply to comment #82)
> The read only access works. 
> Could you add 'rnicolas', so I can work on nebula build ? 

Added.

Once you're happy that this works, we could look at creating a "grid" build, which is one Hudson job that delegates to multiple smaller jobs; with a single shell script (passing in parameters), you could run all the little Nebula pieces from one single job. Something to ponder for future. 
Comment 84 Nicolas Richeton CLA 2009-04-18 07:59:37 EDT
Tests are now executed during gallery build and results are 	available on the download page.

I had to update start.sh to head to be able to use buildUpdate without buildZips. That way, we get rid of SDK & runtime zip files. The nighlty build are now names -ALL- instead of -SDK-.

But, i had to put buildZips back to run the tests since SDK is required in that case :). Gallery test plug-in contains only the required files to run tests  and can be used as template. I'll update the wiki soon. 

I also updated the build script to share it between all nebula plugins and to prevent uploading the whole eclipse directory on build failure. A few changes are required in sh and property files.

I've updated both gallery and examples releng plugins. It's not necessary to update grid, pshelf and pgroup right now if you're happy with the current build. 

I changed the warning message on nebula website, since some downloads are available.

Next step is hudson. I took a quick look at it, but it seems that I cannot add new projets i even when logged. I can configure gallery though, so I'll start with it.  My current problem is that i cannot look at the configuration of other projects due to my nebula-limited authorizations. 
Nick, can I be added to another project which publishes build & test results to downloads.eclipse.org ? I promise that I  won't break anything ;-).

Comment 85 Nick Boldt CLA 2009-04-19 14:36:27 EDT
(In reply to comment #84)
> I had to update start.sh to head to be able to use buildUpdate without
> buildZips. That way, we get rid of SDK & runtime zip files. The nighlty build
> are now names -ALL- instead of -SDK-.
> But, i had to put buildZips back to run the tests since SDK is required in that
> case :). 

Yes, we're working on being able to build from the Update & Master zips instead of the SDK/ALL zip. See bug 272403. I think I have it working locally, but I need to do more testing before I publish my changes to HEAD.

Once it's done, you can omit buildZips entirely if you only care about the Update zip.

> Next step is hudson. I took a quick look at it, but it seems that I cannot add
> new projets i even when logged. I can configure gallery though, so I'll start
> with it.  My current problem is that i cannot look at the configuration of
> other projects due to my nebula-limited authorizations. 
> Nick, can I be added to another project which publishes build & test results to
> downloads.eclipse.org ? I promise that I  won't break anything ;-).

At the moment, the other Athena builds don't do the publication step, only the build. Pubishing is handled separately because the Hudson user can't write into /home/data/httpd/download.eclipse.org/.

See bug 266374 for examples of how you can set up a crontab as you@build.eclipse.org to run the publish steps (copy, fix permissions, unpack update zip). This process will ultimately be scripted so it need not be so per-person configured, but something more like `publish.sh ve N` or `publish.sh nebula I`. May even go so far as to write it as an Ant script so it's more platform portable. :)

I can give you read access to other configs in Hudson, eg., for Linux Tools, VE, or GEF, but the source for those scripts can also be found in CVS or SVN.

http://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/releng/trunk/org.eclipse.linuxtools.releng/tools/
http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.common.releng/hudson/?root=Technology_Project

I've also cached Hudson config.xml files (the storage for the job configuration) here, if you're curious how to set up build parameters and pass them to the script.

http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.dash/athena/org.eclipse.dash.commonbuilder/org.eclipse.dash.common.releng/hudson/gef-3.5.x-nightly/build.eclipse.org/config.xml?root=Technology_Project&view=markup
Comment 86 Nick Boldt CLA 2009-04-20 18:32:29 EDT
I've improved the state of the Hudson build, but there's still something missing.

Specifically, there's no build.properties in the root of the gallery.releng project:

* org.eclipse.nebula.widgets.gallery.releng/build.properties doesn't exist 

Can you create a copy of the build.properties you want to use for running in Hudson on build.eclipse?

You need to set at a minimum these properties, or something similar:

projectid=technology.nebula
zipPrefix=nebula.widgets.gallery
incubation=-incubation
version=0.5.0
buildType=N
mainFeatureToBuildID=org.eclipse.whatever.your.all.feature.is
testFeatureToBuildID=org.eclipse.whatever.your.main.test.feature.is

JAVA_HOME=/opt/public/common/ibm-java2-ppc-50
JAVA50_HOME=/opt/public/common/ibm-java2-ppc-50
dependencyURLs=http://download.eclipse.org/eclipse/downloads/drops/S-3.5M6-200903130100/eclipse-SDK-3.5M6-linux-gtk-ppc.tar.gz,http://other/dependendencies/you/require.zip

build.steps=buildUpdate,buildZips,buildTests,generateDigests,test,publish,cleanup

Comment 87 Nick Boldt CLA 2009-04-21 15:11:37 EDT
(In reply to comment #86)
>build.steps=buildUpdate,buildZips,buildTests,generateDigests,test,publish,cleanup

buildZips step is no longer required for running tests. You can use this:

build.steps=buildUpdate,buildTests,generateDigests,test,publish,cleanup

------

Do you want me to create a new Hudson job for use w/ other Nebula components? If you have a script that wraps start.sh and iterates through the different sub-builds, you could try a matrix-style build to do all the components in series.

http://ericlefevre.net/wordpress/2007/07/29/matrix-project-building-with-hudson/
Comment 88 Nick Boldt CLA 2009-05-01 01:15:15 EDT
I've got nebula.widgets.gallery building in Hudson!

--

To switch it over to use YOUR .releng project (rather than the one I created in Dash's CVS), do the following:

a) configure [1] the project to point at the other CVS location. Use this script instead of what's there now:

export BUILDTYPE="${BUILDTYPE}"
export EXTRAFLAGS="${EXTRAFLAGS}"
export writableBuildRoot="${WORKSPACE}/build"
 . ${WORKSPACE}/org.eclipse.swt.nebula/releng/org.eclipse.nebula.widgets.gallery.releng/build_hudson_build.eclipse.org.sh

Of course, you'll want to copy that script from my version of your .releng project, and update the CVS repo/path details to match your repo.

b) wipe out the workspace [2] to remove the old symlink to my .releng project

c) kick a build [3]

[1]https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/configure

[2]https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/wipeOutWorkspace

[3]https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/build

--

Hudson currently checks every 6hrs for CVS changes & will run a build if necessary; they take about 6 mins (longer if signed). Once done, you can publish the update site zip (and the other files in the build folder) to download.eclipse.org; you may also want to unpack the zip so that you can provide an update URL in your feature(s).

If you want to do signed builds, change the BUILDTYPE value to I, M, S, or R.
Comment 89 Nicolas Richeton CLA 2009-06-01 08:36:37 EDT
Nick, thank you for making the galley widget build on hudson. I'll take a look at the steps you described.

I've added CollapsibleButtons releng and feature projects. CollapsibleButtons is building and available on the download page.
Comment 90 Nicolas Richeton CLA 2009-06-01 12:18:27 EDT
Added calendarcombo and compositetable releng/feature/download
Comment 91 Eric Wuillai CLA 2009-06-18 17:04:25 EDT
I added feature and releng projects for DateChooser and FormattedText.
Thanks to add them in the build and download process.
Comment 92 Nicolas Richeton CLA 2009-06-23 06:57:04 EDT
Added DateChooser and FormattedText.
Comment 93 Nicolas Richeton CLA 2009-07-16 17:27:52 EDT
According to Denis Roy's last mail, we've made it all wrong. I'll have to move the build process from my home directory to /shared but I need to get in the right user group before that. In the meanwhile, I've updated the crontab to build only once a week (I don't think we really need more frequent builds right now).
Comment 94 Nick Boldt CLA 2009-07-16 18:47:00 EDT
(In reply to comment #93)
> move the build process from my home directory to /shared 

All the Hudson-run builds are using /opt/users/hudsonbuild/.hudson/jobs/<jobname>/ as their working directory.

Are you saying we have to move everyone? 
Comment 95 Nicolas Richeton CLA 2009-07-17 01:30:44 EDT
In his mail, Denis says : 
"- Home directories are not working directories.  The Build server's /shared array is the only suitable place to run builds."

I don't know where /opt points to, but Nebula builds currently run from a home directory. 
Comment 96 Eclipse Webmaster CLA 2009-07-17 09:24:35 EDT
To clarify Denis message: '(committers) home directories are not working directories'.  So no building in places like: ~yourcommitteridhere .

The Hudson user is located on a separate partition local to the build machine, so you should be ok.

-M.
Comment 97 Matthew Hall CLA 2009-07-29 19:57:28 EDT
Ping.  I've set up my org.eclipse.nebula.widgets.radiogroup.releng and o.e.n.w.radiogroup_feature projects in CVS.  What next?
Comment 98 Nicolas Richeton CLA 2009-07-30 07:06:36 EDT
Matthew,
I've set up the nightly build job, radioggroup is now available on nebula downloads.

Nick,
I've finaly switched the releng project used by hudson to the gallery one. Building works and I'm currently experimenting with Hudson (signing, publishing). I assume publishing to downloads.eclipse.org can be done by adding a new build step. Do you have any ready-to-use script example for publishing files and update sites. (I'm not allowed to view other project's configuration). 

Ok, so thanks to Nick the Gallery widget is building in Hudson. The only issue is that the build is triggered by any change on Nebula and not only by changes on gallery. This is because of our CVS layout since we don't have a clear separation between widgets. This will probably change with the new project organisation proposed by Tom. 

I'll ask for the creation of new jobs for the other widgets. 

I've also created a bug for a new location for the current build (see bug 285127). We will continue to use this build until the building/publishing of all widgets is working in Hudson.
Comment 99 Nick Boldt CLA 2009-07-30 14:15:34 EDT
(In reply to comment #98)
> I've finaly switched the releng project used by hudson to the gallery one.
> Building works and I'm currently experimenting with Hudson (signing,
> publishing). I assume publishing to downloads.eclipse.org can be done by adding
> a new build step. Do you have any ready-to-use script example for publishing
> files and update sites. (I'm not allowed to view other project's
> configuration). 

Ready to use script? Me? But of course!

http://wiki.eclipse.org/Common_Build_Infrastructure/Publishing

This should be run as a user who has write permission in http://download.eclipse.org/path/to/your/downloads/folder/

> Ok, so thanks to Nick the Gallery widget is building in Hudson. The only issue
> is that the build is triggered by any change on Nebula and not only by changes
> on gallery. This is because of our CVS layout since we don't have a clear
> separation between widgets. This will probably change with the new project
> organisation proposed by Tom. 

Hudson can only handle ONE cvs repo for monitoring in order to know when to build. So, if as you suggest you're planning to reorg into /cvsroot/whatever/org.eclipse.nebula/*everything under here*/, then you can have Hudson respond to a change ANYWHERE in that part of the tree.

Or, switch to SVN and you can add any series of folders you want to monitor. For JBoss Tools and Dev Studio, we monitor up to 6 different repos for source changes (some for which we don't have write permission).

> I'll ask for the creation of new jobs for the other widgets. 
> I've also created a bug for a new location for the current build (see bug
> 285127). We will continue to use this build until the building/publishing of
> all widgets is working in Hudson.

You don't need /opt/users/nebulaBuild. If you use Hudson, it's all done in hudsonbuild's home dir. No need for anything else.

There's another thing you MIGHT like to do w/ your component builds, if they're independent enough. I recently put together a parallel job for EMF Query, Transaction and Validation. This allows three components to build concurrently in response to a change in CVS. So a change to Query will fire a Transaction build (as well as a Query and a Validation build).

Alternatively, you might prefer the set up I did for MWE, Xpand and Xtext, where they will build in series and notify downstream that it's time to build if the upstream project changes. Here's the middle build - you'll notice it has an upstream and a downstream.

https://build.eclipse.org/hudson/view/Athena%20CBI/job/cbi-m2t-xpand-0.7/

In this case, you'll get a new Xtext build if and of these happen:

* Xtext sources change, or
* Xpand does a new *successful* build, or
* MWE does a new *successful* build and Xpand does a new *successful* build
Comment 100 Nicolas Richeton CLA 2009-08-04 10:32:50 EDT
- Gallery is building AND publishing with Hudson. 
- I disabled the previous Gallery build and I requested job creation for all
other widgets. We will be soon building with Hudson only... *Thanks Nick* :-) 
- Current build of other widgets is running in /opt/users/nebulaBuild. Admins
will like it.

Comment 101 Nicolas Richeton CLA 2010-03-15 06:54:03 EDT
The Nebula hudson build is now failing in the test phase, and I can't find out why. It started to fail without any changes to the releng file or the Gallery code/tests. 

The error changed from, at first:
 [exec]    [p2.dir] !ENTRY org.eclipse.equinox.p2.director.app 4 0 2010-02-25 21:06:05.080
    [exec]    [p2.dir] !MESSAGE The installable unit org.eclipse.nebula.widgets.gallery_feature.feature.group has not been found.
    [exec] 
    [exec] BUILD FAILED
    [exec] <https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/ws/build/org.eclipse.dash.common.releng/builder/tests/test.xml>:448: The following error occurred while executing this line:
    [exec] <https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/ws/build/org.eclipse.dash.common.releng/tools/scripts/buildAllHelper.xml>:1336: Java returned: 13

... to a vnc connect error : 
    [exec] _XSERVTransSocketINETCreateListener: ...SocketCreateListener() failed
    [exec] _XSERVTransMakeAllCOTSServerListeners: server already running
    [exec] 
    [exec] Fatal server error:
    [exec] Cannot establish any listening sockets - Make sure an X server isn't already running

I tried to change the build steps from test to testLocal (fix from the dash-dev mailing list). Now I get a large stack trace with class loading errors. See : https://build.eclipse.org/hudson/job/cbi-nebula.widgets.gallery-0.5.x-nightly/147/console 

I still don't know why it is failing. 
I was about to create the one-for-all nebula build, but I'm now stuck with this issue.
Comment 102 Nick Boldt CLA 2010-03-17 16:18:00 EDT
Prolly the usual X port collision issue (too many people running builds in parallel).

Solution:

a) edit your job to enable the Xvnc flag, so your job will run in its own unique display thread

b) switch your build to run the "testLocal" instead of "test" step
Comment 103 Nicolas Richeton CLA 2010-05-02 13:08:29 EDT
Nick, 
The xvnc error is fixed (or gone), but I'm still unable to fix the build. 

I got this error unsing testLocal 
 [p2.dir] !MESSAGE Unable to instantiate authenticator null
   [p2.dir] !STACK 1
   [p2.dir] org.eclipse.core.runtime.CoreException: Plug-in org.eclipse.ui.net was unable to load class org.eclipse.ui.internal.net.auth.NetAuthenticator.
and the buld ends up to   
   [p2.dir] !MESSAGE The installable unit org.eclipse.nebula.widgets.gallery_feature.feature.group has not been found.

With test: 
I only get the last line, without any stacktrace. 

See https://build.eclipse.org/hudson/view/Nebula/job/cbi-nebula.widgets.gallery-0.5.x-nightly/164/console

Any idea on this ?
Comment 104 Wim Jongman CLA 2010-07-30 16:57:39 EDT
Hi, I've set up my org.eclipse.nebula.widgets.oscilloscope.releng and o.e.n.w.oscilloscope_feature projects in CVS.
Comment 105 Wim Jongman CLA 2010-08-15 18:54:53 EDT
ping
Comment 106 Wim Jongman CLA 2012-01-18 18:55:03 EST
This bug has been resolved. Many thanks to all.