Bug 412933 - Create a new (up to date) shared instance and migrate jobs of the outdated one
Summary: Create a new (up to date) shared instance and migrate jobs of the outdated one
Status: CLOSED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: CI-Jenkins (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Mikaël Barbero CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 476312 476325 476341 422507 474423 476306 476307 476308 476309 476310 476313 476317 476318 476321 476322 476323 476328 476329 476330 476331 476338 476342 476343 476344 476347 476353 476354 476355 476356 476359 476360 476362 476428 476439 476440 476445 476450 476453
Blocks: 446251
  Show dependency tree
 
Reported: 2013-07-15 00:11 EDT by David Williams CLA
Modified: 2015-12-23 06:37 EST (History)
7 users (show)

See Also:


Attachments
too many config.xml files for wtp-javaee-config-0.1.x (21.83 KB, text/plain)
2013-10-24 11:01 EDT, David Williams CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2013-07-15 00:11:46 EDT
I'm thinking we should, and thinking this would be a good time to. 

Not sure if its been running in Sandbox? If there's reasons I could not find an existing bug to move? Have issues been found? 

But, I can't help bug think some bug fixes would be good, it would be good to support the Eclipse Hudson Project ... and I think there is a new feature in 3.0 "multi-configuration jobs" that would make may own use of it easier. 

(Naturally, feel free to ask/inform on cross-project list if anyone "is about to release" but I know of course "yearly" release is done so as far as I know it is a good time).
Comment 1 David Williams CLA 2013-07-29 19:21:18 EDT
Another "funny thing" about current installation, is that the job configuration page still executes some script from "jenkins-ci.org" ... don't know what, didn't look, just got a warning about it from a recent 'add-on' I installed ... but that seems kind of odd. 

I'd prefer when I accessed "eclipse pages" that they only needed scripts from eclipse (or ... the other few more common cases where you use google.com, or something more familiar).
Comment 2 Denis Roy CLA 2013-07-30 13:38:13 EDT
(In reply to comment #1)
> Another "funny thing" about current installation, is that the job
> configuration page still executes some script from "jenkins-ci.org" ...

I'm not seeing this.  Can you provide details as to which job you were editing?  Perhaps there's a Jenkins plugin, such as Sonar, which is doing this.
Comment 3 David Williams CLA 2013-07-30 13:49:39 EDT
(In reply to comment #2)
> (In reply to comment #1)
> > Another "funny thing" about current installation, is that the job
> > configuration page still executes some script from "jenkins-ci.org" ...
> 
> I'm not seeing this.  Can you provide details as to which job you were
> editing?  Perhaps there's a Jenkins plugin, such as Sonar, which is doing
> this.

I'm seeing if I just go to the "Eclipse and Equinox" tab. 
Or, select one of our running jobs: 
the https://hudson.eclipse.org/hudson/job/ep4-unit-lin64/
Or, if I press 'configure' (for that job ... only one I've tested). 

[And, assume you know this is from 'firefox' with the 'NoScript' add-on added.]
Comment 4 Denis Roy CLA 2013-07-30 14:04:26 EDT
There's a small JS at the bottom of that page:

<script>Behaviour.addLoadEvent(function() {
            downloadService.download(
                "default",
                "http://updates.jenkins-ci.org/download/plugins/",
                {version:"2.2.1"},
                "/hudson/updateCenter/byId/default/postBack",
                null);
        });</script>

My guess is that comes from a specific plugin.
Comment 5 David Williams CLA 2013-07-30 14:32:13 EDT
Yeah, it seems in either case, the current 2.21 or the 3.0.1-b2 version takes you to a "hudson-ci.org" page, which is an IP address owned by Oracle, and the page itself has a big Oracle logo on it. 

I could have sworn Hudson was now a vendor neutral Eclipse project. What a poor memory I have!
Comment 6 David Williams CLA 2013-09-21 04:22:08 EDT
Changed to "3.1.0", since that is now the latest. 

Seems some things about how to run it have changed, but I did eventually get it running on my test mahince. It may caused trouble that I was installing "into" the old location and that probably didn't help much. 

I just use 'java -jar hudson.war' (plus some parameters) ... in other words, nothing fancy. 

In about two weeks, we will be a a relatively "quiet time" again (this is after Kepler SR1 is out, and wait another week for Luna M2 to come out, and that might be a good time to do it ... week of October 7th. 

It has https support ... if that's enough to tempt you :)
Comment 7 David Williams CLA 2013-09-21 04:24:03 EDT
Oh, thought I'm mention too, that many of the "plugins" we have installed, on hudson.eclipse.org appear pretty jar out of date ... and "updates" hasn't been checked in a year and a half!
Comment 8 David Williams CLA 2013-09-24 12:52:51 EDT
Adding Thanh to CC since I see he's considering this for some HIPP instances. 

I wanted to mention my experience installing this on my home test server.

I tried it first, and had lots of trouble, and THEN read, somewhere, they recommend installing "into a different location" since some "data" is not migrated correctly, or something. As soon as I did that, it worked fine. 

I was able to copy my "jobs" (config.xml's) over to the new location and all seemed to work fine (I did get an initial message that some data in them was not recognized, and while they will ignore such data, it did offer an option to remove it automatically (which I did). 

Not sure how "users" could be "copied over" ... since I'm the only user on my system :) I just "registered" again ... but know your server handles that differently. 

Mostly just wanted to make sure you knew to "do in a different location". 
I just use the simple 
java -jar hudson.war
type of invocation, but, principle would be the same if you have your own "container" to run it in. (that is, apparently some old "classes" and new "classes" conflict with each other). In my case, it would run once ok, but as soon as I closed that web page, could not reconnect to it.
Comment 9 Thanh Ha CLA 2013-09-24 13:08:37 EDT
(In reply to David Williams from comment #8)
> Not sure how "users" could be "copied over" ... since I'm the only user on
> my system :) I just "registered" again ... but know your server handles that
> differently. 
> 

We use LDAP so not much to do here, and the Job configs store the permissions so as long as the person logging in is using the same credential type Hudson will know the correct permission. Migration may be an issue however if we decided to change the credential type "uid", vs "email".
Comment 10 Thanh Ha CLA 2013-10-16 15:20:13 EDT
One thing I discovered that might prevent us from upgrading is that apparently the coverity scan plugin does not load in Hudson 3.0.1 and instead throws a NPE. I'm not sure if this also affects 3.1.0 but I would imagine so.

Not sure how many plugins depend on the coverity scan plugin but it might be a blocker.

This is in reference to bug 416823 comment 14.


I also opened a bug to the developer but have not yet received a reply [1]. I'm not sure if we have a contact at coverity that we can ask to look into the problem?


[1] https://github.com/kaizer113/scan-plugin/issues/1
Comment 11 David Williams CLA 2013-10-24 10:58:16 EDT
(In reply to Thanh Ha from comment #10)
> One thing I discovered that might prevent us from upgrading is that
> apparently the coverity scan plugin does not load in Hudson 3.0.1 and
> instead throws a NPE. I'm not sure if this also affects 3.1.0 but I would
> imagine so.
> 
> Not sure how many plugins depend on the coverity scan plugin but it might be
> a blocker.
> 
> This is in reference to bug 416823 comment 14.
> 
> 
> I also opened a bug to the developer but have not yet received a reply [1].
> I'm not sure if we have a contact at coverity that we can ask to look into
> the problem?
> 
> 
> [1] https://github.com/kaizer113/scan-plugin/issues/1

I think the one installed in shared instance is not "ready for prime time", from its web-page, so maybe it can be uninstalled on shared instance, and people can experiment in their HIPP instances if they want. Their webpage says

= = = =
Hudson plugin to enable Coverity SCAN analysis for open source projects, which are part of Eclipse Foundation. This project is currently in Alpha stage and we're working to enable it for Eclipse Foundation projects. This project is aimed at the Open Source community only.

Current enterprise customer should use the Jenkins plugin https://github.com/jenkinsci/coverity-plugin

For any inquiry please contact scan-admin@coverity.com or visit http://scan.coverity.com
= = = =

Also, from the jobs I have permission to view (config.xml) seems no one is using it on main shared instance of Hudson

       [10:39:12] david_williams@build:/opt/public/jobs

$ grep -r --include config.xml "coverity" * 1>~/temp/jobs.txt 2>~/temp/joberrs.txt

I got zero hits only a few "real" jobs I could see, though I'll attach joberrs.txt, since appears something was mis-copied for wtp-javaee-config-0.1.x

So, in otherwords, I wouldn't take 'coverity' as blocking an upgrade.
Comment 12 David Williams CLA 2013-10-24 11:01:35 EDT
Created attachment 236847 [details]
too many config.xml files for wtp-javaee-config-0.1.x

I don't know what wtp-javaee-config-0.1.x is, but from the large number of config.xml files it (apparently) has ... based on "permission denied", seems that something was mis-copied/configured? 

But ... could me my mis-interpretation of grep results?
Comment 13 Thanh Ha CLA 2013-10-28 09:59:26 EDT
Looks like the Sonar plugin's also not compatible with Hudson 3.x but in this case there is known work to port it [1].

[1] http://dev.eclipse.org/mhonarc/lists/hudson-dev/msg00443.html
Comment 14 Thanh Ha CLA 2013-10-28 10:01:25 EDT
(In reply to Thanh Ha from comment #13)
> Looks like the Sonar plugin's also not compatible with Hudson 3.x but in
> this case there is known work to port it [1].
> 
> [1] http://dev.eclipse.org/mhonarc/lists/hudson-dev/msg00443.html

That's Sonar 2.1 plugin specifically. We do have the older version 2.0.1 working on existing HIPP instances today. (Running Hudson 3.0.1).
Comment 15 David Williams CLA 2014-08-21 17:52:37 EDT
I'd still like to see this updated, preferably to latest level (3.2.0). 
I use 3.1.0 on my test machines, and the "performance machine" is at 3.1.2. 

And when I see a difference in behavior or UI, I always wonder if it is due to the Hudson level. 

(Though, in the most recent example was not ... I'll be opening a separate bug to install Ant 1.9.2).
Comment 16 Matthias Sohn CLA 2014-11-12 04:04:52 EST
+1 to update
according to Prakash Hudson 2.x is out of maintenance
also it's blocking bug 446251
Comment 17 Andreas Sewe CLA 2015-02-04 05:13:46 EST
FWIW, we (Code Recommenders) would like to see an upgrade to a Hudson version that has Bug 390862 fixed; we currently have several old jobs that we would like to delete but cannot due to this bug.

As far as I can tell, the final commit on Bug 390862 was included in 3.1.1 (not 3.1.0).
Comment 18 Mikaël Barbero CLA 2015-02-04 09:22:45 EST
Hi, 

I am taking over the bug 422507 (added as a dependency). This will provide you with a mean to upgrade the version of your HIPP. I am just starting to look at what Thanh did so I am not already able to provide an ETA.
Comment 19 Marc-André Laperle CLA 2015-02-27 15:41:52 EST
(In reply to Matthias Sohn from comment #16)
> +1 to update
> according to Prakash Hudson 2.x is out of maintenance
> also it's blocking bug 446251

We would the update as well for Trace Compass builds (shared instance) on Mac and Windows.
Comment 20 Mikaël Barbero CLA 2015-03-02 04:03:26 EST
I am creating a new shared instance of Hudson (with v3.2.2) because the upgrade from our old shared instance can not be done safely (reason: too old...). Once it is setup and properly running, we can think of migrating jobs from the shared instance to this fresh new one. 

I will keep you posted on this bug (and cross-project) so that people who want to migrate to the new shared instance can do it on a "on-demand" basis.
Comment 21 David Williams CLA 2015-03-05 08:57:42 EST
Just making title more accurate.
Comment 22 Mikaël Barbero CLA 2015-06-26 07:07:43 EDT
The new hudson shared instance is up and running https://hudson.eclipse.org/shared/. I tried to configure it exactly the same way it was on https://hudson.eclipse.org/hudson to make the jobs migration easier. I also re-recreated the slaves as before.

Any volunteer to try to migrate to this new shiny instance?
Comment 23 Marc-André Laperle CLA 2015-07-02 08:57:19 EDT
(In reply to Mikael Barbero from comment #22)
> The new hudson shared instance is up and running
> https://hudson.eclipse.org/shared/. I tried to configure it exactly the same
> way it was on https://hudson.eclipse.org/hudson to make the jobs migration
> easier. I also re-recreated the slaves as before.
> 
> Any volunteer to try to migrate to this new shiny instance?

Can we try to migrate the tracecompass-windows-nightly job? It hasn't worked so far on the shared instance and it would be nice to make it work.
Comment 24 Mikaël Barbero CLA 2015-07-03 04:53:01 EDT
I don't manage to install another slave on the same windows box. As long as we have only a single windows box, I will have to stop the slave from the old shared instance before migrating all the jobs to the new one. 

So, I plan to migrate all these jobs at once (from https://hudson.eclipse.org/hudson/computer/windows7tests/):

checkConfigWin
EclipseSCADA-Installer-Windows
ep-collectResults
ep-unit-win32
ep44M-unit-win32
ep45I-unit-win32
ep45M-unit-win32
ep45N-unit-win32
ep45Y-unit-win32
ep46I-unit-win32
ep46N-unit-win32
jgit-windows-test
tracecompass-windows-nightly

To avoid to disturb projects, my plan is the following:
- copy these jobs to the new shared instance
- disconnect the windows7 slave from the old instance
- connect the windows7 slave to the new instance
- try to have a few successful builds of jobs above
- disconnect windows7 slave from the new instance and reconnect it to the old one
- keep you posted and elaborate a plan for real migration.

When do you think it would be a good timeframe to do that? I would suggest sometime next week.
Comment 25 David Williams CLA 2015-07-03 12:34:49 EDT
(In reply to Mikael Barbero from comment #24)

Next week sounds fine to me. 

Also, you will need the "ep-unit" job copied over, as it's the "master" definition, that all other *unit* jobs cascade from.
Comment 26 Mikaël Barbero CLA 2015-07-07 03:52:13 EDT
Good news everyone. I managed to start the new slave on the windows box! And I managed to run the "jgit-windows-test" job successfully (it fails, but because of test failure, not because of an ArrayIndexOutOfBoundsException in Hudson ;)). 

I will announce on cross-project that I will disconnect the windows7 slave from the current shared instance for the rest of the (european) day and follow the plan of comment #24.
Comment 27 Mikaël Barbero CLA 2015-07-07 09:20:51 EDT
(In reply to Marc-Andre Laperle from comment #23)
> Can we try to migrate the tracecompass-windows-nightly job? It hasn't worked
> so far on the shared instance and it would be nice to make it work.

The job has been migrated and works properly:

https://hudson.eclipse.org/shared/job/tracecompass-windows-nightly/lastBuild/console

It is failing because of test failures.

Any Trace-Compass committer should be able to log on this new shared instance (with their email address) and configure this job. Feel free to update it.
Comment 28 Marc-André Laperle CLA 2015-07-07 10:57:33 EDT
(In reply to Mikael Barbero from comment #27)
> (In reply to Marc-Andre Laperle from comment #23)
> > Can we try to migrate the tracecompass-windows-nightly job? It hasn't worked
> > so far on the shared instance and it would be nice to make it work.
> 
> The job has been migrated and works properly:

Great thank you!

> It is failing because of test failures.

It's proof that we needed this job :) I'll look at fixing the test failures.

> Any Trace-Compass committer should be able to log on this new shared
> instance (with their email address) and configure this job. Feel free to
> update it.

Thanks, I verified that I can log in and configure the job. I will keep an eye on this bug.
Comment 29 Mikaël Barbero CLA 2015-07-08 09:06:48 EDT
I updated the title to make it more accurate.

So everything went more or less fine for ep46N-unit-win32. The test ran properly but the jobs froze at the end:

Warning: c:\shared_hudson\workspace\ep46N-unit-win32\workarea\N20150706-2000\eclipse-testing\results\chkpii does not exist.
     [exec]      [java] Java Result: -1


David, as long as the ep*-unit-win32 are the only jobs effectively running on the outdated shared instance, do you think it would be reasonable to de-activate all the ep* jobs on it and run it exclusively now from the new shared instance? This way, I can permanently disconnect the windows slave from the outdated shared instance and keep it on the new one. 

Other jobs can be migrated step by step after that.
Comment 30 David Williams CLA 2015-08-06 10:12:01 EDT
(In reply to Mikael Barbero from comment #29)
> I updated the title to make it more accurate.
> 
> So everything went more or less fine for ep46N-unit-win32. The test ran
> properly but the jobs froze at the end:
> 
> Warning:
> c:\shared_hudson\workspace\ep46N-unit-win32\workarea\N20150706-2000\eclipse-
> testing\results\chkpii does not exist.
>      [exec]      [java] Java Result: -1
> 

What URL could I look at, to see results? The "warning" you quote would not be related to anything "freezing at end". 

> 
> David, as long as the ep*-unit-win32 are the only jobs effectively running
> on the outdated shared instance, do you think it would be reasonable to
> de-activate all the ep* jobs on it and run it exclusively now from the new
> shared instance? This way, I can permanently disconnect the windows slave
> from the outdated shared instance and keep it on the new one. 
> 
> Other jobs can be migrated step by step after that.

Yes, you plan sounds fine. If there is a problem with the windows jobs, we can address them on new instance.
Comment 31 David Williams CLA 2015-08-06 10:16:45 EDT
(In reply to David Williams from comment #30)
> (In reply to Mikael Barbero from comment #29)
> > I updated the title to make it more accurate.
> > 
> > So everything went more or less fine for ep46N-unit-win32. The test ran
> > properly but the jobs froze at the end:
> > 
> > Warning:
> > c:\shared_hudson\workspace\ep46N-unit-win32\workarea\N20150706-2000\eclipse-
> > testing\results\chkpii does not exist.
> >      [exec]      [java] Java Result: -1
> > 
> 
> What URL could I look at, to see results? The "warning" you quote would not
> be related to anything "freezing at end". 
> 
> > 
> > David, as long as the ep*-unit-win32 are the only jobs effectively running
> > on the outdated shared instance, do you think it would be reasonable to
> > de-activate all the ep* jobs on it and run it exclusively now from the new
> > shared instance? This way, I can permanently disconnect the windows slave
> > from the outdated shared instance and keep it on the new one. 
> > 
> > Other jobs can be migrated step by step after that.
> 
> Yes, you plan sounds fine. If there is a problem with the windows jobs, we
> can address them on new instance.


Ok, I found URL ... https://hudson.eclipse.org/shared/

I assume that is what it will be, moving forward? 

Will you migrate "views" also?
Comment 32 David Williams CLA 2015-08-06 10:20:01 EDT
(In reply to David Williams from comment #31)
> (In reply to David Williams from comment #30)
> > (In reply to Mikael Barbero from comment #29)
> > > I updated the title to make it more accurate.
> > > 
> > > So everything went more or less fine for ep46N-unit-win32. The test ran
> > > properly but the jobs froze at the end:
> > > 
> > > Warning:
> > > c:\shared_hudson\workspace\ep46N-unit-win32\workarea\N20150706-2000\eclipse-
> > > testing\results\chkpii does not exist.
> > >      [exec]      [java] Java Result: -1
> > > 
> > 
> > What URL could I look at, to see results? The "warning" you quote would not
> > be related to anything "freezing at end". 
> > 
> > > 
> > > David, as long as the ep*-unit-win32 are the only jobs effectively running
> > > on the outdated shared instance, do you think it would be reasonable to
> > > de-activate all the ep* jobs on it and run it exclusively now from the new
> > > shared instance? This way, I can permanently disconnect the windows slave
> > > from the outdated shared instance and keep it on the new one. 
> > > 
> > > Other jobs can be migrated step by step after that.
> > 
> > Yes, you plan sounds fine. If there is a problem with the windows jobs, we
> > can address them on new instance.
> 
> 
> Ok, I found URL ... https://hudson.eclipse.org/shared/
> 
> I assume that is what it will be, moving forward? 
> 
> Will you migrate "views" also?

Oh, and we are running some tests today, on old one ... so Friday or Monday would be a good time (from my point of view) if there is ever a "jumping off" point of no return. 

Thanks,
Comment 33 Mikaël Barbero CLA 2015-08-06 10:28:26 EDT
(In reply to David Williams from comment #31)
> Ok, I found URL ... https://hudson.eclipse.org/shared/
> 
> I assume that is what it will be, moving forward? 

Yes, it will be the new URL

> Will you migrate "views" also?

Yes.

(In reply to David Williams from comment #32)
> Oh, and we are running some tests today, on old one ... so Friday or Monday
> would be a good time (from my point of view) if there is ever a "jumping
> off" point of no return. 
> 
> Thanks,

I will copy the jobs tomorrow to the new instance and deactivate them on the old one. There won't be any "point of no return" until we remove the old jobs from the old instance and we won't do it before the dust settles down ;)
Comment 34 Mikaël Barbero CLA 2015-08-07 08:57:31 EDT
I just migrated all jobs to the new instance and deactivated the same jobs on the old shared instance. Could you start builds and let me know how it goes? Thanks!
Comment 35 Marc-André Laperle CLA 2015-08-07 11:01:42 EDT
(In reply to Mikael Barbero from comment #34)
> I just migrated all jobs to the new instance and deactivated the same jobs
> on the old shared instance. Could you start builds and let me know how it
> goes? Thanks!

You migrated *all* jobs? I don't see tracecompass-mac-nightly for example. Maybe I'm misunderstanding.
Comment 36 Mikaël Barbero CLA 2015-08-07 11:06:32 EDT
Sorry for the misunderstanding. I migrated the all the "ep*" jobs. Once they prove to be successful, I will redo a migration of the following jobs:

* jgit-windows-test
* tracecompass-windows-nightly
* EclipseSCADA-Installer-Windows

and deactivate the windows slave on the old instance. After that, I will start the migration of all other jobs, one project at a time. I will contact them individually. HTH
Comment 37 David Williams CLA 2015-08-08 02:46:26 EDT
Can you install "Rebuilder" plugin? (rebuilds job, using same parameters, when looking at the "parameters view" of a job that has ran. 

I find this very handy, and actually thought it came "built in" with the newer Hudson's, but see it does not, so guess at some point I installed it in my own test machines and HIPP instances. 

(It is in "standard list" of plugins to install).
Comment 38 David Williams CLA 2015-08-08 02:52:56 EDT
One relatively minor issue I've encountered is that for "project based security", if I previously had my committer ID in the matrix, it must now be my committer email address. It would be a nice touch if you could "convert" those automatically ... if remaining jobs to migrate use project based security matrix. 


Another relatively minor issue, there is one case where we want Hudson to write a small file in a directory under /shared/eclipse. 
Now the "hudson user" (the userid running the build) is no longer "hudsonBuild" but is "genie,shared". And, found I had to add that "user" to the ACL list of the directly I want to write to.
Comment 39 David Williams CLA 2015-08-08 03:06:10 EDT
(In reply to David Williams from comment #38)

> 
> Another relatively minor issue, there is one case where we want Hudson to
> write a small file in a directory under /shared/eclipse. 
> Now the "hudson user" (the userid running the build) is no longer
> "hudsonBuild" but is "genie,shared". And, found I had to add that "user" to
> the ACL list of the directly I want to write to.

A small "expansion" of this point. It seems that the normal slaves still use "hudsonBuild" ... and it is only "master" that uses "shared.genie" as the "user".
Comment 40 David Williams CLA 2015-08-08 12:55:59 EDT
(In reply to David Williams from comment #37)
> Can you install "Rebuilder" plugin? (rebuilds job, using same parameters,
> when looking at the "parameters view" of a job that has ran. 
> 
> I find this very handy, and actually thought it came "built in" with the
> newer Hudson's, but see it does not, so guess at some point I installed it
> in my own test machines and HIPP instances. 
> 
> (It is in "standard list" of plugins to install).

The shared instance allowed ME to install that plugin, and even "restart" Hudson (from the install plugin page). 

Not sure if this is "just for me" (as part of the 24 hour support team :) 
or, if the security of the shared Hudson instance still needs to be tightened up? 

= = = = 
off topic: the "rebuild" action only shows up on parameterized jobs that are ran after the plugin is installed, not on old ones that already ran, before the plugin was installed.
Comment 41 David Williams CLA 2015-08-09 01:12:12 EDT
Our "platform jobs" are successfully migrated.
Comment 42 Mikaël Barbero CLA 2015-08-10 03:54:34 EDT
(In reply to David Williams from comment #38)
> One relatively minor issue I've encountered is that for "project based
> security", if I previously had my committer ID in the matrix, it must now be
> my committer email address. It would be a nice touch if you could "convert"
> those automatically ... if remaining jobs to migrate use project based
> security matrix. 

I can run scripts from the Hudson console script to change from id to email for committers access. I have to remember to do it for every jobs I will migrate. Thanks for the reminder.

> Another relatively minor issue, there is one case where we want Hudson to
> write a small file in a directory under /shared/eclipse. 
> Now the "hudson user" (the userid running the build) is no longer
> "hudsonBuild" but is "genie,shared". And, found I had to add that "user" to
> the ACL list of the directly I want to write to.

I overlooked at this issue. However, I don't think it will be a problem for other jobs as I don't think many of them write to somewhere in /shared. Sorry for the troubles. 

(In reply to David Williams from comment #39)
> A small "expansion" of this point. It seems that the normal slaves still use
> "hudsonBuild" ... and it is only "master" that uses "shared.genie" as the
> "user".

Yes, the slaves are the same as on the "old" shared instance, so the user did not changed.

(In reply to David Williams from comment #40)
> The shared instance allowed ME to install that plugin, and even "restart"
> Hudson (from the install plugin page). 
>
> Not sure if this is "just for me" (as part of the 24 hour support team :) 
> or, if the security of the shared Hudson instance still needs to be
> tightened up? 

I reproduced the global permissions of the old shared instance on the new one. You were already entitled with the full configuration permissions on it, so you also have it on the new one. If you prefer, I can remove it and only let webmasters do the changes. WDYT?
Comment 43 David Williams CLA 2015-08-10 12:30:59 EDT
(In reply to Mikael Barbero from comment #42)

> I reproduced the global permissions of the old shared instance on the new
> one. You were already entitled with the full configuration permissions on
> it, so you also have it on the new one. If you prefer, I can remove it and
> only let webmasters do the changes. WDYT?

No need to change. As long as it's just me. :) I mostly mentioned it in case things were left "wide open" or something. Thanks for checking and clarifying.
Comment 44 David Williams CLA 2015-08-10 12:37:38 EDT
(In reply to David Williams from comment #41)
> Our "platform jobs" are successfully migrated.

One thing that came up after 3 or so jobs ran is that the Windows machine "ran out of space". I looked at workspace and saw our workspace tmp directory had much more than it usually did, which led me to check the "clean workspace before build" flag. Normally, we always clean workspace, before a test run, but it was unchecked. 

This may be a problem with original definitions, or Hudson itself, since I have always found that flag, to "clean workspace before build" did not seem to be "inherited" quite right when using cascading jobs. That is, I've always found it difficult to get it to "stick". 

I've fixed all our current jobs, on shared instance, and do not think the migration process needs to be changed, I only mention it here in case it helps others who are migrating. You may want to double check your "clean workspace" flag, especially if using cascading jobs.
Comment 45 Mikaël Barbero CLA 2015-08-11 10:01:16 EDT
Marc-André, Matthias, 

are you ok with deactivating the jobs 

jgit-windows-test and
tracecompass-windows-nightly

on the old shared instance (https://hudson.eclipse.org/hudson) to only run them on the new shared instance (https://hudson.eclipse.org/shared)
Comment 46 Marc-André Laperle CLA 2015-08-11 10:36:22 EDT
(In reply to Mikael Barbero from comment #45)
> Marc-André, Matthias, 
> 
> are you ok with deactivating the jobs 
> 
> jgit-windows-test and
> tracecompass-windows-nightly
> 
> on the old shared instance (https://hudson.eclipse.org/hudson) to only run
> them on the new shared instance (https://hudson.eclipse.org/shared)

Yes. Thank you for all this.

BTW, this uncovered a bug in our test suite so it's already useful to us!
Comment 47 Mikaël Barbero CLA 2015-08-11 11:04:18 EDT
(In reply to Marc-Andre Laperle from comment #46)
> (In reply to Mikael Barbero from comment #45)
> > Marc-André, Matthias, 
> > 
> > are you ok with deactivating the jobs 
> > 
> > jgit-windows-test and
> > tracecompass-windows-nightly
> > 
> > on the old shared instance (https://hudson.eclipse.org/hudson) to only run
> > them on the new shared instance (https://hudson.eclipse.org/shared)
> 
> Yes. Thank you for all this.

Done. You're welcome ;)

Next step is migrating the mac specific jobs. Platforms one are already migrated, and the last one is tracecompass-mac-nightly. Is it ok to migrate it also to the new instance?
Comment 48 Marc-André Laperle CLA 2015-08-11 11:08:18 EDT
(In reply to Mikael Barbero from comment #47)
> Next step is migrating the mac specific jobs. Platforms one are already
> migrated, and the last one is tracecompass-mac-nightly. Is it ok to migrate
> it also to the new instance?

Yes. We never got it to work on the old shared instance so feel free to do so whenever.
Comment 49 David Williams CLA 2015-08-11 21:36:35 EDT
(In reply to Mikael Barbero from comment #42)
> 
> > Another relatively minor issue, there is one case where we want Hudson to
> > write a small file in a directory under /shared/eclipse. 
> > Now the "hudson user" (the userid running the build) is no longer
> > "hudsonBuild" but is "genie,shared". And, found I had to add that "user" to
> > the ACL list of the directly I want to write to.
> 
> I overlooked at this issue. However, I don't think it will be a problem for
> other jobs as I don't think many of them write to somewhere in /shared.
> Sorry for the troubles. 
> 
> (In reply to David Williams from comment #39)
> > A small "expansion" of this point. It seems that the normal slaves still use
> > "hudsonBuild" ... and it is only "master" that uses "shared.genie" as the
> > "user".
> 
> Yes, the slaves are the same as on the "old" shared instance, so the user
> did not changed.
> 

I find it odd that as of this evening, genie.shared is no longer on the ACL of the directory where I had it set. And, therefore the "end of the job" failed. 

Is there some cronjob that "fixes" ACLs or something?
Comment 50 Mikaël Barbero CLA 2015-08-12 03:26:14 EDT
(In reply to David Williams from comment #49)
> I find it odd that as of this evening, genie.shared is no longer on the ACL
> of the directory where I had it set. And, therefore the "end of the job"
> failed. 
> 
> Is there some cronjob that "fixes" ACLs or something?

Weird. The idea of a cronjob seems to be a good bet. Unfortunately, I can't find any definition about it. We will have to wait for Denis or Matt to come back of vacations (next week). 

In the meantime, I could add genie.shared to the eclipse.platform.releng group. As long as the shared instance is mainly used by you, it would not allow be a big breach. We will remove the user from the group as soon as the ACL issue is fixed. WDYT?
Comment 51 David Williams CLA 2015-08-12 10:24:48 EDT
(In reply to Mikael Barbero from comment #50)
> (In reply to David Williams from comment #49)
> > I find it odd that as of this evening, genie.shared is no longer on the ACL
> > of the directory where I had it set. And, therefore the "end of the job"
> > failed. 
> > 
> > Is there some cronjob that "fixes" ACLs or something?
> 
> Weird. The idea of a cronjob seems to be a good bet. Unfortunately, I can't
> find any definition about it. We will have to wait for Denis or Matt to come
> back of vacations (next week). 
> 
> In the meantime, I could add genie.shared to the eclipse.platform.releng
> group. As long as the shared instance is mainly used by you, it would not
> allow be a big breach. We will remove the user from the group as soon as the
> ACL issue is fixed. WDYT?

Thanks for the offer. And, I do think needs to be tracked down, eventually. But, for now I've restricted the job that writes to /shared to run on "fastlane". 

fastlane runs with user 'hudsonBuild' and I did have the job configured to run in on master||fastlane. My only "need" is that this job should run with a "high" priority (and, is itself very fast running) ... so, as long as "fastlane" does not get "blocked" by long running jobs, that should work for my needs, and is probably a better overall solution. I think I read somewhere best to avoid having jobs run on 'master'? 

I have opened bug 474809 to track the disappearing genie.shared as a minor issue, to investigate eventually, but, should not be an issue for me, as long as "fastlane" works. 

Thanks again.
Comment 52 Mikaël Barbero CLA 2015-09-01 04:21:02 EDT
(In reply to Marc-Andre Laperle from comment #48)
> (In reply to Mikael Barbero from comment #47)
> > Next step is migrating the mac specific jobs. Platforms one are already
> > migrated, and the last one is tracecompass-mac-nightly. Is it ok to migrate
> > it also to the new instance?
> 
> Yes. We never got it to work on the old shared instance so feel free to do
> so whenever.

Done. See https://hudson.eclipse.org/shared/job/tracecompass-mac-nightly which is currently running farther than any previous one ;)

I deactivated the job on the old shared instance. Please give me your +1 when you're ready to remove it (and for the windows job too).

Thanks
Comment 53 Mikaël Barbero CLA 2015-09-01 04:23:37 EDT
(In reply to David Williams from comment #51)
> Thanks for the offer. And, I do think needs to be tracked down, eventually.
> But, for now I've restricted the job that writes to /shared to run on
> "fastlane". 
> 
> fastlane runs with user 'hudsonBuild' and I did have the job configured to
> run in on master||fastlane. My only "need" is that this job should run with
> a "high" priority (and, is itself very fast running) ... so, as long as
> "fastlane" does not get "blocked" by long running jobs, that should work for
> my needs, and is probably a better overall solution. I think I read
> somewhere best to avoid having jobs run on 'master'? 
> 
> I have opened bug 474809 to track the disappearing genie.shared as a minor
> issue, to investigate eventually, but, should not be an issue for me, as
> long as "fastlane" works. 
> 
> Thanks again.

David, does the dust settled regarding the EP* jobs? Can I remove the jobs on the old instance safely? Thanks.
Comment 54 David Williams CLA 2015-09-01 09:26:06 EDT
(In reply to Mikael Barbero from comment #53)
> (In reply to David Williams from comment #51)
> > Thanks for the offer. And, I do think needs to be tracked down, eventually.
> > But, for now I've restricted the job that writes to /shared to run on
> > "fastlane". 
> > 
> > fastlane runs with user 'hudsonBuild' and I did have the job configured to
> > run in on master||fastlane. My only "need" is that this job should run with
> > a "high" priority (and, is itself very fast running) ... so, as long as
> > "fastlane" does not get "blocked" by long running jobs, that should work for
> > my needs, and is probably a better overall solution. I think I read
> > somewhere best to avoid having jobs run on 'master'? 
> > 
> > I have opened bug 474809 to track the disappearing genie.shared as a minor
> > issue, to investigate eventually, but, should not be an issue for me, as
> > long as "fastlane" works. 
> > 
> > Thanks again.
> 
> David, does the dust settled regarding the EP* jobs? Can I remove the jobs
> on the old instance safely? Thanks.

 Yes. And that can be all those in the "Eclipse and Equinox" View (a few do not start with 'ep*", but you can/should remove those too). 

Thank you.
Comment 55 Mikaël Barbero CLA 2015-09-01 09:38:24 EDT
(In reply to David Williams from comment #54)
>  Yes. And that can be all those in the "Eclipse and Equinox" View (a few do
> not start with 'ep*", but you can/should remove those too). 
> 
> Thank you.

Done. I removed all jobs in the view and the view itself. Thanks for your help.
Comment 56 Marc-André Laperle CLA 2015-09-01 10:14:01 EDT
(In reply to Mikael Barbero from comment #52)
> I deactivated the job on the old shared instance. Please give me your +1
> when you're ready to remove it (and for the windows job too).

+1 thanks!
Comment 57 Mikaël Barbero CLA 2015-09-01 10:20:00 EDT
(In reply to Marc-Andre Laperle from comment #56)
> (In reply to Mikael Barbero from comment #52)
> > I deactivated the job on the old shared instance. Please give me your +1
> > when you're ready to remove it (and for the windows job too).
> 
> +1 thanks!

Done.

Thank you everyone. We now have migrated all platform specific (windows/osx) jobs to the new shared instance. So I disconnected the windows7, mac-tests and mac-tests2 slave from this instance. I will soon remove them.

I started to raise bugs against every projects with one or more job on the old shared instance to know what is their preferred migration path (new shared instance of a HIPP).
Comment 58 Marc-André Laperle CLA 2015-09-02 23:16:15 EDT
I am having issues with the Mac slave so I created bug 476488 (to minimize noise here).
Comment 59 Mikaël Barbero CLA 2015-12-23 06:37:35 EST
The old shared instance has been shutdown. Projects which was still having jobs on this instance have been notified and the configuration of their jobs have been attached to their respective bug. Here is the list of the jobs when the shutdown occurred:

emf-cdo-integration
emf-cdo-maintenance
emft-b3-build
hudson-tm-32x-test
libra
lyo-client-nightly
lyo-oslc4j-nightly
lyo-oslc4j-packaging
lyo-server-nightly
m2t-jet-master
org.eclipse.fmc-nightlybuild
papyrus-0.8-EYY-maintenance
papyrus-master-tests-failures
papyrus-trunk-extra-nightly
papyrus-trunk-extra-nightly-tests
papyrus-trunk-nightly
papyrus-trunk-nightly-tests
wazaabi-nightly

Closing and considering the migration successful. Thank you everyone for all your help and support.