Community
Participate
Working Groups
Currently release builds involve a lot of manual interaction. It would be much better if we could automate this with one (or maybe a few) Jenkins jobs. Such a job could: * Automatically build the updated docu * Automatically create a new version number (using the current time instead of SNAPSHOT) * Maybe have a job parameter for a version suffix like "-m2" or "-r" * Might need a job parameter for the corresponding JGIT_VERSION (the timestamps would differ, so no way to derive it from the EGit version) * Create the tag * Build & publish * Push the tag bypassing Gerrit It could even auto-increment the version if suffix "-r" is given (and commit & push that commit), and create the necessary up-merge(s) automatically and push them as Gerrit changes. The EGit release job could automatically trigger the egit-hub release job with the same version suffix and with the correct EGIT_VERSION.
Some more thoughts: * The tag would not be signed, but I think that'd be no problem? * The version suffix could be extracted from the JGIT_VERSION. * The build would be manually triggered only. Unclear if permissions on Jenkins would need to be restricted, but I think currently only committers can manually trigger and retrigger jobs anyway.
So far creating a JGit and EGit release involves the following steps During development phase when first milestone for release train is created we create a new stable branch for the upcoming release. This helps to avoid races between development continuing on master and creating a milestone, release candidate or release. Before the release the egit documentation needs to be updated (described in contributor guide). This change is pushed for review and submitted when ok. Create release (first JGit, when that's complete EGit and then EGit Github integration). - define version number x.y.z.timestamp-r, timestamp format is yyyymmddhhmm, use timezone of Jenkins to ensure timestamp of release build can be put in relation to timestamps of snapshot builds. - suffix -m1, -m2 etc is used for milestones, -rc1, -rc2 etc is used for release candidates and -r for releases - currently the release scripts assume we use the same version number for all jgit and egit bundles - run $ ./tools/release vx.y.z.timestamp-r - this requires GPG keys for signing release tags to be installed and asks for GPG passphrase interactively during script execution. - the script creates a signed tag with the given tag name and increases version numbers accordingly, then it shows the diff interactively, when quitting the diff it builds the release locally - in egit and egit-github stable build jobs set the JGIT_VERSION or EGIT_VERSION to the chosen release version, this is used to download the correct version of the JGit, EGit p2 repository for these builds - create a folder for the new egit, egit-github release on the download server - set the publish path in the stable job accordingly - push the release commit for review and wait for the validation build to succeed (builds and tests on Jenkins take most of the time, since Jenkins is pretty slow) - review the change and submit it if ok and validation passed - wait for the stable job to build the release, this deploys it to repo.eclipse.org, and to the download server (egit and egit-github only) - push the release tag - we always signed release tags, and I think that's for a good reason. If we automate creation of releases (and I am all for this) Jenkins should get its own GPG keys (to be kept secured) so that it can sign the release tags. Passphrase could be stored as a Jenkins secret - for jgit release generate the maven site (described in contributor guide) and deploy it to the download server - run $ ./tools/version.sh --snapshot=x.y.z+1-SNAPSHOT to increase version numbers to the next snapshot version following this release to deploy jgit release to maven central (can be done after the release): - use ./tools/maven/download.rb <release version number> to download the release artefacts from nexus - use ./tools/maven/deploy.rb <release version number> to stage the release artefacts on sonatype OSS nexus, requires GPG keys/passphrase for signing - close the staging repository on OSS - update https://github.com/msohn/jgit-build-test to the new jgit release version, set the url of the maven staging repository on sonatype oss nexus, and run the maven build of this project to verify the artefacts deployed in the staging repository are complete and not corrupt - release the staging repository, it can take a couple of hours until they appear on repo1.maven.org (Maven central) - update the respective download pages
(In reply to Matthias Sohn from comment #2) > So far creating a JGit and EGit release involves the following steps Thanks a lot for this; with that we can go forward.
Some notes on this as reminders to myself when I'll come back to this: (In reply to Matthias Sohn from comment #2) > During development phase when first milestone for release train is created > we create a new stable branch for the upcoming release. This helps to avoid > races between development continuing on master and creating a milestone, > release candidate or release. Branch creation would still be manual. I'd think we'd get a single egit-release job with two parameters: the branch (validated to be of the format "stable-\d+\.\d+" and to exist), and a choice dropdown with the choices "Milestone", "Release Candidate", "Release"). The tag suffix can be derived by the job automatically by looking at existing tags: 1. Get version from root pom (M.N.P) 2. Get all tags matching vM.N.P<anything>. Optionally error out if any tag found not signed. 3. Check existing suffixes. Milestone not allowed if there is an -rc\d or -r tag; others not allowed if there is a -r tag. 4. Derive the new suffix. Milestone -m1 if no -m\d tag present, otherwise -m(d+1). Release Candidate ditto with -rc\d. Release -r. > Before the release the egit documentation needs to be updated (described in > contributor guide). This change is pushed for review and submitted when ok. Review step needs to be eliminated. Contributor Guide says to check in particular links and images. We can at least run an HTML validator (if the Mylyn WikiText produces valid HTML). Optionally we can probably verify automatically that all internal links can be resolved. Should be done only if the version is M.N.0. > Create release (first JGit, when that's complete EGit and then EGit Github > integration). JGit won't be built by egit-release. Current nightly and stable jobs already can figure out the appropriate JGit version themselves. egit-release would, if successful, trigger egit-github-release. > - define version number x.y.z.timestamp-r, timestamp format is yyyymmddhhmm, > use timezone of Jenkins to ensure timestamp of release build can be put in > relation to timestamps of snapshot builds. Of course a job can determine the timestamp itself. > - suffix -m1, -m2 etc is used for milestones, -rc1, -rc2 etc is used for > release candidates and -r for releases > - currently the release scripts assume we use the same version number for > all jgit and egit bundles Current jobs already can figure that out. They'll build EGit M.N.P against the highest JGit M.N.Q, and likewise for EGit-Github. > - run $ ./tools/release vx.y.z.timestamp-r > - this requires GPG keys for signing release tags to be installed and asks > for GPG passphrase interactively during script execution. GPG key would need to be installed on Jenkins as Credential; interactive passphrase prompt must be eliminated. You already mention how this could be done below. > - the script creates a signed tag with the given tag name and increases > version numbers accordingly, then it shows the diff interactively, when > quitting the diff it builds the release locally Local builds would no longer be supported. (Or rather, everything happening inside the job would be a "local" build, followed by the appropriate push/deploy steps.) > - in egit and egit-github stable build jobs set the JGIT_VERSION or > EGIT_VERSION to the chosen release version, this is used to download the > correct version of the JGit, EGit p2 repository for these builds > - create a folder for the new egit, egit-github release on the download > server > - set the publish path in the stable job accordingly > - push the release commit for review and wait for the validation build to > succeed (builds and tests on Jenkins take most of the time, since Jenkins is > pretty slow) This would happen inside the build; no push to Gerrit. > - review the change and submit it if ok and validation passed No review; assumption is that the build does it correctly. I'd delay this push until after the deploy step (publishing to download server) and do it together with pushing the tag. Job can push directly to git, bypassing Gerrit. Make it fail if upstream branch head changed in the meantime (--force-with-lease). > - wait for the stable job to build the release, this deploys it to > repo.eclipse.org, and to the download server (egit and egit-github only) Existing deploy step, no problem. > - push the release tag Can be done automatically by the job. > - we always signed release tags, and I think that's for a good reason. If we > automate creation of releases (and I am all for this) Jenkins should get its > own GPG keys (to be kept secured) so that it can sign the release tags. > Passphrase could be stored as a Jenkins secret Ok. > - for jgit release generate the maven site (described in contributor guide) > and deploy it to the download server Let's worry about that when JGit moves to the new infrastructure. > - run $ ./tools/version.sh --snapshot=x.y.z+1-SNAPSHOT to increase version > numbers to the next snapshot version following this release Can be done automatically by the job and be pushed for review. Up-merges could also be generated and be pushed for review automatically. (Check out next higher stable branch or master if none and run git merge --log -s recursive -X ours stable-M.N; might need to exclude the doc bundle from that merge; i.e. reset the doc bundle after the merge to what it was before, then amend the merge.) > to deploy jgit release to maven central (can be done after the release): > - use ./tools/maven/download.rb <release version number> to download the > release artefacts from nexus > - use ./tools/maven/deploy.rb <release version number> to stage the release > artefacts on sonatype OSS nexus, requires GPG keys/passphrase for signing > - close the staging repository on OSS > - update https://github.com/msohn/jgit-build-test to the new jgit release > version, set the url of the maven staging repository on sonatype oss nexus, > and run the maven build of this project to verify the artefacts deployed in > the staging repository are complete and not corrupt > - release the staging repository, it can take a couple of hours until they > appear on repo1.maven.org (Maven central) > > - update the respective download pages Up to the deploy to OSS this could be done by a jgit-release job provided it has a GPG setup and is allowed to publish to OSS. I suppose currently that's tied to you personally? I have no idea what "close the staging repository on OSS" means. The rest would still be manual. But those are steps for an eventual jgit-release job anyway. One problem is what to do if pushing the commit(s) and the tag after the deployment to the download server fails.
(In reply to Thomas Wolf from comment #4) > Some notes on this as reminders to myself when I'll come back to this: > > (In reply to Matthias Sohn from comment #2) > > During development phase when first milestone for release train is created > > we create a new stable branch for the upcoming release. This helps to avoid > > races between development continuing on master and creating a milestone, > > release candidate or release. > > Branch creation would still be manual. > > I'd think we'd get a single egit-release job with two parameters: the branch > (validated to be of the format "stable-\d+\.\d+" and to exist), and a choice > dropdown with the choices "Milestone", "Release Candidate", "Release"). > > The tag suffix can be derived by the job automatically by looking at > existing tags: > > 1. Get version from root pom (M.N.P) > 2. Get all tags matching vM.N.P<anything>. Optionally error out if any tag > found not signed. > 3. Check existing suffixes. Milestone not allowed if there is an -rc\d or -r > tag; others not allowed if there is a -r tag. > 4. Derive the new suffix. Milestone -m1 if no -m\d tag present, otherwise > -m(d+1). Release Candidate ditto with -rc\d. Release -r. > > > Before the release the egit documentation needs to be updated (described in > > contributor guide). This change is pushed for review and submitted when ok. > > Review step needs to be eliminated. Contributor Guide says to check in > particular links and images. We can at least run an HTML validator (if the > Mylyn WikiText produces valid HTML). Optionally we can probably verify > automatically that all internal links can be resolved. > > Should be done only if the version is M.N.0. > > > Create release (first JGit, when that's complete EGit and then EGit Github > > integration). > > JGit won't be built by egit-release. Current nightly and stable jobs already > can figure out the appropriate JGit version themselves. egit-release would, > if successful, trigger egit-github-release. > > > - define version number x.y.z.timestamp-r, timestamp format is yyyymmddhhmm, > > use timezone of Jenkins to ensure timestamp of release build can be put in > > relation to timestamps of snapshot builds. > > Of course a job can determine the timestamp itself. > > > - suffix -m1, -m2 etc is used for milestones, -rc1, -rc2 etc is used for > > release candidates and -r for releases > > - currently the release scripts assume we use the same version number for > > all jgit and egit bundles > > Current jobs already can figure that out. They'll build EGit M.N.P against > the highest JGit M.N.Q, and likewise for EGit-Github. > > > - run $ ./tools/release vx.y.z.timestamp-r > > - this requires GPG keys for signing release tags to be installed and asks > > for GPG passphrase interactively during script execution. > > GPG key would need to be installed on Jenkins as Credential; interactive > passphrase prompt must be eliminated. You already mention how this could be > done below. > > > - the script creates a signed tag with the given tag name and increases > > version numbers accordingly, then it shows the diff interactively, when > > quitting the diff it builds the release locally > > Local builds would no longer be supported. (Or rather, everything happening > inside the job would be a "local" build, followed by the appropriate > push/deploy steps.) > > > - in egit and egit-github stable build jobs set the JGIT_VERSION or > > EGIT_VERSION to the chosen release version, this is used to download the > > correct version of the JGit, EGit p2 repository for these builds > > - create a folder for the new egit, egit-github release on the download > > server > > - set the publish path in the stable job accordingly > > - push the release commit for review and wait for the validation build to > > succeed (builds and tests on Jenkins take most of the time, since Jenkins is > > pretty slow) > > This would happen inside the build; no push to Gerrit. > > > - review the change and submit it if ok and validation passed > > No review; assumption is that the build does it correctly. I'd delay this > push until after the deploy step (publishing to download server) and do it > together with pushing the tag. Job can push directly to git, bypassing > Gerrit. Make it fail if upstream branch head changed in the meantime > (--force-with-lease). > > > - wait for the stable job to build the release, this deploys it to > > repo.eclipse.org, and to the download server (egit and egit-github only) > > Existing deploy step, no problem. > > > - push the release tag > > Can be done automatically by the job. > > > - we always signed release tags, and I think that's for a good reason. If we > > automate creation of releases (and I am all for this) Jenkins should get its > > own GPG keys (to be kept secured) so that it can sign the release tags. > > Passphrase could be stored as a Jenkins secret > > Ok. > > > - for jgit release generate the maven site (described in contributor guide) > > and deploy it to the download server > > Let's worry about that when JGit moves to the new infrastructure. > > > - run $ ./tools/version.sh --snapshot=x.y.z+1-SNAPSHOT to increase version > > numbers to the next snapshot version following this release > > Can be done automatically by the job and be pushed for review. Up-merges > could also be generated and be pushed for review automatically. (Check out > next higher stable branch or master if none and run git merge --log -s > recursive -X ours stable-M.N; might need to exclude the doc bundle from that > merge; i.e. reset the doc bundle after the merge to what it was before, then > amend the merge.) > > > to deploy jgit release to maven central (can be done after the release): > > - use ./tools/maven/download.rb <release version number> to download the > > release artefacts from nexus > > - use ./tools/maven/deploy.rb <release version number> to stage the release > > artefacts on sonatype OSS nexus, requires GPG keys/passphrase for signing > > - close the staging repository on OSS > > - update https://github.com/msohn/jgit-build-test to the new jgit release > > version, set the url of the maven staging repository on sonatype oss nexus, > > and run the maven build of this project to verify the artefacts deployed in > > the staging repository are complete and not corrupt > > - release the staging repository, it can take a couple of hours until they > > appear on repo1.maven.org (Maven central) > > > > - update the respective download pages > > Up to the deploy to OSS this could be done by a jgit-release job provided it > has a GPG setup and is allowed to publish to OSS. I suppose currently that's > tied to you personally? I have no idea what "close the staging repository on > OSS" means. The rest would still be manual. But those are steps for an > eventual jgit-release job anyway. > > One problem is what to do if pushing the commit(s) and the tag after the > deployment to the download server fails. Discovered in https://git.eclipse.org/r/#/c/148627/2/org.eclipse.mylyn.github-feature/feature.xml@24 : one additional step on the release day (egit and egit-github only): manually copy the content of the latest update-X.Y to update on the download server. That should become a separate publish-release job. That would become a separate publish_release job.
(In reply to Thomas Wolf from comment #5) > (In reply to Thomas Wolf from comment #4) ... > Discovered in > https://git.eclipse.org/r/#/c/148627/2/org.eclipse.mylyn.github-feature/ > feature.xml@24 : one additional step on the release day (egit and > egit-github only): manually copy the content of the latest update-X.Y to > update on the download server. > > That should become a separate publish-release job. > > That would become a separate publish_release job. +1 instead of copying we could redirect p2 to the latest releases own site, e.g. updates -> updates-5.4.2 AFAIK the download server does not follow symlinks though we can use a composite p2 repo [1] which refers to the latest release. Then we would need to update this composite on release day via publish-release job. We could also add a composite referring to all existing releases, the only downside of such a larger composite is that downloading the index is slower. AFAIK p2 always downloads the complete index so that the client can efficiently search (there is no p2 server). [1] https://help.eclipse.org/2019-06/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fp2_composite_repositories.htm
(In reply to Matthias Sohn from comment #6) > > instead of copying we could redirect p2 to the latest releases own site, e.g. > updates -> updates-5.4.2 > AFAIK the download server does not follow symlinks though we can use a > composite p2 repo [1] which refers to the latest release. Then we would need > to update this composite on release day via publish-release job. > > We could also add a composite referring to all existing releases, the only > downside of such a larger composite is that downloading the index is slower. > AFAIK p2 always downloads the complete index so that the client can > efficiently search (there is no p2 server). > > [1] > https://help.eclipse.org/2019-06/index.jsp?topic=%2Forg.eclipse.platform.doc. > isv%2Fguide%2Fp2_composite_repositories.htm Let's keep things simple for now. A simple ssh running rsync to copy the directory should do the trick. I propose two jobs, one for egit and one for egit-github, taking the version number Major.Minor.Patch as input and triggered manually on the release day. They'd validate the input and then run ssh genie.egit@projects-storage.eclipse.org rsync -zavh ${fromDirectory} ${publishDirectory}
(In reply to Thomas Wolf from comment #7) > ssh genie.egit@projects-storage.eclipse.org rsync -zavh ${fromDirectory} > ${publishDirectory} That already doesn't work; looks like the rsync hangs. :-( See the new (now disabled) egit-github-publish-release job. No idea what's wrong. Does the genie also only get that restricted shell and cannot do a rsync?
(In reply to Thomas Wolf from comment #8) > (In reply to Thomas Wolf from comment #7) > > ssh genie.egit@projects-storage.eclipse.org rsync -zavh ${fromDirectory} > > ${publishDirectory} > > That already doesn't work; looks like the rsync hangs. :-( See the new (now > disabled) egit-github-publish-release job. > > No idea what's wrong. Does the genie also only get that restricted shell and > cannot do a rsync? egit-github-publish-release now works. Not using rsync to copy directly on the server; instead using scp to first download from the updates-X.Y directory into the workspace, and if that worked use the same code we already use to get the site to update-X.Y but this time to updates. (Basically, upload the downloaded files via scp.)
(In reply to Thomas Wolf from comment #9) > egit-github-publish-release now works. Used it to bring https://download.eclipse.org/egit/github/updates/ up to 5.4.0. Also created second new job for EGit: egit-publish-release. Same mechanism, only different base path.
The egit-stable and egit-github-stable jobs could call egit-publish-release or egit-github-publish-relese, respectively, automatically if building a release version (ending in "-r") and the patch level is > 0. Or they could directly publish to both updates-X.Y _and_ to updates. For patch level == 0 we wouldn't want to do this; we first build the release version before release day, and then later on release day we want to sync updates. But for patch releases we could consider doing it automatically.
(In reply to Matthias Sohn from comment #2) > So far creating a JGit and EGit release involves the following steps ... > - for jgit release generate the maven site (described in contributor guide) > and deploy it to the download server uploading maven site manually doesn't work anymore since shell release engineers can use is now heavily restricted $ mvn site:site site:deploy ... [ERROR] Failed to execute goal org.apache.maven.plugins:maven-site-plugin:3.6:deploy (default-cli) on project org.eclipse.jgit-parent: Error uploading site: Error performing commands for file transfer: Exit code: 1 - Not allowed to do 'cd' -> [Help 1] this means we need to move this step to a build job running on Jenkins
Another thing: AFAIK the maven build also produces a zip file of the generated p2 repository. But the egit-stable build only copies the p2 repo to .../updates-X.Y. Should it also copy the zip into the same directory? I suspect it may otherwise be missing, the "deploy-release" jobs don't generate a zip either.
(In reply to Thomas Wolf from comment #13) > Another thing: AFAIK the maven build also produces a zip file of the > generated p2 repository. But the egit-stable build only copies the p2 repo > to .../updates-X.Y. Should it also copy the zip into the same directory? yes, I used to do this manually since I was too lazy to automate this > I suspect it may otherwise be missing, the "deploy-release" jobs don't > generate a zip either. yes
(In reply to Matthias Sohn from comment #14) > (In reply to Thomas Wolf from comment #13) > > Another thing: AFAIK the maven build also produces a zip file of the > > generated p2 repository. But the egit-stable build only copies the p2 repo > > to .../updates-X.Y. Should it also copy the zip into the same directory? > > yes, I used to do this manually since I was too lazy to automate this > > > I suspect it may otherwise be missing, the "deploy-release" jobs don't > > generate a zip either. > > yes OK. I've copied the zip from the 5.5.0 nexus directory [1] over to the download site for updates-5.5 with the help of a my thomas-test Jenkins job. I've also changed the egit-stable job to also push the zip generated in the maven build to the download server, but only for release versions (ending in "-r"). So in the future manually copying that zipped update site shouldn't be necessary anymore. (And I can update the Oomph setup before the release date!) The egit-publish-release job that will copy everything from updates-5.5 over to updates should also transfer that zipped update site. [1] https://repo.eclipse.org/content/repositories/releases/org/eclipse/egit/org.eclipse.egit.repository/5.5.0.201909110433-r/
(In reply to Thomas Wolf from comment #15) > (In reply to Matthias Sohn from comment #14) > > (In reply to Thomas Wolf from comment #13) > > > Another thing: AFAIK the maven build also produces a zip file of the > > > generated p2 repository. But the egit-stable build only copies the p2 repo > > > to .../updates-X.Y. Should it also copy the zip into the same directory? > > > > yes, I used to do this manually since I was too lazy to automate this > > > > > I suspect it may otherwise be missing, the "deploy-release" jobs don't > > > generate a zip either. > > > > yes > > OK. I've copied the zip from the 5.5.0 nexus directory [1] over to the > download site for updates-5.5 with the help of a my thomas-test Jenkins job. > > I've also changed the egit-stable job to also push the zip generated in the > maven build to the download server, but only for release versions (ending in > "-r"). So in the future manually copying that zipped update site shouldn't > be necessary anymore. (And I can update the Oomph setup before the release > date!) > > The egit-publish-release job that will copy everything from updates-5.5 over > to updates should also transfer that zipped update site. great, thanks if the restricted shell allows using the mv command we may consider to always first upload to a temporary directory and then, if upload succeeded, rename the temporary directory to the final one, this would lower the risk to end up with a corrupt update site in case of errors
