Bug 574335 - All new pull requests hang infinite waiting for Travis CI/CD
Summary: All new pull requests hang infinite waiting for Travis CI/CD
Status: CLOSED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: GitHub (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 blocker (vote)
Target Milestone: ---   Edit
Assignee: Eclipse Webmaster CLA
QA Contact:
URL: https://github.com/eclipse-ee4j/jaxrs...
Whiteboard:
Keywords: cross-project
Depends on:
Blocks:
 
Reported: 2021-06-20 03:35 EDT by Markus Karg CLA
Modified: 2021-07-15 04:14 EDT (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Markus Karg CLA 2021-06-20 03:35:18 EDT
All recently filed PR are hanging infitnite waiting for continuous-integration/travis-ci: Expected — Waiting for status to be reported.

I assume the problem is that travis.org was ceised and projects must use travis.com instead.

There is nothing projects can do here, as the continuous-integration/travis-ci check is under sole control of the EF admins. As this means that ALL projects hosted on Github are broken which use this integration, ALL those projects cannot accepts PRs currently!

Hence I urge the EF admins to fix this issue ASAP and restart all the hanging processes, so that it will be possible to have pre-review checks been done and finally merge those hanging PRs!
Comment 1 Mikaël Barbero CLA 2021-06-21 03:37:46 EDT
It's not related to travis.org being ceased. It's related to the fact that travis ci now only offers very limited amount of free credits, and those are not renewable. See https://docs.travis-ci.com/user/billing-overview/#free-plan

It seems that we could engage with travis ci in order to get "OSS credits" (https://docs.travis-ci.com/user/billing-faq/#what-if-i-am-building-open-source). Credits requests have to be renewed every single time we run out of credits:

"How many credits (build minutes) you’d like to request (should your run out of credits again you can repeat the process to request more or to discuss a renewable amount)"

From above, renewable credits seem to be even harder to request. 

I've issued a request nonetheless. Note that should Travis' credits management become too much of a burden to releng's team, we may just stop supporting Travis-ci. 

Finally, you should know that the Eclipse Foundation offers a free CI solution based on Jenkins to all its projects (see https://wiki.eclipse.org/Jenkins and https://wiki.eclipse.org/CBI#Provided_Services). You may want to investigate how you could leverage it.
Comment 2 Mikaël Barbero CLA 2021-06-21 09:53:10 EDT
Travis CI has granted 25k credits to the https://github.com/eclipse-ee4j organization. Those credits don't renew automatically. FYI, 25k credits ~= 40 build hours on Linux agents (much less if you  use macOS or Windows agents).

Please open another ticket when those credits get exhausted. We may then be able to apply for renewable credits.
Comment 3 Markus Karg CLA 2021-06-27 03:13:44 EDT
The Pull Requests still are hanging.
Do we just have to wait, or can you trigger the hanging continuous-integration/travis-ci checks to resume?
Comment 4 Markus Karg CLA 2021-06-27 03:50:35 EDT
You definitively need to do something here.

I have checked the eclipse repositories listed at travis.com and they do not include our eclipse-ee4j/jaxrs-api repository. I checked my personal travis account and was told that I have to enter the migration beta program to migrate my personal fork of jaxrs-api. So I signed that program, migrated from travis.org to travis.com, and pushed a change to one of the hanging MRs. Doing so instantly triggered a build on travis.com with for jaxrs-api with my personal account, and that build was successful.

But still the continuous-integration/travis-ci rule is hanging! So it seems that there IS something that you have to do, and my assumption is that the eclipse account at travis.com must enter the migration beta program at travis.com and request to migrate all repos (it is just a click on the frontend to do that).

I kindly ask to perform all needed steps ASAP, as we cannot make ANY progress with Jakarta REST 3.1 due to the piling stack of PRs waiting for continuous-integration/travis-ci!
Comment 5 Mikaël Barbero CLA 2021-06-28 02:55:38 EDT
Did the additional credits from last week helped or not?
Comment 6 Mikaël Barbero CLA 2021-06-28 03:12:55 EDT
Travis ci is a complete mess. I had to logout / login again a couple of times in order to see organizations again. I think you're all set now. Feel free to reopen if that's not the case.

Note that I've explicitely asked for renewable credits and here is the answer I got:

> Thanks for your reply, at the current moment our OSS credit grants are a manual
> process. When your credits begin running low again, please reach back out to the 
> Support team.

Please consider moving out of Travis CI, their way of handling OSS projects does not scale.
Comment 7 Frederic Gurr CLA 2021-06-28 05:05:17 EDT
The following projects had not been migrated to travis-ci.com yet.

cdi
cdi-tck
common-annotations-api
eclipselink
ejb-api
grizzly
jaxrs-api
jersey
jsonp
krazo
mojarra
security-api
soteria
tyrus
websocket-api
yasson

That's fixed now.
Comment 8 Markus Karg CLA 2021-06-29 13:18:51 EDT
Travis says that there are no credits so the account it locked, see https://travis-ci.com/github/eclipse-ee4j/jaxrs-api/jobs/519877246. As a result the build was cancelled in flight. So still we are blocked in our work.
Comment 9 Frederic Gurr CLA 2021-06-29 13:42:37 EDT
@Radek (copied in CC): The eclipselink project used up 26000 credits just in the last two days. Please check what's sucking up all the credits. This is blocking other projects.
Comment 10 Frederic Gurr CLA 2021-06-29 13:44:13 EDT
@Markus: As Mikael stated above, the current way how OSS projects are handled at Travis CI does not scale. So either move to GitHub actions, other similar services or use the build infrastructure provided by the Eclipse Foundation.
Comment 11 Markus Karg CLA 2021-06-29 13:50:39 EDT
If we move to Github actions won't we run into the same trouble?

If we move to EF-Jenkins how to merge the needed PR (the merge button is blocked as long as Travis doesn't confirm the build).

So we can do that, but first you need to take measures that JAX-RS can build on Travis for the time being.
Comment 12 Radek Felcman CLA 2021-06-29 13:51:33 EDT
Hi,

please remove Travis-CI from https://github.com/eclipse-ee4j/eclipselink project settings (everything webhook, Travis-CI GitHub application.... ).
After switch from Travis-CI.org to Travis-CI.com is almost unable to continue use this CI solution.
We started migration to Eclipse-CI infrastructure:
See https://bugs.eclipse.org/bugs/show_bug.cgi?id=574530
https://ci.eclipse.org/eclipselink/job/eclipselink-pull-request-master/
Comment 13 Markus Karg CLA 2021-06-29 13:52:27 EDT
BTW, Travis scales perfectly well with OSS - but it requires one Travis account per project, not one Travis account per OSS foundation.
Comment 14 Denis Roy CLA 2021-06-29 14:11:56 EDT
(In reply to Markus Karg from comment #13)
> BTW, Travis scales perfectly well with OSS - but it requires one Travis
> account per project, not one Travis account per OSS foundation.

Definition of "does not scale".
Comment 15 Frederic Gurr CLA 2021-06-29 14:17:23 EDT
(In reply to Markus Karg from comment #11)
> If we move to Github actions won't we run into the same trouble?
🤷‍♂️

> If we move to EF-Jenkins how to merge the needed PR (the merge button is
> blocked as long as Travis doesn't confirm the build).
> 
> So we can do that, but first you need to take measures that JAX-RS can build
> on Travis for the time being.
Administrators can merge PRs despite pending checks. Let us know which PRs can be merged and we'll handle it.
Comment 16 Frederic Gurr CLA 2021-06-29 14:20:14 EDT
(In reply to Markus Karg from comment #13)
> BTW, Travis scales perfectly well with OSS - but it requires one Travis
> account per project, not one Travis account per OSS foundation.
Each GitHub organization already has their own Travis CI "plan". So projects in the eclipse-ee4j org do not share credits with projects in the eclipse GitHub org.
Comment 17 Markus Karg CLA 2021-07-01 12:56:04 EDT
(In reply to Frederic Gurr from comment #15)
> (In reply to Markus Karg from comment #11)
> > If we move to Github actions won't we run into the same trouble?
> 🤷‍♂️
> 
> > If we move to EF-Jenkins how to merge the needed PR (the merge button is
> > blocked as long as Travis doesn't confirm the build).
> > 
> > So we can do that, but first you need to take measures that JAX-RS can build
> > on Travis for the time being.
> Administrators can merge PRs despite pending checks. Let us know which PRs
> can be merged and we'll handle it.

Good to know. I will ask the committers to discuss this.
Comment 18 Markus Karg CLA 2021-07-01 12:57:08 EDT
(In reply to Frederic Gurr from comment #16)
> (In reply to Markus Karg from comment #13)
> > BTW, Travis scales perfectly well with OSS - but it requires one Travis
> > account per project, not one Travis account per OSS foundation.
> Each GitHub organization already has their own Travis CI "plan". So projects
> in the eclipse-ee4j org do not share credits with projects in the eclipse
> GitHub org.

Sure, but all the many eclipse-ee4j projects share one single account. If jaxrs-api would be its own account, it would have plenty of free credits.
Comment 19 Mikaël Barbero CLA 2021-07-15 04:14:40 EDT
We're done here.