Thank you Stephan for your balanced, concise, insightful commentary!
More comments below...
On 29.01.2020 23:54, Stephan Herrmann wrote:
> Thanks, Ed, for raising these concerns.
>
> The thread is already at a volume which makes it hard to digest, I can
> only hope I got the essence of what everybody is thinking.
>
Yes, the thread has frayed into multiple smaller threads, some of which
stray far from the original focus. E.g., release cadence and quality
concerns.
>
> One way I could interpret the situation is: It takes a very small
> number of qualified people to herd the train - full-time, and then the
> immediate issue is resolved.
Yes, Fredric Gurr, Ed Merks, and Markus Knauer help to ensure that the
large volume of inputs result in the correct and appropriate outputs.
The simrel inputs are a large sets of p2 repositories comprising an even
larger set of installable units. Fred's work ensures that these inputs
map to a simple p2 repository (the train repo) with a correct, coherent,
consistent set of installable units, organized into categories. Markus
Knauer's work ensures that the EPP inputs (the package/product
definitions), in combination with the train repo, map to a simple p2
repository (the epp repo) as well as a set of canned packages
(zip/tar/dmg). These two simple repos are composed to give us the
overall simrel repo. Ed Merks ensures that the train repo in
combination with the epp repo maps to a set of Product Versions in
Oomph's Eclipse.org Product Catalog, and delivers a set of installers
(*.exe/tar/dmg) that is based on the latest installable units from the
train repo. These installers can be used to install the same "packages"
as are delivered as canned packages (zip/tar/dmg).
Fred is funded by the Foundation, which stepped in when David Williams
funding was cut by IBM. Ed Merks was funded by itemis, but that has
been cut. Markus Knauer is funded by Eclipse Source, but his daily work
is unrelated and so he has stepped back because essentially his work is
unfunded. In the past, both itemis and Eclipse Source were Strategic
Developer Members, but each has stepped back from that long ago. At
this point the Foundation is not able to step in because of its own
budget constraints and is also not willing to step in because it's a
slippery slope to step in purely using existing budget whenever some
part of the community fails. Of course the failure in this case is
cross project and broadly sweeping...
> There are member companies for whom it should not be a problem to fund
> the one appointed train master - perhaps via the foundation. I'd love
> to see Ed's efforts to be compensated in a way which would allow him
> to give us a long term perspective. I don't represent a member company
> so I can't comment on whether and how this simple idea could be put to
> action.
>
The proposed solution is a Working Group with membership fees as the
funding model for this process. Of course for that to work, some member
companies need to get step up and that boils down to the question: why
should I (as a member company) get involved (pay) when I can just wait
for and hope that some other member company will pay the cost instead?
It feels like a game of chicken played by multi-billion revenue
companies, where the amount of money actually required is as a molecule
of water. I remember all too well the days when the Platform was
funded almost exclusively by IBM: that game of chicken lasted for
years. Thank goodness Red Hat joined the party and thank the mighty
techo deities for Tycho.
>
> I assume all of us subscribed to cross-projects have a shared interest
> in providing to our users plenty of options of which software they
> want to install, in various combinations, and all in the latest and
> greatest version. Without bad surprises on their way.
Yes and no. Herein we already see a fundamental split in the
community. There is the "we don't care about the train repo" camp, and
also the "we don't care about the packages" camp...
> Please, those who want to do away with SimRel: tell me, what efforts
> are incurred by SimRel that or not essential for this original goal?
> If SimRel incurs non-essential efforts, produces accidental
> complexity, shouldn't we just name these, so we can fix things
> (process? automation?).
In my opinion, aside from the question how exactly the p2
repo/installable-unit inputs are managed/represented, the train repo is
and must remain one of the primary goals and outputs of the overall
process. Many of us care deeply about this and some of us only care
about this. Producing this is Fred's role and of course it's already
primarily automated. In fact, so automated that we have a daily staging
train repo. Only Fred can comment on how much overall work is involved
in this aspect. I can only comment that we could dramatically improve
the testing of quality metrics and streamline the process of problem
tracking/reporting. Of course there are arguments in favor of reducing
the set of participating projects, but none of that (including
representation changes) will change the nature of the work involved.
Moreover, I don't think this is the primary burden that we need to try
to reduce or eliminate.
For those who argue to reduce this to just the things needed by EPP
package (which is likely those people whose inputs are already in EPP
packages), just consider that I personally don't want to participate in
a process of producing EPP packages if there is no train repo that
contains all the things that I personally want in the train repo. I
expect Stephan has similar concerns about his Object Teams contribution;
it's not in any package so is easily caught a broad sweep of
irrelevance. Please don't do that.
The underlying point here is that every point of view is valid and while
your personal mileage from using the train repo may vary, please
consider carefully the mileage value it has for the community as a
whole: if it has value for someone, it has value.
> (Disclaimer: I don't care about packages because a simrel repo +
> various oomph setups are all I need to get the Eclipses I want, or to
> push the same to colleagues in the company). (Disclaimer 2: I don't
> hate the cbi aggregator, I prefer to fix it when there's a problem).
>
The "we don't care about the train" camp are also arguing that we could
represent the EPP's inputs differently. Instead of representing it via
the aggregator model (in XML), we could represent it as a target
definition model (in XML) and a site definition model (in XML). Of
course this is factually true, but to me it's all just so much XML.
Furthermore, I find this deeply technical level of discussion about
representations pointless, because the fundamental point is that someone
must manage a representation and a large set of committers must edit a
representation. And in any case, at this point, still no one has
stepped up to do anything. Changing the representation introduces a new
set of problems whereas the existing representation is working well and
is fully automated already. So any argument to change the
representation at this point is an argument to create more new work with
a new different set of problems, while diverging from the issue at hand,
i.e., even if someone runs off and prototypes how this could be changed
(of course it's possible in principle!), there will remain the problem
of who will actually deal with all this for the next 10 years?
>
> For a thought experiment wrt the release cadence: is it possible our
> cadence is still too slow? Imagine a truly continuous SimRel.
Yes, in my wildest fantasy we would have the full slate of daily staging
outputs. There is no technical reason I can think of why this isn't
feasible.
> Perhaps, SimRel contributions should happen *always*, i.e., whenever a
> project has any code changes, or buffered to once per day. Some
> contributions will break downstream projects, which have the option to
> react, or drop out after 3 build failures. If your dependency drops
> out, you will notice immediately. Some version bumps, dependency
> changes will need coordination, here on cross-projects, but they will
> just happen at their own timing, not driven by release dates. In a
> world where so many tasks can be automated, isn't the continues flow
> of all seeing each others HEAD the simplest model of all?
This is exactly what I imagine. We can make aggregation fail when
licenses are bad or unsigned content is contributed, but we can only do
that once we've automated that and are in a valid state just once; we're
pretty darned close right now. We can and should add more automated
integration tests. If we all felt comfortable about the quality of this
continue integration's output we could all use it as the basis for our
daily development environment. Each of us could be testing it
continuously every day by actually using it every day. We'd notice
problems much faster and that will most definitely improve quality.
<digression>In fact I have a Platform SDK workspace with 26 Git
repositories that comprise the Platform SDK. I regularly run the setup
so that it updates the installation and TP to the latest IBuild and then
I pull all the Git repos. I see exactly what will be (or soon will be)
in the next IBuild. I notice problems before anyone else, before there
even is a build, and I can easily help fix said problems, e.g., Bug
558313. I.e., I literally see HEAD of everything...</digression>
>
> In this model, SimRel (or should we call it SimFlow) would concentrate
> on ensuring that our components stay on track wrt compatibility with
> each other. The other concern about more regressions due to shorter
> cycles and no maintenance branch would still need to be discussed but
> that's probably a different discussion. That discussion could well
> suggest a much slower cadence, but perhaps that is even possibly *on
> top* of a continues SimFlow.
Yes, these other discussions (cadence and quality) are really important
as well, but you hit the nail on the head with respect to what I want to
see this process become. Continuous, with appropriate checks and
balances in place, along with automated reporting, augmented with a
growing set of automated integration tests to eliminate the onerous +1
process of EPP. And then we each eat our own dog food every day,
savoring it as the world's finest delicacy, ready to be served to the
masses.
>
>
> Just brain storming ...
Indeed, collectively we've called everything into question and have
headed down different threads. That is goodness. As such, while the
"we don't care about the train" camp brainstorms down one thread, let me
brainstorm the converse on behalf of the "we don't care about the
packages" camp. We could most certainly change the representation of
the EPP packages and still produce the same outputs. We can see how
this is factually true just by looking at how Oomph's setup model
represents a package. Each package is essentially just a Product
Version and that essentially boils down to a p2 task that installs a set
of installable units from some p2 repositories. We could represent
package versions primarily in that form rather than generating this form
from the epp repo. Then we could "just eliminate EPP" while still
producing all the same outputs. With this approach, we could streamline
things like JDT's Java beta support so that the moment a new version of
Java is released, all the packages (at least those installed via the
installer as Product Versions) simply contain the latest Java support.
A great many improvements are possible. We are only limited by
resource, not by good ideas.
That is the kind of discussion I like to see. Until there are signs of new resources we have to ask "What is the least needed time consuming task we can stop doing ?" so progress on the good ideas can happen.
>
> Stephan
>
>
> _______________________________________________
> cross-project-issues-dev mailing list
> cross-project-issues-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or
> unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/cross-project-issues-dev
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/cross-project-issues-dev
