Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[eclipse.org-planning-council] One Testers Experience (was re-spin process)

While I would certainly like to be paid for my efforts, I think there are a few of us who end up cross-product testing for free.

This year I was working with BIRT and GMF around the Europa release time. These two products, along with my standard Eclipse Env (EMF, GEF, Mylyn, JDT, PDE, ECF, and occasionally CDT) made for a good cross product test suite. While I am not doing integration testing, I figured I could help out since I had a number of these products installed and working together. However, the problems I ran into where not monetary or even a lack of time, but rather general confusion.

Every 6 weeks, I get the latest Milestone Eclipse build and I consider myself a contributor since I open bugs, track issues, and even help pin-point problems. However, for Europa I was far less useful. For starters I didn't know where / when to get the "latest" build. The Wiki [1] had a number of dates, but I didn't know where to go (for example) on May 18 / 23 / 25 / 28 / 29 to get RC1. I wasn't following the this list until late in the game, but for years I never followed eclipse-dev and I was still able to help out. Once I started to follow the list there seemed to be an endless stream of re-spins. While I understand that getting "green builds" is hard, the constant re-spinning (right up until the 11th hour) needs to be addressed if we want testers to help out.

[1] http://wiki.eclipse.org/index.php/Europa_Simultaneous_Release

Also, "to avoid confusion" 3 mailing lists are used as well as as a weekly conference call. As a user, this is "confusing". The Europa release information seems to be targeted at committers not contributors. Next year, part of the release train site should be directed at contributors / testers.

Finally, I had a number of technical problems [2]. Looking back, some of these were my lack of understanding, but it wasn't clear where I should turn. I tried this list, but the only response I got was from Nick saying that he didn't have those problems but rather a list of other problems [3].

[2] http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01258.html [3] http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg01260.html

This is just my experience, but I think others may help out if we made an effort to ease the testing pain.

Summary:
1) Do we expect the testers to follow the 3 dev lists?
2) Do we expect the testers to join the conference calls?
3) Where can we just "GET" the latest release?
4) Where do the testers turn for mentor / support?

Cheers,
Ian



Gaff, Doug wrote:

Employ Testers. Seriously.

Obviously, the challenge that most projects have is that our plates are full just releasing our own stuff. Doing a lot of cross-project integration testing falls to a close second. If there were a few persona-based testers available, that would really help.

Doug G

*From:* cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] *On Behalf Of *Mike Milinkovich
*Sent:* Monday, July 16, 2007 9:49 AM
*To:* 'eclipse.org-planning-council'; 'Cross project issues'
*Cc:* epp-dev@xxxxxxxxxxx; 'Mylyn developer discussions'
*Subject:* RE: [eclipse.org-planning-council] RE: [cross-project-issues-dev]Re-spin process / DSDP-TM Re-spin request

+1

To be honest, I didn’t even realize this was in question. Projects can decide to be in/out of any particular release train and always have the freedom to push out intermediate releases if it makes sense for their projects and its community.

I would like to pick up on Doug’s desire to see more integration testing between the projects involved in the release train. What can we at the Foundation / EMO do to help make that wish a reality?

Mike Milinkovich

Office: +1.613.224.9461 x228

Mobile: +1.613.220.3223

mike.milinkovich@xxxxxxxxxxx <mailto:mike.milinkovich@xxxxxxxxxxx>

*From:* eclipse.org-planning-council-bounces@xxxxxxxxxxx [mailto:eclipse.org-planning-council-bounces@xxxxxxxxxxx] *On Behalf Of *Jeff McAffer
*Sent:* Thursday, July 12, 2007 10:19 PM
*To:* Cross project issues
*Cc:* 'Mylyn developer discussions'; 'Cross project issues'; eclipse.org-planning-council; epp-dev@xxxxxxxxxxx *Subject:* RE: [eclipse.org-planning-council] RE: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request


I agree as well. The release trains are value add and need to really mean something. Projects should do what they do, release when they release, and, from time to time, may choose to participate in a train. The release trains are a great thing for consumers but, as Doug points our, they are not the only way of releasing something.

Jeff

*Doug Schaefer <DSchaefer@xxxxxxx>*
Sent by: cross-project-issues-dev-bounces@xxxxxxxxxxx

07/09/2007 01:19 PM

Please respond to
Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>

	

To

	

"eclipse.org-planning-council" <eclipse.org-planning-council@xxxxxxxxxxx>, "'Cross project issues'" <cross-project-issues-dev@xxxxxxxxxxx>

cc

	

"'Mylyn developer discussions'" <mylyn-dev@xxxxxxxxxxx>, epp-dev@xxxxxxxxxxx

Subject

	

RE: [eclipse.org-planning-council] RE: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request


	




I agree, Mik. Different projects at different stages have different needs for release frequency. For the CDT, we would really like to stay annually to make sure the vendors adopting the CDT have predictable quality and schedule. However, having said that, we do get tempted to releasing more often but then we would lose the stability that having firm yearly dates gives us.

I think we may be losing sight of what these simultaneous releases were meant to be. Nothing is stopping you from releasing quarterly on your own. We certainly aren’t making these trains the only release vehicle available to projects, just a predictable one that commercial vendors redistributing Eclipse projects can depend on. For their benefit (being one of them ;), I’d love to see our next step in the evolution of the trains be doing actual integration testing between the projects. If the projects are releasing randomly, we would never be able to achieve that.

Doug Schaefer, /QNX Software Systems
Eclipse CDT Project Lead, //http://cdtdoug.blogspot.com/ <http://cdtdoug.blogspot.com/>

------------------------------------------------------------------------


*From:* eclipse.org-planning-council-bounces@xxxxxxxxxxx [mailto:eclipse.org-planning-council-bounces@xxxxxxxxxxx] *On Behalf Of *Mik Kersten*
Sent:* Monday, July 09, 2007 12:55 PM*
To:* 'eclipse.org-planning-council'; 'Cross project issues'*
Cc:* epp-dev@xxxxxxxxxxx; 'Mylyn developer discussions'*
Subject:* RE: [eclipse.org-planning-council] RE: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request

Just thought of a simpler way of stating this.

While the SDK is mature enough for an annual release cycle, other projects can benefit the community by providing more frequent releases. Both biannual and quarterly update schedules overlap well with the annual release train. Since Mylyn is continuing to evolve rapidly, we would like Europa and EPP to consider supporting quarterly updates.

Mik

*From:* eclipse.org-planning-council-bounces@xxxxxxxxxxx [mailto:eclipse.org-planning-council-bounces@xxxxxxxxxxx] *On Behalf Of *Mik Kersten*
Sent:* Monday, July 09, 2007 9:26 AM*
To:* 'Cross project issues'*
Cc:* 'Mylyn developer discussions'; 'eclipse.org-planning-council'; epp-dev@xxxxxxxxxxx* Subject:* [eclipse.org-planning-council] RE: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request

+1 for increased frequency and figuring out how to coordinate this with EPP. Here is Mylyn's point of view.

Since we are part of the EPP distros it would be good to have a well-defined mechanism and process for getting the latest and greatest to users via EPP while supporting the Ganymede train and Europa maintenance releases. Current constraints we see: · Platform and other core projects get adopted every 6 weeks by early adopters, but only yearly by the majority. As long as automatic update checking is not turned on by default we cannot count on majority users getting new versions via update sites. · Some projects like Mylyn are evolving rapidly and ensure that majority users are always best off using the latest. · Many or most integrators don't want to port more than once a year, whereas we foresee the need to make breaking API changes, i.e. moving from 2.0 to 3.0, at some point during the Ganymede cycle.
· Maintaining more than two streams is a pain.

Here's our current idea for how to accommodate these constraints, and I'm wondering how this fits for other projects and Europa/Ganymede/EPP planning. 1) Support Platform 3.3 and 3.4 with a single stream for as long as possible, branch when necessary. This means that we will be releasing the same version for the Ganymede train and Europa updates. (We have used this scheme the past couple of years and ended up branching around Platform M4 each time because we are quite dependent on internals and API additions). 2) Follow the 6 week schedule but make releases available to majority users on a quarterly basis, i.e. every other Ganymede release. For Mylyn this would mean something like the following post-Europa schedule: · Week 6: Mylyn 2.1M (pushed to Mylyn update site only, i.e. can only count on early adopters getting it)
· Week 12: Mylyn 2.1 (pushed to Europa/Ganymede/EPP sites)
· Week 18: Mylyn 2.2M
· Week 24: Mylyn 2.2
· Week 30: Mylyn 2.3M
· Week 36: Mylyn 2.3
· Week 42: Mylyn 3.0 RC0 (incorporate API changes)
· Weeks 48-52: Mylyn 3.0 (final)

Effects of this would be:
· EPP could incorporate Mylyn updates once per quarter ensuring that the majority of users can work from the latest and greatest.
· The “2.x” releases would have the same quality bar as Europa.
· The “2.xM” would allow us to get feedback on UI changes from early adopters before releasing them to majority users. · Integrators only have to port once at the end of the cycle, at the same time that they would be moving to the 3.4 Platform.

Mik

*From:* cross-project-issues-dev-bounces@xxxxxxxxxxx [mailto:cross-project-issues-dev-bounces@xxxxxxxxxxx] *On Behalf Of *John Arthorne*
Sent:* Monday, July 09, 2007 8:33 AM*
To:* Cross project issues*
Subject:* Re: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request


Using the same update site makes sense to me. I guess the question then is how often the staging site is migrated to the release site. Did this only happen twice for Callisto (once in the fall and once in the winter)? Perhaps this frequency needs to be increased to accomodate the various release rhythms of the projects involved. The other important element in the coordinated release is the EPP packages. I would love to see a well known place for staging the EPP bits, so we can invite the community to grab the latest packages and participate in testing prior to release dates.

*Bjorn Freeman-Benson <bjorn.freeman-benson@xxxxxxxxxxx>*
Sent by: cross-project-issues-dev-bounces@xxxxxxxxxxx

07/09/2007 11:17 AM

Please respond to
Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>

	

To

	

Cross project issues <cross-project-issues-dev@xxxxxxxxxxx>, "eclipse.org-planning-council" <eclipse.org-planning-council@xxxxxxxxxxx>

cc

	

Subject

	

Re: [cross-project-issues-dev] Re-spin process / DSDP-TM Re-spin request


	





John,
My plan for the Europa Fall Maintenance Update Site was to use the same update site as the Europa release - I think that's what we did for Callisto and it makes sense for Europa as well. We'll Europa-matic build into the Europa staging site (as we did for the major release) until we are collectively happy with the bits and then we will upgrade those staging site bits to the release site.

I am already doing Europa-matic builds for the maintenance release (or the re-spin, or whatever we decide): the results are in the usual place: http://dash.eclipse.org/~bfreeman/europa/ <http://dash.eclipse.org/%7Ebfreeman/europa/>

If you think we should do something else (a *separate* Fall Maintenance Update Site, separate from the release site), let's talk about that...

- Bjorn

John Arthorne wrote:
I think this debate about respins is only happening because we don't yet have the Europa fall update site set up yet. In the absence of this, Europa respins are the only way to get our critical fixes out there. I suggest we create a Europa fall update site as soon as possible so we can put the Europa June 29th release behind us.
_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev

------------------------------------------------------------------------

_______________________________________________
cross-project-issues-dev mailing list
cross-project-issues-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/cross-project-issues-dev


--
R. Ian Bull
PhD Candidate, University of Victoria
http://www.ianbull.com
http://irbull.blogspot.com/




Back to the top