I’ve been wrestling with this one for a while… Just what is the role of a container project? I asked this question more formally in Bug 301065. I invite you to make your comments. The Eclipse Development Process is, after-all, your process.

Frankly, I think it’s time to do away with the notion of container projects. I think that we can move forward with a single notion of a project. Here’s what I’m thinking.

• A project has exactly one group of committers that have write access to the various resources owned by that project. From the Webmaster’s point-of-view, this means that each project has exactly one UNIX group.
• Projects manage access to their various parts using social convention (this is already happening today on most Eclipse projects)
• A project has zero or one of each of the following resources: a source code repository, a website, some space on the downloads server, some space on the archive server, access to a build server, etc.
• A project must have at least one project leader.
• A project may have one or more subprojects. Subprojects have their own distinct group of committers, and resources. We may refer to the organization of projects using whatever language makes sense to the target audience and broader community (e.g. “parent project”, “child project”, “subproject”, …).
• A project’s source repository should be separate and distinct from the source repository of its “parent” project.

I think that the most controversial thing here is the notion that a project can have its own code repository and subprojects. For what it’s worth, this is already happening in some Eclipse projects. I’ve always found the limitation against this very artificial. The one part of this that bothers me is overlapping repositories. I don’t think that subprojects should exist within the source code repository of the parent project: this just leads to confusion with our Dash tooling and adopters (who have to sort out which parts of the repository belong to which project). Confusion leads to anger. Anger leads to pain. Pain leads to suffering. Of course, some projects are doing this today, so maybe I’m overstating the problem (we should be able to sort out any possible issues with Dash).

Probably the second most controversial thing suggested here is the notion of a single group of committers for all of a project’s assets. Experience has shown us that managing fine-grain control to the various resources just doesn’t work well. Nobody seems to understand who should (and does) have access to what. For me, the responsibility that comes with being a committer includes the responsibility to know where you’re supposed to be working. On the Examples project, for example, committers working on the Toast example, just don’t commit on the Slideshow example. They just don’t. This, despite the fact that they have full access. Frankly, if you’re not sure that you can trust a developer to follow social conventions, then maybe you need to rethink nominating them as a committer.

At least that’s what I think. What do you think?

The Eclipse Development Process (EDP) defines “incubation” as a phase that projects go through on their way to maturity. New projects start in incubation to allow the project’s committers time to familiarize themselves with the state of being an Eclipse project, give them time to start developing a following in the community, and allow them some leeway to make and correct early mistakes. Projects in incubation can make releases, but they must be pre-1.0 releases, and must be labeled as “Incubating”. Frankly, I think that incubation is a fantastic idea and intend to keep it very much a part of the EDP.

An interesting use of the incubation phase as emerged and is the topic of discussion on Bug 300000 (which sounds a little like the title of a really bad sci-fi movie): the perpetual incubator. Many mature Eclipse projects now have projects that are intended to remain in perpetual incubation. These never-grow-up “Peter Pan” projects are an excellent place to innovate, test new ideas, and grow functionality that may one day be moved into the mature project. It is a curious, however, that the very notion of an incubator project is essentially (though not explicitly) forbidden by the EDP in its current form.

With the next revision of the EDP, I’d like to acknowledge the notion of an “Incubator” project. In Bug 300000, Holger Voorman and others propose that we adopt the name “Labs” for this notion to conform with precedents set by other open source organizations. Frankly, though, I believe that we’ve already established our own precedent and intend to run with it (I’m sure you’ll let me know if you disagree).

As I see them, Incubators are operating projects that never have releases; they do not require yearly continuation reviews; and they are not part of the release train. Incubators have builds, and downloads. They conform to the standard incubation branding are subject to the IP due diligence rules outlined for incubating projects. Incubators do not graduate.

I’d like to avoid creating any new process around the notion of Incubators. I’m hopeful that it will be enough to change some of the language in the EDP (and the Incubation HOW TO document) to allow projects to simply declare themselves as an incubator and leave it at that.

Unfortunately, I think that we may have a language issue. An “Incubator” project is in the incubation phase. However, a project in the incubation phase is not necessarily an incubator. A project like Mint, that one day intends to graduate, is in incubation. The Examples and the Equinox Incubator projects never intend to graduate and so are incubator projects in the incubation phase.

Maybe it’ll help if I stop thinking about it so much.

These are heady times in Eclipse project-land. I feel busier than a… well, I’m really, really busy.

We have creation reviews pending for the Modeling Team Framework and Object Teams proposals. Object Teams is an already-established, mature project that’s moving to Eclipse. The logistics of the move are interesting enough (the project itself is darned interesting to boot). This project will enter eclipse.org under incubation, but intends to graduate with their first release; this makes a lot of sense for a project with established code base as it allows their developers time to get used to the Eclipse Development Process, but then quickly establish themselves as mature technology. There is some overlap with this project and some existing Eclipse project, but discussions are underway to see how these technologies can work together, or carve out their own unique corners.

New project proposals are flowing it at an incredible pace.

The Enterprise Modules Project, code-named Gemini, was proposed in late 2009 and is due for creation soon. This project will “provide a home for subprojects that integrate existing Java enterprise technologies into module-based platforms, and/or that implement enterprise specifications on module-based platforms.” As a start, the reference implementations for eight OSGi Enterprise Expert Group specifications will be created as subprojects. Gemini is complemented by Dynamic Enterprise Application Platform Project, Virgo, which intends “to provide a runtime platform for the development of server-side enterprise applications built on top of Equinox, and optionally using modules from the Gemini project.”

Also queued up are the Graphiti, JavaScript Development Tools, Mangrove - SOA Modeling Framework, Mylyn Reviews, and ScalaModules proposals. There are about a dozen other proposals pending that I’d love to tell you about, but will have to wait.

There seems to be no end of interest in bringing projects to Eclipse. This is very gratifying to me and I do love working with the hard-working folks involved. Parts of our process can be frustrating though, and I’m working hard to address them while balancing the important principles that define what it means to be an Eclipse project. I’ll be presenting the Eclipse Board with an overview of the changes to the Eclipse Development Process that I’m working on in mid-February. I intend to have a draft ready for presentation to the Board in time for EclipseCon. In the meantime, I’ll be discussing the changes here and in Eclipse Bugzilla. If there are specific issues that you’d like addressed as part of this revision, either open a bug or email me directly (I’m the only “wayne” at the Eclipse Foundation).

I made a mention to Mike a couple of weeks back that I wanted to make a few changes to the Eclipse Development Process (EDP). He seemed to think that evolving our process was a grand idea and reminded me that changes to the development process require approval from the Eclipse Board. I knew that already, but it was still nice of him to remind me (he’s one of those give-you-gentle-reminders-and-let-you-get-it-done sorts of bosses). But I digress.

I was originally thinking of only a few small changes. But I’ve gathered up a bit of steam and have decided that maybe it’s worth trying to make some more fundamental changes. I’ve been in this role for about nine months; after nine months of immersion in our process, I have some ideas where I think we can do a better job. But before I start making wholesale changes, I though it might be prudent to try and bring some of my ideas before the community to gather feedback. Keep in mind that–ultimately–any changes we make to the development process do have to be approved by the board, are bounded by our bylaws, etc.

There are several key principles that I believe underly the EDP (in no particular order, because they’re all important):

• Transparency
• Openness
• IP cleanliness
• Quality

I believe that most of what we find in the EDP is intended to support one of these key principles. Correct me if I’m wrong.

There’s one more thing that I’ve added to this list:

• “Do the right thing” ahead of process.

Recently, I’ve been thinking a heck of a lot about reviews. Release reviews in particular. I’ve had many conversations with committers about release reviews and why we do them. Years ago, we required that project members “meet” the community on a scheduled call to present their case for release. Attendance on those calls was never particularly strong, and so the call melted away and we’ve effectively changed over to a “review period” of a week that infrequently involves some kind of call. The project still has to provide review “docuware” and the working theory is that interested parties from the community read this carefully prepared documentation and then theoretically pose questions that are theoretically answered. I don’t mean to sound overly theoretical, but my experience is that most of these documents are read by Anne and me, and few others outside of the project. Correct me if I’m wrong.

I’ve written in the past about how valuable I think these documents are. This hasn’t changed. I still think that it’s valuable to a project to capture what they’ve done leading up to the review. But I wonder if the energy that goes into this document could be better directed at something with a little more shelf-life. Like, perhaps, a “new and noteworthy” document for the release.

So here’s what I’m thinking. Keep in mind that we’re still very early in this process and that this is little more than an idea at this point.

At the beginning of a development cycle, projects are required to provide a project plan in the standard format. The project plan document has a handy spot for listing release dates. Through the plan, the release dates are easily accessible by interested parties (users, contributors, adopters, and the broader community). We can even parse out these dates and include them on the “What’s New” page and RSS feed.

So each project can use their plan (in the standard format as required by Board mandate) as the mechanism to notify the community of the release. This gives the community the project’s entire development cycle to get involved, ask questions, challenge assumptions, and what-not.

When a project is ready to complete their release (as documented in their project plan), they get IP Log approval, put the finishing touches on their “new and noteworthy” documentation, and open a bug that states that they’re ready to release. Either the project’s PMC or the EMO (or both) adds their +1 to the bug and Bob’s-your-uncle. The approval is required to ensure that the aforementioned principles (and documentation requirements) have been observed. By this point, however, approval should be little more than a rubber stamp since the PMC and EMO will have been aware of the pending release for some time.

Frankly, if the community hasn’t gotten involved by this point, I’m not sure how adding an additional week of review time will improve matters…

I like this approach because the two documents that are needed (the project plan and “new and noteworthy”) (a) are things a project should be doing anyway, (b) are, at least in the case of the project plan, actual requirements set down by the Eclipse Board, (c) convey the really important information that is contained in the review documentation, and (d) are immediately useful. Correct me if I’m wrong.

Again, this is just an idea right now. It’s something I’d like to explore. In my mind, this should be easier for projects and more valuable for the community. I believe that it is a natural evolution of the process.

Thoughts?

It seems natural to me that the EGit project is doing their developing using Git itself.

With this in mind, the EGit project has created and is managing their own Git instance on one of our vservers at egit.eclipse.org. With this server, the EGit project is helping us at the Eclipse Foundation understand how the Eclipse Development Process (EDP) applies to development in Git. So far, we haven’t noticed much of anything that needs to change in the EDP, but we’re certainly open to that possibility should change be necessary.

We’re particularly concerned about the IP issues surrounding contributions. I’ve been watching the project fairly closely to make sure that I’m comfortable that the Eclipse IP Due Diligence Process is being observed; and it is. Kudos are are certainly due to project lead Shawn Pearce for his patience in dealing with the many questions that I’ve thrown his way (and the small book he wrote for me in e-mail form).

We’ve already asked the EGit team to make a few changes to the way they’re doing things. They’ve fronted their access to Git through Gerrit Code Review which, frankly, looks pretty cool. We asked for changes, for example, that ensure that non-committer contributors explicitly agree to the Eclipse Terms of Use when submitting contributions directly through Gerrit. I like how Gerrit works, am I’m grateful to the EGit team for providing us with a real, live, functioning instance that we can observe and learn from.

Given that the EGit team has been following the principles prescribed by the EDP, I’ve asked them to start producing and distributing a build of their bundles in order to attract more testers and contributors. I’ve been running a build that I created myself a few weeks ago, and it all seems pretty functional. Don’t go looking for a “Git” perspective at this point as there is none. Instead, look to “Import” a repository.

For now, only EGit gets to use this repository (sorry). Having this live repository is an important part of our migration to Git. Denis has created a number of bugs (all dependencies of “master” Bug 257706) to track the various activities that need to be undertaken to make Git a real first-class source code repository solution at Eclipse. You’ll notice that Bug 293192, “Quality team provider and tooling for Git”, is one of the blockers. The EGit team is working hard to meet this requirement. We, as always, invite your participation to make this real.

Having a up-to-date (and correct) IP Log is important for projects and we’re trying to provide as much help as we can to make sure that they’re in order.

I’ve been working with a couple of projects to help them get their IP Logs in order. In the process, I’ve managed to generalize a “review” script that run queries against the Eclipse Bugzilla instance to help identify bugs and attachments that should potentially be referenced as contributions in a project’s IP Log. I’ve already used this script to identify a couple of contributions that I neglected to record for the Examples and Eclipse IDE for Education (ide4edu) projects.

The script is currently pretty rough. I haven’t made any linkage to it from any other eclipse.org page, so you have to manually modify the URL to make it work for your project. It could probably also stand to have a little more prose added to better describe what it provides. You can see the results of running the script for Mylyn, and Examples. To try other projects, change the id parameter value to the id of the project (e.g. tools.mylyn, or technology.examples). FWIW, it doesn’t work on eclipse.platform as it requires too much server memory and it bails out. Close to a decade’s worth of bug reports will do that for you.

Using this script to poke around a bit, I’ve noticed a few projects that still don’t quite get the iplog+ thing. Clearly, I need to do a better job of working with projects to get this right.

The main benefit of using the Bugzilla iplog flag is that we can use this information to automatically generate an IP Log for the project. However, the generation process is only as good as the data it’s provided with. Here are a few pointers:

• Only actual code contributions need to be included in the IP Log.
• Only contributions from non-committers need to be included in the IP Log
• Contributions that come from a developer before they become a committer do need to be included in the IP Log; the tools are smart enough to handle this time-sensitivity (so committer-provided contributions erroneously flagged as iplog+ are ignored).
• Ideally, attachments should be flagged as iplog+ rather than bugs (if an attachment is flagged, the bug itself almost certainly should not be).
• A bug might be flagged as iplog+ if the summary or one of the comments contains code that’s been rolled into the project’s source code repository.
• Marking the bug iplog+ means that the summary and every comment are potential contributions that will need to be manually filtered when the IP Log is generated.

The page that results from the “review” script is read-only. You can’t actually change the iplog flag’s value from the page. But there are handy links that take you right to where you need to be if you decide that you do need to flag something. Hopefully this makes it easier.

As always, I invite your feedback. If this page turns out to be useful, I’ll sort out how to better incorporate it into eclipse.org.

One of the things that I quite like about my new job at the Eclipse Foundation is the feeling that I can make a difference in the lives of committers, adopters, users, and the broader community. Actually… check that. It’s not me that’s making the difference. Individuals and project teams seem to be stepping up all over the place to make things better. Change is happening at the Foundation.

Builds have gotten better with the arrival of the Athena Common Builder. I know first-hand how much easier it is to get a build running as I’ve applied it to some of the projects that I’m working on. Numerous projects have adopted Athena and seem to be thriving on it. This is, of course, largely due to the tireless efforts of Nick Boldt (who has recently been brought into the Architecture Council). We’re still not to the point of having a single Ant or Maven script, but it is getting better. New work around build is on the horizon in the form of the B3 project.

I recall the first answer to the request for Git repository support at eclipse.org: “no way”. Today, we have a Git mirror of our CVS servers and are on our way toward implementing a proper Git repository for a few select projects sometime early in the new year. Denis “Scotty” Roy is, of course, working his usual miracles. It won’t be long before we have first-class support for Git both as a repository and as integration with the Eclipse workbench via the EGit project. On that note, first-class rock-solid Eclipse-based tools for Git is one of the conditions for full-blown Git support at eclipse.org; please get involved with the EGit project and help where you can.

Early discussions about Maven repositories at eclipse.org didn’t get very far. But, thanks to the good folks from the Buckminster project, we are planning to make the Helios release train available as both a p2 and a Maven repository. Over the past couple of weeks, I’ve been chatting with many folks in our community about creating a new über repository where projects that are not participating in the release train can make their code available to both p2 and Maven clients. The best part in all this is that it doesn’t require very much work for projects to participate: it’s looking like they have only to make their update sites known to the aggregator and they’re in. The aggregator will do important things like check dependencies and keep track of previous versions; these are important features for people who depend on bits staying where they are so that they can base real builds on them. Having an über repository will make it far easier for the broader community to find and use Eclipse technology. We’re still early in this process, but I remain hopeful and excited.

And darn it, we’re going to get Bug 280730 resolved.

It’s amazing what a little positive energy can bring.

The existing Automatic IP Log Tool and the new EMF-based one that I’m unveiling next week at Eclipse Summit Europe both make use of the iplog flag in Eclipse Bugzilla. After spending some quality time with the code that scours the various Eclipse Foundation databases to find contributions to a project, I’ve come to realize that many committers don’t know how exactly to use this flag. The log generation tools are only as good as the input they’re provided with, so I thought I’d spend a few minutes describing when, why, and how to use this handy flag.

You can set the iplog flag on an attachment. This is probably the most natural way to use the flag. To set this flag, open the “Details” for the attachment, set the value to “+”, and commit. That’s it. Do this for any attachment that contains something that you’ve committed into an Eclipse source code repository. This includes things like code patches, image files, XML schemas, and pretty much anything else. Only mark those attachments that actually make it into the repository; just leave any other attachments alone.

You can also set the iplog flag on an entire bug. You would do this if the bug’s summary or one of the comments contains code, or some other valuable asset that you commit into the source code repository. Unfortunately, flagging the bug indicates to the automated tools that every comment on the bug is a potential contribution (sadly, it appears that there is no way to set a flag on an individual comment). When you set the flag on the entire bug, you have to manually go through the generated log and remove any comments that shouldn’t be there. Yes, this can be painful. It’s made more painful by the fact that any new comments added to the bug after you’ve done your clean up will appear the next time the log is generated. This is particularly troublesome in the current Automatic IP Log Tool, and only slightly less so with the new one.

The bottom line is that it is always easier if your contributions come in the form of an attachment.

Only those attachments and bugs that contain contributions from developers who are not committers on the project at the time need be flagged. This is time-sensitive. If the contributor eventually does become a committer, those patches they’ve contributed while they were not a committer still need to be accounted for in the log and must therefore be flagged. The tools are smart enough to check the dates. Make sure that each committer’s Bugzilla email address matches the one in the Eclipse Foundation database, or you’ll wind up with some extra stuff in your IP Log.

Openness and transparency are two of the more important principles at Eclipse and outward communication is an important part of both. It is through outward communication that we let the community know what’s going on. I recently sent out an email, for example, declaring to the community that the Doc2Model project proposal is now available for review. This outward communication is an example of transparency as we are using to inform the community of what is happening. It is an example of openness, as it is intended to invite interested parties to bring their ideas and resources to the project. It is just one part of the communication story.

I’ve been thinking a lot about communication lately. More specifically, I’ve been thinking about the utility of sending notification emails. We send an email to notify the membership of new project proposals, up-coming reviews, and other such things. Are these emails actually useful? Do they end up in your “Junk” folder? Do you read them and say “I have got to get me one of those?”

Our “What’s new” page contains all the information you need to keep up with the project proposal and review activities. There’s even an RSS feed for it. I was starting to think that this was enough. But this morning, I received two responses to my recent email announcing the Doc2Model proposal. So, clearly somebody is reading the emails. I suppose that we could also set up a Twitter feed of some sort, and maybe blog more consistently about these events.

In any case, it’s all about the community. It’s all about making sure that everybody gets a chance to participate. I just don’t want to be part of the conspiracy.

I look forward to seeing everyone at Eclipse Summit Europe next week!

On Thursday this week (October 8/2009), I’m presenting Mylyn at the FOSSLC Eclipse Bootcamp in Ottawa. Doug Schaefer will be presenting “Eclipse CDT and developing for Android”, and Boris Bokowski will be introducing e4.

I wish this was happening after Eclipse Summit Europe. If it were after the Summit, I could apply some of the wisdom from Steffen Pingel’s Effective Mylyn talk to my own.

See you all at the Bootcamp and/or Eclipse Summit Europe