Bug 220977 - Standard IP Log format automatically maintained
Summary: Standard IP Log format automatically maintained
Status: RESOLVED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Process (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P1 enhancement (vote)
Target Milestone: ---   Edit
Assignee: Bjorn Freeman-Benson CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 113717 (view as bug list)
Depends on: 113717 234059
Blocks: 119341 214690 231040 232604 233988 239316 239330
  Show dependency tree
 
Reported: 2008-02-29 12:41 EST by Bjorn Freeman-Benson CLA
Modified: 2008-10-29 15:04 EDT (History)
22 users (show)

See Also:


Attachments
mylyn/context/zip (4.30 KB, application/octet-stream)
2008-05-14 11:47 EDT, Bjorn Freeman-Benson CLA
no flags Details
work in progress (10.04 KB, patch)
2008-06-04 01:00 EDT, Bjorn Freeman-Benson CLA
no flags Details | Diff
portal screenshot showing truncated/abbreviated text and TONS of whitespace (84.80 KB, image/png)
2008-06-04 01:22 EDT, Nick Boldt CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bjorn Freeman-Benson CLA 2008-02-29 12:41:04 EST
A standard IP log format (XML?) that is automatically maintained/produced from the project meta-data, bugzillas, and CVS/SVN log information.
Comment 1 Bjorn Freeman-Benson CLA 2008-05-13 17:26:49 EDT
(1) /projects/ip_log.php?projectid=foo.bar is a Phoenix html page showing the current up-to-the-minute dynamically produced ip log for the project (see below)
(2) ?projectid=a,b,c,d
(3) database tables for the exceptions and extensions to the ip log (see below)
(4) portal boxes for emo and legal to manage the database tables
(5) updated ipzilla for the new CQ keywords from Section 3 below
(5) /projects/project_summary.php?projectid=foo.bar links to ip_log.php
(6) emo/legal double checks and merges all the existing ip logs they have received against the automated ip logs
(7) the project meta-data for ip logs are removed
(8) projects are informed of the new automated ip logs and requested to remove their old ip log files and to change all links to the new automated ip logs

The IP Log page has:
(a) a url way to produce an XML version of the page for program consumption: same information, different format
(b) a big giant header at the top that explains that this is dynamic information and not approved and vetted and to see the frozen version for the approved version for each release
(c) a title that includes all the projects aggregated into this ip log
(d) an explanation/link for the project team to correct incorrect information on the page
(e) a button on the page (or in the portal) to freeze a copy and submit it for review via ipzilla
(f) a user interface widget at the bottom of the page to define which projects in the hierarchy to include in the ip log. Selecting a subset and pressing "submit" refreshes the page with a new url of type (2)

The IP Log sections are:

Section 1: a list of all past and present committers who have ever written any lines of code for the project. This list will be generated from the list of all committers in the Foundation DB and the commits activity data from the dashboard.

Sub-Section 1A: a list of all the past and present committers who were/are active committers at one time but do not appear to have committed code. This is the list of people that perhaps our commits activity data collection is insufficient to track but who had the ability to affect the code. This includes an explanation of why the data might be inaccurate.

Exception 1B: if a project lead determines that, truly, a former committer's code has been completely removed from the project, he or she can request to the EMO to remove that person from the IP log. This will be a manual process (lead emails EMO, EMO updates a special database table of exceptions). If the project lead requests this, the request and result will be logged in the IP log, e.g., "Susan was listed as a committer but Zach said on April 1, 1900 that none of Susan code remains in the active source code repository".

Section 2: a list of all non-Committers whose contributions were added to the code base.  This is harder to track unless the projects are good about using bugzilla (most are). We will extract this by listing all the non-Committers people who have submitted patch files to bugs that are RESOLVED, VERIFIED, or CLOSED.

Exception 2B: again, the project lead can, on request to the EMO, exclude selected people from the list. The project lead will use this exception to handle cases where some random person submitted a patch and the bug was fixed, but that patch was not used.  Of course the project team could also "obsolete" the attachment - that will also remove it from section 2. The exception list will be Person x Bug# because someone might have submitted an accepted patch on one bug but not for another.

Section 3: a list of all the approved CQs. This is the set of all possible third-party libraries and dependencies that might be being used by the project. There are actually four lists:

3.1 Third-party libraries that are approved and in use.
3.2 Third-party libraries that were approved and used, but are no longer in use (the code that used them was removed)
	keyword: obsolete-and-no-longer-used
3.3 Third-party libraries that are approved, not currently in use, but might be in the future.
	keyword: not-currently-used
3.4 Third-party libraries that were approved but were never used at all and will not be used.
	keyword: withdrawn-and-never-used

Note 3A: the ip_log generator knows how to follow the (PB) references in ipzilla back to the original CQ. Thus if a CQ says "we want to reuse X that was approved in CQ 12", the ip log will show this CQ (I want to reuse) and below it will show the original CQ (X is approved).

Section 4: a list of all pending CQs. This list is useful when someone looks at one of these automated IP logs so that he or she can see whether there is live code in the repository that uses these pending CQs. Obviously the best situation for a release is an empty section 4. If a release occurs with a non-empty section 4, the ip log needs to include appropriate explanation, i.e., that the code referred to by the pending CQs is not in use or whatever.
Comment 2 Bjorn Freeman-Benson CLA 2008-05-13 23:49:47 EDT
*** Bug 113717 has been marked as a duplicate of this bug. ***
Comment 3 Bjorn Freeman-Benson CLA 2008-05-14 00:10:52 EDT
(5.5) update the documentation http://www.eclipse.org/projects/dev_process/project-log.php and http://www.eclipse.org/legal/committerguidelines.php
Comment 4 Bjorn Freeman-Benson CLA 2008-05-14 11:28:24 EDT
Completed: (1), (2), (b), (c), (d)
Remaining: (3), (4), (5), (5bis), (5.5), (6), (7), (8), (a), (e), (f)

e.g. http://www.eclipse.org/projects/ip_log.php?projectid=technology.dash
Comment 5 Bjorn Freeman-Benson CLA 2008-05-14 11:47:23 EDT
Created attachment 100210 [details]
mylyn/context/zip
Comment 6 Nick Boldt CLA 2008-05-14 12:29:34 EDT
Two nits & a question.

On http://www.eclipse.org/projects/ip_log.php?projectid=technology.dash, the page title is "tenative", not "tentative".

The link to "learn how this IP log is generated and how to correct errors" points to the wiki, which in turn points to bug 221934 instead of this bug (I've fixed the bug link).

Will this log eventually include actual bugs, eg., http://www.eclipse.org/modeling/gmf/project-info/ipquery.php
Comment 7 Bjorn Freeman-Benson CLA 2008-05-14 12:43:09 EDT
(In reply to comment #6)
> Will this log eventually include actual bugs, eg.,
> http://www.eclipse.org/modeling/gmf/project-info/ipquery.php

Yes - see comment #1 (section 3). I'm just making notes in this bug as I incrementally implement each item.
Comment 8 Bjorn Freeman-Benson CLA 2008-05-14 13:00:38 EDT
I should have said:
Completed: (1), (2), (b), (c), (d), 1, 1A
Remaining: (3), (4), (5), (5bis), (5.5), (6), (7), (8), (a), (e), (f), 1B, 2, 2B, 3, 3.1, 3.2, 3.3, 3.4, 3A, 4
Comment 9 Bjorn Freeman-Benson CLA 2008-05-14 14:23:01 EDT
Add: (2.5) ensure that all projects have a project info bugzilla/productname mapping to their correct bugzilla product name
Comment 10 Nick Boldt CLA 2008-05-14 18:49:25 EDT
(In reply to comment #7)
> (In reply to comment #6)
> > Will this log eventually include actual bugs, eg.,
> > http://www.eclipse.org/modeling/gmf/project-info/ipquery.php
> 
> Yes - see comment #1 (section 3). I'm just making notes in this bug as I
> incrementally implement each item.

IPzilla querying is great, but what about when a patch is under 250 lines and there's no CQ? Will you be querying Bugzilla too? I've hit a bit of a brick wall in that some contribs are not attached as patches, and some are attached not by the contributor but by the committer (presumably because the contribution came in via email, mailing list, or newsgroup).  

I have at least 7 bugs / contributions in EMF which don't conform to the 'must be a patch' rule: 

101163, 130525, 131811, 156783, 165770, 166967, 170204

And then there's these bugs, which were modified by a committer before being applied, so the last 'non-obsolete' attached patch is owned by the committer, not the contributor:

147594, 179004, 185971, 215131

Finally, there's the scenario where the email address that the committer THINKS is correct for the contributor has changed or is at least not the same as the one Bugzilla has on file for the patch:

132360, 149770
Comment 11 Bjorn Freeman-Benson CLA 2008-05-14 18:58:40 EDT
(In reply to comment #10)
> Will you be querying Bugzilla too? 

Yes.

> I have at least 7 bugs / contributions in EMF which don't conform to the 'must
> be a patch' rule: 

I'm curious: why are they not patches? Could you fix this by adding a final patch to the bug?

> And then there's these bugs, which were modified by a committer before being
> applied, so the last 'non-obsolete' attached patch is owned by the committer,
> not the contributor:

That should be fine.

> Finally, there's the scenario where the email address that the committer THINKS
> is correct for the contributor has changed or is at least not the same as the
> one Bugzilla has on file for the patch:

In the end, we have to maintain data somewhere. The idea behind this bug is that rather than maintaining the data once in ipzilla + bugzilla + foundation db and then once in some spreadsheet on the project lead's machine, we'll use just the one source of data. Obviously if that data is incorrect, we will need to update the data (in this case, the email address of the contributor as recorded in the bugzilla database).
Comment 12 Nick Boldt CLA 2008-05-14 19:07:19 EDT
> I'm curious: why are they not patches? 

Presumably because the contribution came in via email, mailing list, or newsgroup. Until this week there's been no stated process for capturing contributions.

> Could you fix this by adding a final patch to the bug?

If I must, yes. Does seem like the best solution, except for the whole "more work for me" aspect. :)

> In the end, we have to maintain data somewhere. The idea behind this bug is
> that rather than maintaining the data once in ipzilla + bugzilla + foundation
> db and then once in some spreadsheet on the project lead's machine, we'll use
> just the one source of data. Obviously if that data is incorrect, we will need
> to update the data (in this case, the email address of the contributor as
> recorded in the bugzilla database).

+10. I'm all for centralizing. Especially since it means that instead of having to do this annual audit, we can just update bugzilla and IPzilla and the IP stuff will magically flow out by itself.

Do you think any of the UI I'm playing with for Modeling will be of use to you?

http://www.eclipse.org/modeling/emf/project-info/ipquery.php?showobsolete
http://www.eclipse.org/modeling/emf/project-info/ipquery.php?showbuglist
http://www.eclipse.org/modeling/emf/project-info/ipquery.php?unformatted
Comment 13 Bjorn Freeman-Benson CLA 2008-05-14 19:15:56 EDT
(In reply to comment #12)
> If I must, yes. Does seem like the best solution, except for the whole "more
> work for me" aspect. :)

How about the "automatically produced ip logs = much less work for you, so the small extra work of reminding your contributors to open a patch nets out to 'less work'" ? :-)

> Do you think any of the UI I'm playing with for Modeling will be of use to you?

I'll look at, thanks.
Comment 14 Bjorn Freeman-Benson CLA 2008-05-14 19:53:46 EDT
Completed: (1), (2), (b), (c), (d), 1, 1A, 2
Remaining: (2.5), (3), (4), (5), (5bis), (5.5), (6), (7), (8), (a), (e), (f), 1B, 2B, 3, 3.1, 3.2, 3.3, 3.4, 3A, 4
Comment 15 John Arthorne CLA 2008-05-15 13:41:47 EDT
Having just gone through a manual update of the Equinox IP log, I can say that presence of a patch on a fixed bug is a poor indicator of whether a community contribution was accepted. People frequently submit patches that contain test cases that produce the bug, sample code that illustrates the problem, etc. Or, they submit a patch but it is not used and a different solution is arrived at, or only a portion of the patch is used in combination with additional work from a committer. 

To distinguish all these subtle cases, we in the Eclipse & Equinox projects use the "contributed" bugzilla keyword to indicate that a real community contribution was accepted and released. Having the tool check for this keyword (for those projects who use it), would be a good way to make it more reliable. Or, you could mandate use of this keyword for those who want to use the automated tool - otherwise you'll likely end up with auto-generated incorrect IP information for those who don't take the effort to make this determination. I realize your described solution has an exception process (2B) that allows appeal of the automated log, but my estimate is that the automated tool would have about 50% success rate and this exception process of the project lead needing to appeal to the EMO will prove cumbersome.
Comment 16 Richard Gronback CLA 2008-05-15 13:47:00 EDT
(In reply to comment #15)
> To distinguish all these subtle cases, we in the Eclipse & Equinox projects use
> the "contributed" bugzilla keyword to indicate that a real community
> contribution was accepted and released. Having the tool check for this keyword
> (for those projects who use it), would be a good way to make it more reliable.

Agreed.  We adopted this convention in GMF and are able to generate our log, as referenced in comment #6.
Comment 17 Nick Boldt CLA 2008-05-15 14:32:58 EDT
(In reply to comment #16)
> (In reply to comment #15)
> > To distinguish all these subtle cases, we in the Eclipse & Equinox projects use
> > the "contributed" bugzilla keyword to indicate that a real community
> > contribution was accepted and released. Having the tool check for this keyword
> > (for those projects who use it), would be a good way to make it more reliable.
> 
> Agreed.  We adopted this convention in GMF and are able to generate our log, as
> referenced in comment #6.

There's definitely a need for a way to override the generated content. In EMF, we only recently adopted this convention, so I've had to go back thru the old .csv log and in many cases, tag bugs with a bit of text to allow them to appear as contributions when there's:

* no patch (or an attachment of another non-patch type)

* no current patch (eg., the contributor's patch was overridden by the committer, who attached his modified patch to the bug and marked the original as obsolete)

I also lack a way to filter out committers from the query. For example this query subset (for just EMF Validation) [1] lists Christian (cdamus) as a contributor for his own component. This is invalid, since he's a committer (and has been since the inception of the component).

[1]http://www.eclipse.org/modeling/emf/project-info/ipquery.php?sortBy=bugid&component=Validation

But in this case [2], James (jbruck) was a contributor prior to becoming a committer / component lead, so some (or all?) of these are legitimate contributions by a then-contributor-now-committer.

[2]http://www.eclipse.org/modeling/mdt/project-info/ipquery.php?sortBy=contact&component=UML2

Comment 18 Bjorn Freeman-Benson CLA 2008-05-15 14:35:16 EDT
(in reply to comment #15 and comment #17)
Thanks John. (Note that if a portion of the patch is used, that still qualifies the person as an IP contributor.) I'll work the "contributed" keyword into this tool, but I also have to err on the side of including more information rather than less because it is better to have an IP log that over-declares contributions than one that under-declares them.  I.e., false positive/inclusion is better than false negative/exclusion.

Note to myself: Sub-Section 2A: list the "perhaps contributors" who have attached patches but the bugs are not marked with the "contributed" keyword.
Comment 19 Bjorn Freeman-Benson CLA 2008-05-15 15:42:06 EDT
(in reply to comment #17)
> no patch
There should be a patch for everything - there's got to be data somewhere to generate the ip log from and using patches as the data source seems reasonable, n'est pas? So if there is no patch, one should be added to the bug.

> (or an attachment of another non-patch type)
Could you explain more?  For example the person attaches an image/icon and that icon is included in the CVS (and thus is a contribution) - is that the idea?

> no current patch (eg., the contributor's patch was overridden by the committer, who attached his modified patch to the bug and marked the original as obsolete)
If the original is marked obsolete, my code already handles this case.

> I also lack a way to filter out committers
Fortunately I have a way to do this (just haven't checked in the code yet) :-)

(in reply to nobody, just a note to myself)
Completed: (1), (2), (b), (c), (d), (f), 1, 1A, 2
http://www.eclipse.org/projects/ip_log_selector.php
Remaining: 
(2.5) check existing bugzilla <--> meta data mappings
(3) database tables for the exceptions
(4) portal boxes for emo and legal
(5) updated ipzilla for the new CQ keywords
(5bis) project_summary.php links to ip_log.php
(5.5) update the documentation
(6) double check existing ip logs
(7) the project meta-data for ip logs are removed
(8) projects are informed of the new
(a) a url way to produce an XML version
(e) a button on the page (or in the portal) to freeze a copy and submit it for review via ipzilla
1B Committer exceptions
2A Contributor keyword exceptions
2B Contributor exceptions
3 approved CQs, 3.1, 3.2, 3.3, 3.4, 3A
4 pending CQs
Comment 20 John Arthorne CLA 2008-05-15 15:51:52 EDT
Just for reference, here are a few examples of potentially problematic bugs:

bug 221375 - attachments are icons, no code contributed (but still a noteworthy contribution)

bug 216050 - contribution is embedded in a comment rather than as a patch.

bug 226401 - in this case the contributor had difficulty with the patch wizard, and ended up attaching his complete copy of the files he modified (so attachment is not a patch).
Comment 21 John Arthorne CLA 2008-05-15 15:53:35 EDT
(the above are examples of "false negatives" that the tool will consider to not be a contribution, but the "contributed" keyword correctly flags as contributions)
Comment 22 Janet Campbell CLA 2008-05-15 16:58:17 EDT
I wanted to thank everyone for their comments.  I fully expect that the automation of the IP Logs will relieve a burden for both the project's and the IP Team both at simultaneous release time as well as throughout the year.  It'll also drive standardization making the logs easier to read, more consumable, and provide us with a basis to provide additional benefits to our members.  

That said, it'll obviously be an evolutionary process with improvements made over time.  I encourage your comments and suggestions, just ask for a little patience as we strive towards perfection.
Comment 23 Nick Boldt CLA 2008-05-15 17:42:27 EDT
(In reply to comment #19)
> (in reply to comment #17)
> > no patch
> n'est pas? So if there is no patch, one should be added to the bug.

Ah, but if the contribution came in directly to IPzilla or was written in the bug as a comment, then it's not a patch. (Granted, one could be added.)
 
> > (or an attachment of another non-patch type)
> Could you explain more?  For example the person attaches an image/icon and that
> icon is included in the CVS (and thus is a contribution) - is that the idea?

Yes, that's one option. The other option is that the attached text file was not flagged "isPatch" and thus is of type text/plain instead.
 
> > no current patch (eg., the contributor's patch was overridden by the committer, who attached his modified patch to the bug and marked the original as obsolete)
> If the original is marked obsolete, my code already handles this case.

Right, but does the contributor get credit? And what about the situation where a contributor submitted a patch, then someone else submitted a better one. Does the first contributor get credit if the committer ultimately only used the second one, and how does the tool know which one was used?
 
Comment 24 Bjorn Freeman-Benson CLA 2008-05-15 17:53:08 EDT
(In reply to comment #23)
> Right, but does the contributor get credit? And what about the situation where
> a contributor submitted a patch, then someone else submitted a better one. Does
> the first contributor get credit if the committer ultimately only used the
> second one, and how does the tool know which one was used?

From an IP Log point of view, it only matters whether IP has gotten into the code base.  If the original patch was not used in any IP way, then it would be marked obsolete and the original contributor would not be listed in the IP Log. (The project is, of course, welcome to acknowledge that contributor in other ways.)  If the original patch was used in some IP way (a piece of code, whatever), then the patch would not be marked obsolete and the contributor would be listed in the IP Log, even if the original patch was not used as a whole.
Comment 25 Bjorn Freeman-Benson CLA 2008-05-15 18:30:36 EDT
Completed: (1), (2), (b), (c), (d), (e), (f), 1, 1A, 2
Remaining: 
(2.5) check existing bugzilla <--> meta data mappings
(3) database tables for the exceptions
(4) portal boxes for emo and legal
(5) updated ipzilla for the new CQ keywords
(5bis) project_summary.php links to ip_log.php
(5.5) update the documentation
(6) double check existing ip logs
(7) the project meta-data for ip logs are removed
(8) projects are informed of the new
(a) a url way to produce an XML version
1B Committer exceptions
2A Contributor keyword exceptions
2B Contributor exceptions
3 approved CQs, 3.1, 3.2, 3.3, 3.4, 3A
4 pending CQs

And, of course, other improvements as suggested by comment #10, comment #12, comment #15, comment #17, comment #20, comment #23
Comment 26 Bjorn Freeman-Benson CLA 2008-05-15 19:55:41 EDT
Additional completed: 3, 4
Not going to do (at least not now): (a), 3A
Remaining: 
(2.5) check existing bugzilla <--> meta data mappings
(3) database tables for the exceptions
(4) portal boxes for emo and legal
(5) updated ipzilla for the new CQ keywords
(5bis) project_summary.php links to ip_log.php
(5.5) update the documentation
(6) double check existing ip logs
(7) the project meta-data for ip logs are removed
(8) projects are informed of the new
1B Committer exceptions
2A Contributor keyword exceptions
2B Contributor exceptions
3.1, 3.2, 3.3, 3.4 keywords for cq exceptions
other improvements
Comment 27 Nick Boldt CLA 2008-05-16 11:04:16 EDT
As requested by Bjorn, I'm copying this email here.

-\-\-

Currently, Modeling maintains their IP in one of two ways:

a) static .csv files, as per the old convention
b) generated from Bugzilla

To piggyback onto what Rich has already said re: GMF, the rest of 
Modeling (or at least I) would be happy to be placed in bucket #2, with 
Foundation-provided old log vs. new log reconciliation.

To that end, I've ported our old static .csv files over to use the 
GMF-style query tool, which can also export to the old .csv format if 
you prefer that markup (link on the right-hand side of the page).

http://www.eclipse.org/modeling/emf/project-info/ipquery.php?showobsolete
http://www.eclipse.org/modeling/emft/project-info/ipquery.php?showobsolete
http://www.eclipse.org/modeling/mdt/project-info/ipquery.php?showobsolete
http://www.eclipse.org/modeling/m2t/project-info/ipquery.php?showobsolete
http://www.eclipse.org/modeling/m2m/project-info/ipquery.php?showobsolete
 - and, of course -
http://www.eclipse.org/modeling/gmf/project-info/ipquery.php?showobsolete

-/-/-
Comment 28 Nick Boldt CLA 2008-05-16 12:16:08 EDT
Another scenario & limitation, reported by Jacques Lescot (EMFT - Ecore Tools):

1. the 'must be a patch' rule might be a problem for contributions concerning a complete new plugin (contributed as a zip file in general). For example, in the bug https://bugs.eclipse.org/bugs/show_bug.cgi?id=212896 I had to set the last contribution as a 'patch' whereas the MIME/type should be of type application/zip.
2. In the same bug referenced above, the original contribution was provided by an external person. However while I had to review it and as I contributed a revised one, my name appear in the IP Log page [1] (whereas I am not the real contributor of the contribution).

[1]http://www.eclipse.org/modeling/emft/project-info/ipquery.php
Comment 29 Martin Oberhuber CLA 2008-05-20 16:19:52 EDT
FYI, here's another corner case that we have had to deal with:

where a contributor submits some patches, becomes a committer afterwards, and then applies his own patches. In my understanding, these contributions must be logged because they were made before the contributor became committer, so they are under different legal terms.

Still, it's a (current) committer having applied the patches. Looks like the date when the patch was applied needs to be considered and checked against the date when a committer became a committer.
Comment 30 Martin Oberhuber CLA 2008-05-20 16:22:02 EDT
On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm the tool doesn't like the fact that our Bugzilla Project Name contains a space char, entered as encoded URL element %20
Comment 31 Martin Oberhuber CLA 2008-05-20 16:23:19 EDT
On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm , decommitterized people are shown as committers (e.g. Kushal Munir), shouldn't there be a separate section for those?
Comment 32 Martin Taal CLA 2008-05-22 17:30:59 EDT
Hi,
EMFT consists of multiple projects which (afaik) should send in their separate ip-log. However in the tool I can only select the modeling.emft component and not the individual EMFT projects. The tool therefore generates an ip-log for all EMFT projects in one run.

I am not sure if this is desirable/workable.

gr. Martin
Comment 33 Nick Boldt CLA 2008-05-22 18:10:22 EDT
(In reply to comment #32)
> EMFT consists of multiple projects which (afaik) should send in their separate
> ip-log. However in the tool I can only select the modeling.emft component and
> not the individual EMFT projects. The tool therefore generates an ip-log for
> all EMFT projects in one run.
> I am not sure if this is desirable/workable.

With the migration to standard groups (bug 198541) and the change that components == projects, this will hopefully no longer be a problem in that you'd be able to ask for modeling.emft.teneo rather than just modeling.emft. 

Bjorn or Karl: is that a correct assumption?
Comment 34 Bjorn Freeman-Benson CLA 2008-05-22 21:15:18 EDT
(In reply to comment #33)
> Bjorn or Karl: is that a correct assumption?

100% correct - thanks for helping out here.

(In reply to comment #32)
> EMFT consists of multiple projects 

Officially, until the Board approves the revised Development Process, EMFT is _one_ project. Thus the tool supports the official situation.  After the Board approval (and in conjunction with bug 198541), the tool will support the new official situation of multi-level project nesting. Thanks for your patience.
Comment 35 Jochen Krause CLA 2008-05-28 05:31:11 EDT
There seems to be a problem detecting some of the contributions, e.g.

 https://bugs.eclipse.org/bugs/show_bug.cgi?id=220477 is not showing up in our generated ip log.

Regarding the above mentioned code: Elias has contributed a patch but never shows up in the contributor list. There are more occurrences of this kind, and it seems that if one attachment from a contributor is marked as obsolete then he is excluded from the ip-log even if there are more attachments (patches) that would qualify as contribution.
Comment 36 Nick Boldt CLA 2008-05-28 09:48:19 EDT
(In reply to comment #35)
> There seems to be a problem detecting some of the contributions, e.g.
>  https://bugs.eclipse.org/bugs/show_bug.cgi?id=220477 is not showing up in our
> generated ip log.
> Regarding the above mentioned code: Elias has contributed a patch but never
> shows up in the contributor list. There are more occurrences of this kind, and
> it seems that if one attachment from a contributor is marked as obsolete then
> he is excluded from the ip-log even if there are more attachments (patches)
> that would qualify as contribution.

Sounds like we need a mechanism to show/hide obsoleted patches and to mark them "non-obsolete" or "still a contribution". 

For Modeling, we have two tricks to work around this problem:

1. tagging bugs with [contrib/] comments (details in the Data Inclusion box)

http://www.eclipse.org/modeling/emf/project-info/ipquery.php#Note

2. optionally showing obsolete patches (link in the Data Filters box)

http://www.eclipse.org/modeling/emf/project-info/ipquery.php?showobsolete

Comment 37 Bjorn Freeman-Benson CLA 2008-05-28 20:40:17 EDT
(In reply to many comments)
I am continuing to improve the tool (mostly based on all your suggestions and critiques). Two areas that are definitely being addressed:

1. A way for project leads to manage the incorrectness of the automated tool. In other words, if the tool incorrectly says X is a contributor, there will be a way for you project leads to fix that.  False positives and false negatives.

2. A way for the system to detect more of the corner cases. Obviously I'll never be able to catch them all, but with your help I should be able to find more of them.
Comment 38 Bjorn Freeman-Benson CLA 2008-05-28 20:42:01 EDT
From: Sharon Corbett [mailto:sharon.corbett@eclipse.org]
Sent: Wednesday, May 28, 2008 9:09 AM
Subject: RE: Automatic IP-Log - problem with state "approved"

We recently changed the state for approved CQs to simply "approved".
The automated ip log generator appears not to understand this new state.
Comment 39 Bjorn Freeman-Benson CLA 2008-06-03 21:12:27 EDT
(In reply to comment #31)
> On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm ,
> decommitterized people are shown as committers (e.g. Kushal Munir), shouldn't
> there be a separate section for those?

The IP Log shows all people who are, or were, committers, i.e., that they have/had commit rights to the project and actually wrote code for the project. Thus people who are decommitterized will still show up as "committers" because they were committers.
Comment 40 Bjorn Freeman-Benson CLA 2008-06-04 01:00:12 EDT
Created attachment 103493 [details]
work in progress

After working on putting the "fix errors in ip log" functionality in the portal, I've decided that it is the wrong user-interface approach. Tomorrow I will start over putting the user interface into the ip_log.php page itself, active only if the viewer is a logged-in committer to the project. Putting it in the portal provides nice documentation and tests, but no amount of wishful thinking made it a good user interface to be looking at the data in one place and modifying it in another.
Comment 41 Nick Boldt CLA 2008-06-04 01:22:19 EDT
Created attachment 103496 [details]
portal screenshot showing truncated/abbreviated text and TONS of whitespace

(In reply to comment #40)
> [T]he portal provides [...] no [...] good user interface[.]

While I admit mangling your quote out of context like that is pretty dodgy, I'm afraid I must nonetheless agree. Maintaining any sort of project metadata is a pain in the butt in the portal. See screenshot.

Do you think this approach of editing in-place might be extended to allow pages like http://www.eclipse.org/projects/project_summary.php?projectid=modeling.emf to also be editors for their content?

Will the portal code be made available so patches can be submitted?
Comment 42 Karl Matthias CLA 2008-06-04 12:17:47 EDT
(In reply to comment #41)
> Created an attachment (id=103496) [details]
> portal screenshot showing truncated/abbreviated text and TONS of whitespace
> 
> (In reply to comment #40)
> > [T]he portal provides [...] no [...] good user interface[.]
> 
> While I admit mangling your quote out of context like that is pretty dodgy, I'm
> afraid I must nonetheless agree. Maintaining any sort of project metadata is a
> pain in the butt in the portal. See screenshot.

This is the second new interface for the meta-data editor.  Both of which have been massive improvements over editing raw XML, as was done previously.  We'd like to continue to improve things.  Since the Portal is not at the moment open source, please submit constructive suggestions and we will see what we can do.  Perhaps I'm the problem as I don't see anything wrong in your attached screen shot.  Press "edit" next to the field that concerns you and you will see all of your data with no added whitespace.
Comment 43 Bjorn Freeman-Benson CLA 2008-06-04 16:28:08 EDT
(In reply to comment #41)
> Do you think this approach of editing in-place might be extended to allow pages
> like http://www.eclipse.org/projects/project_summary.php?projectid=modeling.emf
> to also be editors for their content?

Yes, but could you open a separate bug for that?  Thanks.
Comment 44 Nick Boldt CLA 2008-06-04 17:05:35 EDT
(In reply to comment #43)
> Yes, but could you open a separate bug for that?  Thanks.

Done. Bug 235713.
Comment 45 Bjorn Freeman-Benson CLA 2008-06-18 21:33:38 EDT
The automated IP log is 90% complete and ready to use. I say "90% complete" because I am in the process of converting all the Ganymede projects' as-submitted IP logs to the automated format, i.e., adding iplog flags to bugs, etc. As I worked my way through these IP logs, I am/will find corner cases that the automated IP log does not handle. Thus I cannot say "100% complete" until I have found and resolved all those corner cases.

However, that aside, the automated IP log is ready for use by all projects requesting Release Reviews from here on.
Comment 46 Bjorn Freeman-Benson CLA 2008-06-20 12:41:17 EDT
Current status part 1 of 2:

(In reply to comment #17)
> There's definitely a need for a way to override the generated content. 

Any committer on the project can now correct the automatically generated content. The corrections persist in the database and thus only need to be entered once.

> But in this case [2], James (jbruck) was a contributor prior to becoming a
> committer / component lead, so some (or all?) of these are legitimate
> contributions by a then-contributor-now-committer.

The automated IP log algorithm considers the dates that a person is a committer versus a contributor.

(In reply to comment #20)
> bug 221375 - attachments are icons, no code contributed
> bug 226401 - in this case the contributor had difficulty with the patch 

These are handled via an "iplog+" flag on the attachment.

> bug 216050 - contribution is embedded in a comment rather than as a patch.

This is handled via an "iplog+" flag on the bug itself.

(In reply to comment #28)
> 1. the 'must be a patch' rule might be a problem for contributions concerning 

The attachments are no longer required to be patches. Using the new "iplog+" flag, any attachment can be flagged for inclusion in the IP log.

(In reply to comment #29)
> where a contributor submits some patches, becomes a committer afterwards, and
> then applies his own patches. In my understanding, these contributions must be
> logged because they were made before the contributor became committer, so they
> are under different legal terms.

My understanding is that the date that counts is when the code was committed to the repository, so under that interpretation, this would be "committer commits code". The automated IP log handles this case.

Comment 47 Bjorn Freeman-Benson CLA 2008-06-20 12:43:38 EDT
Current status part 2 of 2:

These three cases need to be handled:

(In reply to comment #28)
> 2. In the same bug referenced above, the original contribution was provided by
> an external person. However while I had to review it and as I contributed a
> revised one, my name appear in the IP Log page [1] (whereas I am not the real
> contributor of the contribution).

(In reply to comment #30)
> On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm the tool
> doesn't like the fact that our Bugzilla Project Name contains a space char,

(In reply to comment #36)
> 1. tagging bugs with [contrib/] comments (details in the Data Inclusion box)

And this case needs to be checked (now that the code has changed):

(In reply to comment #35)
> There seems to be a problem detecting some of the contributions, e.g.
Comment 48 Martin Oberhuber CLA 2008-06-30 09:47:05 EDT
(In reply to comment #47)
> > On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm the tool
> > doesn't like the fact that our Bugzilla Project Name contains a space char,

The error messages have changed since the latest incarnation of Portal Metadata which now allow for specifying a bugzilla product name with space char: now, the IP Log whines

"Error: the project meta-data key 'bugzilla/components' "Core" is not a valid Component name in the "Target Management" Product name in bugzilla; use the portal  "[maintain] project info meta-data" to enter the correct bugzilla Component name(s) for dsdp.tm"
Comment 49 Martin Oberhuber CLA 2008-10-07 07:14:46 EDT
(In reply to comment #48)
(In reply to comment #47)
> (In reply to comment #30)
> > On http://www.eclipse.org/projects/ip_log.php?projectid=dsdp.tm the tool
> > doesn't like the fact that our Bugzilla Project Name contains a space char,

This issue seems to be fixed now, we're investigating converting the dsdp.tm IP Log to the automated one as per bug 239316. From my point of view, I'd think that this whole bug can be marked fixed now (probably entering new bugs for work yet to be done).
Comment 50 Bjorn Freeman-Benson CLA 2008-10-29 15:04:49 EDT
Per comment #48, closing this bug.