Bug 324177

Summary:

Several failures in three-way diff, detected using new test suite with an oracle

Product:

[Eclipse Project] Platform

Reporter:

Martin Johansen <MartinFagereng.Johansen>

Component:

Compare

Assignee:

Platform-Compare-Inbox <platform-compare-inbox>

Status:

NEW ---

QA Contact:

Severity:

minor

Priority:

Version:

4.0

Target Milestone:

---

Hardware:

All

OS:

All

Whiteboard:

Attachments:

Description	Flags
Detailed report on the test suite and oracle	none
The four java files of the test suite and oracle implementation	none

Description Martin Johansen

2010-09-01 06:58:07 EDT

Build Identifier: 3.5.2

As a project for a university course, I have written a test suite generator algorithm with an oracle to test Eclipse Compare's three-way diff algorithm.

I generated and ran 59136 tests resulting in the following summary:

Verdict        Number Percentage
--------------------------------
Fail            6110      10,33%
Inconclusive    2160       3,65%
Success        50866      86,02%
Total          59136     100,00%

I wrote a *very* slow diff3 implementatation which resulted in the following:

Verdict      Number percentage
------------------------------
Fail              0      0,00%
Inconclusive   1458      2,47%
Success       57678     97,53%
Total         59136    100,00%

So, the failures found can be fixed.

It is interesting to note that in build version 2010-05-31 the successes improved by 1.1% over 3.5.2:

Verdict    Number Percentage Improvement over 3.5.2
---------------------------------------------------
Fail         4146      7,01%                 33,54%
Inconclusive 3484      5,89%                 78,71%
Success     51506     87,10%                  1,10%
Total       59136    100,00%

See the report (report.pdf) for all the details. See the four attached java-files for the test suite and oracle implementations.

Reproducible: Always

Steps to Reproduce:
1. Use Eclipse for RCP/Plug-in Developers Galileo-SR2
2. Checkout version tag R3_5_2 of the following packages
 - org.eclipse.compare/plugins/org.eclipse.compare
 - org.eclipse.compare/plugins/org.eclipse.compare.core
 - org.eclipse.compare.tests
 - org.eclipse.core.tests.harness
 - org.eclipse.core.tests.resources
 - org.eclipse.test.performance
 - org.eclipse.test.performance.(your OS-identifier)
3. Install the test-suite found in the four attached java files into package org.eclipse.compare.test
4. Run the BBTestSuite.testRandomTests or BBTestSuite.testSystematically as a junit test
5. Read of the results in report.txt placed in the project root of org.eclipse.compare.tests

Comment 1 Martin Johansen

2010-09-01 06:59:56 EDT

Created attachment 177932 [details]
Detailed report on the test suite and oracle

Comment 2 Martin Johansen

2010-09-01 07:00:47 EDT

Created attachment 177933 [details]
The four java files of the test suite and oracle implementation

Comment 3 Tomasz Zarna

2010-09-01 07:54:51 EDT

What kind of files were you comparing? If those files were large our diffing algorithm probably used capping the get the results faster (bug 292831 is about one of such cases). We're currently working on providing a way to disable the capping in the algorithm (bug 307280).

Comment 4 Martin Johansen

2010-09-01 07:57:19 EDT

(In reply to comment #3)
> What kind of files were you comparing? If those files were large our diffing
> algorithm probably used capping the get the results faster (bug 292831 is about
> one of such cases). We're currently working on providing a way to disable the
> capping in the algorithm (bug 307280).

The files used was very small, less than 20 lines.

Comment 5 Eclipse Webmaster

2019-09-06 16:10:02 EDT

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.