Comment 1 Frederic Fusier 2008-01-15 12:48:07 EST

I'm currently investigating...

Comment 2 Frederic Fusier 2008-01-15 12:55:34 EST

It looks like we are at the memory limit after having created all the tests:
1.5 -> 20,425,680
1.6 -> 24,319,280
1.7 -> 27,949,912

This memory consumption is due to the number of tests created for all the JDT/Core test suites:
1.5 -> 28,456
1.6 -> 36,691
1.7 -> 44,926

The problem comes from the fact that rt.jar file is indexed at the beginning of the tests and so, need some memory to run properly. It seems that less memory room due the additional created tests combined with more classes to index as the rt.jar is bigger make the tests run reaching the limit when the default memory is used...

Comment 3 Frederic Fusier 2008-01-15 13:16:07 EST

The obvious workaround is to set -Xmx128M in the VM Arguments of the launch configuration, but looking at AddJarFileToIndex, it seems that all entries of the zip file are processed in one shot.

I think the loop should be split when the memory threshold may be too low. Something like MatchLocator.MAX_AT_ONCE used in locateMatches(JavaProject, PossibleMatchSet, int).

Kent, any thought about this?

Comment 4 Kent Johnson 2008-01-15 13:38:12 EST

How are you going to know that the memory threshold is too low ?

Do we know how many 'index document' jobs are created from this loop :

// iterate each entry to index it
ZipEntry ze = (ZipEntry) e.nextElement();
if (Util.isClassFileName(ze.getName())) {
	final byte[] classFileBytes = Util.getZipEntryByteContent(ze, zip);
	JavaSearchDocument entryDocument = new JavaSearchDocument(ze, zipFilePath, classFileBytes, participant);
	this.manager.indexDocument(entryDocument, participant, index, this.containerPath);
}


Or how much space is used by the classFileBytes ?


It will be tricky to index the first 100 classfiles & defer the rest. Need to reopen the jar file, etc.

Could always look into saving space or reading the classFileBytes later...

Comment 5 Frederic Fusier 2008-01-17 09:19:38 EST

(In reply to comment #4)
> How are you going to know that the memory threshold is too low ?
> 
Using Runtime *Memory methods

> Do we know how many 'index document' jobs are created from this loop :
> 
> // iterate each entry to index it
> ZipEntry ze = (ZipEntry) e.nextElement();
> if (Util.isClassFileName(ze.getName())) {
>         final byte[] classFileBytes = Util.getZipEntryByteContent(ze, zip);
>         JavaSearchDocument entryDocument = new JavaSearchDocument(ze,
> zipFilePath, classFileBytes, participant);
>         this.manager.indexDocument(entryDocument, participant, index,
> this.containerPath);
> }
> 
None, the indexing of class files does not create any new job.

> 
> Or how much space is used by the classFileBytes ?
> 
The problem does not come from the class file bytes as the entryDocument is transient and so, garbaged when more room is necessary. The problem comes from the MemoryIndex which is always growing during this loop and is purged only when the index is saved.

> 
> It will be tricky to index the first 100 classfiles & defer the rest. Need to
> reopen the jar file, etc.
> 
> Could always look into saving space or reading the classFileBytes later...
> 
The solution Jerome proposed and I'm working on is to save the index inside the loop to purge the MemoryIndex and release memory. The first tries I made with this strategy avoided to get the OOME with default VM arguments.

Comment 6 Frederic Fusier 2008-01-17 09:26:22 EST

Created attachment 87154 [details]
Proposed patch

With this patch all JDT/Core pass on a 1.7 VM using only a heap size of 48M!

Comment 7 Kent Johnson 2008-01-17 09:55:04 EST

Ok but the save is quite expensive & merging the additional class files into a partial index is also time consuming.


You may save some space but at what expense ? 3x slower to index ?

Comment 8 Frederic Fusier 2008-01-17 12:15:51 EST

This will be surely slower, I have to measure to be more precise, but otherwise you get an OOME. So, there's no other choice to have a chance to finish the indexing in such a case...

Or, we can prefer get the OOME and so, have a chance to know that it's time to change the VM Heap Size value.

Comment 9 Frederic Fusier 2008-01-21 11:29:56 EST

The solution should first wait for the OOME and catch it before entering into this desperate mode... ;-)

Any way, set currently as P5 because even with this fixed, default memory would be not enough to finish all JDT/Core tests. There would still be an OOME in the last part of the tests.

So, the suggested workaround for now it's just to increase the VM Java Heap size while running the tests on a VM >= 1.6

Comment 10 Markus Keller 2009-04-20 11:09:58 EDT

> The solution should first wait for the OOME and catch it before entering into
> this desperate mode... ;-)

I hope the ;-) means: "Not really, since you never know in which thread the OOME occurs."

But you could try something with a SoftReference and write the index to disk when the reference gets queued and memory is getting low.

Comment 11 Frederic Fusier 2009-04-21 03:54:32 EDT

(In reply to comment #10)
> > The solution should first wait for the OOME and catch it before entering into
> > this desperate mode... ;-)
> 
> I hope the ;-) means: "Not really, since you never know in which thread the
> OOME occurs."
> 
No, the ;-) just meant: "of course, we'll never do that!". Because, in this peculiar case, we definitely knows in which thread the OOME occurs: the Java Indexing one (see the stack trace)... So, one might have been tempted with this so simple solution...

> But you could try something with a SoftReference and write the index to disk
> when the reference gets queued and memory is getting low.
> 
Agreed that it would be a prettier solution rather than the proposed patch, but as Kent said in comment 7, the save operation would be quite expensive, hence we currently prefer let the OOME happen...

Comment 12 Frederic Fusier 2009-04-21 10:50:39 EDT

*** Bug 272655 has been marked as a duplicate of this bug. ***

Comment 13 Eclipse Genie 2019-10-25 19:03:37 EDT

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.