Community
Participate
Working Groups
We've encountered an interesting issue with broken index: instead of complain that something is not OK with the index data and rewrite the index Eclipse opened scary "OutOfMemory" dialog and asked user to exit workbench. I'm pretty sure that this was caused by our unreliable NFS file system, and the real problem was not checking the input arguments in HashtableOfObject (HashtableOfInt, HashtableOfLong, and all other HashtableOf<type> classes) before creating arrays. !ENTRY org.eclipse.jdt.core 4 4 2015-08-05 11:51:55.074 !MESSAGE Background Indexer Crash Recovery !STACK 0 java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:38) at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:643) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:464) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:456) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:550) at org.eclipse.jdt.internal.core.index.Index.save(Index.java:198) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:807) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:849) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:569) at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:388) at java.lang.Thread.run(Thread.java:745) file = file:/nfs_mounted_dirtectory/workspace/.metadata/.plugins/org.eclipse.jdt.core/588816551.index offset = 159 size = 1299541108 Example code demoing that the given size above will always crash VM at HashtableOfObject.java:38 where the char array is created as this.keyTable = new char[extraRoom][]; ######### public class Snippet { public static void main(String[] args) { int size; size = 1299541108; size = (int) (size * 1.75f); testVM(size); size = (int) (Integer.MAX_VALUE + 1); testVM(size); size = Integer.MAX_VALUE/4; testVM(size); size = Integer.MAX_VALUE/3; testVM(size); size = Integer.MAX_VALUE/2; testVM(size); size = Integer.MAX_VALUE - 1; testVM(size); size = Integer.MAX_VALUE; testVM(size); } private static void testVM(int size) { try { char[][] arr = new char[size][]; System.err.format("Successfully initialized an array with %,d elements.\n", size); } catch (Throwable t) { System.err.format("Failed to init an array with %,d elements: %s\n", size, t.getMessage() + " / " + t.getClass()); } } } ######### So it would be nice if JDT HashtableOf<type> code would not blindly accept any values from clients and pass them to the array constructors, but perform a minimal size checks and throw some more & user friendly looking exception. Looking at the code, HashtableOf<type> are widely used in JDT so changing the interface (or the behavior) seems to be not that easy. Also I think there are few duplicates of this bug which were falsely closed as not reproducible - see bug 268365 and bug 426390.
@Manoj: do you plan to follow up on this? The smallest possible fix would be to add checks for the "big" input values, throw IllagalArgumentException in that case, catch those in DiskIndex try/catch blocks and wrap to a user friendly IOException, saying that index read/write was failed.
(In reply to Andrey Loskutov from comment #1) > @Manoj: do you plan to follow up on this? > > The smallest possible fix would be to add checks for the "big" input values, > throw IllagalArgumentException in that case, catch those in DiskIndex > try/catch blocks and wrap to a user friendly IOException, saying that index > read/write was failed. I can take a look at this post M1 - here the catch is how big is "big" - so maybe instead of a blanket size check it may be better to do some sanity checks by cross-verifying - just thinking aloud - would take a look and see whether something can be done.
*** Bug 508860 has been marked as a duplicate of this bug. ***
Created attachment 268998 [details] broken (old 4.6.3) index Just occured on 4.6.3 again. java.lang.OutOfMemoryError: Requested array size exceeds VM limit Dumping heap to java_pid42266.hprof ... Heap dump file created [2560859371 bytes in 10.060 secs] java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:38) at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:663) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:466) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:458) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:560) at org.eclipse.jdt.internal.core.index.Index.save(Index.java:197) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:905) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:947) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:644) at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:377) at java.lang.Thread.run(Thread.java:748) -------------------- DEBUG -------------------- file = file:/data/eclipse_workspaces/ws-cpr_enhancements_stex/.metadata/.plugins/org.eclipse.jdt.core/3328416758.index offset = 4405 size = 1845493760 -------------------- END -------------------- Exception in thread "Java indexing" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:38) at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:663) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:466) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:458) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:560) at org.eclipse.jdt.internal.core.index.Index.save(Index.java:197) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:905) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:947) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:644) at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:377) at java.lang.Thread.run(Thread.java:748) java.lang.OutOfMemoryError: Requested array size exceeds VM limit Dumping heap to java_pid132963.hprof ... Heap dump file created [3175195918 bytes in 12.759 secs] java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:38) at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:663) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:466) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:458) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:560) at org.eclipse.jdt.internal.core.index.Index.save(Index.java:197) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:905) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:947) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:644) at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:377) at java.lang.Thread.run(Thread.java:748) -------------------- DEBUG -------------------- file = file:/data/eclipse_workspaces/ws-cpr_enhancements_stex/.metadata/.plugins/org.eclipse.jdt.core/2323372486.index offset = 26815 size = 1694498815 -------------------- END -------------------- Exception in thread "Java indexing" java.lang.OutOfMemoryError: Requested array size exceeds VM limit at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:38) at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:663) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:466) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:458) at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:560) at org.eclipse.jdt.internal.core.index.Index.save(Index.java:197) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:905) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:947) at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:644) at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:377) at java.lang.Thread.run(Thread.java:748)
Created attachment 268999 [details] Another broken 4.6.3 index @Manoj: any chance for 4.8? BTW, there was old request to re-create index (see bug 213702). Would be this an option?
(In reply to Andrey Loskutov from comment #5) > Created attachment 268999 [details] > Another broken 4.6.3 index > > @Manoj: any chance for 4.8? Will keep this in the radar for 4.8.
*** Bug 532901 has been marked as a duplicate of this bug. ***
For your information, in case it helps: After getting a corrupt index, I deleted completely my <workspace>/.metadata/.plugins/org.eclipse.jdt.core/ folder. After restart and opening the "Open Type" dialog, the index was rebuilt without error. But some time later I got the follow error while interacting with the "Open Type" dialog: eclipse.buildId=4.8.0.I20180322-0645 java.version=1.8.0_141 java.vendor=Oracle Corporation BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US Command-line arguments: -os linux -ws gtk -arch x86_64 org.eclipse.core.jobs Error Tue Mar 27 12:29:25 EDT 2018 An internal error occurred during: "Items filtering". java.lang.ArrayIndexOutOfBoundsException: 1633 at org.eclipse.jdt.internal.core.index.DiskIndex.readStreamChars(DiskIndex.java:944) at org.eclipse.jdt.internal.core.index.DiskIndex.readChunk(DiskIndex.java:748) at org.eclipse.jdt.internal.core.index.DiskIndex.readDocumentName(DiskIndex.java:800) at org.eclipse.jdt.internal.core.index.DiskIndex.addQueryResult(DiskIndex.java:154) at org.eclipse.jdt.internal.core.index.DiskIndex.addQueryResults(DiskIndex.java:239) at org.eclipse.jdt.internal.core.index.Index.query(Index.java:140) at org.eclipse.jdt.internal.core.search.matching.TypeDeclarationPattern.queryIn(TypeDeclarationPattern.java:361) at org.eclipse.jdt.core.search.SearchPattern.findIndexMatches(SearchPattern.java:2389) at org.eclipse.jdt.internal.core.search.matching.MatchLocator.findIndexMatches(MatchLocator.java:293) at org.eclipse.jdt.internal.core.search.PatternSearchJob.search(PatternSearchJob.java:114) at org.eclipse.jdt.internal.core.search.PatternSearchJob.execute(PatternSearchJob.java:69) at org.eclipse.jdt.internal.core.search.processing.JobManager.performConcurrentJob(JobManager.java:262) at org.eclipse.jdt.internal.core.search.BasicSearchEngine.searchAllTypeNames(BasicSearchEngine.java:1848) at org.eclipse.jdt.core.search.SearchEngine.searchAllTypeNames(SearchEngine.java:1199) at org.eclipse.jdt.internal.ui.dialogs.FilteredTypesSelectionDialog.fillContentProvider(FilteredTypesSelectionDialog.java:519) at org.eclipse.ui.dialogs.FilteredItemsSelectionDialog$FilterJob.filterContent(FilteredItemsSelectionDialog.java:2050) at org.eclipse.ui.dialogs.FilteredItemsSelectionDialog$FilterJob.internalRun(FilteredItemsSelectionDialog.java:1998) at org.eclipse.ui.dialogs.FilteredItemsSelectionDialog$FilterJob.doRun(FilteredItemsSelectionDialog.java:1970) at org.eclipse.ui.dialogs.FilteredItemsSelectionDialog$FilterJob.run(FilteredItemsSelectionDialog.java:1957) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:60)
Bulk move out of 4.9
Still occurs in 4.13: java.lang.ArrayIndexOutOfBoundsException: Index 1846 out of bounds for length 1846 at org.eclipse.jdt.internal.core.index.DiskIndex.readStreamChars(DiskIndex.java:947) at org.eclipse.jdt.internal.core.index.DiskIndex.readChunk(DiskIndex.java:751) at org.eclipse.jdt.internal.core.index.DiskIndex.readDocumentName(DiskIndex.java:803) at org.eclipse.jdt.internal.core.index.DiskIndex.addQueryResult(DiskIndex.java:157) at org.eclipse.jdt.internal.core.index.DiskIndex.addQueryResults(DiskIndex.java:211) at org.eclipse.jdt.internal.core.index.Index.query(Index.java:147) at org.eclipse.jdt.internal.core.search.matching.TypeDeclarationPattern.queryIn(TypeDeclarationPattern.java:364) at org.eclipse.jdt.core.search.SearchPattern.findIndexMatches(SearchPattern.java:2390) at org.eclipse.jdt.internal.core.search.matching.MatchLocator.findIndexMatches(MatchLocator.java:296) at org.eclipse.jdt.internal.core.search.PatternSearchJob.search(PatternSearchJob.java:117) at org.eclipse.jdt.internal.core.search.PatternSearchJob.execute(PatternSearchJob.java:72) at org.eclipse.jdt.internal.core.search.processing.JobManager.performConcurrentJob(JobManager.java:265) at org.eclipse.jdt.internal.core.search.BasicSearchEngine.searchAllTypeNames(BasicSearchEngine.java:1851) at org.eclipse.jdt.core.search.SearchEngine.searchAllTypeNames(SearchEngine.java:1100) at org.eclipse.jdt.internal.ui.InterfaceIndicatorLabelDecorator.addOverlaysWithSearchEngine(InterfaceIndicatorLabelDecorator.java:185) at org.eclipse.jdt.internal.ui.InterfaceIndicatorLabelDecorator.addOverlays(InterfaceIndicatorLabelDecorator.java:148) at org.eclipse.jdt.internal.ui.InterfaceIndicatorLabelDecorator.decorate(InterfaceIndicatorLabelDecorator.java:131) at org.eclipse.ui.internal.decorators.LightweightDecoratorDefinition.decorate(LightweightDecoratorDefinition.java:251) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager$LightweightRunnable.run(LightweightDecoratorManager.java:105) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.decorate(LightweightDecoratorManager.java:360) at org.eclipse.ui.internal.decorators.LightweightDecoratorManager.getDecorations(LightweightDecoratorManager.java:346) at org.eclipse.ui.internal.decorators.DecorationScheduler$1.ensureResultCached(DecorationScheduler.java:386) at org.eclipse.ui.internal.decorators.DecorationScheduler$1.run(DecorationScheduler.java:362) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
*** This bug has been marked as a duplicate of bug 578211 ***
