Bug 551356 - ArrayIndexOutOfBoundsException in MementoTokenizer.nextToken
Summary: ArrayIndexOutOfBoundsException in MementoTokenizer.nextToken
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.14   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: 4.14 M1   Edit
Assignee: Stephan Herrmann CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-09-23 02:26 EDT by Christian Dietrich CLA
Modified: 2019-10-04 17:36 EDT (History)
3 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Christian Dietrich CLA 2019-09-23 02:26:12 EDT
Have a a gradle buildship based project where we call

IJavaElement javaElement = JavaCore.create(containerHandle);

with containerhandle = 

=org.xtext.example.mydsl2/\/home\/dietrich\/.gradle\/caches\/modules-2\/files-2.1\/org.eclipse.xtext\/org.eclipse.xtext.xtext.generator\/2.20.0-SNAPSHOT\/d13f21d195a538d55759b1fc6f9c7fe9c288f50b\/org.eclipse.xtext.xtext.generator-2.20.0-SNAPSHOT.jar=/gradle_used_by_scope=/

this no longer works and crashes with a ArrayIndexOutOfBoundsException in MementoTokenizer.nextToken

java.lang.ArrayIndexOutOfBoundsException: 273
	at org.eclipse.jdt.internal.core.util.MementoTokenizer.nextToken(MementoTokenizer.java:57)
	at org.eclipse.jdt.internal.core.JavaProject.getHandleFromMemento(JavaProject.java:2008)
	at org.eclipse.jdt.internal.core.JavaElement.getHandleFromMemento(JavaElement.java:348)
	at org.eclipse.jdt.internal.core.JavaModel.getHandleFromMemento(JavaModel.java:159)
	at org.eclipse.jdt.internal.core.JavaElement.getHandleFromMemento(JavaElement.java:348)
	at org.eclipse.jdt.core.JavaCore.create(JavaCore.java:3504)
	at org.eclipse.jdt.core.JavaCore.create(JavaCore.java:3479)
	at org.eclipse.xtext.ui.containers.JavaProjectsStateHelper.initContainedURIs(JavaProjectsStateHelper.java:81)

the array looks like

[=, o, r, g, ., x, t, e, x, t, ., e, x, a, m, p, l, e, ., m, y, d, s, l, 2, /, \, /, h, o, m, e, \, /, d, i, e, t, r, i, c, h, \, /, ., g, r, a, d, l, e, \, /, c, a, c, h, e, s, \, /, m, o, d, u, l, e, s, -, 2, \, /, f, i, l, e, s, -, 2, ., 1, \, /, o, r, g, ., e, c, l, i, p, s, e, ., x, t, e, x, t, \, /, o, r, g, ., e, c, l, i, p, s, e, ., x, t, e, x, t, ., x, t, e, x, t, ., g, e, n, e, r, a, t, o, r, \, /, 2, ., 2, 0, ., 0, -, S, N, A, P, S, H, O, T, \, /, d, 1, 3, f, 2, 1, d, 1, 9, 5, a, 5, 3, 8, d, 5, 5, 7, 5, 9, b, 1, f, c, 6, f, 9, c, 7, f, e, 9, c, 2, 8, 8, f, 5, 0, b, \, /, o, r, g, ., e, c, l, i, p, s, e, ., x, t, e, x, t, ., x, t, e, x, t, ., g, e, n, e, r, a, t, o, r, -, 2, ., 2, 0, ., 0, -, S, N, A, P, S, H, O, T, ., j, a, r, =, /, g, r, a, d, l, e, _, u, s, e, d, _, b, y, _, s, c, o, p, e, =, /]

and the tokenizer seems incable of handling it.
i have no idea where the problem is

Xtext / Buildship / JDT (no changes on Xtext side)

do you have any clue what happes?
Comment 1 Christian Dietrich CLA 2019-09-23 02:38:27 EDT
maybe this is a buildship issue

the gradle_used_by_scope list is empty,
maybe the problem is that jdt does not treat this case correct
Comment 2 Christian Dietrich CLA 2019-09-23 02:40:12 EDT
mementos like this one see to work

=org.xtext.example.mydsl2/\/home\/dietrich\/.gradle\/caches\/modules-2\/files-2.1\/org.eclipse.xtext\/org.eclipse.xtext.xbase\/2.20.0-SNAPSHOT\/2002581a29987364e0d0d33e4f6a61016bed13bf\/org.eclipse.xtext.xbase-2.20.0-SNAPSHOT.jar=/gradle_used_by_scope=/main,test
Comment 3 Stephan Herrmann CLA 2019-09-23 06:22:07 EDT
I will check if JDT can encode empty attribute values in a way which it can later decode.
Comment 4 Donat Csikos CLA 2019-09-24 03:22:55 EDT
It's a valid use-case for Buildship to have an empty scope field. I'd be surprised if this caused any problems on our side, but I'll take a look.
Comment 5 Eclipse Genie CLA 2019-09-24 14:38:11 EDT
New Gerrit change created: https://git.eclipse.org/r/150064
Comment 6 Stephan Herrmann CLA 2019-09-24 14:39:33 EDT
(In reply to Eclipse Genie from comment #5)
> New Gerrit change created: https://git.eclipse.org/r/150064

Our encoding of classpath attributes was indeed insufficient, I had to add a trailing delimiter.

Thanks for bringing this up, Christian.
Comment 8 Stephan Herrmann CLA 2019-09-24 16:47:37 EDT
(In reply to Eclipse Genie from comment #7)
> Gerrit change https://git.eclipse.org/r/150064 was merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> ?id=79083c3cdeb2a4d0d81bf944be0d26b02be61ef5

Released for 4.14 M1
Comment 9 Andrey Loskutov CLA 2019-09-25 03:05:26 EDT
(In reply to Eclipse Genie from comment #7)
> Gerrit change https://git.eclipse.org/r/150064 was merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> ?id=79083c3cdeb2a4d0d81bf944be0d26b02be61ef5

This causes massive regression in my existing SDK workspace, I saw error dialog and ~20 errors on startup.

Two different error types:

1) Without stack trace from PDE API tooling (3-4 times):

org.eclipse.pde.api.tools
Error
Wed Sep 25 08:58:39 CEST 2019
Unable to restore package: =org.eclipse.osgi/osgi\/src=/ignore_optional_problems=/true<org.osgi.framework.launch
...
Unable to restore package: =org.eclipse.jgit.junit.http/src=/test=/true<org.eclipse.jgit.junit.http
...
Unable to restore package: =org.eclipse.swt/Eclipse SWT PI\/gtk=/org.eclipse.jdt.launching.CLASSPATH_ATTR_LIBRARY_PATH_ENTRY=/org.eclipse.swt.gtk.linux.x86_64<org.eclipse.swt


2) With stack trace, repeats with different offsets ~20 times (I guess for every working set I have):

eclipse.buildId=4.14.0.I20190924-1800
java.version=11.0.2
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -data /data/4x_platform_workspace -os linux -ws gtk -arch x86_64

org.eclipse.ui.workbench
Error
Wed Sep 25 08:58:12 CEST 2019
Problems occurred when invoking code from plug-in: "org.eclipse.ui.workbench".

java.lang.ArrayIndexOutOfBoundsException: Index 66 out of bounds for length 66
	at org.eclipse.jdt.internal.core.util.MementoTokenizer.nextToken(MementoTokenizer.java:57)
	at org.eclipse.jdt.internal.core.util.MementoTokenizer.getStringDelimitedBy(MementoTokenizer.java:159)
	at org.eclipse.jdt.internal.core.JavaProject.getHandleFromMemento(JavaProject.java:2013)
	at org.eclipse.jdt.internal.core.JavaElement.getHandleFromMemento(JavaElement.java:348)
	at org.eclipse.jdt.internal.core.JavaModel.getHandleFromMemento(JavaModel.java:159)
	at org.eclipse.jdt.internal.core.JavaElement.getHandleFromMemento(JavaElement.java:348)
	at org.eclipse.jdt.core.JavaCore.create(JavaCore.java:3504)
	at org.eclipse.jdt.core.JavaCore.create(JavaCore.java:3479)
	at org.eclipse.jdt.internal.ui.PersistableJavaElementFactory.createElement(PersistableJavaElementFactory.java:63)
	at org.eclipse.ui.internal.WorkingSet$1.run(WorkingSet.java:168)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
	at org.eclipse.ui.internal.WorkingSet.restoreWorkingSet(WorkingSet.java:163)
	at org.eclipse.ui.internal.AbstractWorkingSet.getElementsArray(AbstractWorkingSet.java:166)
	at org.eclipse.ui.internal.AbstractWorkingSet.getElements(AbstractWorkingSet.java:154)
	at org.eclipse.jdt.internal.ui.workingsets.WorkingSetModel.isSupportedAsTopLevelElement(WorkingSetModel.java:684)
	at org.eclipse.jdt.internal.ui.workingsets.WorkingSetModel.getActiveAndAllWorkingSetsFromManagers(WorkingSetModel.java:402)
	at org.eclipse.jdt.internal.ui.workingsets.WorkingSetModel.restoreState(WorkingSetModel.java:607)
	at org.eclipse.jdt.internal.ui.workingsets.WorkingSetModel.<init>(WorkingSetModel.java:268)
	at org.eclipse.jdt.internal.ui.packageview.PackageExplorerPart$9.run(PackageExplorerPart.java:1477)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:45)
	at org.eclipse.jdt.internal.ui.packageview.PackageExplorerPart.createWorkingSetModel(PackageExplorerPart.java:1474)
	at org.eclipse.jdt.internal.ui.packageview.PackageExplorerPart.init(PackageExplorerPart.java:408)
	at org.eclipse.ui.internal.ViewReference.initialize(ViewReference.java:128)
	at org.eclipse.ui.internal.e4.compatibility.CompatibilityPart.create(CompatibilityPart.java:340)
Comment 10 Andrey Loskutov CLA 2019-09-25 04:12:03 EDT
(In reply to Andrey Loskutov from comment #9)
> (In reply to Eclipse Genie from comment #7)
> > Gerrit change https://git.eclipse.org/r/150064 was merged to [master].
> > Commit:
> > http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> > ?id=79083c3cdeb2a4d0d81bf944be0d26b02be61ef5
> 
> This causes massive regression in my existing SDK workspace, I saw error
> dialog and ~20 errors on startup.

Re-starting the IDE didn't show the same errors again (and I don't see any issues with existing working sets), so probably the change is simply incompatible with existing workspaces.

PS The PDE API errors are coming from org.eclipse.pde.api.tools.internal.ApiDescriptionManager.restoreNode(), so I assume that could cause some strange API issues.
Comment 11 Stephan Herrmann CLA 2019-09-25 09:55:10 EDT
If some plugin persists handle mementos produced by JDT between bug 466299 and this bug then, yes, some incompatibility can occur. Going from latest release to HEAD should not produce this issue, since the new element in the memento is optional, so old mementos are still valid.

Are you saying after re-start all is OK?

If the problem were in JDT build state, I would simply increase the version of that state, but for API tools I don't now how to force ignoring persistent handle mementos.
Comment 12 Andrey Loskutov CLA 2019-09-25 10:03:14 EDT
(In reply to Stephan Herrmann from comment #11)
> If some plugin persists handle mementos produced by JDT between bug 466299
> and this bug then, yes, some incompatibility can occur. Going from latest
> release to HEAD should not produce this issue, since the new element in the
> memento is optional, so old mementos are still valid.

OK.

> Are you saying after re-start all is OK?

Yep.

> If the problem were in JDT build state

I haven't checked this.
Comment 13 Stephan Herrmann CLA 2019-09-25 17:18:31 EDT
(In reply to Andrey Loskutov from comment #12)
> (In reply to Stephan Herrmann from comment #11)
> > If some plugin persists handle mementos produced by JDT between bug 466299
> > and this bug then, yes, some incompatibility can occur. Going from latest
> > release to HEAD should not produce this issue, since the new element in the
> > memento is optional, so old mementos are still valid.
> 
> OK.
> 
> > Are you saying after re-start all is OK?
> 
> Yep.
> 
> > If the problem were in JDT build state
> 
> I haven't checked this.

I had searched the code around persistent JDT build state before, and could not find any way how this memento could end up there.

With this, @Andrey, do you still see a need for action? Perhaps just a mail to eclipse-dev with the hint to restart, if this happens?
Comment 14 Stephan Herrmann CLA 2019-09-26 17:17:01 EDT
Re-closing on the assumption that all trouble from comment 9 ff. is just the result from updating from one I-build to another, and will not even affect milestone-based updates, lest the release.

Plus, workaround exists: restart Eclipse.
Comment 15 Christian Dietrich CLA 2019-09-28 02:49:11 EDT
thx for the fix Stephan