Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jdt-dev] Performance issues using IType interface

Hi, Jay

I attach sample scanner for external jars from classpath.
Method ClasspathScanner.buildJars(String[] externalJars, IJavaProject javaProject) iterates jars and all IType objects in them. It checks IType.isInterface() and then does some specific work for interfaces and other types. After IType.isInterface() is once called, then yes, the Java model caches element info for the type and all subsequent queries go fast. But it looks like at the moment of this very first call there is no cached element info yet, and Java model builds it by reading jar entry. Typical stack trace is given in attached Stacktrace.txt - it starts at my ClasspathScanner.buildJars goes through JavaModelManager.getZipFile and reaches java.util.zip.ZipFile.open. So, for every type new ZipFile object is created and opened.

Method ClasspathScanner.buildJarsImproved(String[] externalJars, IJavaProject javaProject) wraps the call to buildJars() with JavaModelManager.cacheZipFiles() and JavaModelManager.flushZipFiles() (I took this pattern from org.eclipse.jdt.internal.core.hierarchy.IndexBasedHierarchyBuilder.build(boolean)). In this case ZipFile is created only once for a jar and scan goes 100 times faster for large jars.

Yes, if some other code had scanned a jar before my scanner, I would not need wrapper method buildJarsImproved(), but it seems that my builder going after Java builder finds jars unscanned yet by another party.

I see that I might start my scan with a call to org.eclipse.jdt.core.JavaCore.newTypeHierarchy(IRegion,
>> WorkingCopyOwner, IProgressMonitor)
thus forcing initialization of Java model for jars to be scanned. But that does not look nice, to run intentionally some big unnecessary job to use its side effect as a condition of good performance of my own code. I would prefer an API method that would wrap my scan as I do it in buildJarsImproved().

Best regards,
Slava Kabanovich

On 09/09/2015 10:11 PM, Jayaprakash Arthanareeswaran wrote:

I must admit I don't understand your requirement fully, but let me state that once the Java model (such as BinaryType etc.) is constructed, these are cached by the
Java model manager. May be there is a way you can do what you are doing by using the cached java model?

Regards,
Jay

Inactive
          hide details for Slava Kabanovich ---09/09/2015 11:16:02
          AM---Hi, Stephan I have found the solution to my problem by sSlava Kabanovich ---09/09/2015 11:16:02 AM---Hi, Stephan I have found the solution to my problem by studying why

From: Slava Kabanovich <vkabanovich@xxxxxxxxxx>
To: "Eclipse JDT general developers list." <jdt-dev@xxxxxxxxxxx>
Date: 09/09/2015 11:16 AM
Subject: Re: [jdt-dev] Performance issues using IType interface
Sent by: jdt-dev-bounces@xxxxxxxxxxx





Hi, Stephan

I have found the solution to my problem by studying why
IndexBasedHierarchyBuilder.build(boolean) works fast. It turns out that
when I iterate over types in a jar, the first call to IType (e.g.
IType.isInterface()) results in creating new ZipFile object for each
type. That can be easily prevented by

        JavaModelManager manager = JavaModelManager.getJavaModelManager();
        try {
            manager.cacheZipFiles(this);
            //All build over jars goes here
        } finally {
            manager.flushZipFiles(this);
        }

JDT works fine and fast. Caching ZipFile objects makes all the difference.

The only concern remaining now is that JavaModelManager is not a part of
API. May I request to consider providing API methods for doing similar
searches?

Thank you.

Best regards,
Slava Kabanovich

On 06/06/2015 08:32 PM, Slava Kabanovich wrote:
> Hi, Stephan
>
> Thank you for the reply.
>
>
> > What requests exactly to you make?
>
> For instance, the very first call IType.isInterface() takes too much
> time. If after that I call any other methods of IType they perform
> much faster. That is probably because creating element info takes much
> time as it probably prepares data for all requests to IType.
> Suppose, I need to look into all classes, but i know that I do not
> need interfaces and abstract classes, only concrete classes. Then, I
> try to be smart and get done with interfaces by a single check
> IType.isInterface(). I expect it to be much faster than getting all I
> need for a concrete class. But it does not work as expected. As I have
> said, (the first) call to isInterface() takes almost as long as
> everything else I need to get from a concrete class.
>
>
> > Not sure what you mean by Java build, since org.eclipse.jdt.core
> does implement building Java programs.
>
> I may be mistaken, is not Java building implemented with packages
> org.eclipse.jdt.internal.core.jdom and *.internal.compiler.* packages?
> With 'org.eclipse.jdt.core' I meant not plugin (yes, the plugin
> implements Java build), but package that includes IType and other
> handles for querying Java model. When Java builder completes, element
> info for IType is not yet created. Then I conclude that Java builder
> does not need requesting anything from IType and uses the internal
> model that is more efficient.
>
>
> > Also, I'm not sure I understand what exactly your tool is doing?
>
> It is ok with incremental building, certainly I do not need looking
> into a jar at every build, but when a project is imported into
> workspace, the full build is performed for it, and Batch tool needs to
> determine which jars are Batch archives. Unfortunately, there is no
> required resource like batch.xml (it may be present, but is ok to be
> missing). So I do not see another way than scanning, just once, the
> jar in search for Batch artifacts. If none found, I mark the jar as
> 'not a Batch archive', and it is ignored after that.
> The same with CDI 1.1. While in CDI 1.0 archives resource beans.xml
> was required, it is not so since 1.1.
> Once more, the performance issue is critical for the case of importing
> into workspace a large set of projects with many dependencies. It may
> be critical also at workspace refresh if dependencies are changed
> (e.g. versions of all jars are changed).
>
> >Have you considered, instead of iterating lot's of potentially
> irrelevant elements,
> >to perform a search (e.g., for all types with a given annotation)?
>
> In CDI 1.1+, there are many bean defining annotations and most
> confusing is that they include custom (defined by application) scopes
> and stereotypes, which the tool may either collect as the result of
> build, or by checking annotations found at the type during iteration.
>
>
> >Or just use a subtype hierarchy of a well-known interface?
>
> At present, in Mars M7, when I start 'Open Type Hierarchy' for IType,
> it takes about 1 minute to show in the view. In Luna it took only a
> few seconds.
>
>
> >Have you seen the following API?
> >org.eclipse.jdt.core.JavaCore.newTypeHierarchy(IRegion,
> WorkingCopyOwner, IProgressMonitor)
>
> Thank you, I will try it to see if it may give gain in performance.
>
>
> Best regards,
> Slava Kabanovich
>
> On 06/06/2015 05:39 AM, Stephan Herrmann wrote:
>> Hi Slava,
>>
>> A few points in your description are not clear to me, so it's
>> difficult to advise.
>>
>>
>> Questions:
>>
>> > At present this is implemented with requests to IType interface.
>>
>> What requests exactly to you make?
>> Creating an IType handle has virtually no cost.
>> Some requests require to read and analyze the underlying compilation
>> unit or class file etc.
>>
>> > makes one to assume that Java build uses internally more efficient
>> model than org.eclipse.jdt.core.
>>
>> Not sure what you mean by Java build, since org.eclipse.jdt.core does
>> implement building Java programs.
>>
>>
>> Also, I'm not sure I understand what exactly your tool is doing? Why
>> do you have to inspect all
>> entries of all jars in the classpaths? When speaking of incremental
>> buildings I'd assume
>> inspecting all compilation units in (affected projects in) the
>> workspace should be enough.
>> Why isn't it?
>>
>> Have you considered, instead of iterating lot's of potentially
>> irrelevant elements,
>> to perform a search (e.g., for all types with a given annotation)? Or
>> just use a
>> subtype hierarchy of a well-known interface?
>> If a lot of hierarchy lookup is needed, perhaps doing this in one big
>> bulk instead of
>> thousands of individual invocations might also speed up things.
>> Have you seen the following API?
>>
>> org.eclipse.jdt.core.JavaCore.newTypeHierarchy(IRegion,
>> WorkingCopyOwner, IProgressMonitor)
>>
>>
>> best,
>> Stephan
>>
>>
>> On 05/27/2015 02:38 AM, Slava Kabanovich wrote:
>>> Hello,
>>>
>>> I work on Batch and CDI models for JBossTools in Eclipse.
>>> Incremental builders for the models look for types that either
>>> implement
>>> some interfaces (Batch), or have bean defining annotations (CDI). At
>>> present this is implemented with requests to IType interface.
>>> In a large workspace, with a lot of jars in classpath, at first
>>> build (or at a change in class path) jar entries have to be checked
>>> for being Batch or CDI archive, there is no simple way to do that
>>> (in CDI since 1.1, CDI 1.0 had required beans.xml that made life
>>> much easier) other than checking all types in jar. Tests show that
>>> in average about 2000 IType objects may be checked in one second,
>>> which results in unacceptably long build time for large workspaces.
>>> Batch and CDI build take 2 to 3 times longer than Java build. It
>>> makes one to assume that Java build uses internally more efficient
>>> model than org.eclipse.jdt.core.
>>>
>>> Could it be possible to get faster methods for checking inheritance
>>> and annotations? It may be still ok to work with IType interface
>>> for objects verified as Batch artifacts or CDI beans/annotations,
>>> but it is of great importance to be able to quickly rule out
>>> irrelevant types. Could you please advise me already existing ways
>>> to solve this problem, or consider implementing in JDT new high
>>> performance requests?
>>>
>>> Thank you.
>>>
>>> Best regards,
>>> Viacheslav Kabanovich
>>> _______________________________________________
>>> jdt-dev mailing list
>>> jdt-dev@xxxxxxxxxxx
>>> To change your delivery options, retrieve your password, or
>>> unsubscribe from this list, visit
>>>
https://dev.eclipse.org/mailman/listinfo/jdt-dev
>>
>> _______________________________________________
>> jdt-dev mailing list
>> jdt-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or
>> unsubscribe from this list, visit
>>
https://dev.eclipse.org/mailman/listinfo/jdt-dev
>

_______________________________________________
jdt-dev mailing list
jdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-dev




_______________________________________________
jdt-dev mailing list
jdt-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jdt-dev

package sample;

import org.eclipse.core.runtime.CoreException;
import org.eclipse.jdt.core.IClassFile;
import org.eclipse.jdt.core.IJavaElement;
import org.eclipse.jdt.core.IJavaProject;
import org.eclipse.jdt.core.IPackageFragment;
import org.eclipse.jdt.core.IPackageFragmentRoot;
import org.eclipse.jdt.core.IType;
import org.eclipse.jdt.internal.core.JavaModelManager;

public class ClasspathScanner {

	public void buildJars(String[] externalJars, IJavaProject javaProject) throws CoreException {
			for (String jar: externalJars) {
				IPackageFragmentRoot root = javaProject.getPackageFragmentRoot(jar);
				if (root == null || !root.exists()) {
					continue;
				}
				IJavaElement[] es = root.getChildren();
				for (IJavaElement e : es) {
					if (e instanceof IPackageFragment) {
						IPackageFragment pf = (IPackageFragment) e;
						IClassFile[] cs = pf.getClassFiles();
						for (IClassFile c : cs) {
							IType type = c.getType();
							if(type.isInterface()) {
								// Process interfaces here
							} else {
								// Process other types here
							}
						}
					}
				}
			}
	}

	public void buildJarsImproved(String[] externalJars, IJavaProject javaProject) throws CoreException {
		JavaModelManager manager = JavaModelManager.getJavaModelManager();
		try {
			manager.cacheZipFiles(this);
			buildJars(externalJars, javaProject);
		} finally {
			manager.flushZipFiles(this);
		}
	}

}
"Worker-2186" #3405 prio=5 os_prio=0 tid=0x000000000112f800 nid=0x4094 runnable [0x00007f60a8af0000]
   java.lang.Thread.State: RUNNABLE
  at java.util.zip.ZipFile.open(Native Method)
  at java.util.zip.ZipFile.<init>(ZipFile.java:220)
  at java.util.zip.ZipFile.<init>(ZipFile.java:150)
  at java.util.zip.ZipFile.<init>(ZipFile.java:164)
  at org.eclipse.jdt.internal.core.JavaModelManager.getZipFile(JavaModelManager.java:2678)
  at org.eclipse.jdt.internal.core.JavaModelManager.getZipFile(JavaModelManager.java:2644)
  at org.eclipse.jdt.internal.core.JarPackageFragmentRoot.getJar(JarPackageFragmentRoot.java:156)
  at org.eclipse.jdt.internal.core.ClassFile.getJarBinaryTypeInfo(ClassFile.java:355)
  at org.eclipse.jdt.internal.core.ClassFile.getBinaryTypeInfo(ClassFile.java:290)
  at org.eclipse.jdt.internal.core.ClassFile.getBinaryTypeInfo(ClassFile.java:284)
  at org.eclipse.jdt.internal.core.ClassFile.buildStructure(ClassFile.java:93)
  at org.eclipse.jdt.internal.core.Openable.generateInfos(Openable.java:259)
  at org.eclipse.jdt.internal.core.SourceRefElement.generateInfos(SourceRefElement.java:107)
  at org.eclipse.jdt.internal.core.JavaElement.openWhenClosed(JavaElement.java:579)
  at org.eclipse.jdt.internal.core.BinaryType.getElementInfo(BinaryType.java:287)
  at org.eclipse.jdt.internal.core.JavaElement.getElementInfo(JavaElement.java:302)
  at org.eclipse.jdt.internal.core.BinaryType.isInterface(BinaryType.java:725)
  at sample.ClasspathScanner.buildJars(ClasspathScanner.java:27)



Back to the top