Re: [Dltk-dev] Retained memory can be huge in FullBuildChange

[Alexey Panchenko <alex.panchenko@xxxxxxxxx>]

Hi Kaloyan,

I agree, that a list of 440k instances requires some significant amount of memory.

What happens there is scanning the whole project to collect all the files, then select source modules by calling DLTKCore.create() on each file. Probably this code is also involved in causing Full GC, high memory usage is not necessary a problem, but this needs to be profiled.

Returning a list of the changed resources could be quite convenient, e.g. in the DLTK codebase it is used to clear problem markers on them - to handle cases like resource was source module before, but then was removed/excluded from buildpath, this happens in org.eclipse.dltk.internal.core.builder.StandardScriptBuilder.build(IBuildChange, IBuildState, IProgressMonitor).

Also, this method can be used by DLTK adopters to process some associated resources, etc.

Overall, an interesting case, I don't think I had ever seen so many files in a project. Honestly, I would recommend refactoring that file structure, and at the same time I don't mind improving DLTK to handle it better.

Regards,

Alex

On Tue, Apr 29, 2014 at 7:07 PM, Kaloyan Raev <kaloyan.r@xxxxxxxx> wrote:

Hi, 

I am currently investigating a performance issue of one of our customers. The customer's project has ~445,000 files from which only ~13,000 are source modules. The rest of the files are mostly media files. 

If the customer import the complete project in the IDE then he suffers a huge performance issue with IDE almost not responding during the build. If the customers removes the media files and the project is reduced to ~50,000 files total (still all the ~13,000 source modules remain) then the issue is not observed.

My analysis shows that the IDE is not responding due to excessive garbage collecting. At some point almost all of the CPU time is spent in Full GC work. I did a memory profiling and found out that one of the dominators is one org.eclipse.dltk.internal.core.builder.FullBuildChange object. The FullBuildChange class builds a list of all IFile-s in the project and keeps it throughout the complete build process. It seems that keeping references to IFile objects is quite expensive. The retained size of this list of ~445,000 IFile objects is ~220 MB. This is quite a lot and reduces the available memory heap for the actual build execution. 

As far as I understand this list of IFile objects is built for the sole purpose to be returned from the IProjectChange.getResources() method. I wonder why DLTK builders need this list of resources at all. Shouldn't DLTK builders work completely on ISourceModule objects instead?

Greetings,

Kaloyan

_______________________________________________
dltk-dev mailing list
dltk-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/dltk-dev