Bug 283113 - [fiximprove]: Preverification is extremely slow / loops when automatic build is enabled
Summary: [fiximprove]: Preverification is extremely slow / loops when automatic build ...
Status: RESOLVED FIXED
Alias: None
Product: MTJ (Archived)
Classification: Tools
Component: Project Builder (show other bugs)
Version: 1.0   Edit
Hardware: PC Windows XP
: P3 normal with 10 votes (vote)
Target Milestone: 1.1   Edit
Assignee: Gorkem Ercan CLA
QA Contact:
URL:
Whiteboard:
Keywords: investigate
Depends on:
Blocks:
 
Reported: 2009-07-10 04:28 EDT by Christian CLA
Modified: 2010-06-11 08:04 EDT (History)
6 users (show)

See Also:


Attachments
Fix for StandardPreverifier (4.43 KB, patch)
2010-02-16 05:21 EST, Marc Wilhelm CLA
gorkem.ercan: iplog+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Christian CLA 2009-07-10 04:28:14 EDT
Build ID: 20090619-0625

Steps To Reproduce:
1. Create a larger project for the Nokia S40 device
2. Change something which triggers the automatic build
3. After the build finished it just restarts about 5 seconds later looping forever
4. If I do disable automatic building this loop won't happen but each build caused by changing some relevant project settings will take virtually ages (4-5 Minutes), but if I build using the context menu (Mobile tools for Java -> Create Package) on the project it will finish after a few moments (10 seconds).


More information:
I was able to reproduce this behaviour with eclipse 3.4.2 build M20090211-1700 and Eclipse Pulsar build 20090619-0625 but not if I install version 0.9.1 of the mtj plugin. Unfortunately running midlets using 0.9.1 and the sun wtk 3.0 does not work since the produced command line seems to be invalid.
I do run eclipse using Windows XP with an encrypted file system.
Comment 1 Christian CLA 2009-07-10 05:26:59 EDT
Ok I'm now able to launch the wtk 3 emulator from inside 0.9.1 - so I can just switch back to 0.9.1 ...
Comment 2 Gustavo de Paula CLA 2009-07-10 11:30:40 EDT
are you saying that the issue exists on 1.0 version and that you can only use 0.9.1?

:)
gep

(In reply to comment #1)
> Ok I'm now able to launch the wtk 3 emulator from inside 0.9.1 - so I can just
> switch back to 0.9.1 ...
> 

Comment 3 Christian CLA 2009-07-16 03:45:16 EDT
(In reply to comment #2)
> are you saying that the issue exists on 1.0 version and that you can only use
> 0.9.1?
> 
> :)
> gep
> 

The infinite loop for the automatic build only happens for 1.0 - it works fine with 0.9.1. The same is true for the general slowness of the preverification process - since the preverifier itself has not changed it is perhaps invoked several times? 
My problem with running the wtk 3.0 were only a side note which should explain why I needed version 1.0 badly but in the end I was able to solve that problem but not the problems with 1.0 - that's why I (and all the colleagues as well) switched back to 0.9.1 ...
Comment 4 David Marques CLA 2009-07-20 07:49:25 EDT
Hi Christian,

Please answer the questions bellow:

1 - What is the sdk you are using ?
2 - Would you have a project you could send us in order to reproduce the issue ?
3 - How many runtime configurations (devices) do you have on your project ?

Regards,

David Marques

(In reply to comment #3)
> (In reply to comment #2)
> > are you saying that the issue exists on 1.0 version and that you can only use
> > 0.9.1?
> > 
> > :)
> > gep
> > 
> 
> The infinite loop for the automatic build only happens for 1.0 - it works fine
> with 0.9.1. The same is true for the general slowness of the preverification
> process - since the preverifier itself has not changed it is perhaps invoked
> several times? 
> My problem with running the wtk 3.0 were only a side note which should explain
> why I needed version 1.0 badly but in the end I was able to solve that problem
> but not the problems with 1.0 - that's why I (and all the colleagues as well)
> switched back to 0.9.1 ...
> 

Comment 5 Christian CLA 2009-07-21 11:08:57 EDT
> 1 - What is the sdk you are using ?
I'm using the 6th edition of the Nokia S40 SDK

> 2 - Would you have a project you could send us in order to reproduce the issue
> ?

That could be difficult - I could try to set up a demo project that has the same issues but I don't think that we can attach our project here. I'll see how complex a project has to be to produce this problem.

> 3 - How many runtime configurations (devices) do you have on your project ?

Currently I do have 5 devices in the workspace (WTK 2.5.2) but only 1 device in the project (the device from the Nokia S40 SDK).
Comment 6 Christian CLA 2009-07-24 07:57:19 EDT
Well after I got a new machine I retried to update to mtj 1.0 and this time I had no problems why so ever. Perhaps this is thanks to the fact that I used the update button directly instead of installing directly or perhaps I just had a corrupt plugin the last time I installed it. However it works now for me. 
Comment 7 Christian CLA 2009-07-24 09:24:57 EDT
Sorry but I have to take back my last statement- the problem is still here it just did not show up because the "Preverification Builder" was not checked. When I checked it again the loop is back - however for now it is fine to just disable it and do the preverification just in the release builds done by ant. 
Comment 8 Diego Madruga Sandin CLA 2009-07-28 10:02:04 EDT
Hi Christian,

are you using any plugin that auto generate code or resources? The only reason i see for the preverification be triggered so many times is that you workspace is being changed automatically. The previrification builder would only be called in case of changes in any of the source folders.

Could you launch your eclipse with the following vmargs and send us the console output?

-vmargs -Dmtj.build.logging=all

This will enable tracing in MTJ Build console;

Thanks
Diego

(In reply to comment #7)
> Sorry but I have to take back my last statement- the problem is still here it
> just did not show up because the "Preverification Builder" was not checked.
> When I checked it again the loop is back - however for now it is fine to just
> disable it and do the preverification just in the release builds done by ant. 
> 

Comment 9 Christian CLA 2009-07-29 04:24:43 EDT
(In reply to comment #8)
> Hi Christian,
> 
> are you using any plugin that auto generate code or resources? The only reason
> i see for the preverification be triggered so many times is that you workspace
> is being changed automatically. The previrification builder would only be
> called in case of changes in any of the source folders.

We are auto-generating code only with a ant-build which is not used by any plugin and eclipse should not be aware of it - so no plugin doing it during the normal eclipse build process. 

> 
> Could you launch your eclipse with the following vmargs and send us the console
> output?
> 
> -vmargs -Dmtj.build.logging=all

I'll do that and attach the resulting log here. 

> 
> This will enable tracing in MTJ Build console;
> 
> Thanks
> Diego
Comment 10 Christian CLA 2009-07-29 04:45:18 EDT
Since it is really short, I'll just paste it here.

> PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> ResourceDeltaBuilder.build
> ResourceDeltaBuilder.handleClassAddsAndChanges; classFiles count = 0
< ResourceDeltaBuilder.handleClassAddsAndChanges
< ResourceDeltaBuilder.build
< PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
======================== Launching Preverification =========================
======================== Preverification exited with code: 0
< PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
< PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> ResourceDeltaBuilder.build
> ResourceDeltaBuilder.handleClassAddsAndChanges; classFiles count = 0
< ResourceDeltaBuilder.handleClassAddsAndChanges
< ResourceDeltaBuilder.build
< PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
======================== Launching Preverification =========================
======================== Preverification exited with code: 0
< PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
< PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> ResourceDeltaBuilder.build
> ResourceDeltaBuilder.handleClassAddsAndChanges; classFiles count = 0
< ResourceDeltaBuilder.handleClassAddsAndChanges
< ResourceDeltaBuilder.build
< PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
======================== Launching Preverification =========================
======================== Preverification exited with code: 0
< PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
< PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> ResourceDeltaBuilder.build
> ResourceDeltaBuilder.handleClassAddsAndChanges; classFiles count = 0
< ResourceDeltaBuilder.handleClassAddsAndChanges
< ResourceDeltaBuilder.build
< PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
======================== Launching Preverification =========================
======================== Preverification exited with code: 0
< PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
< PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.build project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> ResourceDeltaBuilder.build
> ResourceDeltaBuilder.handleClassAddsAndChanges; classFiles count = 0
< ResourceDeltaBuilder.handleClassAddsAndChanges
< ResourceDeltaBuilder.build
< PreverificationBuilder.preverifyProject project = P/MedosLite-Trunk
> PreverificationBuilder.preverifyLibraries project = P/MedosLite-Trunk
======================== Launching Preverification =========================
======================== Preverification exited with code: 0


What I did was to startup eclipse with automatic build turned off and then just turned it on. That was the result. The log just seemed to go on for ever like this so does the workspace building. If I shut down eclipse and restart it with auto build enabled the log looks alike (I have to insert a space into a file and save it to trigger the build, it does not build right after startup).




Comment 11 Christian CLA 2009-07-30 11:10:54 EDT
Btw: If you want to add logging to the code, I could just use a patched version of the mtj 1.0 if this helps you to identify the problem. Currently I'm using two installations and workspaces so it won't cause much trouble to patch one with a more informative logging.
Comment 12 Marc Wilhelm CLA 2010-02-16 05:21:31 EST
Created attachment 159159 [details]
Fix for StandardPreverifier

Hi,

I've found the reason for the time consuming preverification process in some environments:
If the j2me classpath is very long (especially longer than MAX_COMMAND_LENGTH=2000), the preverifier will be called for nearly each single class.
The attached patch changes the preverifier invocation by writing all classes to preverify into a parameter file and invoke only once (!) the preverifier.

Cheers,
Marc
Comment 13 Gorkem Ercan CLA 2010-04-08 15:46:04 EDT
Marc, the patch looks OK but before I proceed and use the patch I need you to state that you have developed the patch yourself, and have the right to contribute it to eclipse under EPL.
Comment 14 Marc Wilhelm CLA 2010-04-10 19:32:11 EDT
Gorkem,
the patch was developed by myself and is free to contribute under EPL.

Is this statement as bugzilla comment enough to you?

Marc
Comment 15 Gorkem Ercan CLA 2010-04-12 11:19:15 EDT
Applied the patch with some additional fixes/cleanup.
Comment 16 Gokhan Berberoglu CLA 2010-06-10 16:09:13 EDT
(In reply to comment #15)
> Applied the patch with some additional fixes/cleanup.

Hi Gorkem,

I'm still having endless preverification loops when automatic build is enabled.

Cause of the loops seems to be Floggy persistance framework. (http://floggy.sourceforge.net/) 

Once the project is configured for floggy (by selecting "add Floggy nature" from right click menu of project) preverification loops starts. Once its removed (by selecting "remove Floggy nature" from right click menu of project) automatic build works fine.

I'm using floggy 1.3.1 with clean installation of helios rc3..

I can provide more information about issue..
Comment 17 Gorkem Ercan CLA 2010-06-11 05:33:11 EDT
This is probably a different issue. I am not very familiar with floggy but I suspect that floggy has its own builder that works after the MTJ preverification builder. Can you check in your project with Floggy nature if there is a floggy builder. What is its order? Can you quickly change the order of floggy builder to be before preverification builder but after Java builder. 

You can check the builders on the right click menu of your project ->Properties -> Builders
Comment 18 Gokhan Berberoglu CLA 2010-06-11 07:31:18 EDT
(In reply to comment #17)

Yes, problem is related with floggy's builder, its gone once i uncheck "floggy project builder".

default build order is:

java builder
floggy builder
preverification builder
package buidler


Regards,
Gokhan
Comment 19 Gorkem Ercan CLA 2010-06-11 08:04:12 EDT
created bug 316593 for the new case