Bug 117302 - Clean build of large project gives unresolved type errors
Summary: Clean build of large project gives unresolved type errors
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 3.2   Edit
Hardware: PC Windows XP
: P3 major (vote)
Target Milestone: 3.2.1   Edit
Assignee: Kent Johnson CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2005-11-21 06:56 EST by Neil Swingler CLA
Modified: 2006-09-12 08:55 EDT (History)
0 users

See Also:


Attachments
debug console log (358.30 KB, text/plain)
2005-11-21 07:01 EST, Neil Swingler CLA
no flags Details
.classpath (5.39 KB, text/plain)
2005-11-30 12:00 EST, Neil Swingler CLA
no flags Details
.project (528 bytes, text/plain)
2005-11-30 12:01 EST, Neil Swingler CLA
no flags Details
.classpath (4.54 KB, text/plain)
2005-11-30 17:34 EST, Neil Swingler CLA
no flags Details
.project (379 bytes, text/plain)
2005-11-30 17:34 EST, Neil Swingler CLA
no flags Details
Compile errors from full build (2.25 KB, text/plain)
2006-06-19 18:52 EDT, Neil Swingler CLA
no flags Details
Errors after touch of StrategyuIOnstrumentTablePopupMenu (1.90 KB, text/plain)
2006-06-19 18:55 EDT, Neil Swingler CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Neil Swingler CLA 2005-11-21 06:56:26 EST
I have a large project with 2196 source files. When I do a clean and build I get various unresolved type errors. I then force an incremental build by touching the files reporting the errors and it builds cleanly.

e.g.
"The import com.csfb.foa.business.eurex.strategy.builder cannot be resolved" and
"StategyLegsSelectorPanel cannot be resolved to a type" in foa/src/com/sfb/foa/business/eurex/ErxStrategyType.java

I do have a workaround hack to the JDT. I changed MAX_AT_ONCE in AbstractImageBuilder from 1000 to 10000. This eliminates the problem and I can do a clean and build.

There are cyclic dependencies between e.g. ErxStrategyType and StategyLegsSelectorPanel 

builder debug output to follow
Comment 1 Neil Swingler CLA 2005-11-21 07:01:05 EST
Created attachment 30307 [details]
debug console log

console output with .options

org.eclipse.jdt.core/debug=true
org.eclipse.jdt.core/debug/cpresolution=true
org.eclipse.jdt.core/debug/builder=true
Comment 2 Kent Johnson CLA 2005-11-29 14:28:15 EST
I have tried creating a project with 2 inter-connected types from different packages and source folders, while setting MAX_AT_ONCE to 1... but I have not found any problems with unresolved imports or types.

Please attach your project's .classpath & .project files so we can see how your project is setup.
Comment 3 Neil Swingler CLA 2005-11-30 06:58:39 EST
I also hoped to recreate the problem in the same way but could not.

Project files to follow.
Comment 4 Neil Swingler CLA 2005-11-30 12:00:28 EST
Created attachment 30874 [details]
.classpath
Comment 5 Neil Swingler CLA 2005-11-30 12:01:06 EST
Created attachment 30875 [details]
.project
Comment 6 Kent Johnson CLA 2005-11-30 15:25:36 EST
Ahhh your output folder is linked.

Is the problem repeatable if you do not make bin a linked folder?

Are any other tools also writing to the output folder?

What's your current setting for the option Java->Compiler->Building->'Scrub output folders when cleaning projects' ?
Comment 7 Neil Swingler CLA 2005-11-30 17:33:46 EST
So I simplified the project a bit. I will attach the new .classpath and .project

I eliminated the linked folder -> no difference.
There are no other tools writing to bin.
The scrub output setting was checked (at the project level). So I unchecked it and then a clean would rebuild OK.
However I deleted the project, deleted the output directory and reimported it -> same problems as before.
Comment 8 Neil Swingler CLA 2005-11-30 17:34:24 EST
Created attachment 30912 [details]
.classpath
Comment 9 Neil Swingler CLA 2005-11-30 17:34:59 EST
Created attachment 30913 [details]
.project
Comment 10 Neil Swingler CLA 2005-12-01 02:38:02 EST
Are there some other debug settings I can try. I tried enabling the compiler debug but the output exceeded the 9999 line maximum of the dos window.

I should also add that the problem can also be demonstrated with a 3.1 release version.

The problem only started happening when the offending classes were added to the codebase. For sure, we had plenty of cyclic dependencies before these classes were checked in.
Comment 11 Kent Johnson CLA 2005-12-01 11:35:48 EST
So you don't have any Ant builders?

Because there are tons of large projects out there that do not have this problem.

With your changes, you now have a single large project with 1 source folder and 1 output folder and ~50 jar files.

Is your workspace stored on your local machine?

Does it include any other projects? Do any of these projects have linked folders into the large project?

Plenty of disk space available?
Comment 12 Neil Swingler CLA 2005-12-02 04:54:20 EST
>So you don't have any Ant builders?
No

>Because there are tons of large projects out there that do not have this
>problem.
Ours did not for the last 3 years until the offending files were checked in recently.

>With your changes, you now have a single large project with 1 source folder and
>1 output folder and ~50 jar files.
Yes

>Is your workspace stored on your local machine?
Yes. My N: drive is a share on my own hard disk

>Does it include any other projects?
No.

>Plenty of disk space available?
Yes.
Comment 13 Kent Johnson CLA 2005-12-02 09:49:32 EST
What are all the errors you get from a full build?

Then how many errors are left when you incrementally touch some of the source files?

How many additional source files were checked in recently?
Comment 14 Neil Swingler CLA 2005-12-02 12:03:14 EST
Do you want a 20MB zip with the whole project + jars?

I will need to clear it first of course
Comment 15 Kent Johnson CLA 2005-12-02 13:20:31 EST
Sure
Comment 16 Neil Swingler CLA 2005-12-07 02:39:41 EST
I am not permitted to send you our code :-(

I could try to debug it myself if you have some hints where to look. Or could point me to any documentation on understanding the jdt compiler/builder.
Comment 17 Kent Johnson CLA 2005-12-07 11:38:23 EST
Before we try that can you answer the questions from comment #13
Comment 18 Kent Johnson CLA 2006-01-16 16:09:39 EST
Please reopen if the problem still exists.
Comment 19 Kent Johnson CLA 2006-06-16 12:20:47 EDT
Neil, its possible your problem is the same as bug 146324 .

Can you please try a nightly build of HEAD to see if its fixed.
Comment 20 Neil Swingler CLA 2006-06-17 13:25:06 EDT
Kent, I tried N20060617-0010 the problem still exists
Comment 21 Kent Johnson CLA 2006-06-19 10:51:08 EDT
I realize you cannot give me all of your project, but based on the errors that you have, can you give me little pieces that may help us find the case... much like bug 146324 .

If we know which errors you're getting then we be able to figure it out even without the code.
Comment 22 Neil Swingler CLA 2006-06-19 18:52:10 EDT
Created attachment 44879 [details]
Compile errors from full build
Comment 23 Neil Swingler CLA 2006-06-19 18:55:54 EDT
Created attachment 44881 [details]
Errors after touch of StrategyuIOnstrumentTablePopupMenu
Comment 24 Neil Swingler CLA 2006-06-19 18:59:06 EDT
Touching StrategyBuilderPanel did not get rid of any errors
Then touching ErxStrategyType got rid of all of them

I will experiment with MAX_AT_ONCE = 1 to see if there is some trivial way to reproduce it but I already tried this before without success.
Comment 25 Kent Johnson CLA 2006-06-20 09:51:45 EDT
So what exactly are the errors? Where are the type references from (superclass, fields, method parameters/body)?

Which filenames define the types StrategyBuilderPanel and ErxStrategyType ?

Are they both public types and if not, do the problems persist if you make them public?
Comment 26 Kent Johnson CLA 2006-06-20 15:19:55 EDT
Sorry - didn't see the attachments - dumb email!
Comment 27 Kent Johnson CLA 2006-06-20 15:50:06 EDT
What filenames define StrategyBuilderPanel and StrategyLegsSelectorPanel ?

Are they both public types ?
Comment 28 Neil Swingler CLA 2006-06-21 03:18:59 EDT
StrategyBuilderPanel, StrategyLegsSelectorPanel and ErxStrategyType are all public classes in their own correctly named source files. You can see the source files involved in the debug output I attached with comment #1.

The source files do not contain any other top level classes.
Comment 29 Neil Swingler CLA 2006-06-21 04:21:28 EDT
Looking at the log:

ErxStrategyType and StrategyInstrumentTablePopupMenu get built indirectly in the 1st pass.

StrategyBuilderPanel and StrategyLegsSelectorPanel get built explicitely in the 2nd pass although they should have been built indirectly in the 1st pass.
Comment 30 Kent Johnson CLA 2006-06-21 11:20:04 EDT
I think the missing type errors are not the root cause of your problem.

Below are the 3 types defined in the package com.csfb.foa.business.eurex.strategy.builder which is compiled in the second group of source files.

com/csfb/foa/business/eurex/strategy/builder/StrategyLegsSelectorPanel.java
com/csfb/foa/business/eurex/strategy/builder/StrategyLegSelectorComponents.java
com/csfb/foa/business/eurex/strategy/builder/StrategyBuilderPanel.java

And both of your files that have errors after a full build report that they cannot resolve the import to com.csfb.foa.business.eurex.strategy.builder

What do the import statements look like?

import com.csfb.foa.business.eurex.strategy.builder.StrategyBuilderPanel;
import com.csfb.foa.business.eurex.strategy.builder.StrategyLegsSelectorPanel;
or
import com.csfb.foa.business.eurex.strategy.builder.*;

What happens if you rename the package? Do the errors still happen?
Comment 31 Neil Swingler CLA 2006-06-21 12:09:26 EDT
The imports are fully qualified. There are no *s

All 3 files from that src/com/csfb/foa/business/eurex/strategy/builder package are in the 2nd compile pass

Will have a play later to see if renaming, using * etc. makes any difference
Comment 32 Kent Johnson CLA 2006-06-21 14:03:35 EDT
I find it very odd that this is the only package that cannot be resolved.

Please watch to see if its created and then deleted.

And double check that you do not have any exclusion filters that may hide that package - which I doubt you do since we do compile its source files.
Comment 33 Kent Johnson CLA 2006-06-26 10:18:36 EDT
Neil, have you had any luck with the package rename ?
Comment 34 Neil Swingler CLA 2006-06-28 08:08:59 EDT
Kent, exciting news. 

I changed the package name to foo - and the problem goes away!
Comment 35 Neil Swingler CLA 2006-06-28 08:29:05 EDT
builder1 => fails
b234567 => works (so nothing to do with the package name length)
Comment 36 Neil Swingler CLA 2006-06-28 08:37:57 EDT
build => fails
buil => works

renaming the package to just "build", (i.e. not com.csfb....build) also does not work
Comment 37 Neil Swingler CLA 2006-06-28 09:04:23 EDT
I changed the imports to use * It still fails to build.
Comment 38 Kent Johnson CLA 2006-06-28 10:46:03 EDT
Well, we either have a hash problem with our tables that is causing us to miss this package.

Or your project has filters set that are excluding this package from us in some cases.

Can you search your entire project, including all .* files, for 'build*' ?
Comment 39 Neil Swingler CLA 2006-06-28 10:59:56 EDT
Kent, I have build* in the filtered resources. I realise you asked me about this before and can only apologise for not checking earlier.
Comment 40 Kent Johnson CLA 2006-06-28 11:44:27 EDT
No problem.

Remove it if you don't need it.

But I still think its our bug that we do not create the package folder even tho this filter exists - I'll look into a fix for 3.2.1
Comment 41 Kent Johnson CLA 2006-06-28 15:34:35 EDT
Released for 3.3M1 in HEAD.
Released for 3.2.1 in TARGET_321 branch.

Added CopyResourceTests.testFilteredResources()

We did not create package folders in the output folder when the package matched a filtered resource pattern that stopped its extra resources from being copied to the output folder.

This change creates the package folders in the output folder while we collect the source files instead of the extra resource copy walk.
Comment 42 Frederic Fusier CLA 2006-08-07 10:59:14 EDT
Verified for 3.3 M1 using build I20060807-0010.
Comment 43 Maxime Daniel CLA 2006-09-12 08:55:03 EDT
Verified for 3.2.1 using build M20060908-1655.
Opening separate bug for a variation (bug 157019).