Community
Participate
Working Groups
The 'derived' setting on files and folders is currently stored in workspace metadata. The problem with this is that if an existing project is imported into a new Eclipse workspace using 'File' -> 'Import...' -> 'Existing Project into Workspace', the derived information is lost and builders can no longer tell whether a generated file or folder is in fact generated or is a user artifact (because derived flags are missing). A "leaking" of folders and files can occur. Especially if builders use time stamp information to generate derived content.
I understand the desire for this, but it's not something we're likely to do. Similar arguments could be made for storing most resource metadata in the project content area. However, there are trade-offs involved in storing data within the project. It would be more expensive to store the derived bit there because it would generate resource changed events whenever derived state changed. The format on disk would be more verbose, causing serialization to be more expensive - currently the derived state is stored in a single bit within a flag integer associated with each resource in the workspace tree file. The derived state would also then be shared in repositories, which doesn't make sense because the derived resources themselves are not checked in. Ideally, a builder should be able to figure out what resources it created without resorting to the derived bit. After import when the builder first runs, it could then set the derived bit again for all those resources. This is roughly what the Java builder does in this scenario.
(In reply to comment #1) > I understand the desire for this, but it's not something we're likely to do. > Similar arguments could be made for storing most resource metadata in the > project content area. However, there are trade-offs involved in storing data > within the project. It would be more expensive to store the derived bit there > because it would generate resource changed events whenever derived state > changed. The format on disk would be more verbose, causing serialization to be > more expensive - currently the derived state is stored in a single bit within a > flag integer associated with each resource in the workspace tree file. The > derived state would also then be shared in repositories, which doesn't make > sense because the derived resources themselves are not checked in. > Ideally, a builder should be able to figure out what resources it created > without resorting to the derived bit. After import when the builder first > runs, it could then set the derived bit again for all those resources. This is > roughly what the Java builder does in this scenario. There are real-life scenarios where figuring out what resources were created is not possible. Also, persistent properties that are stored for files in a workspace are not imported. I see this as an issue - if somebody could always figure out derived flags or persistent properties (I know, you only answered the derived flag, but I see this as one special persistent property), why should he ever use this feature? Otherwise persistent properties and/or derived flags are only usable as performance enhancement for e.g. builders, but without having the assurance that these properties are available later, these properties are useless.
> if somebody could always > figure out derived flags or persistent properties (I know, you only answered > the derived flag, but I see this as one special persistent property), why > should he ever use this feature? The purpose of the derived flag is to allow builders to indicate to other plugins, specifically version and configuration management systems, that the resource has been derived from some other resources. This allows other plugins to omit such a resource from repository checkin, filter it from the UI, etc. If it was just used to allow a builder to track derived resources for its own benefit then the API would not be needed - a simple persistent property would have sufficed.
(In reply to comment #3) Ok, you are right if you speak about the 'derived' flag seperately. In my comment I spoke about both the persistent properties and the derived flag, because both information are lost during import. I can accept that the derived flag is lost, but I cannot accept that the persistent properties are lost generally. Losing them during import makes them useless.
+1 for me. It is a nuisance for us to keep remembering to set derived resources. BTW, the setting seems to be forgotten by eclipse quite often too.
Won't fix. Persistent properties and attributes such as the derived bit are metadata that live only within a workspace. If clients such as builders need to have data survive across export/import they will have to store it in the project area themselves (for example in the project scope of the preference store).
(In reply to comment #6) > Won't fix. Persistent properties and attributes such as the derived bit are > metadata that live only within a workspace. Why do you treat encoding different then? > If clients such as builders need > to have data survive across export/import they will have to store it in the > project area themselves (for example in the project scope of the preference > store). If you need to tell Code Versioning plugins that a file does not need to be checked in, you cannot use other preferences. The derived flag is the only attribute that can be used for that. Usually, a user should unmark the flag "Include derived files." when exporting a project interchange file. This would solve the issue, too. Unfortunately, many customers don't do this. If you then subsequently import e.g. a project interchange file containing derived files, these files appear to be authored.
*** Bug 187769 has been marked as a duplicate of this bug. ***
(In reply to comment #3) > The purpose of the derived flag is to allow builders to indicate to other > plugins, specifically version and configuration management systems, that the > resource has been derived from some other resources. This allows other plugins > to omit such a resource from repository checkin, filter it from the UI, etc. Several plugins generate files that should not always be considered for configuration management because they are generated files. Some projects want these files under source control, some do not want these files under version control. Examples are: .findbugs, SQLJJavaSource, an ant build that generates temp files etc. In my opinion the derived flag marks everything that should not be considered for configuration management issues in this particular project (not specific to the workspace). This is a project setting and should be shared.
> In my opinion the derived flag marks everything that should not be considered > for configuration management issues in this particular project (not specific to > the workspace). This is a project setting and should be shared. I agree on your definition of the derived flag, but I don't see why this means the setting should be shared. In other words, why do you need to share the list of files that are not shared? Each client will never share these files, because they are marked as derived every time they are generated...
(In reply to comment #10) > > In my opinion the derived flag marks everything that should not be considered > > for configuration management issues in this particular project (not specific to > > the workspace). This is a project setting and should be shared. > I agree on your definition of the derived flag, but I don't see why this means > the setting should be shared. In other words, why do you need to share the > list of files that are not shared? Each client will never share these files, > because they are marked as derived every time they are generated... I am not talking about derived marks that get set by plugins but derived marks i set manually. We have teams of 10 developers (300 in total) - about 700 rsa projects in clearcase. Every time a project gets imported, the derived marks have to be set manually to define which resources are not clearcase relevant for this project. IBM recommends to create a new workspace every few weeks (performance hint) - so even 1 delevoper has to mark the derived sources again and again.
(In reply to comment #11) > IBM recommends to create a new workspace every few weeks > (performance hint) > I've never heard about this, does this apply to Eclipse, or just RSA?
Reopening for further discussion. The derived file support was originally intended as a mechanism for automated tools such as builders to flag files as generated artifacts. Your usage of manually marking files as derived is quite different.
>> IBM recommends to create a new workspace every few weeks > I've never heard about this, does this apply to Eclipse, or just RSA? This certainly doesn't apply to basic Eclipse. I have used the same workspace for primary development for over five years, spanning releases 2.0 to 3.3. Nils, is there a document online that makes this suggestion, or was this recommendation made through word of mouth? This could be an issue for a specific version of a specific product.
(In reply to comment #14) > Nils, is there a document online that makes this suggestion, or was this > recommendation made through word of mouth? This could be an issue for a > specific version of a specific product. Thanks for reopening the bug. See http://www-128.ibm.com/developerworks/rational/library/05/517_radtip/ section "fresh workspaces" (Gary Karasiuk). For the original issue this is only an additional argument. Every developer who imports a project into a new workspace would benefit from sharing manually set derived flags.
It sounds like what you really want is the equivalent of the .cvsignore file in CVS. This file records the set of files that should not be shared in a CVS repository. There is a preference page under Preferences > Team > Ignored Resources where you can specify patterns of file names to exclude from repository control. Your particular repository may or may not support this.
(In reply to comment #16) > It sounds like what you really want is the equivalent of the .cvsignore file in > CVS. This file records the set of files that should not be shared in a CVS > repository. There is a preference page under Preferences > Team > Ignored > Resources where you can specify patterns of file names to exclude from > repository control. Your particular repository may or may not support this. Ignored Resources would be an acceptable alternative but ignored Resources can not be specified as file name pattern including folders (e.g. SqljJavaSource/**/*.*). Only filenames can be specified there (see bug 39239). These settings should be exchanged in the epf preferences file (is there an eclipse bug already opened for this feature?).
(In reply to comment #17) > These settings should be exchanged in the epf preferences file (is there an > eclipse bug already opened for this feature?). See bug 194232.
I'm marking this as a duplicate of bug 39239 as that seems to be the best solution to this problem. *** This bug has been marked as a duplicate of bug 39239 ***
(In reply to comment #19) > I'm marking this as a duplicate of bug 39239 as that seems to be the best > solution to this problem. > *** This bug has been marked as a duplicate of bug 39239 *** I don't see this as an duplicate of 39239, because the base discussion has nothing to do with 39239. I rather want to focus on the initial issue of this bug. The general request of this bug is that the 'derived' information should be stored in the project metadata like e.g. encoding is done. Currently it is saved in the workspace metadata (independent if it is used to circumvent 39239 or for other things). What is bad if the data is stored in the workspace metadata? The information is lost if using the function "Import existing project". Now there were arguments that derived information may be reconstructed by the builders. This is true for builders like java compilers, but in at least one product based on Eclipse there are sophisticated builders which cannot always reconstruct this 'derived' flag. But also for the Java Builder, I see some strange situations: Create a new workspace, disable automatic build, and import an existing project that contains java files and classes. After the import, the 'derived' flag is not available and no builder runs that could reconstruct this. If I now use CVS, I check in all java files and all classes. Yes, this is a rare situation, but what is the derived flag for if plugins like CVS cannot rely on it?
Fair enough. I guess I should have said that bug 39239 is fixed so you can accomplish what you want using the Ignored Resources preference page. I marked this as a duplicate because I don't see the need to have two mechanisms that do the same thing.
I would also like to see this implemented, as it would be useful to have a way to version the fact that certain directories are built by external builders. In our projects the "build" and "publish" directories are built by ant and should be marked as derived so their contents don't show up in searches. It's a pretty soul-crushing task to mark the "build" and "publish" folders of 135 projects as derived, as there's no programmatic way of doing so and the enhancement for it is not upcoming (bug 30440). Combined with Eclipse forgetting the derived bit randomly and the inability to share it with others, it's not something that you particularly want to spend your time doing. It would be ideal if there was either an external way to mark folders as derived, or if the derived bit could be committed to version control so it only has to be done once.
Using ignored resources for this (bug 39239) is not sufficient as ignored resources still show up in searches.
> It's a pretty soul-crushing task to mark the "build" and "publish" folders of > 135 projects as derived, as there's no programmatic way of doing so IResource.setDerived(true) is the programmatic way to mark resources as derived.
By programmatic I mean scriptable external to Eclipse, i.e. using ant, a bash script, or the like.
I've been thinking about writing import export like behaviour for derived in given project or (workspace) but for now i could trace derived flag state in resources. other idea is to gather this information during export.
(In reply to comment #25) > By programmatic I mean scriptable external to Eclipse, i.e. using ant, a bash > script, or the like. > Maybe Eclipse Monkey (http://www.eclipse.org/dash/) can help you?
(In reply to comment #6) > Won't fix. Persistent properties and attributes such as the derived bit are > metadata that live only within a workspace. I could say that derived resources live only within a workspace too, so why we export them at all? If one wanted to export them (and import later), he should clear the derived flags. This could be also achieved via an option in the export wizard. This option could be used to choose whether we want to export derived resources (derived flags will be cleared) or not. In my opinion the one of possible solutions would be an option to export a part of the workspace (not project(s)). The result of the export would be projects' folders and .metadata with all those properties that we want to export.
(In reply to comment #28) > (In reply to comment #6) > > Won't fix. Persistent properties and attributes such as the derived bit are > > metadata that live only within a workspace. > > I could say that derived resources live only within a workspace too, so why we > export them at all? If one wanted to export them (and import later), he should > clear the derived flags. I agree. But there is also another issue. Please consider the following scenario: 1. Create a project and generate a few source files, let's say java files using external tool like xjc from JDK. 2. Refresh your project/workspace so that you can see those files in Package Explorer. 3. Mark manually those files as derived - derived information is saved in workspace. 4. Commit the project to repository. Generated java files are not committed because of their derived flag and that is fine. 5. Create new workspace and import the project from repository. 6. Run the generator to recreate those java files locally. 7. Refresh your project/workspace. 8. Try to commit the project - you will see that generated files are recommended for commit because they don't have derived flag. In such case I think it would be better to store derived flag within project so when we checkout a project from repository (or just import a project) and then regenerate all files that are needed but not stored within repo, derived state is "visible" and generated files are set as derived just after generation. Please note that generating files can be done by external tool like xjc that cannot set derived flag for generated files. Possibly, this could be supported (or even solved) by running xjc as Ant task with additional task setting derived state (see bug 30440). > This could be also achieved via an option in the export wizard. This option > could be used to choose whether we want to export derived resources (derived > flags will be cleared) or not. > > In my opinion the one of possible solutions would be an option to export a part > of the workspace (not project(s)). The result of the export would be projects' > folders and .metadata with all those properties that we want to export. I think exporting .metadata is not a good idea. If we need some information to be exported or committed to repository, then such information should be moved from workspace .metadata to project .settings or anywhere else inside project so that everything that is needed to create a clone of the project in another workspace is stored within project.
I would love to see this issue resolved. I can help with development, if someone points me in the right direction. (i.e., I Volunteer!) A few things, first: - Correcting this issue would also handle Bug 154089, "Add ability to "remember" that a resource is derived when said resource is deleted". - I think many people have provided valid use cases of storing this information in the project settings. It seems the most common reason is that the resource is created by an outside build process (e.g., ant, maven, javadoc, your-favorite-codegen tool, etc.), and it should always be 'Derived', even if the resource is deleted and then re-created. I'm not sure what the best place to remember this information is. Looks like we have a few certain candidates: 1. .settings 2. .metadata 3. .project 4. .derived (new file) It appears some of those are bad choices, but I wouldn't know. If someone could point me in the right direction (i.e., give me some class names in the Eclipse source tree), I can write a patch. Or work with someone, since this issue is in ASSIGNED status anyway. P.S. A very specific use case is for the maven-eclipse plugin, which generates .project and .settings, and it wants to be able to do IResource.setDerived() on the 'target' directory (http://jira.codehaus.org/browse/MECLIPSE-126).
*** Bug 345419 has been marked as a duplicate of this bug. ***
(In reply to comment #30) > P.S. A very specific use case is for the maven-eclipse plugin, which generates > .project and .settings, and it wants to be able to do IResource.setDerived() on > the 'target' directory (http://jira.codehaus.org/browse/MECLIPSE-126). I'd like to plug this use case too. It would be so helpful for Maven users to be able to generate Eclipse project files that contain the correct derived settings. This becomes increasingly important when the projects contain large numbers of automatically generated files (e.g. hundreds of XMLBeans-produced classes). To further compound this, many source control plugins will create markers upon spotting files that are not under version control (and are not in derived folders). It becomes a race between the source control plugin and my ability to right-click and choose "derived". It's a race I've lost on occasions (or forgetting to participate in), resulting in the manual deletion of a markers file containing thousands of "helpful" reminders!
I'm using Eclipse Java EE, Indigo SR1, build 20110916-0149 with EGit. When I set this flag manually, it's lost during normal usage in the same workspace during the same day. What's the point of search option "Consider derived resources" when I cannot rely on this flag? My project has many files, but I have many thousands of generated files. When using search, it's great to have option to exclude some directories with generated files. I cannot use *.ext filter because I have *.html files as templates in projects and generated *.html files (and many similar cases, that's not the only one).
Just to let you know, there is (quite simple) alternative about this problem (in case someone would be interested, even if it has nothing to do with the bug itself). You just have to configure Maven to *not* delete the target/ directory but, instead, to clean what's inside. By doing this : you put the Eclipse "derived" flag on your target/ folders as usual. As Maven won't delete it, the flag will stay forever. The Maven configuration for that is pretty straightforward (it's a good idea to have a common parent root between projects in order to mutualize this configuration). <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-clean-plugin</artifactId> <version>${versionPluginClean}</version> <configuration> <excludeDefaultDirectories>true</excludeDefaultDirectories> <filesets> <fileset> <directory>target</directory> <includes> <include>**/*</include> </includes> </fileset> </filesets> </configuration> </plugin> Hope this could help.
(In reply to comment #34) It does help, thanks Serge.
I could provide one case when it's needed to reuse derived flag in a project-specific way. It's about using maven nested projects. Assume we have following project structure parent-module/pom.xml parent-module/module1.1/pom.xml parent-module/module1.1/... parent-module/module1.2/pom.xml parent-module/module1.2/... Importing existing maven project into eclipse we get a new project for each pom.xml: parent-module/pom.xml parent-module/.project parent-module/module1.1/pom.xml parent-module/module1.1/.project parent-module/module1.1/... parent-module/module1.2/pom.xml parent-module/module1.2/.project parent-module/module1.2/... So if we look for "some file" into workspace we'll get something like this: parent-module/module1.1/some_file.txt module1.1/some_file.txt Of course there is no use in the first match, so we have to manually set module1.1 and module1.2 as derived resources in parent-module to avoid such things. But if there a lot of modules in our project it's very boring to set all of them as derived each time there is need to (re)import projects. I see two solutions for this: 1. Fix this bug 2. Change default eclipse behaviour in such cases - eclipse could automatically mark folder as derived if it's the root folder of the existing project in workspace. Please, let me know if there is an existing issue.
Mikhail: The derived flag doesn't need to be used for this. A better solution is to use resource filters: - Open properties of parent-module - Resource > Resource Filters - Add filters to exclude module1.1 and the others This hides these directories in parent-module. The configuration is stored in .project, so it can be shared.
I really miss this feature (i.e.: storing the "Derived" state in project metadata, so it could be shared) or an equivalent shareable one. Our use case is somewhat similar to those stated here: folders that are filled in by external programs (Gradle in our case). Their contents can be generated Java source files, the output of the Google GWT compiler or the output of a process to to do inter-project WAR overlaying. The need to mark such folders as "Dervived" is because: 1. their contents isn't meant to be shared in the SCM repository 2. their contents isn't meant to be validated 3. their contents isn't meant to be found in searches 4. for some of them, their contents must not be hidden (on the contrary, they must be regularly refreshed!) because they must take part to other Eclipse processes (like WTP publishing, for instance) Because of 4. we can't use resource filters on all of such folders. Because of 2. and 3., it isn't just a problem of marking those folders as "ignored" by the version control system. The "Dervied" state is about project setup, hence I would expect it to be shared along with all the other project settings (like source folders, build path, source encoding, compiler compliance, and so on).
There's a pretty decent plugin that addresses this issue: http://nodj.github.io/AutoDeriv/
Are there any possibility to have this features in photon successors? it is very useful to exclude folders (for example build folder)
AutoDeriv (mentioned above) works fine for me (and is also in the Eclipse Marketplace), but there's another possible workaround which doesn't require a plugin: Choose "Properties" of your project(s) in the project explorer; in "Resource -> Resource Filters" click on "Add Filter..." and choose an "Exclude all" filter on the directories/files you don't want.
(In reply to Andreas Fischl from comment #41) > Choose "Properties" of your project(s) in the project explorer; in "Resource > -> Resource Filters" click on "Add Filter..." and choose an "Exclude all" > filter on the directories/files you don't want. See my previous comment as to why adding a filter is not equivalent to setting the "derived" flag. So, this is a workaround that covers just a subset of use cases.
