Bug 271281 - Enable resource sharing between web projects
Summary: Enable resource sharing between web projects
Status: NEW
Alias: None
Product: WTP Java EE Tools
Classification: WebTools
Component: jst.j2ee (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 enhancement with 12 votes (vote)
Target Milestone: Future   Edit
Assignee: Chuck Bridgham CLA
QA Contact: Chuck Bridgham CLA
URL:
Whiteboard: ProjectStructure
Keywords: helpwanted
Depends on:
Blocks:
 
Reported: 2009-04-06 03:16 EDT by Philippe Marschall CLA
Modified: 2013-12-06 09:09 EST (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Philippe Marschall CLA 2009-04-06 03:16:59 EDT
WTP does not allow for modular web projects where resources and JSPs come from several web projects. This means that all JSPs and static resources have to come from the "final" web project. Sharing JSPs and static resources between projects is not possible.

The main problem is that only Java utility modules and be added to web modules, not web modules itself.

This problem often surfaces in Maven projects. There work arounds like WAR overlays [1] exist. Maven integration projects are facing the same issue [2].

 [1] http://maven.apache.org/plugins/maven-war-plugin/overlays.html
 [2] http://jira.codehaus.org/browse/MNGECLIPSE-599
Comment 1 Mauro Molinari CLA 2009-04-06 04:44:24 EDT
This could be a duplicated of bug #107039.
Comment 2 Carl Anderson CLA 2009-04-06 13:57:09 EDT
We are going to revisit this area in WTP 3.2, but any help would be most appreciated.
Comment 3 Philippe Marschall CLA 2009-04-07 02:30:05 EDT
(In reply to comment #1)
> This could be a duplicated of bug #107039.
> 

It's similar but IMHO a different issue. As far as I understand the main idea behind bug #107039 is to have a flexible mapping. Sharing between projects is more an afterthought.

The idea behind this issue is to have sharing between projects. Flexible mapping is out of the picture for this one.
Comment 4 Philippe Marschall CLA 2009-04-07 02:32:22 EDT
(In reply to comment #2)
> We are going to revisit this area in WTP 3.2, but any help would be most
> appreciated.
> 

Help in what area? Code contribution? Testing? Guidelines?
Comment 5 Mauro Molinari CLA 2009-04-07 09:00:38 EDT
(In reply to comment #3)
> The idea behind this issue is to have sharing between projects. Flexible
> mapping is out of the picture for this one.

But if you had a flexible resource mapping support as described in that enhancement request you could share resources between projects, couldn't you?
Comment 6 Philippe Marschall CLA 2009-04-08 02:31:47 EDT
(In reply to comment #5)
> (In reply to comment #3)
> > The idea behind this issue is to have sharing between projects. Flexible
> > mapping is out of the picture for this one.
> 
> But if you had a flexible resource mapping support as described in that
> enhancement request you could share resources between projects, couldn't you?
> 

If it worked across projects and supported JSPs. My understanding is that cross project support in bug #107039 is more an afterthought or "nice to have" feature. Also bug #107039 seems to focus on having a UI ("will help us click and choose files ...") whereas this bug focuses on having the capability and ignores UI support.
Comment 7 Rob Stryker CLA 2009-08-20 22:13:37 EDT
First, I'd like to mention that there's a big contribution / UI change going in very shortly that allows you to have a UI to finally add / remove wb-resource mappings and dependencies in a way thats less weird and seems to interact directly with the underlying component.xml. That's first. This portion has already been done and all that's left is working it in properly.  (See but 277482 and feel free to look at the past JEE Status Meetings.)

There is also a BUNCH of discussion regarding what amounts to filesets to be added to the component.xml. Obviously we already have them with the wb-resource tag, but that's obviously very limited as they're project-relative paths. You can't map in resources from other projects. You can't map in out-of-workspace filesets. You can't map in, for example, a fileset from your plugin's secret-location, or what have you.

Our goal is to add an extensible uri / handle (similar to what dependencies has, but *actually* extendable via extension points) which can "map in" any set of resources you want. We'll try to provide some default types (classpath container, workspaceFolder, fsFolder, etc), but if those aren't enough for you, there will be an extension point.

Warning: This is still just in 'chatter' mode. We are *definitely* targeting 3.2 with it, but aside from adding "handle-uri" to the wb-resource tag's attribute list, no definitive agreements have come about yet. 

Assuming your concerns have mostly to do with deployment structure, this will attempt to solve that problem. 
Comment 8 Mauro Molinari CLA 2010-10-11 11:21:47 EDT
Hi Rob,
what's the state of this now?
Comment 9 Philippe Marschall CLA 2010-10-11 11:54:58 EDT
Can be closed IMHO. Linked resources and the new deployment assembly solve this for me.
Comment 10 Mauro Molinari CLA 2010-10-11 12:03:18 EDT
(In reply to comment #9)
> Can be closed IMHO. Linked resources and the new deployment assembly solve this
> for me.

Only partially. Moreover, although they are REALLY useful, for this particular use case they provide a solution which is much more complicated than it could be. I explained my point of view better in bug #107039 comment #5.
Comment 11 Rob Stryker CLA 2010-10-12 02:29:43 EDT
I'm not inclined to go adding functionality that's already possible via linked folders. It seems that linked folders help satisify all the equations. 

The only thing I could / would do is add another mapping type in the Deployment Assembly page which allows you to map to any folder in the workspace rather than just folders in your project. But, that would duplicate behaviour available via linked folders. 

Unless someone can create a compelling case why what's there now isn't sufficient, I'm inclined to let this one sit until I have more time.
Comment 12 Mauro Molinari CLA 2010-10-12 03:22:35 EDT
(In reply to comment #11)
> The only thing I could / would do is add another mapping type in the Deployment
> Assembly page which allows you to map to any folder in the workspace rather
> than just folders in your project. But, that would duplicate behaviour
> available via linked folders. 

This would be very useful, actually.

> Unless someone can create a compelling case why what's there now isn't
> sufficient, I'm inclined to let this one sit until I have more time.

With the improvement above, I think that this request would be almost satisfied. In fact, my main current concern is that by now a new user would find it hard to figure out how to assemble a webapp with resources coming from different projects: he has to be smart enough to know about linked resources. Imagine if something similar would have been done for the Java classpath... With your suggested improvement this would be solved.

The only remaining doubts that I can think of are the following ones. First, suppose you map many folders for webapp resources in the Deployment Assembly page: which is the main one? I mean, which is the folder that is searched for a web.xml and/or for a WEB-INF/lib folder? Does WTP search in all of them?

Moreover, if the same resource is available in more than one mapped folder, which copy will be considered/published? In other words, is there an implicit order in the mapped fodlers or is it random?
Comment 13 Rob Stryker CLA 2010-10-12 03:35:29 EDT
We've never allowed pulling resources in from other projects, so a lot of your questions are seriously undefined. For deployment, if multiple folders had a web.xml, I have no idea what would happen. 

This is one of the reasons why the servertools and jeetools teams are hesitant about allowing this much freedom for the user. It just opens up more and more questions that we, most likely, don't have the time to sort out and design properly. 

If I were to add in the ability to map to random workspace folders, I can almost guarantee the behaviour would be "dumb" and would be used solely for packaging and if there are multple of the same files, it'd be undefined which one wins out. 

Which is why we're hesitant to add it ;)
Comment 14 Mauro Molinari CLA 2010-10-12 04:27:38 EDT
(In reply to comment #13)
> We've never allowed pulling resources in from other projects, so a lot of your
> questions are seriously undefined. For deployment, if multiple folders had a

If you have a Dynamic Web Project you can create linked folders (or even virtual folders with linked resources in it) that point to other projects folders. With the additional benefits of variables, you can link resources with workspace-relative paths. In this way, you can mimic the desired result.

> web.xml, I have no idea what would happen. 

What about implementing a "mark as main webapp folder" feature?

> If I were to add in the ability to map to random workspace folders, I can
> almost guarantee the behaviour would be "dumb" and would be used solely for
> packaging and if there are multple of the same files, it'd be undefined which
> one wins out. 

What about implementing the "ordering" of the mapped folders in the Deployment Assembly page? I mean, the order of publishing would be the one defined by the mapped folders list in the Deployment Assembly page. Of course, a "move up"/"move down" feature would then be needed in order to let people rearrange items in the desired way.

I think that this bug may be considered closed when the Deployment Assembly page will allow to map folders from other projects, while bug #107039 (which I still think is striclty related to this one) could be closed after the implementation of the two features above ("main webapp folder" and ordering of mapped resources).

This is just my humble opinion, of course. It comes from an every day use of WTP for managing a complex web application.
Comment 15 Jamie Cramb CLA 2010-10-12 04:35:26 EDT
Hi Folks,

First post on here so bear with me! :)

This has been a bit of a stumbling block for us in our company as we work with Software Product Lines (http://www.sei.cmu.edu/productlines/).  We essentially build up a core set of reusable components over-time and when it comes time to construct a new product / solution we share in the existing components and fill in the gaps with product / solution specific functionality.

Here's a crude example... we could have developed:
*  A component called "login" which provides a Login Controller that lets a user log into the system & the a JSP as a view (e.g. login.jsp) to render the login form.
*  A component called "registration" which provides a Registration Controller that lets a user sign up for an account with the system,  and a set of JSPs to provide the associated views (e.g. registration.jsp, registrationconfirm.jsp, accountactivation.jsp).

We then get a requirement to build a product that has login & registration functionality so we:
* Create a new SCM archive for the product
* Share in the code for the login & registration components
* Create a new bootstrap application (e.g. "myproduct") that has our WEB/INF folder, web.xml, etc, and then glue all of the bits together to make a working application.

To achieve this with eclipse projects we would do the following:
* The login component would be its own project with a "Utility Module" project facet
* The registration component would be its own project with a "Utility Module" project facet
* The myproduct bootstrap application would also be its own project with a "Dynamic Web Module" project facet
* the myproduct project would then specify the login & registration projects as "Java EE Module Dependencies"

At the moment we have this sort of setup working great with the exception of components that are supposed to have static resources (JSPs, image resources, CSS, JavaScript, etc) version controlled with them; we need to version control the static resources with the components so they can be easily shared into new products / solutions... for example, we would want the JSP files from all of the components to be picked up and appear under the classpath of the Dynamic Web Module project at runtime.  We can achieve this at build time with things like ANT builds that copy & collapse the static resources together; however, we want to be able to take advantage of things like hot-deploy to speed up development productivity so it would be great if the Eclipse tooling could deal with this.  The only easy workaround we've found so far is manually copying the static resources into the Dynamic Web Modules project structure in the IDE but this isn't very pretty & can be rather error prone.

We have the following restrictions / requirements:
* Be able to dynamically deploy static resources from multiple Utility Module projects & support hot-deploy of those resources to aid in productivity.
* Avoid a solution that requires fixed paths (e.g. c:\a\b\c) as this will vary on a per product/solution/developer environment basis 
* Be able to collapse static resources into the same location in the Dynamic Web Modules classpath at runtime (i.e. collapse the contents of the src/main/webapp/WEB-INF folders in all of the Utility Modules into the WEB-INF directory on the Dynamic Web Modules classpath).

Anyway, I hope my ramblings help to highlight a valid use case for this feature and don't confuse too much :).  Also, if there's another way to solve this problem then please feel free to point us in the right direction.

Thanks in advance!
Comment 16 Mauro Molinari CLA 2010-10-12 05:02:18 EDT
(In reply to comment #15)
> Hi Folks,
> 
> First post on here so bear with me! :)

Hi Jamie,
this is exactly my own use case, too. As I said in my previous comments, with WTP 3.2 and Helios you can achieve this by using linked resources (and virtual folders) and variables. In your Dynamic Web Project you may create a virtual folder "login", then create linked resources in it that point to WORKSPACE_LOC/LoginUtilityProject/WebContent... files (where WORKSPACE_LOC is a variable that points to the workspace directory in your local filesystem). Then, in the Deployment Assembly page of your Dynamic Web Project, you just add the "login" virtual folder to the deployed folders and you are done. This actually works, but forces you to create a potentially complex virtual folder structure. Moreover, you should better work with the login component source code from its owning project, rather than from the Dynamic Web Project, because of the fact that the source control linking for that component is kept on the utility project, not on the Dynamic Web Project. Also, I don't know if there could be some side effects on refreshing on one side for changes made on the other side. You may have to do some tests (I did, but with simple scenarios; for production development I'm still grouping components in a single project and using a linked folder just for the JARs I want in WEB-INF/lib).
Comment 17 Rob Stryker CLA 2010-10-12 05:07:50 EDT
Honestly, even though it's convoluted and a bit confusing, I think using linked folders is probably the best way to handle this. A lot of the error checking is already done in the platform for this stuff, and I wouldn't want us to duplicate the work. It seems like it works fine now that I've given it a try. 

We've only started experimenting with new reference types in the deployment assembly section and already one of them was pulled or severely cut back. So we're currently hesitant to just start throwing new reference types all over the place which then need to be kept in sync and error-checked properly. 

It seems to me that if linked resources work, this is the proper solution. It's a little convoluted in that you have to map Proj1/randomfolder to Proj2/WebContent, and then in Project1 map that randomFolder should be deployed just like Proj1/WebContent, and I'd like to hear more discussion on it.

Problems at the API level just make this quite a bit of work for what might be limited gain on this issue. But if other adopter products decided this was the best idea, I might be able to get it in.
Comment 18 Mauro Molinari CLA 2010-10-12 05:18:41 EDT
(In reply to comment #17)
> Problems at the API level just make this quite a bit of work for what might be
> limited gain on this issue. But if other adopter products decided this was the
> best idea, I might be able to get it in.

I'm sure it would be a lot of work (especially at the API level, as you said) in order to improve this. However, to be honest, I think that any new user may expect to be able in Eclipse to map static web folders just like he/she does for Java source folders. The concept is pretty much the same, just the type of resources is different. Using linked resources for this specific use case sounds like a workaround to me. And one thing that really surprises me is that there seem to be not so many people that have such kind of needs about web application development and deployment. I may guess that they solve this in other (more or less effective) ways...

Anyway, I respect any decision you will take on this.
Comment 19 Chuck Bridgham CLA 2010-10-12 14:26:56 EDT
Hi - Carl just pointed me to this activity.  And in general, I agree with Rob.  The existing linked folder support in base eclipse allows for this type of shared resources.  

The one big limitation is any server won't have any idea what linked resources are, and it would depend on the server adapter publish to always copy these files to a temp location, fully assembling the module contents on each publish... this can be painfully slow.

Just curious - have you guys looked at the new servlet 3.0 shared resources folder(META-INF/resources)?  It was specifically created for your scenario sharing static contents.  and of course - web fragments for dynamic components.
Comment 20 Rob Stryker CLA 2010-10-12 23:58:21 EDT
>The one big limitation is any server won't have any idea 
>what linked resources are, and it would depend on the 
>server adapter publish to always copy these
>files to a temp location, fully assembling the module 
>contents on each publish... this can be painfully slow.

First of all, at least theoretically, if everyone were following servertools API only and not digging into JEE-specific code, Server adapters shouldn't *NEED* to have any idea what linked resources are ;) The servertools API gives you a list of resource folders and under that, files. There's no requirement in this API that this match any actual configuration on disk or in the workspace. 

As a proof of concept I could make you a module factory which has some list of resources from wildly different parts of the users disk ;) 

Second, some servers (like the generic servers) already do this intense copying and then assembling. Others have found libraries to help assemble without copying everything all over the place ;) 

I still don't really have any position at this moment and will need to discuss it with my team as well as you guys, whether the functionality seems redundant or not.
Comment 21 Jamie Cramb CLA 2010-10-14 12:57:45 EDT
Hi Mauro / all, thanks for the pointers so far.  I had a quick play with the workaround Mauro suggested using an STS (SpringSource Tool Suite) milestone release which is based on 3.6 (we are currently using a 3.5-based version of STS for day-to-day development).

I was able to:
* Create a spring webapp which had the webapp bootstrap configs (a Dynamic Web Module project)
* Create a component (a java project with a Utility Module facet) which held a simple Spring MVC controller & a folder structure with a JSP to render a view
* Use the linked resources option on the creation of a folder to share the JSP file into the directory structure of the webapp project & have it deploy & work on tcServer (tomcat-based).

Mauro, I'm not sure how the virtual folder in your example plays into the picture, can you elaborate a bit further on that part? (I've not used the virtual folder functionality before)

Based on my tests so far, and assuming that I've not missed a trick with the virtual folder mechanism, I think the linked resources mechanism will become unmanageable for projects with a large number of components that have static resources that need to be shared into the main web application project... consider the following:

My main web application project is going to have the following directory structure for its webapp folder:

- src/main/webapp
    + images
    + js
    + css
    - WEB-INF
        + tags
        + views

Each of my components is then going to contribute their own sets of static resources that are associated with the functionality they provide... for example an account component could have:

- src/main/webapp
    - images
        * accounticon.png
        * accountdetailsbanner.png
    - js
        * account.js
    - css
        * account-layout.css
    - WEB-INF
        - tags
	    * accountoverview.tag
            * accountdetails.tag
        - views
            * accounthome.jsp
            - accountsupport
                * accountsupporthome.jsp
                * accountcancellation.jsp

The only way I could see this working is to create linked resources using the following pattern:

webapplication/src/main/webapp/images/account -> WORKSPACE_LOC/component/src/main/webapp/images
webapplication/src/main/webapp/js/account -> WORKSPACE_LOC/component/src/main/webapp/js
webapplication/src/main/webapp/css/account -> WORKSPACE_LOC/component/src/main/webapp/css
webapplication/src/main/webapp/WEB-INF/tags/account -> WORKSPACE_LOC/component/src/main/webapp/WEB-INF/tags
webapplication/src/main/webapp/WEB-INF/views/account -> WORKSPACE_LOC/component/src/main/webapp/WEB-INF/views

This has a few drawbacks:
* There number of linked resources is going to get out of hand pretty quickly
* There is no way to collapse the static resources from multiple components into a single folder (i.e. sub-folders have to be created for each component)

Anyway, just my thoughts so far, I'm happy to keep discussing / testing alternatives going forward :)
Comment 22 Mauro Molinari CLA 2010-10-15 04:22:36 EDT
In order to avoid the need of subfolders, you may create links to single files:
File | New File | Advanced | Link to file in the filesystem.

Of course, in this way your project structure would become even more complicated and hard to maintain, also because when you add a new file in the component project you have to also create a new link in the webapp project.

Virtual folders can help you to create more complex structures, but at a first thought I fear they won't help you to get rid of the subfolders.

Virtual folders work in this way: you may create a virtual folder in your project. A virtual folder can contain only other virtual folders or linked resources (folders, files). So, with a virtual folder you are able to create a totally virtual path in your project, that has no physical counterparts on the filesystem.

For instance, in the webapp project you may create a virtual folder named "myvirtualfolder" and create a structure link the following

- myvirtualfolder (virtual folder)
   - component1 (linked folder, points to WORKSPACE_LOC/component1/account)
   - component2 (linked folder, points to WORKSPACE_LOC/component2/account)

In this way, when you deploy your webapp it will be deployed just like it had a physical folder structure with a "myvirtualfolder" folder and two folders inside it, one called "component1" (whose contents are those coming from WORKSPACE_LOC/component1/account) and another called "component2" (whose contents are those coming from WORKSPACE_LOC/component2/account).
Comment 23 Philippe Marschall CLA 2012-03-01 05:30:07 EST
With servlet 3.0 this is no longer an issue

  https://blogs.oracle.com/alexismp/entry/web_inf_lib_jar_meta

I believe this can be closed
Comment 24 Mauro Molinari CLA 2012-03-01 06:06:01 EST
(In reply to comment #23)
> I believe this can be closed

I do not fully agree. IMHO this is more about the flexibility the IDE provides to organize projects in the workspace, rather than a limit in the servlet specifications < 3.0.

So, I don't think an enhancement here would be useless just because the existence of the servlet 3.0 specification now.
Comment 25 Stanislav Marencik CLA 2013-12-06 09:09:23 EST
My workaround is to open .settings/org.eclipse.wst.common.component in war project and add such line:
<wb-resource deploy-path="/" source-path="../prjholder/src/main/resources"/>

It points to other project - jar project with web resources. It is impossible to add relative path starting with .. by Eclipse GUI. This should be improved.

For java code I'm using Deployment Assembly in war project.

Hotswap for both xhtml and java code works.