Community
Participate
Working Groups
It would be great to have a mechanism for importing plug-ins from the plug-ins tab through CVS. Fortunately, I have a patch in the waiting ...
Created attachment 73261 [details] Source and runtime of a plugin that allows importing of CVS code by highlighting in Plug-ins tab This is not great code, and ideally should be in the "Import As" group; unfortunately, that isn't going to work since there's no way of extending it. To get people who might want to use this under Europa, I'm attaching it as an installable bundle, along with the source code. This will give a menu icon (http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.pde.ui/icons/obj16/plugin_mf_obj.gif?view=co) in the toolbar, and when you open the PlugIns view and click on it, will attempt to load in the plugin (HEAD) into your repository. The way that it finds out where to get the plugin is as follows: From a known list of prefixes (calculated by hand), it consults the set of extensions which look like: <repository prefix="org.eclipse.core" location=":pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse"/> (there's a bunch of them in the bundle) It then constructs a PSF file (/tmp/def12345.psf) with that as includes: project="1.0,:pserver:anonymous@dev.eclipse.org:/cvsroot/eclispe,org.eclipse.core.whatever,org.eclipse.core.whatever,HEAD" Finally, it runs the ProjectSetImporter to bring the code into the workspace. Many, many things can go wrong. The plugin might not be stored there, for a start; so it's possible to give a different location: <repository prefix="org.eclipse.mylyn" location=":pserver:anonymous@dev.eclipse.org:/cvsroot/tools,org.eclipse.mylyn/"/> This will expand to: project="1.0,:pserver:anonymous@dev.eclipse.org:/cvsroot/tools,org.eclipse.mylyn/org.eclipse.mylyn.whatever,org.eclipse.mylyn.whatever,HEAD" It's pretty much guaranteed to fail if your final project level isn't the name of the plugin. Also, you're going to get that name in your workspace whether you like it or not. In the highly likely event of it not being able to find the right location, there's a couple of things you can do: 1) Add an extension <repository prefix="com.example" location=":pserver:..."/>. It will get consulted when you import something. The prefix can be as long or as short as you want; the examples have 3 levels because that was easiest for me to work with, but you could have 1 character if you so wanted. (Zero is a special case. No, you can't have that.) 2) Add X-Eclipse-RepositoryLocation: :pserver:anonymous@dev.eclipse.org:/cvsroot/technology,the-foo-bar-project/ to your Manifest.MF. This overrides any pattern that might be found in the registry, but at least gives you the flexibility to say where to get things from. Plus, it might catch on. NB although I've called this 'org.eclipse.pde.ui.team', it's not sponsored by PDE, UI, Team, or anyone else that you might think of. It's just something I whacked together. If you like this, tell your friends. If not, tell me by leaving a bug report here. Any reports of non-working imports would be good (but by and large, I'm just intersted in the org.eclipse.* subset at this stage) so that I can add extra entries in for others to not trip over. I'm pretty sure there's going to be a lot of them; you can test by importing 'org.eclipse.equinox.net', which is a fairly small plugin and will validate that it works. Oh, and I know the UI is horrible. Hopefully, if it works out, it will get adopted in the standard Plug-Ins view, but that's not going to happen in any of the Europa releases. PS No, it doesn't do subversion. Technically no reason why it couldn't, except that the PSF generation assumes CVS through-and-through. I suspect PSF could handle SVN, so it might be possible.
Mik, these extension points might be something you need your voice on. There is some similarity with mapping to the issue tracker repositories.
Completely irrelevant; it's a different extension point.
(In reply to comment #3) > Completely irrelevant; it's a different extension point. Alex, this is at least slightly related because Mylyn is planning an extention point that will bind plug-ins to task repositories (see bug 150278 and its driver, bug 183606). Whether or not it makes sense to combine these two is another question, but we do have some related behavior encapsulated in the org.eclipse.mylyn.team.ui plug-in which abstracts Mylyn's coupling to repositories such as CVS and SVN, although there is no project-set specific support.
I think there's a difference between 'where the code is located' versus 'which bugs do I raise against'. The former is a CVS/SVN repository, whereas the latter is a bugzilla/jira instance (or whatever). One thing I'd like to see is encouraging people to record this information in the Manifest, rather than having to provide specifics in mapping files as I've done here. For example, if you put: X-Eclipse-RepositoryLocation: :pserver:dev.eclipse.org:/cvsroot/eclipse,/path/to/location in the bundle, it will use that directly. One could imgaine Mylyn doing similar things, like: X-Eclipse-IssueRepository: bugzilla,https://bugs.eclipse.org/ That way, there'd be no need for a mapping and people could supply the information themselves and Mylyn could introspect it. whether it catches on or not is a different matter ... Alex.
The problem with this approach is that it does not respect the separation producer / consumer. When companies are building products on top of eclipse, they actually want bugs to be entered against them rather than against eclipse, then they do the triaging and report eclipse bugs accordingly. In fact this separation is crucial for the scalability of our ecosystem since otherwise committers would be dying under bugs.
That's assuming that those products are open-source. If they're not, you can't import the plug-ins anyway. The difference would be if the companies in question shipped open-source contents (e.g. the equinox.common) but wanted bugs reported back to them, rather than Eclipse. I'd argue that PDE in itself is something that companies generally don't want to support. In any case, they can easily not ship this plug-in if it's something they don't want to do, right?
> The difference would be if the companies in question shipped open-source contents (e.g. the equinox.common) but wanted bugs reported back to them, rather than Eclipse. This is exactly the case! Which is why you can't put the bug tracking info in the bundle. As for having for having the code repo info in the manifest, I'm really mitigated but I guess it depends on what you are trying to achieve: a) getting the latest code, b) getting the code that corresponds to this binary.
The bug-tracking-in-bundle isn't really what this bug is about; it is just something that Mik/Eugene picked up on. It's not something that I care about particularly either for this bug. The purpose of this bug was to allow a PDE -> Import As -> Checkout from CVS, in much the same way that you can import binary content/source etc. Given that I don't know where all bundles came from, I provided a couple of extension mechanisms to do this. One was the repository extension point; another was an entry in the manifest. I can imagine that there's some 'core' entries that you wouldn't want them in, but other open-source people shipping bundles (e.g. the Scala plugin, for example) might well want to advertise that fact. If it's thought that this wouldn't need to be part of the main UI, then I''m happy for it to be in a different bundle. However, I'd like to propose a patch in that case that would allow actions to be contributed ot the Plug-ins action view, such that if this plugin (or something like it) was shipped and available, it was possible to import the items from CVS. I could also extend the format to allow a particular version identifier, so that the exact same version was used. That would be the best solution, but it was a quick-and-dirty way of demonstrating it in the hope it might be useful to others.
I'm with Pascal. People who sell products based on Eclipse should be supporting their products. Eclipse is in essence an implementation detail. We are not the world's support team. To Alex's point, knowing the location of the source for a bundle is interesting. It would be good to understand more about the usecase that requires the source to be checked out as opposed to simply being available. (not questioning that there are usecases, just want to hear what Alex has in mind). For additional context, we are going to start shipping source in individual bundles rather than the source plugins that we currently ship. Orbit has already started doing this and the Eclipse project will follow suit soon. As part of that work we have been talking about ways of decoupling binary and the source. This bug may be relevant there.
Primarily, if I find a bug and fix it, I can't do so easily from the source that gets bundled with the plug-in itself. I seem to recall that the layout in the source bundle isn't the same as the layout if you check it out from version control, what with some teams (e.g. JDT UI) using multiple source folders. I could be wrong on that one, however. The biggest problem is that the source has no version information, so I can't contribute a patch (other than copy-n-paste of the examples) and even when I do, the code might have moved on so the patch can't be (easily) applied by the receivers. All of this means that the source, whilst useful for looking and seeing how things are done, is basically useless when it comes to supplying patches. I appreciate that most committers have their projects checked out full time and are working with that on a daily basis, but for those bungee contributors such as myself, it cuts down on the amount of time it takes me to turn around a bug fix. To some extent, having the team data in the source (of which there's an enhancement request separately) would limit the benefit of this bug. However, with both your and Pascal's points about supportability; I also think this is useful from a non-Eclipse plugin's point of view (or for that matter, some of the smaller projects like ECF). I could ship uber-plugin_1.0.0.jar, not bother to package up the source, but have the information in the manifest which would allow me to check out a version so that I could obtain the source that I wanted. I may like to provide a bug tracking instance in the manifest so that I could encourage others to raise bugs. The point is, I see this as an infrstructure piece that may be useful to others too, not just Eclipse.org projects. So whilst I take your comments on board (and indeed, that wasn't the purpose of this bug in the first place), there may be benefit in having the option to contribute an issue URL, even if some (all?) of the Eclipse.org projects choose not to use it. The code I've thrown together here just checks out HEAD, and obviously, you'd want a way of specifying a specific version (so you know you get the same code base). Having the CVS info in the plug-in's source would also be another way of doing that. NB this was originally posed by a question on EclipseZone by someone wanting to more easily contribute fixes back into the Eclipse infrastructure; and anything that makes that easier has to be good, right? However; Mik et al; can you raise a separate bug relating to the use of issue-tracking-urls-in-the-manifest, so that this one can concentrate on providing-a-source-repository-url as well as the checkout-plugin action.
thanks for the extra information Alex. Yes this is something that we and others have struggled with. Indeed it is as you say. The source bundles are there for viewing/debugging, not developing/patching. For the latter you need to check out the source. Pascal points out that embedding this information in the manifest itself is unfortunate since you, as a producer, are not necessarily aware of the context in which people will consume your bundle. This is true throughout the eco-system. Perhaps a different approach that used say look aside files or markup in the provisioning data used to install the plugins would be better suited to maintaining hte separation of concerns? Or some sort of discovery mechanism where you appeal to a server with the bundle information and receive back information on how to get the source or log a bug. BTW, I seem to recall that Maya has some support in this direction but do not recall exactly what.
The example of the X-Eclipse-RepositoryLocation is something that can be used to override behaviour, but doesn't have to be used. Indeed, in the plug-in I've contributed, I've coded the relationship between plugin and repository location as extension points, which could easily be deployed in fragments or additional bundles (I seem to recall you can't supply extensions in fragments ...) The attached plug-in reads a map between plug-in name (e.g. org.eclipse.europa.*) and where it lives (e.g. dev.eclipse.org...) so simple prefixes can cover a number of plug-ins. In the case of the producer-vs-consumer debate, I don't think this necessarily needs to be supplied as part of the PDE plug-ins view. This information can be contributed dynamically through the installation of subsequent extension points, which could be supplied as a separate bundle (like .source.) that downstream users of Eclipse don't normally distribute. The main reason for putting the code to handle the other cases is where other bundles from open-source suppliers (e.g. Apache, Log4J etc.) might want to specifically advertise where they come from. My point is that whilst I personally think encoding the information in the manifest is a good way to go, I can see the objections that some might have and it's not the only way; the extension point mechanism I've provided can be supplied by 3rd party bundles or in-bundle. The plug-in just does the importing from where it's been told to import, nothing more or less :-) I'd suggest that fragmenting the attached into the plug-in that does the code (move into plugins view) and a separate one or ones for the extension points that currently define the mapping. Those who choose to create their own bundles have the option of adding extra information to the manifest, adding an extension point in-bundle, or adding an extension point in a separate bundle. Alex.
*** Bug 208998 has been marked as a duplicate of this bug. ***
In reponse for bug 208998 comment 1: I am not saying I am opposed to Project from CVS. But personally for me it would be such a shortcut like "Revert to BASE" insted of Replace With-> Another Branch or Version and so on. I am just tired with checking out plugins one by one when I want to have minimalistic set in my workspace, and eclipse plugins are outdated. Every time I have to launch the wizard, browse for projects... and then I see, that fresh plugins have dependencies, that are outdated... And again the whole fun. What I would like to achieve is *fast* converting plugin dependencies into the usbale source code.
If I understand this issue properly, it seems that there are two main stumbling blocks here: 1) How do you properly calculate the the PSF entry for a particular project? 2) How do you deliver this with an Eclipse build? I agree with Pascal that this information should not be in the binary bundle. I also don't like the idea of using extension points for the same reason. However, it could be in the source bundle or, better yet, delivered in a separate file either associated with the feature or product. Having it separate would allow products to ship without source but still let consumers easily obtain the source. The form of the mapping could simply be a project to PSF file entry for that build. There are two ways you could acheive this: a) The file could be generated by PDE build. This is the preferable solution since PDE build knows exactly how it got the source and could easily create the proper manifest. b) You could create a PSF entry for a particular consumable that had a variable where the qualifier goes. For an Eclipse CVS project, it would look something like: :pserver:anonymous@dev.eclipse.org:/cvsroot/eclispe,org.eclipse.core.runtime,org.eclipse.core.runtime,%QUALIFIER% The source importer would need to substitute the %QUALIFIER% variabe with the bundle qualifier before attempting the import. This solution would not require any changes to PDE build but would not work if there wasn't a direct mapping from a qualifer to a repository tag. This is not a problem for CVS but may be for other repository types A nice property of either approach a) or b) is that it will work for a repository provider that handles project sets. The PDE Import Page wouldn't even need a separate tab. It could just have another option that was "Import source from code repository if available". If there was a project to source code repository mapping, the code would be fetched, otherwise, it would be imported from the source bundle.
Note that as an extension point, it can be contributed outside of the binary bundle. It doesn't have to be in the same bundle as the code I showed above; I merely demonstrated this as a possibility. However, there may be many open-source projects that would like to have the code/URL made available through their manifest. Choice is good, right? Why should this work only for Eclipse-hosted bundles? The {manual~automated} PSF approach is something that's only going to apply to the Eclipse packaging of the product. I'm not suggesting that Eclipse should do the same as an open-source project might want to, but having a marker field in the manifest is a great way of communicating that information if people want to use it. We already have Bundle-UpdateLocation in the spec; what's the difference between that and (say) X-Bundle-SourceLocation? And, in much the same way that Eclipse don't use the Bundle-UpdateLocation, it wouldn't have to be the same for this requirement either.
Created attachment 160522 [details] patch This patch adds support to PDE to read SCMURLs from Eclipse-SourceReferences headers. Specifically for 'scm:cvs:" headers. Extensibility is provided by two new extension points: * org.eclipse.pde.core.bundleImporters which allows delegates to be contributed to handle importing bunldes (from a repository to a workspace project) based on manifest entries * org.eclipse.pdde.ui.bundleImportPages which allows a wizard page to be contributed to the plug-in import wizard for a specific bundle importer In PDE, I have provided a bundleImporter extension for scm:cvs SCMURLs, as well as a simple wizard page (needs polish), to allow users to import from HEAD instead of specific versions. I need to update PDE build side to create scm:cvs: urls rather than :psf format URLs.
Created attachment 160533 [details] updated patch A few updates.
Applied to HEAD.
NB this could be equally well used for standalone JARs, not just bundles. Is there any way of making the importer more generic, by moving to a Team plugin rather than PDE/UI? Obviously, PDE/UI is needed (to select the bundle from the bundles list) but one could equally well argue that such a header could be present in e.g. an ant.jar file included by a JDT project.
String project = element.getAttribute(ATTR_PROJECT); 80 if (project == null) { 81 String bsn = (String) manifests[i].get(Constants.BUNDLE_SYMBOLICNAME); 82 if (bsn != null) { 83 ManifestElement[] bsnElement = ManifestElement.parseHeader(Constants.BUNDLE_SYMBOLICNAME, bsn); 84 project = bsnElement[0].getValue(); 85 } 86 } This defaults to using the bundle name if project isn't present. However, it would be better to default to the last segment of the module, as noted in the other bug. Not only is it likely to be the case that the two match anyway, but there's nothing (necessarily) bundle-specific about the import mechanism.
This patch is adding a whole lot of new API without much time for reviewing it. I really like the functionality, but wouldn't it make sense to have that new API as provisional in Eclipse 3.6, such that it can be hardened in the next release cycle? Or is this API already in a shape that we'll want to maintain even into the e4 era? From a high level, it seems odd that the extension points are in PDE. In my understanding, the team provider (such as subclipse or subversive) or a 3rd party plugin should know how to marshal / unmarshal a repository reference to an SCMURL. Why does this knowledge have to be registered into PDE? Shouldn't that be on a lower level? It would feel more natural if PDE knew how to extract the SCMURL from the Manifest, then the team provider knew how to materialize the SCMURL into a team project set or something else on disk, and then PDE again knew how to import this stuff on into a PDE project.
>I really like the functionality, but wouldn't it make sense to have that new >API as provisional in Eclipse 3.6 Once we freeze the API for a release there is no such thing like provisional. For details please see http://wiki.eclipse.org/Provisional_API_Guidelines. We can either: - go ahead and freeze the API with M6 if we feel condifent - ask PMC to extend the freeze for this until M7 and mark it provisional for now - make everything internal (with some friends) so we can test that out and make a good API for 3.7. Interested parties can test and comment on the support using the internal code but with the knowledge that this will change for 3.7.
(In reply to comment #23) > From a high level, it seems odd that the extension points are in PDE. In my > understanding, the team provider (such as subclipse or subversive) or a 3rd > party plugin should know how to marshal / unmarshal a repository reference to > an SCMURL. Why does this knowledge have to be registered into PDE? Shouldn't > that be on a lower level? > > It would feel more natural if PDE knew how to extract the SCMURL from the > Manifest, then the team provider knew how to materialize the SCMURL into a team > project set or something else on disk, and then PDE again knew how to import > this stuff on into a PDE project. Yes, ideally this should be in the Team plugin. As noted elsewhere, if I can obtain a SCMURL from anywhere else (non-bundle Manifest, even CDT Makefile) then it would be nice to use the same generic functionality outside of PDE. PDE UI does need to be involved for the selection of a bundle, but the responsibility should stop at the acquisition of the SCMURL. (In reply to comment #24) > Once we freeze the API for a release there is no such thing like provisional. > For details please see http://wiki.eclipse.org/Provisional_API_Guidelines. > > We can either: > - make everything internal (with some friends) so we can test that out and make > a good API for 3.7. Interested parties can test and comment on the support > using the internal code but with the knowledge that this will change for 3.7. I'd vote for internal. That way, we get the functionality in PDE ahead of 3.6 but still allow (internal) uses. Then we can extract out the providers into Team Core in 3.7 if needed. I hope the experience with this will lead to ratification of OSGi bug 48, in which case we might need to change the header to Bundle-SCMURL anyway (or equivalent).
Having generic function for importing SCMURLs in team providers sounds like the right thing to do. There is already an API in team that could support this: org.eclipse.team.core.ProjectSetCapability.asReference(URI, String) Currently, the implementation does not support SCM style URI/URLs, but this could be implemented, and would provide headless support for importing a set of SCMURLs. However, from and end user point of view, we still need a UI to allow the user to modify/configure the import - i.e. a wizard page where the user can say "import all this stuff from HEAD" or "import all tihs stuff from a branch" instead of using specific versions. The API provided by team does not currently allow any of this manipulation gernerically (likely because it is provider specific). So, if team were to provide a corresponding UI extension point to contribute a wizard for importing a set of URIs, then each team provider could have provider-specific features in their wizards, and PDE could re-use these wizards in the plug-in import wizard (as could other tooling, such as JDT). Based on the current time line, this is, of course, not realistic. To proceed, we could mark the existing extension points as internal, and move the implementations to internal.provisional packages for 3.6. This would still allow for extensibility if needed, but allow us to change the implementation in 3.7. The API in PDE build fetch factories for generation of SCMURLs should be able to remain as API, since that part of the story does not change.
(In reply to comment #26) > However, from and end user point of view, we still need a UI to allow the > user to modify/configure the import Do we? - I think I could live with always importing what's in the Manifest. I can still "Team > Switch to another branch" afterwards if needed. Note that in the bundles that come with e.g. Eclipse 3.5 SR1, not all the sources are released from the maintenance branch, so entering such branch information in a wizard would be problematic anyways. There might be a need for mapping repositories (e.g. to address different :extssh: user on export versus import), like CVS allows on project set import today. But if I'm not mistaken that's an internal callback from below the PSF import so there might not be a need for API. > To proceed, we could mark the existing extension points as internal, and move > the implementations to internal.provisional packages for 3.6. This would still > allow for extensibility if needed, but allow us to change the implementation > in 3.7. Given that your current patch works but there's still some open questions whether it's right, this seems to be the right approach to me. And Alex already agreed on this course of moving forward :) > The API in PDE build fetch factories for generation of SCMURLs should be able > to remain as API, since that part of the story does not change. Ok for me, since the PDE/build ant tasks don't have access to the Team component ... they are operating in another realm, correct?
(In reply to comment #27) > (In reply to comment #26) > > However, from and end user point of view, we still need a UI to allow the > > user to modify/configure the import > Do we? - I think I could live with always importing what's in the Manifest. I > can still "Team > Switch to another branch" afterwards if needed. Note that in > the bundles that come with e.g. Eclipse 3.5 SR1, not all the sources are > released from the maintenance branch, so entering such branch information in a > wizard would be problematic anyways. You can check out a version and then replace, but it takes twice as long, and if you're importing a lot of projects it would be nice to offer this option to the user. Most SDK developers/committers will want this, where as most patch providers will just want to get a version. > There might be a need for mapping repositories (e.g. to address different > :extssh: user on export versus import), like CVS allows on project set import > today. But if I'm not mistaken that's an internal callback from below the PSF > import so there might not be a need for API. Yes, I think team will do this for us already. > > To proceed, we could mark the existing extension points as internal, and move > > the implementations to internal.provisional packages for 3.6. This would still > > allow for extensibility if needed, but allow us to change the implementation > > in 3.7. > Given that your current patch works but there's still some open questions > whether it's right, this seems to be the right approach to me. > And Alex already agreed on this course of moving forward :) > > The API in PDE build fetch factories for generation of SCMURLs should be able > > to remain as API, since that part of the story does not change. > Ok for me, since the PDE/build ant tasks don't have access to the Team > component ... they are operating in another realm, correct? Correct, PDE build operates on the file system - i.e. no workspace/resources or team support.
(In reply to comment #28) > the user. Most SDK developers/committers will want this, where as most patch > providers will just want to get a version. Ok, I had only thought about patch providers. I do see how a single HEAD OVERRIDE checkbox would make sense (note that HEAD OVERRIDE can be implemented independent of the provider by just stripping the "tag" from the SCMURL). I'm still not so sure about other branches. But I could imagine that at least some team providers might have "interesting" options to set in a UI.
Created attachment 160644 [details] moves the API to internal.provisional
(released to HEAD)
Created attachment 160671 [details] Use the last part of the url, rather than the bundle name, for the default project Changes the default to use the last part of the URL as the project name if not specified, rather than the bundle symbolic identifier. This is applicable for all SCMURL importer types (not just CVS) and can be applied to any project type in the future (Java, C/C++ etc.).
This will carry over to M7. The releng builder build won't actually be updated to use the new PDE build until after M6... so there won't be any headers to use until then.
Created attachment 165142 [details] patch Fix to import wizard. The CVS page was importing all bundles rather than just the bundles selected in the second page.
Created attachment 165144 [details] patch Updated to work with the express page as well.
Fixed. Please open new bugs as required.
