Bug 569687 - [tycho-packaging-plugin] Parametrize the location of the source MANIFEST.MF file
Summary: [tycho-packaging-plugin] Parametrize the location of the source MANIFEST.MF file
Status: NEW
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Tycho (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 10
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-12-13 13:16 EST by Jose M. Arnesto CLA
Modified: 2021-04-28 16:51 EDT (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jose M. Arnesto CLA 2020-12-13 13:16:56 EST
Currently, the location of the source MANIFEST.MF file cannot be configured. The PackagePluginMojo class, in its updateManifest() method, always tries to get that file from the META-INF folder in the project base dir:

> File mfile = new File(project.getBasedir(), "META-INF/MANIFEST.MF");

After loading the manifest and modifying its Bundle-Version property, it is written to the target directory and then used for the creation of the jar file itself. 

Should the location of the source MANIFEST.MF file be configurable, you could use a MANIFEST.MF file in the PDE containing Maven properties, filter them using maven and then pass the result to the packaging plugin.
Comment 1 Mickael Istria CLA 2020-12-13 14:15:52 EST
(In reply to Jose M. Arnesto from comment #0)
> Currently, the location of the source MANIFEST.MF file cannot be configured.

Which specific other location would you like to use? Can you please elaborate about the use-case here?
Comment 2 Jose M. Arnesto CLA 2020-12-13 18:51:02 EST
My intention is to be able to use variables in the MANIFEST.MF file in the source folder. Something like this:

> Bundle-ClassPath: .,{$dtp.manifest.mavenclasspath}

Then, I would have Maven filter that file, replacing these variables with their values and then have tycho-packaging-plugin use the resulting MANIFEST.MF, instead of the one in the source folder.
Comment 3 Jose M. Arnesto CLA 2020-12-13 19:10:20 EST
The purpose of all of this is to be able to declare Maven dependencies in the POM.xml and having them bundled with my plugin without having to modify the manifest by hand.

Say I declare this dependency:

<dependencies>
   <dependency>
       <groupId>org.apache.commons</groupId>
       <artifactId>commons-collections4</artifactId>
       <version>4.1</version>
   </dependency>
</dependencies>	

Then, to have these dependencies bundled with my plugin, I use the copy-dependencies goal of the maven-dependency-plugin to have them copied to a lib directory in the target folder and, then use the additionalFileSets configuration option of the tycho-packaging-plugin plugin to have them included in my jar:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy-libraries</id>
            <phase>validate</phase>
            <goals>
                <goal>copy-dependencies</goal>
            </goals>
            <configuration>                    	
                <excludeScope>system</excludeScope>
                <outputDirectory>${project.build.directory}/lib</outputDirectory>
            </configuration>
        </execution>
    </executions>
</plugin> 

<plugin>
    <groupId>org.eclipse.tycho</groupId>
    <artifactId>tycho-packaging-plugin</artifactId>		
    <configuration>
        <additionalFileSets>
            <additionalFileSet>
                <directory>${project.build.directory}</directory>
                <includes>
                    <include>lib/*</include>
                </includes>
            </additionalFileSet>
        </additionalFileSets>
    </configuration>	
</plugin>

So far so good, but then the problem of adding those libraries to the manifest remains, and that is why I would like to be able to filter the MANIFEST. I would set something like this in the source MANIFEST.MF:

Bundle-ClassPath: .,{$dtp.manifest.mavenclasspath}

I would then use the maven-dependency-plugin plugin again to assign a proper value to that variable:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
		<execution>
            <id>build-compiler-mavenclasspath</id>
            <goals>
                <goal>build-classpath</goal>
            </goals>
            <configuration>
                <excludeScope>system</excludeScope>
                <fileSeparator>/</fileSeparator>
                <prefix>target/lib</prefix>
                <outputProperty>dtp.compiler.mavenclasspath</outputProperty>
            </configuration>
        </execution> 
    </executions>
</plugin> 

And then, if I could tell tycho-packaging-plugin to use the filtered version of the MANIFEST.MF file, all the process would be fully automatic and there would be no need to edit the MANIFEST.MF file by hand when adding more Maven dependencies or when changing their versions.
Comment 4 Christoph Laeubrich CLA 2020-12-14 01:32:17 EST
The problem is that Tycho reads the manifest in a very early stage and requires its information for dependency resolution so filtering it afterwards will not always work as expected.

Why do you need to embeed the dependencies into your plugin anyways? That should be avoided whenever possible.
Comment 5 Mickael Istria CLA 2020-12-14 01:46:29 EST
(In reply to Jose M. Arnesto from comment #2)
> My intention is to be able to use variables in the MANIFEST.MF file in the
> source folder. Something like this:
> 
> > Bundle-ClassPath: .,{$dtp.manifest.mavenclasspath}
> 
> Then, I would have Maven filter that file, replacing these variables with
> their values and then have tycho-packaging-plugin use the resulting
> MANIFEST.MF, instead of the one in the source folder.

So what you're interested in is filtering, not really specifying an alternative location, Please retitle the bug accordingly.

(In reply to Christoph Laeubrich from comment #4)
> The problem is that Tycho reads the manifest in a very early stage and
> requires its information for dependency resolution so filtering it
> afterwards will not always work as expected.

That's right. Some "non-payload" fields could be filtered because they are mostly ignored by Tycho, but some more important fields that are used by the dependency management or by tycho-compiler-plugin are read very early. Bundle-Classpath is one of those important fields.
Comment 6 Jose M. Arnesto CLA 2020-12-14 06:38:05 EST
(In reply to Christoph Laeubrich from comment #4)
> The problem is that Tycho reads the manifest in a very early stage and
> requires its information for dependency resolution so filtering it
> afterwards will not always work as expected.

If that is the case, perhaps adding a new parameter to the tycho-packaging-plugin whose contents were appended to the Bundle-Classpath would be a valid alternative, isntead of filtering the whole file.
 
> Why do you need to embeed the dependencies into your plugin anyways? That
> should be avoided whenever possible.

I know that is to be avoided, but sometimes you have no other alternative but to do that.
Comment 7 Jose M. Arnesto CLA 2020-12-14 06:45:32 EST
(In reply to Mickael Istria from comment #5)
> (In reply to Jose M. Arnesto from comment #2)
> > My intention is to be able to use variables in the MANIFEST.MF file in the
> > source folder. Something like this:
> > 
> > > Bundle-ClassPath: .,{$dtp.manifest.mavenclasspath}
> > 
> > Then, I would have Maven filter that file, replacing these variables with
> > their values and then have tycho-packaging-plugin use the resulting
> > MANIFEST.MF, instead of the one in the source folder.
> 
> So what you're interested in is filtering, not really specifying an
> alternative location, Please retitle the bug accordingly.

I am neither interested in specifying a different MANIFEST.MF file nor in filtering the manifest per se, but in a way to add additional contents to the Bundle-Classpath field.

> (In reply to Christoph Laeubrich from comment #4)
> > The problem is that Tycho reads the manifest in a very early stage and
> > requires its information for dependency resolution so filtering it
> > afterwards will not always work as expected.
> 
> That's right. Some "non-payload" fields could be filtered because they are
> mostly ignored by Tycho, but some more important fields that are used by the
> dependency management or by tycho-compiler-plugin are read very early.
> Bundle-Classpath is one of those important fields.

And what does Tycho do when it finds an unintelligible token (the variable to be replaced) in that classpath? According to the behaviour I have seen here, it just ignores it. Having said that, I only added that variable at the end, so it may behave differently should that token be in the middle of a more complex path. In any case, maybe allowing for appending additional contents to that field would be preferable as opposed to a more general filtering capability.
Comment 8 Jose M. Arnesto CLA 2020-12-16 05:53:51 EST
All in all, if I implement a new property so that the plugin may get the MANIFEST from a different location while preserving the current behavior when that property is absent and then submit it as Gerrit change, would there be interest into merging it?
Comment 9 Mickael Istria CLA 2020-12-16 06:44:16 EST
(In reply to Jose M. Arnesto from comment #8)
> All in all, if I implement a new property so that the plugin may get the
> MANIFEST from a different location while preserving the current behavior
> when that property is absent and then submit it as Gerrit change, would
> there be interest into merging it?

Probably yes, if it really helps. However, as Christoph mentioned, it's very likely to not be sufficient to cover your use-case. We'd avoid merging it into Tycho if it doesn't actually resolve or improve an existing user-story; such existing user-story and how the proposed change fixes it needs to be explicited first.
Comment 10 Jose M. Arnesto CLA 2020-12-20 07:10:36 EST
(In reply to Mickael Istria from comment #9)
> 
> Probably yes, if it really helps. However, as Christoph mentioned, it's very
> likely to not be sufficient to cover your use-case. We'd avoid merging it
> into Tycho if it doesn't actually resolve or improve an existing user-story;
> such existing user-story and how the proposed change fixes it needs to be
> explicited first.

Perhaps I shouldn't have gone to such detail, explaining the filtering part and so on. The only thing I actually need is to be able to tell the plugin to get its MANIFEST.MF from a different location. That is it. And, from the Tycho point of view, I think that should be it, as that added flexibility adds value out of itself (or, at least, does not reduce the capabilities of the plugin, as its use is optional and, when absent, the previous behavior remains). 

Why a given user needs to get the MANIFEST.MF from a different place should not be something Tycho cares about. I mean, I want to filter it before the plugin uses it, but other users could want to apply a different processing to it.

At the end of the day, I am a user. I know my user story. I know that modification covers it (I already implemented it here and it works). If that is not enough, I don't know what else I can do.
Comment 11 Mickael Istria CLA 2020-12-20 08:24:41 EST
> added flexibility adds value out of itself 

Adding extensibility also comes with a maintenance and usage cost that makes it worth evaluation whether a given user story can already have different more straightforward and easier to maintain solutions.

> Why a given user needs to get the MANIFEST.MF from a different place should
> not be something Tycho cares about. I mean, I want to filter it before the
> plugin uses it, but other users could want to apply a different processing
> to it.
> 
> At the end of the day, I am a user. I know my user story. I know that
> modification covers it (I already implemented it here and it works). If that
> is not enough, I don't know what else I can do.

I disagree here. one user willing something and providing one single approach to fix it doesn't mean that this user is willing something the project should natively support and that the proposed solution is the best one. We need more context and more analysis before we can just welcome yet-another-new-feature-to-maintain in the project.
Comment 12 Christoph Laeubrich CLA 2020-12-20 09:00:07 EST
I wonder if this can already be archived, tycho:package upports Archiver Config[1] and that config seems to allow specification of manifest file[2] already?


[1] https://www.eclipse.org/tycho/sitedocs/tycho-packaging-plugin/package-plugin-mojo.html#archive
[2] https://maven.apache.org/shared/maven-archiver/examples/manifestFile.html
Comment 13 Mickael Istria CLA 2021-04-08 18:09:38 EDT
Eclipse Tycho is moving away from this bugs.eclipse.org issue tracker to https://github.com/eclipse/tycho/issues/ instead. If this issue is relevant to you, your action is required.
0. Verify this issue is still happening with latest Tycho 2.4.0-SNAPSHOT
  if issue has disappeared, please change status of this issue to "CLOSED WORKFORME" with some details about your testing environment and how you did verify the issue; and you're done
  if issue is still present when latest release:
* Create a new issue at https://github.com/eclipse/tycho/issues/
  ** Use as title in GitHub the title of this Bugzilla ticket (may include the bug number or not, at your own convenience)
  ** In the GitHub description, start with a link to this bugzilla ticket
  ** Optionally add new content to the description if it can helps towards resolution
  ** Submit GitHub issue
* Update bugzilla ticket
  ** Add to "See also" property (up right column) the link to the newly created GitHub issue
  ** Add a comment "Migrated to <link-to-newly-created-GitHub-issue>"
  ** Set status as CLOSED MOVED
  ** Submit

All issues that remain open will be automatically closed next week or so. Then the Bugzilla component for Tycho will be archived and made read-only.