aether-users

Martin Koci wrote:
Can you detail all chains of objects/fields holding the artifact instances?

The holding structure is
[...]
org.apache.maven.project.MavenProject ->
org.sonatype.aether.util.DefaultRepositorySystemSession ->
org.sonatype.aether.graph.Dependency

Am I right to assume that the omitted part of the chain is
MavenProject.projectBuilderConfiguration
-> DefaultRepositorySystemSession.cache
?
If so, I suggest to have m2e reset/clear the cache once dependency 
resolution is done or otherwise use a size-restricted cache impl.
Sorry, it's really chaotic. Let's have Dependecy A. Then:
project B: depends on A
project C: depends on B and A
As result, Project C has dependency A twice: 1) direct dependency 2)
transitive dependency from B. When I remove the direct dependency, will
be resolution faster?

Potentially, it depends on whether the paths leading to A employ the 
same remote repositories, dependency management etc. If all factors that 
could affect the transitive dependencies of A are equal, the sub graph 
rooted at A is shared and not build again.
Note however, that removing a direct dependency when a project actually 
uses its code, is conceptually wrong. The better approach would rather 
be to exclude the transitive dependency if you really want to go down 
that path.
Theoreticaly, when we minimize dependencyManagement section, will be
resolution faster?

Theoretically, yes. dependencyManagement translates into some HashMap's 
that get consulted for each dependency to see whether its version or 
scope are managed and the dependency object needs to get updated. 
However, whether this processing has actually significant contribution 
to your case or not is something I can't tell. From what you said 
earlier, it seems conflict resolution is the biggest CPU task.

Benjamin
Hi, 

thanks for your answer. Comments below.

Benjamin Bentmann pÃÅe v Ät 19. 04. 2012 v 01:04 +0200:
> Martin Koci wrote:
> 
> > 1) creates  many instances of
> > org.sonatype.aether.util.artifact.DefaultArtifact: ~ 1 290 948 instances
> > org.sonatype.aether.graph.Dependency  ~ 1 283 202 instances
> > Those instances cannot be GCed, relations come from
> > org.apache.maven.MavenProject
> 
> Can you detail all chains of objects/fields holding the artifact instances?

The holding structure is
(simplified)org.eclipse.osgi.framework.internal.core.Framework ->
org.eclipse.m2e.core.internal.MavenPluginActivator ->
org.eclipse.m2e.core.internal.project.registry.ProjectRegistryManager ->
org.eclipse.m2e.core.internal.project.registry.ProjectRegistry.workspacePoms->
workspacePoms is Map ->
org.eclipse.m2e.core.internal.project.registry.MavenProjectFacade ->
org.apache.maven.project.MavenProject ->
org.sonatype.aether.util.DefaultRepositorySystemSession ->
org.sonatype.aether.graph.Dependency 

> Also, how many dependencies does your workspace have in total?
> 
Hard to say, but many. How to count them? In workspace are many project
with 100 dependencies or more. The number of instances from aether can
be correct, the workspace and its dependecies is really big. I'm not
reporting a bug in aether, just asking ;)
> > The problem is that process takes too much time and consumes too much
> > memory.
> 
> What are the concrete numbers for "too much"?
in terms of time : It depends on CPU of the computer: update
dependencies takes 20min - 45 min; we  use already m2e with option "do
not automatically contact remote repositories" so (I hope) that process
does not include a communication over net. Yourkitprofiler shows the
method NearestVersionConflictResolver.selectVersion as hotspot.

In terms of memory: Dependency instances retain 591MB, DefaultArtifact
550MB, together with other plugins allocates eclipse over 1400MB in
heap.

> 
> > B) Then: I noticed that when the same dependency is in "client" project
> > and also as a transitive dependency,  then is this DependencyNode
> > processed 2x: when I removed dependecies already present as transitive,
> > it improved performance a little bit.
> >
> > Question: Is this assumption correct? Is better to have use "optimize"
> > pom.xml in this way?
> 
> Even after reading this twice, I have no clue what this paragraph means. 
> Can you re-phrase?
> 
Sorry, it's really chaotic. Let's have Dependecy A. Then:
project B: depends on A
project C: depends on B and A

As result, Project C has dependency A twice: 1) direct dependency 2)
transitive dependency from B. When I remove the direct dependency, will
be resolution faster?

> > Question: are there any tips how to optimize pom.xml to avoid
> > unnecessary object created during dependency resolving?
> 
> No.
> 
> > For example, creates section  new instances from
> > aether domain? We have a big   section in parent
> > (for all 600 project) pom.xml.
> 
> Of course, dependency management is relevant for dependency resolution 
> so at some point manifests in corresponding objects to process it. Those 
> should however only be temporary and not persist once dependency 
> resolution is done.
> 
Theoreticaly, when we minimize dependencyManagement section, will be
resolution faster?

Many thanks,

Martin

> 
> Benjamin
> _______________________________________________
> aether-users mailing list
> aether-users@xxxxxxxxxxx
> http://dev.eclipse.org/mailman/listinfo/aether-users

Martin Koci wrote:
1) creates  many instances of
org.sonatype.aether.util.artifact.DefaultArtifact: ~ 1 290 948 instances
org.sonatype.aether.graph.Dependency  ~ 1 283 202 instances
Those instances cannot be GCed, relations come from
org.apache.maven.MavenProject

Can you detail all chains of objects/fields holding the artifact instances?
Also, how many dependencies does your workspace have in total?
The problem is that process takes too much time and consumes too much
memory.

What are the concrete numbers for "too much"?
B) Then: I noticed that when the same dependency is in "client" project
and also as a transitive dependency,  then is this DependencyNode
processed 2x: when I removed dependecies already present as transitive,
it improved performance a little bit.
Question: Is this assumption correct? Is better to have use "optimize"
pom.xml in this way?

Even after reading this twice, I have no clue what this paragraph means. 
Can you re-phrase?
Question: are there any tips how to optimize pom.xml to avoid
unnecessary object created during dependency resolving?

No.
For example, creates section  new instances from
aether domain? We have a big   section in parent
(for all 600 project) pom.xml.

Of course, dependency management is relevant for dependency resolution 
so at some point manifests in corresponding objects to process it. Those 
should however only be temporary and not persist once dependency 
resolution is done.

Benjamin
Hi,

I'm solving performance problems with maven "update dependecies"
process, mainly in Eclipse, but easy reproducible with maven from
command line:

enviroment: Eclipse with m2e, maven runtime 3.0.4 with latest aether
from sonatype 1.13.1, about 600 project in workspace (Question: is
anywhere maven updated to  aether form org.eclipse package?)

process update dependecies for whole workspace (after fresh eclipse
start) 

1) creates  many instances of 
org.sonatype.aether.util.artifact.DefaultArtifact: ~ 1 290 948 instances
org.sonatype.aether.graph.Dependency  ~ 1 283 202 instances
Those instances cannot be GCed, relations come from
org.apache.maven.MavenProject

2) invokes 1 mil. - 3mil method
org.sonatype.aether.util.graph.transformer.NearestVersionConflictResolver.selectVersion per one project, i.e . about 850 mil invocations pro whole workspace

The problem is that process takes too much time and consumes too much
memory.

A) First, I detected, that many instances of Dependency type are created
when too wide version range is used. I removed some unnecessary version
ranges - this helped a lot, but results are still slow.

B) Then: I noticed that when the same dependency is in "client" project
and also as a transitive dependency,  then is this DependencyNode
processed 2x: when I removed dependecies already present as transitive,
it improved performance a little bit. 

Question: Is this assumption correct? Is better to have use "optimize"
pom.xml in this way?

C) But I see still any instances of aether.* types in memory. This was
reported also in m2e mailing list as [1] and by me [2]

Question: are there any tips how to optimize pom.xml to avoid
unnecessary object created during dependency resolving?

For example, creates section  new instances from
aether domain? We have a big   section in parent
(for all 600 project) pom.xml.

Many thanks,

Martin

[1] http://dev.eclipse.org/mhonarc/lists/m2e-users/msg02275.html
[2] http://dev.eclipse.org/mhonarc/lists/m2e-users/msg02459.html

Yes, this looks like exactly what I had in mind, thanks!
On 04/17/2012 06:02 PM, Benjamin Bentmann wrote:

Is it possible to include a project's generated Jar on a local file
system as part of the Aether resolution? The project jar would have to
have precedence over the local and remote repositories.

What you describe sounds similar to Maven's reactor resolution which 
is implemented via a WorkspaceReader [0] set on the 
RepositorySystemSession.

Benjamin

[0] 
http://download.eclipse.org/aether/aether-core/0.9.0-SNAPSHOT/apidocs/org/eclipse/aether/repository/WorkspaceReader.html

_______________________________________________

aether-users mailing list

aether-users@xxxxxxxxxxx

http://dev.eclipse.org/mailman/listinfo/aether-users

Is it possible to include a project's generated Jar on a local file
system as part of the Aether resolution? The project jar would have to
have precedence over the local and remote repositories.

What you describe sounds similar to Maven's reactor resolution which is 
implemented via a WorkspaceReader [0] set on the RepositorySystemSession.

Benjamin

[0] 
http://download.eclipse.org/aether/aether-core/0.9.0-SNAPSHOT/apidocs/org/eclipse/aether/repository/WorkspaceReader.html
Hello Aetherians,
Is it possible to include a project's generated Jar on a local file 
system as part of the Aether resolution? The  project jar would have to 
have precedence over the local and remote repositories.
I started looking at SimpleLocalRepositoryManagerFactory and 
EnhancedLocalRepositoryManagerFactory with the thought that an Artifact 
could be registered to use the project jar vs standard resolution. I am 
having a bit of trouble reverse engineering the steps of execution and 
was hoping someone might have some insight and/or a better idea.
Any help is greatly appreciated.
thanks,
Michael
Carlo de Wolf wrote:
Whenever a version range, RELEASE or LATEST is requested (either in the
pom or through fmvn rewriting the request),
RepositorySystem.resolveMetadata is called. Then other components
(DefaultVersionResolver, DefaultPluginVersionResolver etc) will try to
open the file returned and parse it to get the versions overview.
Since JPP uses a different format it can't return any File which adheres
to maven-metadata.xml requirements.

AFAIU, you have basically two options: You either rewrite all those 
components reading metadata to deal with your style of metadata or you 
have your repository system generate the usual metadata files on the fly.
The latter case is the reason why repository connectors do not take URLs 
as input but artifacts/metadata. When the existing Maven components ask 
for for maven-metadata.xml which isn't already cached locally, a 
connector has the chance to either fetch the item as-is from its 
destination or generate it from other info.
And if neither a custom connnector nor your custom 
JPPLocalRepositoryManager fit your case, a custom MetadataResolver is 
another possibility to generate the metadata upon request.

Benjamin
On 03/26/2012 04:43 PM, Benjamin Bentmann wrote:

Carlo de Wolf wrote:
I'm looking at creating an alternative RepositorySystem

Why or what is the use case you are trying to solve?

Fedora currently uses mvn-rpmbuild to build Maven based projects. 
mvn-rpmbuild is a modified Maven which in essence ignores version 
designations. It can only be run on a system which has a JPP repository 
and adheres to a different directory layout (so Fedora and some RPM 
based friends).
We want to change this, so Fedora has a true Maven repository which 
allows multiple versions installed. But at the same time we also don't 
want to ship each and every version of each and every component (/ The 
World), so for almost all components only the latest and greatest is 
installed.
This in term means that most poms won't be able to find the version they 
request. mvn-rpmbuild solves this by ignoring the version part, fmvn 
however rewrites the request to use LATEST and pushes it back into the 
Maven infrastructure. This in turn allows fmvn to be run on other 
platforms besides Fedora.

I'm stuck at resolving a problem with returning metadata requests.

How exactly do those problems manifest themselves?

Whenever a version range, RELEASE or LATEST is requested (either in the 
pom or through fmvn rewriting the request), 
RepositorySystem.resolveMetadata is called. Then other components 
(DefaultVersionResolver, DefaultPluginVersionResolver etc) will try to 
open the file returned and parse it to get the versions overview.
Since JPP uses a different format it can't return any File which adheres 
to maven-metadata.xml requirements. Thus I have to return null, which 
ultimately leads to:

[DEBUG] Could not find metadata 
org.apache.maven.plugins:maven-install-plugin/maven-metadata.xml in ...
Personally I think it is bad to use a File as a means to convey data. 
The contents is 'soft-typed' and it entails extra resources just for 
some piece of data, which might as well be in memory. In that light JPP 
is just an example, other repository systems which want to hook into 
Aether (if there are any) would suffer the same problem.
Carlo
There is no such file in some cases, because the backing repository
system (JPP in my case) has a different format.

I have no clue what JPP is. In case this is relevant to your issue, I 
would appreciate if you could outline some of its specifics that are 
of importance here.
I also have no knowledge about Fedora. That's just to say I was not 
joking when I asked you to post here "detailing your use case". Until 
I understand what you try to achieve and what is currently blocking 
you, I cannot give proper advice.

Benjamin
_______________________________________________
aether-users mailing list
aether-users@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/aether-users

Carlo de Wolf wrote:
I'm looking at creating an alternative RepositorySystem

Why or what is the use case you are trying to solve?
I'm stuck at resolving a problem with returning metadata requests.

How exactly do those problems manifest themselves?
There is no such file in some cases, because the backing repository
system (JPP in my case) has a different format.

I have no clue what JPP is. In case this is relevant to your issue, I 
would appreciate if you could outline some of its specifics that are of 
importance here.
I also have no knowledge about Fedora. That's just to say I was not 
joking when I asked you to post here "detailing your use case". Until I 
understand what you try to achieve and what is currently blocking you, I 
cannot give proper advice.

Benjamin

aether-users

Re: [aether-users] performance: creation of DependencyNode, DefaultArtifact and Dependency

Re: [aether-users] performance: creation of DependencyNode, DefaultArtifact and Dependency

Re: [aether-users] performance: creation of DependencyNode, DefaultArtifact and Dependency

[aether-users] performance: creation of DependencyNode, DefaultArtifact and Dependency

Re: [aether-users] Using a projects generated Jar as part of the resolution

Re: [aether-users] Using a projects generated Jar as part of the resolution

[aether-users] Using a projects generated Jar as part of the resolution

Re: [aether-users] MetadataResult.metaData.file as a means to convey available versions

Re: [aether-users] MetadataResult.metaData.file as a means to convey available versions

Re: [aether-users] MetadataResult.metaData.file as a means to convey available versions