[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Newsgroup Home]
|
[news.eclipse.technology.buckminster] Re: buckminster and maven repositories
|
Hi Thomas,
I'm attaching a first cut at a patch for this. The changes I made are
localized to
org.eclipse.buckminster.maven.internal.Maven2VersionFinder.java. I am
attaching the new class as well as a diff against r2881. I have tested
this against http://repo1.maven.org/maven2 and against Artifactory. It
works fine for release versions of artifacts, but doesn't work properly
for SNAPSHOT versions at the moment. The correct SNAPSHOT artifact URLs
are being generated, but somehow it still fails to match against the
snapshot versions. I'm guessing that this has something to do with the
way the version comparison mechanism works, which I don't really
understand that well so I'd appreciate some guidance on that front. I'd
appreciate any other comments you had on this as well.
Thanks,
Edwin
Thomas Hallgren wrote:
Hi Edwin,
Our original intent was to write a mechanism that would do downloads and
resolution from the Maven repositories. Our first implementation was for
Maven 1. We later added support for Maven 2 but we didn't make it
perfect for the simple reason that by then, discussions had commenced
with people that were committers in the Maven project. They informed us
about the maven embedder technology that would soon be released in Maven
2.1 and they created a patch for a Buckminster maven provider that would
use this embedder. This was back in March 2007.
Unfortunately, Maven 2.1 has not yet been released and when it is, it
has to go through a Eclipse Intellectual Property review. Since it has a
dependency to 40 or so different jars, some with different licenses,
that is likely to take some time.
Buckminster is planning a graduation review on January 30. In
conjunction with that, we also plan to move from technology to tools.
This means that we will no longer be in incubation and hence, no longer
able to bundle things under what's called "parallel IP". This
effectively rules out everything that hasn't been fully IP approved by
the Eclipse EMO and hence, we are stuck with our own home-brewed maven
provider.
So question is, what do we do now. Do we wait until we can get an
approval? Or do we wait even longer until the proposed IAM project is
approved so that they will provide the needed API's? From the looks of
it, not much is happening on that front at present. Perhaps the wait is
over and we must improve our current Maven support. After all, it
wouldn't be that much work.
Would you be interested in having a go at it and perhaps submit a patch?
Regards,
Thomas Hallgren
Edwin Park wrote:
Hi,
I noticed that when buckminster tries to read information from a maven
repository, it assumes that the repository is a browseable apache
directory, and parses the html in order to lookup artifact versions.
This works for maven repositories that are implemented as browseable
apache directories, such as the central http://repo1.maven.org/maven2
repository, but it does not work for many other proxy repositories
(e.g. artifactory). It's good practice NOT to use the central repo,
however, and to use a proxy instead in order to avoid slamming the
central repo which is often slow, to insulate yourself from external
connectivity issues, control version changes, etc. etc.
The right way to address this would be to change the Buckminster
behavior to lookup artifact version information the same way the mvn
command does: by looking up appropriate maven-metadata.xml files in
the repository. These metadata files must be available and accessible
in any maven repository. For instance, if I go to the central repo and
want to find the available versions of org.springframework:spring
artifacts, rather than going to
http://repo1.maven.org/maven2/org/springframework/spring/ and parsing
the directory listing, the correct thing to do would be to GET the
http://repo1.maven.org/maven2/org/springframework/spring/maven-metadata.xml
file and look through its contents. It is much easier to parse and has
the same structure for every repository.
Using the maven-metadata.xml file will also help you support correct
behavior when accessing snapshots. Maven snapshots are identified by
versions with a trailing 'SNAPSHOT', e.g.
commons-dbcp:commons-dbcp:1.3-SNAPSHOT. However, the artifacts
actually stored in the repository will be named something like
http://people.apache.org/repo/m2-snapshot-repository/commons-dbcp/commons-dbcp/1.3-SNAPSHOT/commons-dbcp-1.3-20071125.225953-2.jar
- note that instead of SNAPSHOT, a timestamp is used. The
maven-metadata.xml file to the rescue again: when a client asks for
the 1.3-SNAPSHOT version, look at the maven-metadata.xml file:
http://people.apache.org/repo/m2-snapshot-repository/commons-dbcp/commons-dbcp/maven-metadata.xml
and you will see info about the timestamp and build number which you
can use to construct the proper url for your artifact GET request.
I have all my maven artifacts in an Artifactory repo and can't get to
them via Buckminster right now, so I'd love to see the proper behavior
implemented. I'd be happy to help with the implementation, but I'm not
that familiar w/the Buckminster internals so I'd need some guidance
there; and given that this is pretty straightforward thing, it would
probably be faster for someone more familiar w/the internals to do it
instead. If you do decide you'd like my help though, please let me know.
Cheers,
Edwin
/*****************************************************************************
* Copyright (c) 2006-2007, Cloudsmith Inc.
* The code, documentation and other materials contained herein have been
* licensed under the Eclipse Public License - v 1.0 by the copyright holder
* listed above, as the Initial Contributor under such license. The text of
* such license is available at www.eclipse.org.
*****************************************************************************/
package org.eclipse.buckminster.maven.internal;
import java.net.URI;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
import javax.xml.parsers.DocumentBuilderFactory;
import org.eclipse.buckminster.core.CorePlugin;
import org.eclipse.buckminster.core.ctype.IComponentType;
import org.eclipse.buckminster.core.resolver.NodeQuery;
import org.eclipse.buckminster.core.rmap.model.Provider;
import org.eclipse.buckminster.core.version.IVersionDesignator;
import org.eclipse.buckminster.core.version.VersionMatch;
import org.eclipse.buckminster.runtime.MonitorUtils;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.IProgressMonitor;
import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.NodeList;
/**
* @author Thomas Hallgren
*/
public class Maven2VersionFinder extends MavenVersionFinder
{
public Maven2VersionFinder(MavenReaderType readerType, Provider provider, IComponentType ctype, NodeQuery query)
throws CoreException
{
super(readerType, provider, ctype, query);
}
@Override
URL[] createFileList(IVersionDesignator designator, IProgressMonitor monitor) throws CoreException
{
Maven2ReaderType readerType = (Maven2ReaderType)getReaderType();
URI uri = getURI();
StringBuilder pbld = new StringBuilder();
readerType.appendFolder(pbld, uri.getPath());
readerType.appendEntryFolder(pbld, getMapEntry());
ArrayList<URL> fileList = new ArrayList<URL>();
String rootPath = pbld.toString();
int rootLen = rootPath.length();
String space = getProvider().getSpace();
monitor.beginTask(null, 2000);
try
{
NodeQuery query = getQuery();
for(String version : getVersions(query.getVersionDesignator(), readerType, uri, rootPath, MonitorUtils.subMonitor(monitor, 1000)))
{
VersionMatch versionMatch = MavenComponentType.createVersionMatch(version, space, null);
if(versionMatch != null && query.isMatch(versionMatch))
{
pbld.setLength(rootLen);
pbld.append(version);
pbld.append('/');
pbld.append(getMapEntry().getArtifactId());
pbld.append('-');
if (isSnapshotVersion(version))
appendSnapshotIdentifier(pbld, readerType, uri, rootPath, version, monitor);
else
pbld.append(version);
pbld.append(".jar");
fileList.add(readerType.createURL(uri, pbld.toString()));
}
}
return fileList.toArray(new URL[fileList.size()]);
}
finally
{
monitor.done();
}
}
private List<String> getVersions(IVersionDesignator versionDesignator, Maven2ReaderType readerType, URI uri, String path, IProgressMonitor monitor)
{
List<String> versionList = new ArrayList<String>();
try
{
Document doc = getMetadataDocument(readerType, uri, path + "maven-metadata.xml", monitor);
if (doc != null)
{
Element versioningElement = (Element) doc.getElementsByTagName("versioning").item(0);
Element versionsElement = (Element) versioningElement.getElementsByTagName("versions").item(0);
NodeList versions = versionsElement.getElementsByTagName("version");
for (int i = 0; i < versions.getLength(); i++)
{
String version = versions.item(i).getTextContent();
if (isSnapshotVersion(versionDesignator) == isSnapshotVersion(version))
versionList.add(version);
}
return versionList;
}
}
catch (Exception e)
{
CorePlugin.getLogger().warning(e, e.getMessage());
}
return new ArrayList<String>();
}
private void appendSnapshotIdentifier(StringBuilder pbld, Maven2ReaderType readerType, URI uri, String rootPath, String version, IProgressMonitor monitor)
{
try
{
Document doc = getMetadataDocument(readerType, uri, rootPath + version + "/" + "maven-metadata.xml", monitor);
if (doc != null)
{
Element versioningElement = (Element) doc.getElementsByTagName("versioning").item(0);
Element versionsElement = (Element) versioningElement.getElementsByTagName("snapshot").item(0);
Element timestampElement = (Element) versionsElement.getElementsByTagName("timestamp").item(0);
Element buildNumElement = (Element) versionsElement.getElementsByTagName("buildNumber").item(0);
pbld.append(version.substring(0, version.indexOf("SNAPSHOT")));
pbld.append(timestampElement.getTextContent());
pbld.append('-');
pbld.append(buildNumElement.getTextContent());
return;
}
}
catch (Exception e)
{
CorePlugin.getLogger().warning(e, e.getMessage());
}
throw new RuntimeException("Unable to read snapshot metadata");
}
private boolean isSnapshotVersion(String version)
{
return version.endsWith("SNAPSHOT");
}
private boolean isSnapshotVersion(IVersionDesignator versionDesignator)
{
if (versionDesignator != null)
return isSnapshotVersion(versionDesignator.getVersion().toString());
return false;
}
private Document getMetadataDocument(Maven2ReaderType readerType, URI uri, String path, IProgressMonitor monitor) throws Exception
{
URL url = readerType.createURL(uri, path);
Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(CorePlugin.getDefault().openCachedURL(url, monitor));
// Make sure groupId and artifactId in metadata document match map entry
String mapEntryGroupId = getMapEntry().getGroupId();
String mapEntryArtifactId = getMapEntry().getArtifactId();
String docGroupId = doc.getElementsByTagName("groupId").item(0).getTextContent();
String docArtifactId = doc.getElementsByTagName("artifactId").item(0).getTextContent();
if (mapEntryGroupId.equals(docGroupId) && mapEntryArtifactId.equals(docArtifactId))
return doc;
CorePlugin.getLogger().warning("Metadata at " + url + " has groupId:artifactId " + docGroupId + ":" + docArtifactId + " that does not match map entry " + mapEntryGroupId + ":" + mapEntryArtifactId);
return null;
}
}
Index: src/java/org/eclipse/buckminster/maven/internal/Maven2VersionFinder.java
===================================================================
--- src/java/org/eclipse/buckminster/maven/internal/Maven2VersionFinder.java (revision 8378)
+++ src/java/org/eclipse/buckminster/maven/internal/Maven2VersionFinder.java (working copy)
@@ -10,18 +10,22 @@
import java.net.URI;
import java.net.URL;
import java.util.ArrayList;
+import java.util.List;
+import javax.xml.parsers.DocumentBuilderFactory;
+
+import org.eclipse.buckminster.core.CorePlugin;
import org.eclipse.buckminster.core.ctype.IComponentType;
-import org.eclipse.buckminster.core.reader.URLCatalogReaderType;
import org.eclipse.buckminster.core.resolver.NodeQuery;
import org.eclipse.buckminster.core.rmap.model.Provider;
import org.eclipse.buckminster.core.version.IVersionDesignator;
import org.eclipse.buckminster.core.version.VersionMatch;
import org.eclipse.buckminster.runtime.MonitorUtils;
import org.eclipse.core.runtime.CoreException;
-import org.eclipse.core.runtime.IPath;
import org.eclipse.core.runtime.IProgressMonitor;
-import org.eclipse.core.runtime.Path;
+import org.w3c.dom.Document;
+import org.w3c.dom.Element;
+import org.w3c.dom.NodeList;
/**
* @author Thomas Hallgren
@@ -34,12 +38,6 @@
super(readerType, provider, ctype, query);
}
- private void appendFilesInFolder(URL folder, ArrayList<URL> fileList, IProgressMonitor monitor) throws CoreException
- {
- for(URL url : URLCatalogReaderType.list(folder, monitor))
- fileList.add(url);
- }
-
@Override
URL[] createFileList(IVersionDesignator designator, IProgressMonitor monitor) throws CoreException
{
@@ -56,31 +54,26 @@
monitor.beginTask(null, 2000);
try
{
- // Add entries from all matching folders
- //
NodeQuery query = getQuery();
- boolean defaultIsMatched = query.isMatch(null, null, getProvider().getSpace());
- for(URL versionURL : URLCatalogReaderType.list(readerType.createURL(uri, rootPath), MonitorUtils.subMonitor(monitor, 1000)))
+ for(String version : getVersions(query.getVersionDesignator(), readerType, uri, rootPath, MonitorUtils.subMonitor(monitor, 1000)))
{
- IPath versionPath = new Path(versionURL.getPath());
- int segCnt = versionPath.segmentCount();
- if(segCnt < 1)
- continue;
-
- String folderName = versionPath.segment(segCnt - 1);
- if(!defaultIsMatched)
+ VersionMatch versionMatch = MavenComponentType.createVersionMatch(version, space, null);
+ if(versionMatch != null && query.isMatch(versionMatch))
{
- // No use scanning this folder if the version is incompatible with
- // the query
- //
- VersionMatch versionMatch = MavenComponentType.createVersionMatch(folderName, space, null);
- if(versionMatch == null || !query.isMatch(versionMatch))
- continue;
+ pbld.setLength(rootLen);
+ pbld.append(version);
+ pbld.append('/');
+
+ pbld.append(getMapEntry().getArtifactId());
+ pbld.append('-');
+ if (isSnapshotVersion(version))
+ appendSnapshotIdentifier(pbld, readerType, uri, rootPath, version, monitor);
+ else
+ pbld.append(version);
+ pbld.append(".jar");
+
+ fileList.add(readerType.createURL(uri, pbld.toString()));
}
-
- pbld.setLength(rootLen);
- readerType.appendFolder(pbld, folderName);
- appendFilesInFolder(versionURL, fileList, MonitorUtils.subMonitor(monitor, 1000));
}
return fileList.toArray(new URL[fileList.size()]);
}
@@ -89,4 +82,94 @@
monitor.done();
}
}
+
+ private List<String> getVersions(IVersionDesignator versionDesignator, Maven2ReaderType readerType, URI uri, String path, IProgressMonitor monitor)
+ {
+ List<String> versionList = new ArrayList<String>();
+
+ try
+ {
+ Document doc = getMetadataDocument(readerType, uri, path + "maven-metadata.xml", monitor);
+ if (doc != null)
+ {
+ Element versioningElement = (Element) doc.getElementsByTagName("versioning").item(0);
+ Element versionsElement = (Element) versioningElement.getElementsByTagName("versions").item(0);
+ NodeList versions = versionsElement.getElementsByTagName("version");
+ for (int i = 0; i < versions.getLength(); i++)
+ {
+ String version = versions.item(i).getTextContent();
+ if (isSnapshotVersion(versionDesignator) == isSnapshotVersion(version))
+ versionList.add(version);
+ }
+
+ return versionList;
+ }
+ }
+ catch (Exception e)
+ {
+ CorePlugin.getLogger().warning(e, e.getMessage());
+ }
+
+ return new ArrayList<String>();
+ }
+
+ private void appendSnapshotIdentifier(StringBuilder pbld, Maven2ReaderType readerType, URI uri, String rootPath, String version, IProgressMonitor monitor)
+ {
+ try
+ {
+ Document doc = getMetadataDocument(readerType, uri, rootPath + version + "/" + "maven-metadata.xml", monitor);
+ if (doc != null)
+ {
+ Element versioningElement = (Element) doc.getElementsByTagName("versioning").item(0);
+ Element versionsElement = (Element) versioningElement.getElementsByTagName("snapshot").item(0);
+ Element timestampElement = (Element) versionsElement.getElementsByTagName("timestamp").item(0);
+ Element buildNumElement = (Element) versionsElement.getElementsByTagName("buildNumber").item(0);
+
+ pbld.append(version.substring(0, version.indexOf("SNAPSHOT")));
+ pbld.append(timestampElement.getTextContent());
+ pbld.append('-');
+ pbld.append(buildNumElement.getTextContent());
+
+ return;
+ }
+ }
+ catch (Exception e)
+ {
+ CorePlugin.getLogger().warning(e, e.getMessage());
+ }
+
+ throw new RuntimeException("Unable to read snapshot metadata");
+ }
+
+ private boolean isSnapshotVersion(String version)
+ {
+ return version.endsWith("SNAPSHOT");
+ }
+
+ private boolean isSnapshotVersion(IVersionDesignator versionDesignator)
+ {
+ if (versionDesignator != null)
+ return isSnapshotVersion(versionDesignator.getVersion().toString());
+ return false;
+ }
+
+ private Document getMetadataDocument(Maven2ReaderType readerType, URI uri, String path, IProgressMonitor monitor) throws Exception
+ {
+ URL url = readerType.createURL(uri, path);
+
+ Document doc = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(CorePlugin.getDefault().openCachedURL(url, monitor));
+
+ // Make sure groupId and artifactId in metadata document match map entry
+ String mapEntryGroupId = getMapEntry().getGroupId();
+ String mapEntryArtifactId = getMapEntry().getArtifactId();
+
+ String docGroupId = doc.getElementsByTagName("groupId").item(0).getTextContent();
+ String docArtifactId = doc.getElementsByTagName("artifactId").item(0).getTextContent();
+
+ if (mapEntryGroupId.equals(docGroupId) && mapEntryArtifactId.equals(docArtifactId))
+ return doc;
+
+ CorePlugin.getLogger().warning("Metadata at " + url + " has groupId:artifactId " + docGroupId + ":" + docArtifactId + " that does not match map entry " + mapEntryGroupId + ":" + mapEntryArtifactId);
+ return null;
+ }
}