[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [rdf4j-dev] Newer Lucene Version

Hi,

the pull request now also contains a commit changing URI to IRI.

Side question: I used to amend my branch and force push the updates (i.e. to avoid too many commits and have a clean history). Unfortunately this is now reflected in the conversation stream of https://github.com/eclipse/rdf4j-storage/issues/71 (even though I did the changes in my own fork of the repository).

What is the best practice to avoid this? Should I avoid referencing #71 in my commit messages, and reference it only in the PR?

Besides that the change is now good from my side, except for the ElasticSearch Integration test. Unfortunately I cannot get it run from Maven (even with the workaround thing that works from Eclipse). Likely there are differences in the classpath or the way system properties are passed?


@Bart: If we do the upgrade we should also update jackson to 2.9.x, as far as I know there is a security vulnerability in one of the jackson libraries. We are already using 2.9.5 at the moment, so this upgrade should be straight forward. I might provide a change for this in the next week.

Best,
ÂAndreas






2018-05-24 20:14 GMT+02:00 Bart Hanssens (BOSA) <bart.hanssens@xxxxxxxxxxxx>:

Hello Andreas,


yeah, IMHO a few dependenciesÂ(not necessarily related to ES only) should be updated :-/


If time permits (knocking on wood), I'm planning to take a look at the logging libraries and maybe the tomcat/spring-stuff,

perhaps that might solve some of your issues as well...


Bart



From: rdf4j-dev-bounces@xxxxxxxxxxx <rdf4j-dev-bounces@xxxxxxxxxxx> on behalf of Andreas Schwarte <aschwarte10@xxxxxxxxx>
Sent: Thursday, May 24, 2018 5:10:12 PM
To: rdf4j developer discussions

Subject: Re: [rdf4j-dev] Newer Lucene Version
Â
Hi Bart, Jeen,

it is also my understanding that a new major version of Lucene (and others) can read the indices of the previous version, for version jumps this is not guaranteed. I vaguely remember that for some lucene version even two major versions were supported, however, in general the best practice is to re-index when upgrading to a new major version.

For the Lucene case I will see compatibilty when I do tests in our platform.


Regarding my change: from my point of view the upgrade on the code level is fine, all integration tests are working locally.

However, I have had a hard time of getting the ElasticSearch Integration test to run: it always complains a JarHell. I tried to follow the path (and resolved quite some inconsistencies), but this is a never ending story with all the load of dependencies. For example it complains with mockito and securemock being on the test classpath at the same time, the same for logging frameworks, hamcrest and its dependencies, http client minor versions (which are easy to resolve).

In the end I have given up and used the workaround described on https://stackoverflow.com/questions/38712251/java-jar-hell-runtime-exception (i.e. overriding JarHell on the classpath with noops, and disabling the security manager). With this I could successfully run all ElasticSearch Compliance tests in Eclipse.

Any ideas on how to resolve this for the integration build? The transitive dependencies are really a mess.

What is the proposal to continue?

I will also povide a commit in my change to update from "URI" to "IRI".

As a side note: the migration step to the next major version of the frameworks are simpler (except for elasticsearch where a Rest Client needs to be used)

Best,
ÂAndreas


2018-05-24 7:27 GMT+02:00 Bart Hanssens (BOSA) <bart.hanssens@xxxxxxxxxxxx>:

Hi Jeen,


> Good stuff, thanks Andreas. I'm not really familiar with the various lucene and solr versions, will this have any influence on existing > users with existing indexes? Or will it seamlessly upgrade?Â


it is my understanding that users are encouraged to reindex whenever there's a major (N+1) Lucene/Elasticsearch update...

And reindexing is probably required for upgrading to major version N+2 afterwards, because Lucene (or ES) is not guaranteed to be ableÂread index from version N-2...

(I forgot which one, or maybe both, I have to re-read my own mails :-)

So it would be a good thing to mention reindexing in the release notes



Best regards


Bart



From: rdf4j-dev-bounces@xxxxxxxxxxx <rdf4j-dev-bounces@xxxxxxxxxxx> on behalf of Jeen Broekstra <jeen.broekstra@xxxxxxxxx>
Sent: Thursday, May 24, 2018 1:11:09 AM
To: rdf4j-dev@xxxxxxxxxxx
Subject: Re: [rdf4j-dev] Newer Lucene Version
Â
On 23/5/18 18:42, Andreas Schwarte wrote:
> Hi Bart, all, > > the issue is now tracked in > https://github.com/eclipse/rdf4j-storage/issues/71 > > Also I provided an initial pull request with the current status to > https://github.com/eclipse/rdf4j-storage/pull/72 > > The updates of the Lucene Sail and Solr (Lucene 6.6.4 and Solr 6.6.4) > look solid already and tests are passing.
Good stuff, thanks Andreas. I'm not really familiar with the various lucene and solr versions, will this have any influence on existing users with existing indexes? Or will it seamlessly upgrade?

> Updating elasticsearch seems a bit more difficult. The code change > should be roughly fine (though I still left some preliminary comments > and TODO markers), however, I am still facing issues with the > integration test. It fails with errors complaining about a "JarHell", > i.e. different versions of transitive dependencies being loaded. I am > currently looking into the maven descriptors to see if I can resolve > this. > > Could someone of you already check the current PR and give feedback? > Especially the elasticsearch part, as I have never worked with it > before and do not have a testing environment. Lucen Sail I will test > in our test installation, so that I can definitely cover.
I'll have a closer look this weekend.

> Also one more question: the lucene based modules still use "URI" > instead of "IRI", should I also do the refactoring here? If you're up for it, by all means do.

Thanks,

Jeen



_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev



_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev