Re: [rdf4j-dev] Newer Lucene Version

[Andreas Schwarte <aschwarte10@xxxxxxxxx>]

Hi Bart, Jeen,

it is also my understanding that a new major version of Lucene (and others) can read the indices of the previous version, for version jumps this is not guaranteed. I vaguely remember that for some lucene version even two major versions were supported, however, in general the best practice is to re-index when upgrading to a new major version.

For the Lucene case I will see compatibilty when I do tests in our platform.

Regarding my change: from my point of view the upgrade on the code level is fine, all integration tests are working locally.

However, I have had a hard time of getting the ElasticSearch Integration test to run: it always complains a JarHell. I tried to follow the path (and resolved quite some inconsistencies), but this is a never ending story with all the load of dependencies. For example it complains with mockito and securemock being on the test classpath at the same time, the same for logging frameworks, hamcrest and its dependencies, http client minor versions (which are easy to resolve).

In the end I have given up and used the workaround described on https://stackoverflow.com/questions/38712251/java-jar-hell-runtime-exception (i.e. overriding JarHell on the classpath with noops, and disabling the security manager). With this I could successfully run all ElasticSearch Compliance tests in Eclipse.

Any ideas on how to resolve this for the integration build? The transitive dependencies are really a mess.

What is the proposal to continue?

I will also povide a commit in my change to update from "URI" to "IRI".

As a side note: the migration step to the next major version of the frameworks are simpler (except for elasticsearch where a Rest Client needs to be used)

Best,

 Andreas

2018-05-24 7:27 GMT+02:00 Bart Hanssens (BOSA) <bart.hanssens@xxxxxxxxxxxx>:

Hi Jeen,

> Good stuff, thanks Andreas. I'm not really familiar with the various lucene and solr versions, will this have any influence on existing > users with existing indexes? Or will it seamlessly upgrade? 

it is my understanding that users are encouraged to reindex whenever there's a major (N+1) Lucene/Elasticsearch update...

And reindexing is probably required for upgrading to major version N+2 afterwards, because Lucene (or ES) is not guaranteed to be able read index from version N-2...

(I forgot which one, or maybe both, I have to re-read my own mails :-)

So it would be a good thing to mention reindexing in the release notes

Best regards

Bart

---

From: rdf4j-dev-bounces@xxxxxxxxxxx <rdf4j-dev-bounces@xxxxxxxxxxx> on behalf of Jeen Broekstra <jeen.broekstra@xxxxxxxxx>
Sent: Thursday, May 24, 2018 1:11:09 AM
To: rdf4j-dev@xxxxxxxxxxx
Subject: Re: [rdf4j-dev] Newer Lucene Version

 

On 23/5/18 18:42, Andreas Schwarte wrote:
> Hi Bart, all, > > the issue is now tracked in > https://github.com/eclipse/rdf4j-storage/issues/71 > > Also I provided an initial pull request with the current status to > https://github.com/eclipse/rdf4j-storage/pull/72 > > The updates of the Lucene Sail and Solr (Lucene 6.6.4 and Solr 6.6.4) > look solid already and tests are passing.
Good stuff, thanks Andreas. I'm not really familiar with the various lucene and solr versions, will this have any influence on existing users with existing indexes? Or will it seamlessly upgrade?

> Updating elasticsearch seems a bit more difficult. The code change > should be roughly fine (though I still left some preliminary comments > and TODO markers), however, I am still facing issues with the > integration test. It fails with errors complaining about a "JarHell", > i.e. different versions of transitive dependencies being loaded. I am > currently looking into the maven descriptors to see if I can resolve > this. > > Could someone of you already check the current PR and give feedback? > Especially the elasticsearch part, as I have never worked with it > before and do not have a testing environment. Lucen Sail I will test > in our test installation, so that I can definitely cover.
I'll have a closer look this weekend.

> Also one more question: the lucene based modules still use "URI" > instead of "IRI", should I also do the refactoring here? If you're up for it, by all means do.

Thanks,

Jeen

_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev