Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [rdf4j-dev] #220 lucene maches
Hi Mark,


I have created mock function2  (Test1Function and Test2Function) and suit tests for them. The functions are described but theirs jobs are trivial.  Everything is available in branch jgrzebyta:issues/#220-lucene-matches .

I have some questions:
1. Why do you want to keep QuerySpec if everything will be done in the TupleFunction.
2. QuerySpec are lauched in the beginning of the evaluation. What about if the query argument is unknown during the query construction. Will it work?

eg:

[ a:rawclass ;
  rdfs:label ?label ] .
(?subject ?score ?snippet) search:search(?label search:allMatches search:property x:myprop search:score search:snippet)



On 23 July 2016 at 12:40, Mark Hale <mj_hale@xxxxxxxxx> wrote:
I have worked out the details and propose we write a triplefunction that accepts the following parameters:
search:search(query, subject, propertyPredicate, property, scorePredicate, snippetPredicate)      [functional, not spin syntax]
where
query is the query string
subject is the query subject or the constant search:allMatches
propertyPredicate is the constant search:property or not present
property is present if only propertyPredicate is present and is the property to query or the constant search:allProperties
scorePredicate is the constant search:score or not present
snippetPredicate is the constant search:snippet or not present

the output should be in this order
subject, property, score, snippet
where
subject is included if the subject parameter is search:allMatches else omitted
property included if the propertyPredicate parameter is present and the property parameter is search:allProperties else omitted
score is included if the scorePredicate is search:score else omitted
snippet is included if the snippetPredicate is search:snippet else omitted

As a spin magic property, examples would be
(?subject ?score ?snippet) search:search("test" search:allMatches search:property x:myprop search:score search:snippet)
(?property ?snippet) search:search("test" x:me search:property search:allProperties search:snippet)

Now for the search syntax, the syntax stays the same. We still produce a QuerySpec, but we will convert the QuerySpec into a TupleFunctionCall of the above form. We can then replace the search syntax StatementPatterns in the query tree with this TupleFunctionCall and send it for (late) evaluation instead of the current early evaluation and replacement with a BindingSetAssignment.

Now for evaluation, the bean injection problem can be solved via ThreadLocal as I think you mentioned, or at evaluation time, creating a private TupleFunctionRegistry instance, create a SearchTupleFunction instance and passing the search index beans in the constructor, and adding the function to the registry. I believe this registry can then be passed in to TupleFunctionEvaluationStrategy rather than using the default static TupleFunctionRegistry. The ThreadLocal approach would have the advantage that the TupleFunction could be used as a spin magic property with spin as well as with the search syntax in the lucene sail (and any other magic property syntax). And, I think we can re-use the spin QueryContext thread-local by having a QueryPreparer subclass that includes access to the LuceneIndex. We can then cast to this subclass within the SearchTupleFunction.


On Friday, 22 July 2016, 12:39, Jacek Grzebyta <grzebyta.dev@xxxxxxxxx> wrote:


TupleFunction is for rdf function and for property functions

On 22 July 2016 at 12:11, Jacek Grzebyta <grzebyta.dev@xxxxxxxxx> wrote:
Hi Mark,

Thanks a lot for clarification. Yesterday I have read SPIN documentation and understand it more now. It seems the current search syntax is ok (according spin) but not valid for ARQ spec. I would no bother that because spin is more standardised. :-)

However I found another issues:
 1. Passing LuceneIndex from the lucene layer to the spin support layer. There is no bean injection mechanism. Also it is not like lisp where you have global variables,
 2. TupleFunction accepts in the evaluate method an array of Values. For some reason it is reduced somewhere in-between to array of literals. I have checked and BNodes should be accepted - after all they are instances of values.
 3. There is no support for spin's function call like it is showed in you example code. TupleFunction is a class for ....
 

Jacek Grzebyta





On 18 July 2016 at 21:46, Mark Hale <mj_hale@xxxxxxxxx> wrote:
Let me outline how things work in the spin function case, and then how this can be applied to search.

spin:
statements of the form
   (a b c) x:prop (? ? ?)
are replaced with a TupleFunctionCall to a TupleFunction x:prop with arguments a, b, c. You could express that with the notation x:prop(a,b,c). This is what SpinMagicPropertyInterpreter() does.
TupleFunctionCall is currently considered an optional algebra node (for reasons of backwards compatibility) so there is no core support for it. There are three possible ways to handle its evaluation:
 - the base sail has native support for it (most optimal)
 - use a "SERVICE" coprocessor. TupleFunctionCalls are expressed as
     SERVICE <spin:/> { (a b c) x:prop (? ? ?) }
   and the virtual federated endpoint <spin:/> knows how to evaluate it.
 - treat the base sail as a dumb TripleSource and handle all query evaluation at the sail wrapper layer using TupleFunctionEvaluationStrategy. TupleFunctionEvaluationStrategy can wrap any EvaluationStrategy to add support for TupleFunctionCalls.

search:
here we have statements of the form
?subj search:matches [
               search:query "search terms..."; 
               search:property my:property;

               search:score ?score;
               search:snippet ?snippet ] .
so this entire structure should be replaced by a TupleFunctionCall with arguments "search terms...", my:property and output ?subj, ?score, ?snippet.
The key here is that it is just the format of the statements that is different from spin, everything from there on down is the same. Hope that clarifies things. If you like, if you generate a TupleFunctionCall and produce TupleFunctions for search, I'll abstract out the evaluation mechanism from spin for you to re-use.

Some subtleties to be aware of, search:property I think is both an input and output and some of the output bindings are optional, e.g. if search:score is not present then ?score should not be bound. I would imagine then there would be boolean args like includeScore and includeSnippet so a call would be along the lines of  search:matches("search terms...", my:property, true, true)


On Monday, 18 July 2016, 20:31, Jacek Grzebyta <grzebyta.dev@xxxxxxxxx> wrote:


Hi,

I have tried to understand how TupleFunction implemetations works internally and created a MockFunction with suittests (commit fda04 in my branch issues/#220-lucene-matches). It seems that the syntax currently supported by the whole evaluation machinery is different than syntax of the lucene searching.

I have tried to find differences between simple and complex requests and found that
 simple case works:

?pred <urn:mockfunction/request> ("Foo" "Testmessage")

But complex request doesn't:

 ?pred <urn:mockfunction/request> ([ <urn:mockfunction/value> "test" ] "Testmessage")


The complex case was parsed into:

Projection
   ProjectionElemList
      ProjectionElem "pred"
   Join
      Join
         Join
            Join
               Join
                  StatementPattern
                     Var (name=_anon_bd318df4_a0e6_4893_9b6e_fe6aebb3b63c, anonymous)
                     Var (name=_const_2e12292d_uri, value=urn:mockfunction/value, anonymous)
                     Var (name=_const_364492_lit_e2eec718_0, value="test"^^<http://www.w3.org/2001/XMLSchema#string>, anonymous)
                  StatementPattern
                     Var (name=_anon_21882642_2409_40da_9a31_00011ddf0883, anonymous)
                     Var (name=_const_c5f92d90_uri, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#first, anonymous)
                     Var (name=_anon_bd318df4_a0e6_4893_9b6e_fe6aebb3b63c, anonymous)
               StatementPattern
                  Var (name=_anon_21882642_2409_40da_9a31_00011ddf0883, anonymous)
                  Var (name=_const_f5e424f4_uri, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#rest, anonymous)
                  Var (name=_anon_f0492b9f_91b9_46a1_83e6_715ac8bd40a8, anonymous)
            StatementPattern
               Var (name=_anon_f0492b9f_91b9_46a1_83e6_715ac8bd40a8, anonymous)
               Var (name=_const_c5f92d90_uri, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#first, anonymous)
               Var (name=_const_21b57b15_lit_e2eec718_0, value="Testmessage"^^<http://www.w3.org/2001/XMLSchema#string>, anonymous)
         StatementPattern
            Var (name=_anon_f0492b9f_91b9_46a1_83e6_715ac8bd40a8, anonymous)
            Var (name=_const_f5e424f4_uri, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#rest, anonymous)
            Var (name=_const_62c53cf1_uri, value=http://www.w3.org/1999/02/22-rdf-syntax-ns#nil, anonymous)
      StatementPattern
         Var (name=pred)
         Var (name=_const_25ad8b4b_uri, value=urn:mockfunction/request, anonymous)
         Var (name=_anon_21882642_2409_40da_9a31_00011ddf0883, anonymous)



So it does not loose information about the complexity. My question are:

1. Why the evaluators chain is trying to solve the request on theirs own way rather than give everything to <urn:mockfunction/request>? Is it because the Value argument which is expected by the TupleFunction interface and some of the evaluator is trying to get the value what is not possible?
2. If that is the way let us take the TupleFunction as a special case of more generic interface which would accept Statements, eg:

public CloseableIteration<? extends List<? extends Statement>, QueryEvaluationException> evaluate(ValueFactory valueFactory, Statement... args)

3. What for is TupleEvaluatioStrategy and when it is used in the evaluation chain?


Thanks a lot,
Jacek


_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev



_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev




_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev



_______________________________________________
rdf4j-dev mailing list
rdf4j-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/rdf4j-dev


Back to the top