Description Johannes Dorn 2014-05-22 02:25:16 EDT

We may want to consider using stopwords in Snipmatch.
While "if" should certainly not be a stopword, others like "a", "are", etc. are probably not useful for the search.

We would need a handcrafted list of stopwords. Alternatively, we could filter the standard set of stopwords for java keywords.

There are two places where we can use them: During indexing and for queries.
Indexing stopwords prevent those words to be added to the index.
Query stopwords ignore those words during the search.


There are a couple of issues for our use case, which argue both for- and against using stopwords for both queries and indexing.

Snippet 1: arrayadd
Snippet 2: create a button
Snippet 3: something else
Stopwords: "a", "the"

Example 1:
Query: a

Result when not using stopwords: Snippet1, Snippet 2 - not ideal
Result when using stopwords for indexing and queries: nothing - bad - this is due to "a" being filtered, leaving the query empty.
Result when using stopwords for indexing only: Snippet 1 - perfect

Example 2:
Query: a button
Result when not using stopwords: Snippet 2 - perfect
Result when using stopwords for indexing and queries: Snippet 2 - perfect
Result when using stopwords for indexing only: nothing - bad - due to the AND connection, the query insists on the word/prefix "a", but the index doesn't contain the word or prefix "a".