fix: change ts_query definition to avoid double-stemming #19581
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #19580
When performing sparse queries with words ending in double-E's (such as refugee), relevant database entries would not be returned. This is because the Snowball stemmer is not idempotent on double-E words (i.e.
stem(stem('refugee')) 'refug' != stem('refugee') = 'refugee', and the query was being stemmed twice while the database entries were stemmed only once.This is a consequence of the definition of ts_query, which first transformed the string into a ts query, then to text so the AND operators could be replaced with OR operators, and finally back into a ts query.
Instead, we can replace the spaces with "|"'s in the initial query string, so that the to_tsquery transformation need only be applied once. This will ensure an OR query while stemming only one time.
Note: this assumes English stemming and may have unintended consequences for other languages/encodings, depending on how spaces work in the corresponding Snowball stemmers.
New Package?
Did I fill in the
tool.llamahubsection in thepyproject.tomland provide a detailed README.md for my new integration or package?Version Bump?
Did I bump the version in the
pyproject.tomlfile of the package I am updating? (Except for thellama-index-corepackage)Type of Change
Please delete options that are not relevant.
How Has This Been Tested?
Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.
Suggested Checklist:
uv run make format; uv run make lintto appease the lint gods