-
Notifications
You must be signed in to change notification settings - Fork 342
speed up chunking & add separator chunking #48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
20fed70
to
9900d35
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #48 +/- ##
=======================================
Coverage 94.36% 94.36%
=======================================
Files 11 11
Lines 1189 1189
=======================================
Hits 1122 1122
Misses 67 67 ☔ View full report in Codecov by Sentry. |
NOW I GUESS ALL SHOULD BE WELL : ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great works! Few typing errors I think
Hey, so this PR's got two main changes:
We're now converting docs to tokens in bulk, which is giving us a sweet 3X speed boost when dealing with a ton of docs (we tested it with 30k) by chunking_by_token_size. It's not gonna make much difference for small-scale stuff, but 30k is still pretty much toy-level (both industry and research usually work with way more). So yeah, this is definitely a solid upgrade.
We've added support for separator-based splitting without needing any extra dependencies. This splitting method tries to keep the grammar structure intact, meaning you'll always get complete clauses or sentences (if without any overlap). We tweaked the logic from langchain, so it might not be exactly the same, but it does what it says on the tin.