Skip to content

Create a high-performance ORC reader #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 147 commits into
base: master
Choose a base branch
from
Open

Create a high-performance ORC reader #16

wants to merge 147 commits into from

Conversation

ddrinka
Copy link
Owner

@ddrinka ddrinka commented Feb 8, 2023

The current ORC reading code is slow and allocated often. These changes focus on reducing allocations and batching rows efficiently to produce higher performance and scalability.

Thanks @akasarto for all your work on this.

ddrinka and others added 30 commits March 22, 2022 15:05
@ddrinka
Added initial ByteRangeProvider functionality.
c8f063f
@ddrinka
Adjust our target to .net 5.
d9ae610
@ddrinka
Added infrastructure for a Span-based file-tail reader.
97a5b9a
@ddrinka
Corrected class to interface.
2fab36f
@ddrinka
Added a statistics comparison for various data types.
45b6d16
@ddrinka
Initial work on index search.
3e64aba
@ddrinka
Clarified that GetChunkLength returns the compressed block length a…
3434651 
…nd not the decompressed length.
@ddrinka
Added support for parsing Stripes and initial support for indexes.
5c8e543
@ddrinka
Completed initial iteration of the index parser.
ddb62da
@ddrinka
Added unit tests.
48dbd77 
Fixed bugs discovered by testing.
Converted POCOs into records.
Made DataFileHelper usable in other assemblies.
@ddrinka
Adjusted the layout of RowGroupStatistics.
f91b2dc
@ddrinka
Include stubs for next development steps.
4941f49
@ddrinka
Added StripeColumnStatistics from Metadata section.
84cec81
@akasarto
Add initial integer RLE V2 optimized reader infrastructure.
7813ed3 
Add initial test app.
@akasarto
Fix tracking of bytes read and offsets.
b011923
@akasarto
Add raw optimized column readers infrastructure.
efcc0f2
@akasarto
Add missing readers for ints, dates, doubles, etc..
bfd11df
@akasarto
Proper handle PRESENT stream individual bits.
09f5292
@akasarto
Fix double reader.
20726b4
@akasarto
Fix timestamp reader.
1358c61
@akasarto
Fix binary reader.
405ef6e
@akasarto
Map positions for byte reader.
d25f347
@akasarto
Reuse internal stream buffers.
a1ae061
@akasarto
Move reader data acquisition to top level classes.
c8598a3
@akasarto
Add support for parallel reads.
10ad90a
@akasarto
Use parallel execution only for byte range call.
cadccbb
@akasarto
Add Amazon S3 byte range provider.
bf85221
@akasarto
Remove caching from byte range provider.
6798207
@akasarto
Update optimized reader unit tests.
d4bc7ae
@akasarto
Fix optimized reader unit test method names.
3ed3ba4
akasarto and others added 29 commits July 26, 2022 13:45
@akasarto
Reduce decompression buffer default length.
5fc8b41
@akasarto
General code improvements and cleanup.
37f65fc
@akasarto
Simplify filling buffers with byte range providers.
e75bdcc
@akasarto
Add smaller length for metadata decompression buffers.
02a93a1
@akasarto
Calculate remaining bits only once.
48298f2 
Fix RLE max values number calculation.
@akasarto
Integrate filters into columns.
5b47b6d
@akasarto
Revert "Integrate filters into columns."
797dd52 
This reverts commit 5b47b6d.
@akasarto
Remove unnecessary decompression length parameter.
044b850
@akasarto
Dynamically resize byte range and decompression buffers.
e7362d6
@akasarto
Remove last byte logic from booleans.
f06291e 
Update unit tests.
@akasarto
Refactor TradeDataSource classes.
35a6d3d
@akasarto
Add sample to read entire file.
214c825 
Update sample app names.
@akasarto
Improve range calculations.
a6a7a19
@akasarto
Fix decompression issue when reading entire file.
6515896
@akasarto
Fix reader not tracking correct number of values.
56fd6ab
@akasarto
Move RLE buffers ownership to top classes.
14a7688
@akasarto
Refactor byte RLE skip functionality for remaining bits.
477e318
@akasarto
Add comparison of all values between readers.
b0e8474
@akasarto
Ensure http byte range buffers are filled.
807ed44
@akasarto
Update TradeDataSource read sequence behavior.
f69d150
@akasarto
Boolean RLEs minor changes and cleanup.
1ade727
@akasarto
Fix partial zlib buffer decompression.
1c9fa42
@akasarto
Fix byte range and decompression caching.
0c7391e
@akasarto
Remove unnecessary buffer slicing.
3e7eb69
@akasarto
Remove external class when tracking last stream ranges.
9391009
@akasarto
Update number of last range possible streams.
148d501
@ddrinka
Merge pull request #9 from akasarto/optimizedReaderReviewed
20aec07 
Optimized reader reviewed
@akasarto
Review sample file columns and apps.
10d0999
@ddrinka
Merge pull request #11 from akasarto/refsCleanup
ab93abd 
Review sample file columns and apps.
@akasarto akasarto mentioned this pull request Aug 17, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ddrinka @akasarto