Skip to content

Conversation

@klauspost
Copy link
Owner

If we did not get a single match, assume that the data cannot be entropy compressed.

It is very likely that the data is random, and in the rare case it is not, we can live with it.

Before:

BenchmarkRandomEncodeAllFastest-12           100          10475190 ns/op        1001.01 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           100          11075653 ns/op         946.74 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             200           8126950 ns/op        1290.25 MB/s         139 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             200           8059989 ns/op        1300.96 MB/s          90 B/op          2 allocs/op

After:

BenchmarkRandomEncodeAllFastest-12           300           4425967 ns/op        2369.15 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           300           5108791 ns/op        2052.49 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             300           5085627 ns/op        2061.84 MB/s          89 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             300           5947031 ns/op        1763.19 MB/s          65 B/op          2 allocs/op

If we did not get a single match, assume that the data cannot be entropy compressed.

It is very likely that the data is random, and in the rare case it is not, we can live with it.

Before:
```
BenchmarkRandomEncodeAllFastest-12           100          10475190 ns/op        1001.01 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           100          11075653 ns/op         946.74 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             200           8126950 ns/op        1290.25 MB/s         139 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             200           8059989 ns/op        1300.96 MB/s          90 B/op          2 allocs/op
```

After:
```
BenchmarkRandomEncodeAllFastest-12           300           4425967 ns/op        2369.15 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncodeAllDefault-12           300           5108791 ns/op        2052.49 MB/s           0 B/op          0 allocs/op
BenchmarkRandomEncoderFastest-12             300           5085627 ns/op        2061.84 MB/s          89 B/op          2 allocs/op
BenchmarkRandomEncoderDefault-12             300           5947031 ns/op        1763.19 MB/s          65 B/op          2 allocs/op
```
@klauspost klauspost merged commit f82c96c into master Aug 14, 2019
@klauspost klauspost deleted the zstd-skip-random-faster branch August 14, 2019 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant