How do I properly use zstd BuildDict() to generate a dictionary?

[ktime1]

Hi there,

I'm trying to build a zstd dictionary from a bunch of samples. However, I'm not sure what the history field refers to in the BuildDictOptions. It's a required field, as if you leave it out, BuildDict() returns the error dictionary of size 0 < 8.

So I believe it represents the actual dictionary contents, so I passed it an unpopulated byte slice of size 1024, to represent a max dictionary size of 1Kb. I was thinking BuildDict() would populate the slice.

	history := make([]byte, 1*1024) // 1 Kb
	options := zstd.BuildDictOptions{
		ID:       12345,
		Contents: samples,
		History:  history,
		Level: zstd.SpeedDefault,
	}
	dict, err = zstd.BuildDict(options)

However, this results in the error panic: BUG: unexpected error in ZSTD_compress_usingCDict_wrapper: Dictionary mismatch .

Can anyone help me understand how to properly use BuildDict()? Thanks for any help, I appreciate it!

[klauspost]

Try sending real data instead of just 0s.

[klauspost]

No it is a collection of examples of the files for which you want to build a dictionary. So if it is a JSON type you want it to compress you give it 1000 different JSON files that could be the output.

Maybe if you explain why you want to build a dictionary it would be easier to help.

[ktime1]

We want to build a dictionary for the typical reasons, improving compression ratios on our data. I have enough sample data to build the dictionary, but my confusion here lies in that the type of History is a []byte: https://github.com/klauspost/compress/blob/master/zstd/dict.go#L173

So I'm unclear on how to pass 1000 JSON files into a single byte slice.

[ktime1]

Oh I think I understand now. History is a previous dictionary. Contents is the new samples to enhance the previous dictionary. But it looks like currently there is no way to build a dictionary from scratch (besides the experimental builder).

[klauspost]

it looks like currently there is no way to build a dictionary from scratch (besides the experimental builder).

A "dictionary from scratch" doesn't make sense - you build it once and don't change it. You will only gain compression if you have common data that looks alike.

To quote https://facebook.github.io/zstd/

Training works if there is some correlation in a family of small data samples. The more data-specific a dictionary is, the more efficient it is (there is no universal dictionary). Hence, deploying one dictionary per type of data will provide the greatest benefits. Dictionary gains are mostly effective in the first few KB. Then, the compression algorithm will gradually use previously decoded content to better compress the rest of the file.

[klauspost]

Also I have no idea what the wrapper you are referencing is doing, since I obviously didn't write it.