Fold testvectors/ dir into testvectors_v1/

[cpu]

For a variety of historic reasons not particularly interesting to downstream projects Wycheproof currently has two directories with test vector data, testvectors/ (hereafter referred to as v0) and testvectors_v1/ (hereafter referred to as v1). This is a confusing situation and requires downstream projects to make a decision about which to use (v0, v1, both?) without clear documentation on the differences between them, or which will be receiving updates into the future. Similarly, new contributors aren't certain where to place new algorithm vectors, or updates to existing vectors. We also have to maintain two schema files for the cases where there is a divergence in format between v0 and v1 (e.g. dsa_p1363_verify).

This meta issue tracks work required to consolidate the vector data to a single unified directory, and motivates the choice to make that directory the v1 directory, discarding the v0 directory/structure. Within this issue discussion should be scoped specifically to the C2SP/Wycheproof test vector data as it exists in repo today, and concrete suggestions for a consolidation plan.

v0 or v1?

I think we should centralize on v1, principally for the reason that it maximizes the test vector coverage with the minimal amount of conversion work.

Here's my initial scoping, based more on algorithm coverage than test case coverage within algorithms. Please correct me if I've missed anything pertinent, there's a lot of files to go through :-)

From my first pass, there's not much in v0 that isn't present in v1/:

• ECDH w/ sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1
• Stand-alone JOSE tests (json_web_crypto_test.json, json_web_encryption_test.json, json_web_key_test.json, json_web_signature_test.json) - I think some of these are folded into per-algorithm v1 tests.
• RSA PKCS1 signature generation (rsa_sig_gen_misc_test.json) - originally reported in No RsassaPkcs1Generate tests in testvectors_v1 #89
• Tag-in-ciphertext AES-GCM-SIV DaeadTests (aes_siv_cmac_test.json)

Conversely, in v1 there's a much broader set of algorithms that aren't present in v0/:

• ML-KEM (mlkem_test.json)
• ML-DSA (mldsa_*_sign_[seed|noseed]_test.json, mlds_*_verify_test.json, ~9 files)
• ARIA (aria_gcm_test.json, aria_cbc_pkcs5_test.json, aria_wrap_test.json, aria_cmac_test.json, aria_ccm_test.json, aria_kwp_test.json)
• Camelia (camellia_wrap_test.json, camellia_ccm_test.json, camellia_cmac_test.json, camellia_cbc_pkcs5_test.json)
• SEED (seed_wrap_test.json, seed_gcm_test.json, seed_ccm_test.json)
• SM4 (sm4_gcm_test.json, sm4_ccm_test.json)
• SM3 (hmac_sm3_test.json)
• ASCON (ascon80pq_test.json, ascon128_test.json, ascon128a_test.json)
• MORUS (morus640_test.json, morus1280_test.json)
• AES-FF1 (aes_ffi_*_test.json ~22 files)
• PBES2 (pbes2_hmac*_aes_*_test.json, ~15 files)
• PBKDF2 (pbkdf2_hmacsha[1|512|224|256|384)
• KMAC (kmac128_no_customization_test.json, kmac256_no_customization_test.json)
• SipHash (siphashx_2_4_test.json, siphashx_4_8_test.json, siphash_2_4_test.json, siphash_1_3_test.json, siphash_4_8_test.json)
• SHAKE (ecdsa_secp*_shake_*_p1363_test.json, ecdsa_secp*_shake_*_test.json, rsa_pss_*_shake*_params_test.json, rsa_pss_*_shake*_test.json, ~21 files)

In sum I think the difference in algorithm support between the two makes consolidating on v1 seem most appealing.

Tasks

First, the v0 to v1 missing algorithm conversion work:

• Add/convert ECDH sect283k1, sect283r1, sect409k1, sect409r1, sect571k1, sect571r1 coverage to v1 - Convert missing v0 ECDH sect* test vectors to v1 #158
• Add/convert RSA PKCS1 sig gen to v1- Convert v0 RSA PKCS1 sig gen to v1, improve v1 notes schema #154
• Add/convert tag-in-ciphertext AES-GCM-SIV DaedTests to v1 - Convert aes_siv_cmac_test to v1 format  #159
• Add/convert stand-alone JOSE tests to v1 - Convert json_web_* vectors to v1 format #160

Afterwards, there are some misc. documentation/tooling updates to be made:

• Delete unused schema files post-consolidation.
• Update tools/vectorlint/main.go to update -vectors-dir flag.
• Update CONTRIBUTING.md to describe adding new vectors to the consolidated dir.
• Update README.md to update/remove the FAQ element about v0 vs v1.
• Update README.md with comprehensive algorithm coverage, adding missing algorithms

Resolves #149

[jpgoldberg]

Just as a "me, too", as I was mistaken about which set of vectors to prefer.

Note that the pyca cryptography project uses testvectors/ as the default subdirectory in their tooling. I copied that practice in something I am trying to develop because I figured that they knew which as preferred and actively maintained.

They have

def wycheproof_tests(*paths, subdir="testvectors"):
    ...

I will update what I am doing to default to testvectors_v1

[bleichenbacher-daniel]

I'm sorry about the confusion.
The latest test vectors are here: https://github.com/bleichenbacher-daniel/Rooterberg/tree/main/test_vectors
I'm still in the process of testing and simplifying them. Currently I'm using about 100 libraries to check if the formats
are appropriate and I expect still some changes.
pyca is one of the libraries tested. Current coverage is here:
https://github.com/bleichenbacher-daniel/Rooterberg/blob/main/doc/libraries/py-pyca.md

[jpgoldberg]

That looks like a great project. Some of what I see there may be annoying for my peculiar use case, but I see several possible work-arounds for me.

[FiloSottile]

@bleichenbacher-daniel we appreciate your contributions to the project while at Google, but we ask that you please stop intervening in every conversation to redirect users to your new personal project. That's disruptive and causes additional confusion.

When C2SP accepted Google's request to take Wycheproof into community management we did that with the explicit goal of making the project a repository of test vectors from multiple sources. That obviously means you'd be very welcome to contribute your new vectors, as well!

[bleichenbacher-daniel]

@FiloSottile The test vectors in https://github.com/bleichenbacher-daniel/Rooterberg/ are the next version of project Wycheproof and are a direct extension of my work on Wycheproof. They incorporate the changes to the test vector format that we discussed more than one year ago.

Publishing progress on the project is unfortunately a necessity because you are blocking me from continuing my work on Wycheproof. Not replying to my request for a meeting to organize this project better is not helping either.

Pointing out that there is already a new version of the project in development is not a distraction. It absolutely important to make
developers aware that there is actually some progress going on and hence that this project is still alive.

Let me also tell you that it is very important to run test vectors against multiple libraries, just to ensure that the formats make sense and that the test vectors are not overly paranoid. I'm currently doing this by running the test vectors against a number of popular libraries. These libraries are listed here: https://github.com/bleichenbacher-daniel/Rooterberg/blob/main/doc/libraries.md

Additionally, Wycheproof is and has never been just a repository of test vectors. Many of the libraries suffer from issues that can not be tested with test vectors. Timing attacks is just one example. Wycheproof did have such tests before it even was a project. I'm currently extending these tests to other languages. There are other goals of the project, such as simply determining the compatibility between different libraries or providers. Especially in Java it is very easy to break a project simply be using non-compatible algorithm names and then change the order of the installed providers. Evaluating if interfaces are robust, and detecting supported parameter sets are other important goals. These goals have always existed in Wycheproof and I continue working on them.

[FiloSottile]

@bleichenbacher-daniel a year ago we discussed making it easy to merge test vectors from multiple sources into Wycheproof, including your new vectors. I still hope you'll choose to do that! After #130 all vectors are tagged with their source, as we discussed, so you can send PRs anytime adding, modifying, or deleting any vectors with source Rooterberg (or github.com/bleichenbacher-daniel/Rooterberg, or any string you prefer).

The focus on test vectors going forward was discussed at length in #105 and #127.

I understand you feel differently, but when C2SP accepted to take over Wycheproof from Google, we committed to making it a community project, so again I ask that you please contribute in that context, if you wish to contribute.

[bleichenbacher-daniel]

@FiloSottile you have not addressed the points I raised above. I have been working on Wycheproof and its predecessor for much more than a decade. I think I do know what kind of tests are relevant and can be used to detect vulnerabilities. One crucial observation is that libraries quite frequently repeat old mistakes even mistakes one thinks have been eliminated for a long time.

So the removal of the code testing for timing attacks etc is a crucial step backwards. Even if collecting new test vector were the only useful goal it would still be useful for third parties to have code for generating some new test cases quickly and actual tests against a number of libraries to check the new test vectors. The project does already have these components, but they were removed rsp. rejected. In my mind that is not a winning strategy.

Efficiency is an important factor for the addition of new test vectors. Right now I can push a new version of the test vectors in about 6 hours. This time includes both generation and tests against 100 libraries. I certainly will move to some other site if there is some significant benefit from doing so. But for now the latest test vectors are still here: https://github.com/bleichenbacher-daniel/Rooterberg/ I'm still determining what formats are best. I.e. the need to convert key formats and other formats before calling tests should be minimized. Thus input there is welcome. I'm also working on finding relevant algorithms and providing test vectors for such relevant cases. Often there are just too many parameter combinations.

[cpu]

Within this issue discussion should be scoped specifically to the C2SP/Wycheproof test vector data as it exists in repo today, and concrete suggestions for a consolidation plan.

As requested in the description, please keep the discussion on this issue scoped to its topic.