Relationship between testvectors and testvectors_v1 is unclear

[davidben]

What's the intended relationship between testvectors and testvectors_v1? I.e. if I'm trying to integrate Wycheproof into my project, which directory should I look at? Should I be looking at both? Can this be documented somewhere?

I see that new stuff is getting added to v1, but then v1 doesn't have some test vectors that the old (?) set has (issue #89). Is the intent to align them? If there's some missing cases, where should they be added?

[cpu]

I don't think there's a conclusive answer to these questions so from my perspective it seems best to try and hash something out in this issue and then put a conclusion into docs.

Since transitioning to a community maintenance model I think only one PR has added new vectors (#143), and did so under testvectors_v1. There's one unmerged PR in the queue (#110) that is proposing adding content to testvectors, and another (#146) that is adding to testvectors_v1.

I think as one decision point to get things started it makes sense to say new stuff should go into testvectors_v1. I think with the common "source" schema element we can distinguish within the v1 vectors which testcases were from the no-longer-accessible generator code from the Google wycheproof project and which have been added from new external sources.

[bleichenbacher-daniel]

The latest versions of the test vectors are here:
https://github.com/bleichenbacher-daniel/Rooterberg/tree/main/test_vectors

I'm no longer working on testvectors_v1. I have not had access to the generation code for over two years and have essentially
rewritten everything in the mean time.

The main difference between the old and new versions of the test vectors is that the new test vectors no longer use TestGroups.
These TestGroups just complicate the code used for testing. Some of the files with test vectors have been divided into multiple files.
E.g. symmetric algorithms have now one file per key size. This helps a lot to test libraries that require compile-time constants for key sizes (a lot of rust crates have this property).

Currently I'm testing the new versions of the test vectors against a number of libraries, hence I'm still expecting that some changes to the test vector formats may still be necessary.

[cpu]

The latest versions of the test vectors are here: https://github.com/bleichenbacher-daniel/Rooterberg/tree/main/test_vectors

Are you interested in contributing the vector updates to this repo?

[bleichenbacher-daniel]

Sure, there are already a number of open issues that have not yet been answered.
Obviously it is far from ideal to have write the same code twice and to generate a new repo just to update my own project.
At the moment I'm continuing Wycheproof in my free time, hence I can't afford any kind of blockade.

[cpu]

Let's try to make headway on this issue as a place to start.

If you were contributing updates to the v1 vectors would it be via diffs to testvectors_v1/* or would your preference be to freeze the testvectors and testvectors_v1 directories and move new additions to a separate directory based on some common understanding of a new unified schema (e.g. w/o test groups)?

[bleichenbacher-daniel]

Just to answer the original question:

• testvectors_v1 is dead. I have been blocked from continuing to work on these test vectors for over two years.
• The latest version of the test vectors is here: : https://github.com/bleichenbacher-daniel/Rooterberg/tree/main/test_vectors
• This is still work in progress. I'm currently testing these test vectors against a number of libraries. The main goal is to find out, which algorithms are relevant, what algorithm are actually implemented (e.g. not all algorithms have well defined byte order), restrictions (e.g. PBES-based algorithms often cannot be used with arbitrary byte arrays), special cases (e.g. RSA PKCS 1 signature verification may still contain fallback code for ancient OpenSSL bugs)
• These tests are an essential part of the project.
• I'm currently using python, java, javascript, typescript, swift, rust and C#.

[cpu]

testvectors_v1 is dead.

We recently added new test vectors in there so I don't think that's an accurate characterization.

As mentioned I'm open to the idea that we should instead create a new directory for added vectors but you're side-stepping my question about the topic.

I have been blocked from continuing to work on these test vectors for over two years.

I'm not sure what you're talking about but it certainly doesn't involve me. If you opened a PR updating testvectors I would review it and work to see it merged as appropriate.

Let's stick to the topic of this issue and limit discussion to C2SP/wycheproof and how its vector directories should be handled.

[davidben]

From the perspective of a library maintainer, what would be useful is one place to pick up test vectors, in one format. I'm not particularly partial to which format, but, as the format evolves, there should be consistently one place to get the test vector from.

I'm not sure there's a reason to have multiple redundant directories[*] as the format evolves, TBH. If you all change the format and projects haven't adapted, they'll presumably still be pinned to the older version. Obviously changes to the format cause some downstream churn, but that's true no matter the organization. Having multiple redundant directories just makes it harder to tell what's going on, and whether they're merely fully redundant or semi-redundant.

The other thing that would be useful as a library maintainer would be an easy way to contribute test vectors. For a community-maintained set of test vectors, it should be possible for, well, the community to contribute. (I started this issue because I wanted to add some cases for the ECDSA DER parsing, and remembered that BoringSSL was unable to use new Wycheproof vectors ever since testvectors_v1 showed up and made everything confusing.)

[*] Did testvectors vs testvectors_v1 come out of the old version skew issue between BoringSSL and Wycheproof in google3? That was because of some misapplied policy that made no sense in context. BoringSSL did not want it either and we've have since gotten an exception for that policy. Honestly, if folks had just told us it was causing massive headache, I'd have pushed on the policy sooner. Regardless, that sort of nonsense is very much not Wycheproof's problem now.

[cpu]

I resonate with all of your points here.

One option is to enumerate the missing coverage compared to v0 (testvectors/), generate equivalent v1 vectors, and then remove v0 entirely. Afterwards we can continue to add new vectors to testvectors_v1 in the shape of pre-existing ones (like we did for ML-KEM). I think that's a path towards a world where there's one place to pick up vectors in one format. Making more vectors available easily to consuming projects is more of a priority to me personally than changing the format to make incompatible improvements.

Was BoringSSL hesitating to take testvector_v1 vectors for any reasons other than bifurcation with v0?

The other thing that would be useful as a library maintainer would be an easy way to contribute test vectors.

100% agree. That's the direction I've been marching towards but there are lots of rough edges still (like this directory situation). Is there something besides more documentation & centralizing the test vector directories we could do to make this easier from your perspective?

[davidben]

One option is to enumerate the missing coverage compared to v0 (testvectors/), generate equivalent v1 vectors, and then remove v0 entirely. Afterwards we can continue to add new vectors to testvectors_v1 in the shape of pre-existing ones (like we did for ML-KEM).

That sounds like a great plan to me. I see from #138 (comment) that the situation with the generators might finally be improving? I never really understood why the generators lived separate from the test vectors. Even when it was Google-maintained, I often had to go dig around for the generators to understand what the intent was with some test vector.

Was BoringSSL hesitating to take testvector_v1 vectors for any reasons other than bifurcation with v0?

Really just a huge lack of clarity about whether we were meant to import both, just v1, etc. Whether the missing vectors were an intentional drop, now covered by other tests instead, or just a work in progress, etc. Instead, I had to try to guess. I filed #89 back before all the turbulence and never got an answer, and then forgot about it.

Is there something besides more documentation & centralizing the test vector directories we could do to make this easier from your perspective?

I guess the things that one would expect to improve as a consequence of documenting. If someone adds a one-off test case and it disappears with the next run of the generator, that's no good. Whatever the preferred form is for adding to a given set of tests, it should live in the repo and be clear how to add to it.

[FiloSottile]

From the perspective of a library maintainer, what would be useful is one place to pick up test vectors, in one format. I'm not particularly partial to which format, but, as the format evolves, there should be consistently one place to get the test vector from.

The other thing that would be useful as a library maintainer would be an easy way to contribute test vectors. For a community-maintained set of test vectors, it should be possible for, well, the community to contribute.

Indeed, those are our goals.

@bleichenbacher-daniel I'm sorry if you felt blocked from contributing, we are also maintaining this on essentially a volunteer basis (@cpu works for Geomys, but no one pays Geomys for this), so we've had limited time for it. In case it wasn't clear, we'd be very happy to take your PRs, and I encourage you to send as many as you'd like. We want Wycheproof to be open to vectors from multiple sources (@davidben's second point above), so PRs can't be in the form of just replacing all the existing vectors every time. That's the idea of the source field: it partitions the namespace and lets you change any vectors with source Rooterberg freely.

I never really understood why the generators lived separate from the test vectors. Even when it was Google-maintained, I often had to go dig around for the generators to understand what the intent was with some test vector.

I think this is in conflict with being a community repository of test vectors: we can't maintain a potpourri of generators from different people in the same repo. I think Wycheproof should have just the vectors, and the PRs can link to the generators, so if necessary one can "git blame".

Anyway, back on topic of testvectors vs testvectors_v1: let's not overthink it. There should not be two. We should make it a priority to port missing vectors to testvectors_v1, and delete testvectors (or vice-versa if it's particularly easier).

[davidben]

Re generators: Fair enough. I guess the idea then would be that we'd be able to handle, say, ECDSA tests from different sources without them conflicting? My main concern is whether one can contribute a test vector to a set that was generated by some other random tooling, without the risk of it getting clobbered later.

It can also be useful to have the otherwise secret values available for refreshing test inputs, so that people can easily add variations on something, or reconstruct them with different parameters. But this is probably rarer so I don't mind the git blame approach.

[FiloSottile]

My main concern is whether one can contribute a test vector to a set that was generated by some other random tooling, without the risk of it getting clobbered later.

That's the idea of the source field, and the reason to have documented and automatically checked schemas: when regenerating, you only replace vectors tagged with your source.

[bleichenbacher-daniel]

I'm not sure there's a reason to have multiple redundant directories[*] as the format evolves, TBH. If you all change the format and projects haven't adapted, they'll presumably still be pinned to the older version. Obviously changes to the format cause some downstream churn, but that's true no matter the organization. Having multiple redundant directories just makes it harder to tell what's going on, and whether they're merely fully redundant or semi-redundant.

testvectors_v1 was essentially a preview version generated for OSS Fuzz. The project wanted additional fields to determine if a failure is a vulnerability or not. I also added a lot of references, whenever possible. The test vectors got published too early because of miscommunication.

The other thing that would be useful as a library maintainer would be an easy way to contribute test vectors. For a community-maintained set of test vectors, it should be possible for, well, the community to contribute. (I started this issue because I wanted to add some cases for the ECDSA DER parsing, and remembered that BoringSSL was unable to use new Wycheproof vectors ever since testvectors_v1 showed up and made everything confusing.)

Additional test cases for DER parsing should generally go into the asn_fuzzer, so that everything depending on DER can be tested.
ECDSA is becoming a monster, with all the variants added by different crypto currencies. Here it makes a lot of sense to add new cases to the generation code.

[bleichenbacher-daniel]

From the perspective of a library maintainer, what would be useful is one place to pick up test vectors, in one format. I'm not particularly partial to which format, but, as the format evolves, there should be consistently one place to get the test vector from.

The other thing that would be useful as a library maintainer would be an easy way to contribute test vectors. For a community-maintained set of test vectors, it should be possible for, well, the community to contribute.

Indeed, those are our goals.

@bleichenbacher-daniel I'm sorry if you felt blocked from contributing, we are also maintaining this on essentially a volunteer basis (@cpu works for Geomys, but no one pays Geomys for this), so we've had limited time for it. In case it wasn't clear, we'd be very happy to take your PRs, and I encourage you to send as many as you'd like. We want Wycheproof to be open to vectors from multiple sources (@davidben's second point above), so PRs can't be in the form of just replacing all the existing vectors every time. That's the idea of the source field: it partitions the namespace and lets you change any vectors with source Rooterberg freely.

If accessibility is the issue then why am I not the owner of the repo? After all, I started the project and have written the lion share of the tests and test vectors. The current structure simply does a lot of unnecessary damage to the project.

I never really understood why the generators lived separate from the test vectors. Even when it was Google-maintained, I often had to go dig around for the generators to understand what the intent was with some test vector.

One of the efforts is to add more information to the test vectors, describing the generation method, goal and give pointers to
CVEs or other references. This is typically done by adding flags. One of the tests I'm running simply checks that each test vector has at least one flag. I'm also removing the "acceptable" status and replace the status with flags. The tests that I'm running may have a list of flags that can be skipped. Test results include the flags that were actually skipped during the tests.

I think this is in conflict with being a community repository of test vectors: we can't maintain a potpourri of generators from different people in the same repo. I think Wycheproof should have just the vectors, and the PRs can link to the generators, so if necessary one can "git blame".

Anyway, back on topic of testvectors vs testvectors_v1: let's not overthink it. There should not be two. We should make it a priority to port missing vectors to testvectors_v1, and delete testvectors (or vice-versa if it's particularly easier).

As already pointed out all my test vectors in those directories are outdated.