Speed up the creation of source code bundles

[sschuberth]

Currently, source code bundles are created by (re-)compressing scanned files that probably have been compressed before (e.g. as source artifacts). In the case of source artifacts we could consider to take them as-is and just put them into a wrapping archive with "store" compression / a tarball to speed things up.

A different approach (that could be combined with the one above) would be to cache / store source code bundles in a similar way like we already store scan results, and reuse them. However, a naive implementation would only gain something if none of the contained dependencies changed, which is rather unlikely.

[sschuberth]

Yet another totally different idea would be to not create archives that contain source code in the first place, but only references to the source code on Software Heritage.

[tsteenbe]

Yet another totally different idea would be to not create archives that contain source code in the first place, but only references to the source code on Software Heritage.

Great idea to use Software Heritage, would love an check&auto-upload to Software Heritage. However I don't think it replaces creating archives just be a different option - ORT users might not want to publish their archives to SH. Also bring up the old topic that we need an easy way in ORT to mark "your" package from the "third party" ones.

[sschuberth]

ORT users might not want to publish their archives to SH.

Hmm, if source code would go into the bundle, it's Open Source anyway, and should already be harvested by SH, or?

[tsteenbe]

Hmm, if source code would go into the bundle, it's Open Source anyway, and should already be harvested by SH, or?

No, the code in an archive is not always open source. For example, two parties may agree to a escrow software agreement where the supplier periodically deposits a copy of the software source code and associated materials in a secure storage managed by independent third party. Under certain pre-agreed conditions, say the supplier goes bankrupt or stops supporting the software, the independent third party releases the code to the customer so they can continue using the software.

[sschuberth]

That's fair, but seems like pretty much of an edge-use-case for ORT.

[heliocastro]

Guys, references is the answer.

Mostly the same thing about the download part we (as Cariad ) want to change in ORT, at least from our side to download the pristine tarball / git code and push through some storage server, then the bundle should come from that storage, and
ORT model will store the reference for the storage, can it be local, server, artifactory, database, etc..

Mostly, ORT downloader push through the storage we want, then that wil be the reference, unless we decide that the upstream origin is the desired one

We need guarantee that the source origin match, and sometime if we recompress, the source is not anymore the same.