Have a way to repair broken remote stores

[arianvp]

Describe the bug

A lot of CI systems offer local cache APIs over simple REST API. Using these with nix is problematic however.

The problem is that they will remove files that are old. If a nar file gets deleted but the corresponding narinfo doesn't; we end up in a broken state. It'd be great if nix can somehow "Detect" this?

Similarly many people would like to set some lifecycle rule on S3 bucket and delete old files. But this again doesn't work due to Nix only checking .narinfo files for "does this file exist"

Steps To Reproduce

Lets copy to a store; but remove the NAR files. simulating a dumb cache invalidator.

#
$ nix copy --to file://$PWD/cache1 /nix/store/8gndzbdrkc4a66s14yhk5m0jlfypji0p-nginx-1.28.0 -vvv  --narinfo-cache-positive-ttl 0 --narinfo-cache-negative-ttl 0
rm  ./cache1/nar/*

Now lets remove the path from my local disk and try to re-susbtitute. this will fail

$ nix-collect-garbage
$ nix copy --from file://$PWD/cache1 /nix/store/8gndzbdrkc4a66s14yhk5m0jlfypji0p-nginx-1.28.0   --narinfo-cache-positive-ttl 0 --narinfo-cache-negative-ttl 0    
error: file 'nar/0xncsxrn44zan3935czdg6zr4jdagk3dzq9akc7isr0k4h6jl6d2.nar.xz' does not exist in binary cache

Now rerun nix copy --to to try to fix it

$ nix copy --to file://$PWD/cache1 /nix/store/8gndzbdrkc4a66s14yhk5m0jlfypji0p-nginx-1.28.0 -vvv  --narinfo-cache-positive-ttl 0 --narinfo-cache-negative-ttl 0
querying info about '/nix/store/133d066jnk2bqx8k61r44x9kg8yr6qn9-libxslt-1.1.43' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/3g3sgwpnig8sd0w9wbs7d5gy512r43cq-bash-5.3p3' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/8gbj2w4yr605h1az4lhw0jfvdcmq6f8m-zlib-ng-2.2.4' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/8gndzbdrkc4a66s14yhk5m0jlfypji0p-nginx-1.28.0' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/8kzqz379ffbx14kd8ipr9s5s14ldih1r-gettext-0.25.1' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/a1h94bqvw9213n7ax3cp6a5c7ly3gy2f-pcre2-10.44' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/l619460n5l5jgg2y3b3y6k11q4l462d5-libcxx-19.1.7' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/pff8c9a4l487r0jaipibq21mkww5f1yw-libiconv-109' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/qz8sy946bfh14jb90wagc43q5clnzxp4-libxml2-2.14.5' on 'file:///Users/arian/scratch/cache1'...
querying info about '/nix/store/vhpaiaa94mqzivqkzjrad6af72irp89j-openssl-3.5.1' on 'file:///Users/arian/scratch/cache1'...
copying 0 paths...

Expected behavior

The second nix copy should "fix" the binary cache. But it doesn't. To decide whether a store path exists in a binary cache it only looks at the .narinfo file but I think it should also check if the nar files that the .narinfo points to exists. We could do a GET on the .narinfo and then HEAD requests on all the NARs referenced in the narinfo and reupload any missing NARs

Metadata

Additional context

Checklist

• checked latest Nix manual (source)
• checked open bug issues and pull requests for possible duplicates

---

Add 👍 to issues you find important.

[arianvp]

There is probably already a ticket for this. But there are 3000 issues and github search is impossible to use.

[arianvp]

Basically if any of the NARs that the narinfos points to are non-existent we should treat the whole narinfo as non-existent. Then I think this should just magically all work as expected.

[arianvp]

I was looking at https://cirrus-runners.app/setup/ as github actions mac runners

and noticed they have a local simple REST API for storing local blobs across runs a which would mean we can just use their local cache as a super fast nix substituter but the problem is they delete old paths and if a NAR gets deleted but the NARINFO doesn't you get into a weird state that doesn't fix itself.

For most cases the regular cirruslabs/cache action mechanism is more than enough. But modern build systems like Gradle, Bazel and Pants can take advantage of remote caching. Remote caching is when a build system uploads and downloads intermediate results of a build execution while the build itself is still executing.
Cirrus Runners starts a local caching server and exposes it via CIRRUS_HTTP_CACHE_HOST environments variable. Caching server supports GET, POST, HEAD and DELETE requests to upload, download, check presence and delete artifacts.
For example running the following command:
curl -s -X POST --data-binary @myfolder.tar.gz http://$CIRRUS_HTTP_CACHE_HOST/name-key
will upload myfolder.tar.gz file to the cache with the key name-key.

I also feel like this kind of behaviour is kind of a must for garbage collecting cache.nixos.org. if we make a mistake and accidentally corrupt the cache we can't fix it by rebuilding with hydra unless we fix this particular behavior