A better cache clean-up and expiration policy

[terrajobst]

edit by @zivkan: a first step has been implemented, allowing package "last used in restore" date to be updated: NuGet/NuGet.Client#4222

edit by NuGet Client Triage:

• Restore driven - A minimally impactful approach might be to run a "gc" periodically, like say once a week or once a month.
• Additional command - Community projects like NuGet Cleaner combine with updatePackageLastAccessTime

original post below:

---

The user local NuGet cache has a tendency to grow unconditionally. Which results in funny tweets like this from @davidfowl:

David Fowler‏ @davidfowl 9m
How big is your nuget cache? Mine is 14 gigs #nuget

Can we track when an item was last hit by package restore and apply some expiration policy? I don't have good data, but I bet a good chunk of the bloat aren't individual packages with a single version but a small number of packages with many versions. If were to just delete older versions that weren't needed for restore for a while, we're probably getting to something saner.

Thoughts?

[emgarten]

Related: #2308

[terrajobst]

Yeah, I'm not a fan of an explicit clean as the sole mechanism. We have other discussions on how to hide package restore even more by making dotnet build behave like VS, i.e. running restore if necessary.

I think having a nuget gc command as @davidfowl suggested would be OK, but I really really want nuget restore to apply some cache expiration policy automatically. It doesn't have to be perfect, but something that doesn't fill up my drive indefinitely unless I invoke some special command.

[anangaur]

@terrajobst You still would need some process to clean the cache up? I think and explicit command to purge is the best way. Some of the requirements:

• Should be able to clean unused packages older than a date (yyyymmdd), time(1m,3m,6m, etc.) with defaults (may be 3m?)
• Should be able clean up everything except packages used in a project/solution/repo/dir
• [P2] Should be able to clean up packages from a source.
• [P2] Should be able to clean up all unsigned. /cc: @rido-min

Today we do not keep any metadata around source, signature or lastUsed information. May be we should start persisting these info.

[terrajobst]

The way I'd design the cache is like this:

1. Every time package restore resolves to a cache location, bump one of the dates (e.g. touch the .nuspec file)
2. Every N days (default: 7), delete folders from the package cache that weren't used during the last M months (default: 2 months)

Not sure (1) is viable for perf reasons. If it's not, only bump the dates every so often.

The meta point is that a cache without expiration control isn't a cache -- it's a leak.

[anangaur]

Perf is one consideration. restore could just touch the file (like in linux touch command) which should be more perfomant (or my belief :) ). But I am still not sure which command/process does the clean up i.e. delete of the expired or old packages from the cache. Again restore cannot be the one to do this (performance criterion).

[terrajobst]

It has to be automatic which means it has to be part of restore.

[anangaur]

Ideally yes but without impacting restore perf :)

[terrajobst]

Agreed, but it seems there must be a compromise between spamming my drive and impacting my package restore times.

Maybe we can address 90% of this if .NET Core wouldn’t be modeled as a large number of packages though.

[anangaur]

Maybe we can address 90% of this if .NET Core wouldn’t be modeled as a large number of packages though.

@terrajobst - what's the current thinking here? Is there anything beyond early discussion in this direction? All ears.