yarn focus and --immutable

[cakoose]

We have a Yarn 4.x workspaces monorepo, with TypeScript and the node_modules linker.

To reduce the size of the container image, I'm considering replacing yarn install with yarn focus and passing in just the names of the workspaces that I need in my container.

We currently do yarn install --immutable to ensure (I think?) that the installed packages match what's in my "yarn.lock". It appears that yarn focus does not have such an option.

I found an old GitHub issue (#6277) where someone is saying that --immutable doesn't make sense for yarn focus:

A focused install, almost by definition, would require the lockfile to be updated. A focused install literally mutates the in-memory data structures like parsed package.jsons, then runs a normal install. That would produce a different lockfile, we just don't write that back to disk.

I think I just want it to be have the same as yarn install, except exclude any packages that aren't needed by the workspaces that I have focus'ed. Is that possible?

[clemyan]

Specifically for your use case, you should just use yarn install --immutable to validate the lockfile, then yarn workspaces focus ... to create the final image.

You wouldn't put your linter in the final image, would you? No, you run the linter while you still have your dev-dependencies, then build the final image without dev-dependencies. By the time you build the final image, you should already know the project passes the linter, so you don't need the lint again in the final image.

In the same sense, you shouldn't need a immutable focused install. By the time you focus, you should already know the project passes the --immutable check, so you don't need to check again in the final image.

---

Now let's talk more generally about immutable focused installs. After the part you quoted, I also said

So you don't want an immutable focused install, you want a focused install where some mutations are okay and other are not. The question then becomes what mutations are okay and how to determine if a given change is okay or not.

Immutable installs are fundamentally about detecting mutations (i.e. changes) between two states: the state immediately after the last unfocused install ("before") when the lockfile was generated, and the state that is currently on-disk ("after"). Any mutation that would affect resolution is rejected. Such mutation may be external (e.g. registry went down) or caused by changes on-disk (e.g. a modified package.json).

(Note: While we don't have the full "before" state of the project, the lockfile on-disk is a good enough proxy because it records all resolution that happend during the last unfocused install)

The problem with immutable focused installs is that mutations can also be caused by the focusing itself. A immutable focused install needs to ignore those mutations while still being able to reject external and on-disk mutation. However it is not always possible to distinguish between the two.

And "exclude any packages that aren't needed" probably isn't what you want. I can take a workspace and remove all its dependencies. Now all packages are unneeded, so a immutable focused install should pass?

For some less extreme cases, say we start with a project that passes yarn install --immutable. Which of these should pass?

1. I manually change the package.json to remove a dependency, then focus --production --immutable
2. I manually change the package.json to remove a devDependency, then focus --production --immutable
3. I manually change the package.json to move a package from dependency to devDependency, then focus --production --immutable
4. I manually change the package.json to move a package from dependency to devDependency, then focus --immutable, then focus --production --immutable
5. I manually change the pacakage.json of an unfocused workspace to remove a dependency, then focus --immutable

Some of these aren't distinguishable from one another, or distinguisable from "run focus --immutable without changing anything". I even suspect not all people agree what is "correct" in some more extreme edge cases.

Most people who ask for immutable focused install think that's just "immutable install, but ignore these changes". But no, it is "ignore this change if it is caused by this reason, but don't ignore the exact same change if it is caused by that reason"

At the very least, we need to record more information about the project state during unfocused installs so that a future focused install can detect why mutations happen. But it isn't clear what needs to be recorded. Depending on the exact semantics of immutable focused installs (which, again I don't think there is a consensus on), the complexity can range from "lockfile format changes" to "re-architecting the entire core" to "literally impossible without a time machine"

In any case, if you have a specific project that need a specific form of immutable focused installs, you can implement it as a plugin. But at this point, I'm don't think I can be convinced that a first-party immutable focused install is a good idea until I see a PR that people can point to and say "that's what I'm talking about when I say immutable focused installs"

[cakoose]

I think when people (like me) are asking about --immutable, we're not experts in Yarn concepts. It's coming from the perspective of:

• To make sure there aren't inconsistencies between the package.json and the yarn.lock in CI, we're told to yarn install --immutable.
• To only install the dependencies for a subset of workspaces, we're told to use yarn focus instead of yarn install.

So I end up wondering: if I use yarn focus instead of yarn install --immutable, do I lose the consistency check?

What I gather from your message:

• Yes, I lose the consistency check.
• To recover it, I should do a separate yarn install --immutable
• If it fails, then the whole build should fail.

If the yarn install --immutable succeeds, can I assume that a yarn focus on a fresh copy of the same repo will install the exact same package versions as yarn install --immutable except without the packages that aren't needed by the focused workspaces?

And "exclude any packages that aren't needed" probably isn't what you want. I can take a workspace and remove all its dependencies. Now all packages are unneeded, so a immutable focused install should pass?

Are you referring to me saying "except exclude any packages that aren't needed by the workspaces that I have focus'ed"?

I don't follow how removing all dependencies makes sense there. Is there another way I should have phrased it?