[Bug] Workspaces foreach always exits with zero even on error

[MarsianMan]

• I'd be willing to implement a fix

Describe the bug

When a build or test fails using workspaces foreach run test yarn should exit with a non-zero status.

To Reproduce

Checkout https://github.com/marcneander/yarn-foreach-error.git

From root directory run yarn run test && echo "done" or yarn workspaces foreach run -pvai test && echo "done"

Expectation: in the output below, done should not output when using foreach

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"
/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ 

Note that if I run the test on the workspace directly by name, then the exit code is correct.

Screenshots

Environment if relevant (please complete the following information):

• OS: [OSX, Linux]
• Node version [15]
• Yarn version [2.4.0]

Additional context

In a non-sample project, I am trying to use yarn to run cucumber.js tests in workspaces. The tests are failing, but the build is passing.

yarn workspaces foreach -pviA  run test && echo "done"
➤ YN0000: [workspace A]: Seeing profile=XYZ
➤ YN0000: [common-test]: Seeing profile=XYZ
➤ YN0000: [workspace A]: ..F-
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: Failures:
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: 1) Scenario: Can reach API Proxy # test/features/apigee.feature:9
➤ YN0000: [workspace A]:    ✔ Given I have a REST client # ../../common-test/test/features/step-definitions/apickli.ts:28
➤ YN0000: [workspace A]:    ✔ Given I set "User-Agent" header to 'XXXXXXXX' # ../../common-test/test/features/step-definitions/apickli.ts:34
➤ YN0000: [workspace A]:    ✖ When I GET "/ping" # ../../common-test/test/features/step-definitions/apickli.ts:88
➤ YN0000: [workspace A]:        Error: Error: socket hang up
➤ YN0000: [workspace A]:            at /Volumes/HDD2/apigee-config/src/common-test/test/features/step-definitions/apickli.ts:91:22
➤ YN0000: [workspace A]:            at Request._callback (/Volumes/HDD2/apigee-config/.yarn/cache/apickli-npm-2.3.3-e973675e47-469a7ec0f4.zip/node_modules/apickli/apickli.js:481:14)
➤ YN0000: [workspace A]:            at self.callback (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:185:22)
➤ YN0000: [workspace A]:            at Request.emit (node:events:329:20)
➤ YN0000: [workspace A]:            at Request.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at Request.onRequestError (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:877:8)
➤ YN0000: [workspace A]:            at ClientRequest.emit (node:events:329:20)
➤ YN0000: [workspace A]:            at ClientRequest.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at TLSSocket.socketOnEnd (node:_http_client:502:9)
➤ YN0000: [workspace A]:            at TLSSocket.emit (node:events:341:22)
➤ YN0000: [workspace A]:            at TLSSocket.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [workspace A]:            at endReadableNT (node:internal/streams/readable:1294:12)
➤ YN0000: [workspace A]:            at processTicksAndRejections (node:internal/process/task_queues:80:21)
➤ YN0000: [workspace A]:    - Then response code should be 200 # ../../common-test/test/features/step-definitions/apickli.ts:164
➤ YN0000: [workspace A]: 
➤ YN0000: [workspace A]: 1 scenario (1 failed)
➤ YN0000: [workspace A]: 4 steps (1 failed, 1 skipped, 2 passed)
➤ YN0000: [workspace A]: 0m00.590s (executing steps: 0m00.578s)
➤ YN0000: [common-test]: ..F-
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: Failures:
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: 1) Scenario: Can reach API Proxy # test/features/apigee.feature:9
➤ YN0000: [common-test]:    ✔ Given I have a REST client # test/features/step-definitions/apickli.ts:28
➤ YN0000: [common-test]:    ✔ Given I set "User-Agent" header to 'XXXXX' # test/features/step-definitions/apickli.ts:34
➤ YN0000: [common-test]:    ✖ When I GET "/ping" # test/features/step-definitions/apickli.ts:88
➤ YN0000: [common-test]:        Error: Error: socket hang up
➤ YN0000: [common-test]:            at /Volumes/HDD2/apigee-config/src/common-test/test/features/step-definitions/apickli.ts:91:22
➤ YN0000: [common-test]:            at Request._callback (/Volumes/HDD2/apigee-config/.yarn/cache/apickli-npm-2.3.3-e973675e47-469a7ec0f4.zip/node_modules/apickli/apickli.js:481:14)
➤ YN0000: [common-test]:            at self.callback (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:185:22)
➤ YN0000: [common-test]:            at Request.emit (node:events:329:20)
➤ YN0000: [common-test]:            at Request.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at Request.onRequestError (/Volumes/HDD2/apigee-config/.yarn/cache/request-npm-2.88.2-f4a57c72c4-7a74841f30.zip/node_modules/request/request.js:877:8)
➤ YN0000: [common-test]:            at ClientRequest.emit (node:events:329:20)
➤ YN0000: [common-test]:            at ClientRequest.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at TLSSocket.socketOnEnd (node:_http_client:502:9)
➤ YN0000: [common-test]:            at TLSSocket.emit (node:events:341:22)
➤ YN0000: [common-test]:            at TLSSocket.EventEmitter.emit (node:domain:467:12)
➤ YN0000: [common-test]:            at endReadableNT (node:internal/streams/readable:1294:12)
➤ YN0000: [common-test]:            at processTicksAndRejections (node:internal/process/task_queues:80:21)
➤ YN0000: [common-test]:    - Then response code should be 200 # test/features/step-definitions/apickli.ts:164
➤ YN0000: [common-test]: 
➤ YN0000: [common-test]: 1 scenario (1 failed)
➤ YN0000: [common-test]: 4 steps (1 failed, 1 skipped, 2 passed)
➤ YN0000: [common-test]: 0m00.476s (executing steps: 0m00.464s)
done

I expect a non-zero exit code and "done" to be missing from the output.

[arcanis]

I don't understand - done is not printed when using foreach, as shown in the output you shared (your prompt makes it fairly difficult to parse btw):

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ 

[arcanis]

Also note that we purposefully have code setting the exit code to 1 if anything fails, so I suspect your test framework doesn't actually exits with a non-zero exit code:

berry/packages/plugin-workspace-tools/sources/commands/foreach.ts

Lines 323 to 328 in 909a125

	const errorCode = exitCodes.find(code => code !== 0);
	// The order in which the exit codes will be processed is fairly
	// opaque, so better just return a generic "1" for determinism.
	if (finalExitCode === null)
	finalExitCode = typeof errorCode !== `undefined` ? 1 : finalExitCode;

[MarsianMan]

I don't understand - done is not printed when using foreach, as shown in the output you shared (your prompt makes it fairly difficult to parse btw):

You have misread the commandline output.
Steps in output

1. check out repo (not shown)
2. Set version berry
3. Run workspace test directly (fails as expected)
4. Run workspace test via foreach (unexpectedly passes)

Cleaned up some output

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn set version berry
➤ YN0000: Downloading https://github.com/yarnpkg/berry/raw/master/packages/yarnpkg-cli/bin/yarn.js
➤ YN0000: Saving the new release in .yarn/releases/yarn-2.4.0.cjs
➤ YN0000: Done in 2s 116ms

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn workspace failure run test && echo "done"

/Volumes/HDD2/GitHub/yarn-foreach-error on master*
$ yarn run test && echo "done"
done

Running test using the workspace the test fails correctly (no done)

$ yarn workspace failure run test && echo "done"
$ 

Even running the test explicitly with foreach does NOT fail correctly

$ yarn workspaces foreach run test -A && echo "done"
done
$

Also note that we purposefully have code setting the exit code to 1 if anything fails, so I suspect your test framework doesn't actually exits with a non-zero exit code:

🤷🏾‍♂️ That is nice and well, but this issue is easily reproducible for me. The test fails when running without foreach so unlikely to be a test framework issue. My test framework is Cucumber.js which is pretty battle tested and also NOT part of the reproducible example 😉

[MarsianMan]

In case the problem was the way the test was boostrapped...I tried using false and exit 1 as the failing "test" command (for workspace failure) and both have the same behavior.

NOTE: I did not create the linked repo, but reproducing the same/similar issue with it.

[MarsianMan]

Just following up to check if any further clarification is needed?

[DanielOrtel]

yeah, having the same problem with yarn workspaces foreach, for both jest and eslint. With the following workspace:

{
  "name": "@my-scope/package",
  "version": "1.0.0",
  "private": true,
  "main": "src/index.js",
  "devDependencies": {
    "eslint": "^7.20.0",
    "jest": "^26.6.3"
  },
  "scripts": {
    "lint": "eslint",
    "unit-test": "jest --coverage",
    "test": "yarn lint && yarn unit-test",
  }
}

Running either yarn workspaces foreach lint, yarn workspaces foreach unit-test or yarn workspaces foreach test will produce a successful outcome, even if any/all of these are failing. It does log that, Process exited without output (exit code 0), but still exits with the same 0 exit code.

[bfricka]

We use yarn workspaces foreach -piv run test and it exits w/ the expected non-zero code. I tested this w/ roughly the same echo test. Very odd. That said, I did test it w/ from sources not berry, since we're using TS 4.2.

[jasonwilliams]

Also having the same issue but only on Node v15 or above.
On an environment where Node v14 is installed i get 1 as the exit code (which is correct).

Switching to Node v15 i get 0 (incorrect as the child command fails), also the output is cut off like it didn't finish properly.

@MarsianMan you mentioned Node 15 in your issue, could you try downgrading to v14 and running it again?

[merceyz]

Switching to Node v15 i get 0 (incorrect as the child command fails), also the output is cut off like it didn't finish properly.

Try using yarn from master - #2255

yarn set version from sources

[jasonwilliams]

Thanks @merceyz

I just get

Usage Error: This plugin cannot access the package referenced via yup which is neither a builtin, nor an exposed entry (when initializing @yarnpkg/plugin-workspace-tools, defined in /path/to/project/.yarnrc.yml)

━━━ Yarn Package Manager - 2.4.0-git.20210311.hash-9705c122 ━━━━━━━━━━━━━━━━━━━━━

  $ yarn <command>

You can also print more details about any of these commands by calling them with 
the `-h,--help` flag right after the command name.

When switching to the sources version