You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Run CI on a single machine by generating cloudbuild files. (#4147)
FEATURE
INTERNAL
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Modularize Sum and Softmax. (#4148)
DEV
* Lint files
* Fix package dependencies for inference and vis
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Lint files
* Fix package dependencies for inference and vis
* modularize node kernels (#4154)
Modularizes onesLike, zerosLike, unpack/unstack, batchMatMul, slice, isFinite, isInf, IsNan, reciprocal, squaredDifference
DEV
* Enable incremental typescript builds (#4092)
FEATURE
INTERNAL
Enable incremental builds in the root tsconfig.json to increase build speed during development. For example, running `yarn build` in tfjs-backend-cpu takes ~43 seconds without incremental builds and ~30 seconds with incremental builds (after the first build).
Co-authored-by: Ping Yu <[email protected]>
* Fix typo in .github/stale.yml (#3642)
INTERNAL
* Address some of the comments
* Re-lint everything with VSCode.
* Hoist consts and use UPPER_SNAKE_CASE.
* Require each step to have an id and add ids to all steps in converter and vis.
* Use argparse for 'generate_cloudbuild_for_packages.js'
* Remove inference, vis, and react-native dependencies in package_dependencies.json
tfjs-inference, vis, and react-native use pinned versions of other tfjs packages, so they do not need to be tested when those pinned packages change.
* Fix generate_cloudbuild_for_packages
generate_cloudbuild_for_packages.js was using argparse functions from a later version of argparse that are not available in 1.0.10.
* [webgl] Modularize batchMatMul, sum. (#4140)
FEATURE
* Add Prod kernel to WASM backend. (#4138)
FEATURE
Co-authored-by: Ann Yuan <[email protected]>
* Write tests for generate_cloudbuild.js
Write tests to detect if generate_cloudbuild's output changes. These tests are not unit tests, but they alert us if a change unexpectedly changes the project structure or the generation of cloudbuild files.
* Run CI on a single machine by generating cloudbuild files.
Instead of running each CI test on a separate Google Cloud machine, generate a cloudbuild.yml file to test all affected packages on a single machine. The benifits of this strategy include the following:
* Fewer machines are required for a single test (2 instead of up to 12), so less queueing is required.
* Packages only need to be built once, instead of once per machine.
* Multithreaded steps have access to more cores.
When CI is run, the root cloudbuild.yml is launched on Google Cloud. This file has only two steps: Install yarn dependencies and run `yarn test-affected-packages`. The latter step finds what packages were affected by the change and combines their cloudbuild.yml files into a single `cloudbuild_generated.yml` file. The steps in this file are set to `waitFor` each other according to the dependency tree `scripts/package_dependencies.json`, and each package's `build-deps` step is removed in favor of waiting for its dependencies to be built.
Each package's cloudbuild file still works as expected, and there are a few new yarn commands at the root:
`yarn test-packages` tests all packages affected by a change. It's run during CI.
`yarn test-packages [PACKAGE]...` tests a given set of packages, including all packages they might affect if changed.
* Split generating the cloudbuild from running it
Google Cloud uses a separate image for node and for spawning more gcloud instances. The node image does not have the gcloud command, so we have to generate the cloudbuild file in one step and launch gcloud in another step.
Also fix the `rm` command in find_packages_with_diff.js to remove recursively.
* Add `yarn` to the `waitFor` field of webgpu tests
* Lint files
* Fix package dependencies for inference and vis
* Address some of the comments
* Re-lint everything with VSCode.
* Hoist consts and use UPPER_SNAKE_CASE.
* Require each step to have an id and add ids to all steps in converter and vis.
* Use argparse for 'generate_cloudbuild_for_packages.js'
* Remove inference, vis, and react-native dependencies in package_dependencies.json
tfjs-inference, vis, and react-native use pinned versions of other tfjs packages, so they do not need to be tested when those pinned packages change.
* Fix generate_cloudbuild_for_packages
generate_cloudbuild_for_packages.js was using argparse functions from a later version of argparse that are not available in 1.0.10.
* Write tests for generate_cloudbuild.js
Write tests to detect if generate_cloudbuild's output changes. These tests are not unit tests, but they alert us if a change unexpectedly changes the project structure or the generation of cloudbuild files.
Co-authored-by: Na Li <[email protected]>
Co-authored-by: Yannick Assogba <[email protected]>
Co-authored-by: Ping Yu <[email protected]>
Co-authored-by: Rajeshwar Reddy T <[email protected]>
Co-authored-by: Ann Yuan <[email protected]>
Co-authored-by: miaowzhang <[email protected]>
0 commit comments