Skip to content

Conversation

davidgamez
Copy link
Member

@davidgamez davidgamez commented Jan 20, 2025

Summary:

As mentioned in #1938, the latest ubuntu runner has a difference default python version that is not compatible with some old python dependencies. In this PR we are upgrading python dependencies in the scripts and also preparing the Github actions to bump up Python and Java versions by adding versions as a variable within the action.

Fixes #1938

From our AI friend

This pull request includes several updates to the GitHub Actions workflows and dependencies in the project. The changes primarily focus on updating the environment configurations and dependency versions used in various workflows.

Workflow Environment Updates:

  • Updated the runs-on attribute from ubuntu-22.04 to ubuntu-latest in multiple jobs across various workflow files to ensure the use of the latest Ubuntu version. (.github/workflows/acceptance_test.yml, .github/workflows/docker.yml, .github/workflows/end_to_end.yml, .github/workflows/formatting.yml, .github/workflows/notice_migration_generation.yml, .github/workflows/package_installers.yml, .github/workflows/publish_assets.yml, .github/workflows/stg_web_client_merge.yml, .github/workflows/stg_web_svc_merge.yml, .github/workflows/web_client_pr.yml, .github/workflows/web_release.yml, .github/workflows/web_svc_pr.yml) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28]

Dependency Version Updates:

  • Updated dependencies in scripts/mobility-database-harvester/requirements.txt to newer versions for improved compatibility and performance.
  • Updated dependencies in scripts/notice-migration-generator/requirements.txt to newer versions to ensure compatibility with the latest libraries.

Specific Tool Setup Updates:

  • Introduced environment variables for python_version, java_version, and java_distribution to standardize the setup across various workflows. (.github/workflows/acceptance_test.yml, .github/workflows/docker.yml, .github/workflows/end_to_end.yml, .github/workflows/formatting.yml, .github/workflows/notice_migration_generation.yml, .github/workflows/package_installers.yml, .github/workflows/publish_assets.yml, .github/workflows/stg_web_client_merge.yml, .github/workflows/stg_web_svc_merge.yml, .github/workflows/web_client_pr.yml, .github/workflows/web_release.yml, .github/workflows/web_svc_pr.yml) [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] [25] [26] [27] [28]

These changes collectively improve the consistency, maintainability, and compatibility of the project's CI/CD workflows.

Expected behavior:

Explain and/or show screenshots for how you expect the pull request to work in your testing (in case other devices exhibit different behavior).

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with gradle test to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@davidgamez davidgamez changed the title add python setup to e2e test fix: Fix python scripts failing in ubuntu-latest GitHub workflows Jan 20, 2025
@davidgamez davidgamez changed the title fix: Fix python scripts failing in ubuntu-latest GitHub workflows fix: python scripts failing in ubuntu-latest GitHub workflows Jan 20, 2025
@davidgamez davidgamez marked this pull request as ready for review January 20, 2025 20:13
Copy link
Contributor

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 5760d8c
Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1792 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1792 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1792 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1792 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1792 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

Time Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 3.59 3.71 ⬆️+0.12
Median -- 1.32 1.38 ⬆️+0.06
Standard Deviation -- 10.16 10.38 ⬆️+0.22
Minimum in References Reports us-oregon-hut-airport-shuttle-gtfs-635 0.47 0.48 ⬆️+0.02
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 270.92 278.76 ⬆️+7.84
Minimum in Latest Reports us-oregon-hut-airport-shuttle-gtfs-635 0.47 0.48 ⬆️+0.02
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 270.92 278.76 ⬆️+7.84
📜 Memory Consumption
Metric Dataset ID Reference (s) Latest (s) Difference (s)
Average -- 461.00 MiB 464.29 MiB ⬆️+3.30 MiB
Median -- 333.24 MiB 331.92 MiB ⬇️-1.32 MiB
Standard Deviation -- 748.67 MiB 774.09 MiB ⬆️+25.41 MiB
Minimum in References Reports us-colorado-town-of-telluride-gtfs-2050 39.17 MiB 391.92 MiB ⬆️+352.75 MiB
Maximum in Reference Reports gb-unknown-uk-aggregate-feed-gtfs-2014 10.48 GiB 10.70 GiB ⬆️+228.25 MiB
Minimum in Latest Reports us-massachusetts-marthas-vineyard-transit-authority-gtfs-420 415.92 MiB 39.73 MiB ⬇️-376.19 MiB
Maximum in Latest Reports gb-unknown-uk-aggregate-feed-gtfs-2014 10.48 GiB 10.70 GiB ⬆️+228.25 MiB

Copy link
Contributor

@jcpitre jcpitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good change, using variables

@davidgamez davidgamez merged commit 45d7988 into master Jan 21, 2025
149 of 150 checks passed
@davidgamez davidgamez deleted the fix/ubuntu-latest-python branch January 21, 2025 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix python scripts failing in ubuntu-latest GitHub workflows

2 participants