[Feature Contribution]: Initial DevOps and github actions clean up

[danuw]

What happened?

This is both an issue and a new feature.
It aims to address duplication in Github actions as well as bring consistency of approach.

First the audit with initial thoughts:

• build-cli-dotnet: build everything and run all tests - remove if duplicated
• build-packages: major functionality as library (Cli + webapi) - package integration test to ensure the library is working under happy path conditions => maybe publically github package as Beta
• build-webapi: build test validate and compare => generated swagger changes should be part of the repo/tracked so obvious as part of the PR
• codeQL-anamlysis: scheduled - security check - static analysis to check for vulnerabilities for updated dependencies
• dev_carbon-aware-api: build test deploy DEV to public api endpoint
• run-sdkCLI-githubaction: CI/CD carbon aware github action => use for CodeQL ACTION
• verify-azure-function-with-packages: function validate the packages / solution deploys correctly and can I request the endpoint successfully => merge with others (e.g. build-packages)

Initial recommendation of first changes to address:

• Rename to more meaningful actions and merge actions where needed
• Separate into the following themes (main build actions):
• - PR build and integration tests
  • do we separate per medium (cli vs webAPI)?
  • do we have separate build for each sample or only some?
• - build (& deploy?) on PR merge
• - Validate dependencies and integration tests (cron - can we use the CI/CD pipeline for this?)

Also (further topics to cover in github actions):

• stale issues: implement git action to auto label and close old tickets (older than X days where X needs to be confirmed - may start with 60 or 90 days minimum and reduce overtime)
• create releases?
• highlight Swagger updates in release notes
• I am also keen to version the builds and make sure it surfaces on the swagger as a relevant build version so people know if things have changed or that they are simply looking at the right thing (hoping that can be used in the analytics)

Thank you @bderusha for your inputs
cc @vaughanknight @Willmish

Code of Conduct

• I agree to follow this project's Code of Conduct

Feature Commitment

• I commit to contributing this feature as a PR and working with the GSF to merge this feature into the Carbon Aware SDK.

[danuw]

To be discussed on Tuesday morning...

[YaSuenag]

I hope following topics are implemented on GHA - they are accepted in ADR-0011:

• Ship WebAPI container images from GitHub packages
• Ship client libraries which are supported by GitHub packages

I think they should be kicked on release-created event on GHA.

[danuw]

@YaSuenag , worth linking #149 here

[YaSuenag]

@danuw I think it is better to upload OpenAPI YAML because SDK user can implement their clients without WebAPI container.

In addition, it is also better to upload documents of client bindings. In case of Java, we can generate Javadoc (API document) from source code comments, and we can integrate generating process into build process.

In my personal Java project, I use GHA to generate both release binary and API documents, then they will be uploaded to GitHub packages and GitHub pages. I'm happy if it helps you. https://github.com/YaSuenag/ffmasm/blob/main/.github/workflows/release.yml#L26-L38

[danuw]

Storing for work in progress diagrams...

How to use?

How to use in standalone solutions

flowchart TD
    %% How to use in standalone solutions
    StandaloneProject[Standalone Dotnet project \n  - eg Function] --> |Included as nuget package| SDK[Carbon Aware SDK \n as .net library]
    SDK --> |built for linux| github[Github packages]
    SDK --> |built for arm64| github[Github packages]
    SDK --> |built for amd64| github[Github packages]

Loading

How to use with Hosted APIs

flowchart TD
    %% How to use with Hosted APIs
    %% Hosting options and methods of deployment
    Project -->|connects via REST| API[Carbon Aware API]
    Project --> |uses| RestClient{Carbon Aware \n OpenAPI Generated \n REST Clients}
    RestClient --> |?| java[JAVA client]
    RestClient --> |NPM| node[node client]
    RestClient --> |Pypi| python[python client]
    RestClient --> |?| rust[rust client]
    RestClient --> |?| go[go client]

Loading

Used in processes

flowchart TD
   %% Used in processes
    Pipeline --> |Command line| CLI

Loading

Hosting options and methods of deployment

flowchart TD
    %% -----------------------------------------
    %% Hosting options and methods of deployment
    CAAPI[Hosted API] --> GSF[GSF Hosted]
    CAAPI --> Self[Self Hosted]
    Self --> Cloud
    Self --> Local
    Cloud --> Container[Container]
    Cloud --> WebApp[Web App]
    Local --> Container2[Container]
    Local --> WebApp2[Web App]

Loading

TODO

• Update to reflect what the GSF may cover and status (planned, in progress, done, not planned)
• Other DevOps processes missing (generic for now)

[YaSuenag]

Thank @danuw for sharing current progress! I have some comments:

• How to use in standalone solutions
  • Can we use "AnyCPU" for .Net library? I think it is prefer to single binary if we can.
  • Can we add workflow to publish WebAPI container to GitHub container registry? It relates to ADR-0011.
• How to use with Hosted APIs
  • Client library for Java conforms with Maven.
  • Would they be uploaded to GitHub packages? They are mentioned in ADR-0011, and Java / JS libraries should be uploaded at least.

[danuw]

Hi @YaSuenag , the above is just work in progress. I think the idea was to help visually explain what the repo enables and what we may or may not cover (that part is not yes illustrated on those graphs)
These are mainly to support and summarise what we are aiming for and will need to document further.

Anyway below to answer your questions.

Can we use "AnyCPU" for .Net library? I think it is prefer to single binary if we can.

yes (I assume this also relates to stated ADR right?)

Can we add workflow to publish WebAPI container to GitHub container registry? It relates to ADR-0011.

yes, do you have anything (documented steps) to test, so we use as first success criteria and covered scope please?

How to use with Hosted APIs

I believe this would not necessarily cover at first but rather document further how to generate if needed (such as through videos etc...). Do you believe more documentation is needed and if yes where shall we start? - we can probably discuss this on Tuesday

Hope above helps... Let me know

[Willmish]

My suggestion is: To also add workflows for testing Code samples of those samples we can test - that way we will keep them up to date with dev branch and not realise when someone tries the CA SDK for the first time that the sample is actually broken. Current samples: https://github.com/Green-Software-Foundation/carbon-aware-sdk/tree/dev/samples

[Willmish]

I hope following topics are implemented on GHA - they are accepted in ADR-0011:

* Ship WebAPI container images from GitHub packages

* Ship client libraries which are supported by GitHub packages

I think they should be kicked on release-created event on GHA.

@YaSuenag Would you be willing to collaborate on this with @danuw , as I can see you originally proposed that ADR?

[YaSuenag]

@YaSuenag Would you be willing to collaborate on this with @danuw , as I can see you originally proposed that ADR?

Sure, I'd like to collaborate on this!
However I think #149 is needed at first because OpenAPI document is needed to build client libraries. So I will start to work after merging #149 .