Skip to content

Workflow

Jump to bottom
Ken MacDonald edited this page Aug 25, 2025 · 20 revisions

General Notes

  • All changes to the repository are made in branches on the developer's fork of this repository and submitted via a pull request (PR).
  • Each PR should be labeled with a step label (e.g. a PR to refpix should have a label refpix) and any other relevant labels. These are auto-assigned, but make sure they look correct!
  • Each PR should be assigned a milestone (a build number or a future release version number).
  • Each PR should have a changelog entry (or the no-changelog-entry-needed label as appropriate).
  • A PR is merged after someone other than the submitter has reviewed and approved it. Usually this is one of the maintainers familiar with a particular step or pipeline. Trivial changes can be merged by the submitter of the PR (use your best judgement). If the changes may affect another step, consider asking that step maintainer to review them.
  • A CI workflow will run on the PR branch, which attempts to make a clean install of the repository, run all the unit tests in several architectures and Python versions, and check for compliance to our code style rules.
  • There are additional details on the CONTRIBUTING.md page; please read that too!

Setting up the Development Environment

  1. (optional) Create a clean Conda environment (or similar, if you use a different environment manager).

  2. Fork the main jwst repository.

  3. Create a local copy ("clone") of the repository you just forked and let it know where the main repository is.

    git clone https://github.com/<your-git-user-name>/jwst.git
cd jwst
git remote add upstream https://github.com/spacetelescope/jwst.git

    NOTE: If you're cloning GitHub repositories using HTTPS, you can use a credential helper to tell Git to remember your GitHub username and password every time it talks to GitHub:

    https://help.github.com/articles/caching-your-github-password-in-git/

    If you clone GitHub repositories using SSH, then you authenticate using SSH keys instead of a username and password:

    https://help.github.com/articles/generating-an-ssh-key/

    You should now be able to look at all remotes now and see something like 

    % git remote -v
upstream   https://github.com/spacetelescope/jwst.git (fetch)
upstream   https://github.com/spacetelescope/jwst.git (push)
origin     https://github.com/<your-git-user-name>/jwst.git (fetch)
origin     https://github.com/<your-git-user-name>/jwst.git (push)

    The above operations to fork and clone the repo are normally performed once (not necessary every time you want to create a new PR branch).

  4. Before making changes, it's useful to install the jwst package in "editable" mode with the contributor (test, documentation, and code style) dependencies:

    pip install -e .[contrib]

    This configuration allows calls to the pipeline to pick up any changes you've made to the pipeline without having to re-install jwst every time you make a change.

  5. It's strongly recommended to add our pre-commit hook into the repository. pre-commit should have been installed as part of the [contrib] dependencies, so just run

    pre-commit install

    from the top level of the jwst repository. This step ensures that all of our code style checks will run every time you make a commit.

Contribution Workflow

It's finally time to start work on a new feature/change by making a separate branch that tracks upstream/main.

  1. First, always fetch the upstream branch to get the latest changes into your clone.

    git fetch upstream

  2. Make a new branch called feature1 off of upstream/main.

    git checkout -b feature1 upstream/main

    You will see a message like the following:

    Branch feature1 set to track upstream/main

  3. Work on this branch until the new feature is ready. Then commit the changes:

    git add <file_names_to_commmit>
git commit -m "JP-nnnn: Adding feature 1" file_names_to_commit

    Note that it's common (and encouraged) practice to include the Jira ticket number as a prefix in your commit message. This sets up an automatic link between the Jira ticket and the PR.

    Sometimes development of a new feature takes a while, and perhaps upstream/main has passed you by, or you need a bug fix in your feature branch that someone else committed to upstream/main. So to update your branch with the latest from upstream:

    git fetch upstream
git rebase upstream/main

    It is better to use rebase than merge, because the commit log of upstream/main remains cleaner once your branch is merged back in.

  4. Run any relevant unit tests and regression tests to ensure the changes actually work. If there are no tests that cover the changes, write one!

  5. When all changes are made, push to your forked repository (note: you are pushing the new branch to your own forked copy of the main repository):

    git push origin feature1

  6. Now go to your forked copy (http://github.com/<your_github_username>/jwst.git) to find the branch you just pushed. (There's a link to your github account in the upper right corner of any github page. Clicking on it will take you to your github profile and you can find the repository under the tab repositories.)

    If it's a new branch you will see Compare and pull tab next to the branch name.

    You can look at the changes online. If everything looks good, create a PR against upstream/main by clicking on the Create pull request button. This will take you to the main JWST repository on github.

    • Add relevant descriptions, labels, milestones, and ask for a review.
    • After the review and when CI and regression tests pass, merge using the web interface.

Code style

We use several tools to help us maintain PEP8-compliant code style and Numpy-style docstrings. These tools are all run as part of our pre-commit hook and as part of the CI workflow that runs on every pull request. There is additional information about these tools on our CONTRIBUTING.md page.

If you've set up pre-commit as described above, these should all be running automatically every time you make a commit, and they should be picking up our repository's custom configurations from the pre-commit-config.yaml, pyproject.toml, and .ruff.toml files.

Updating a module for code style

At time of writing (January 2025), new code style rules and checks have been added, but not all of the submodules in the repository have been updated with the requisite changes. Updating a module to align with the style rules requires a slightly more involved workflow, as follows.

  1. Hopefully you're already set up as described above. If you're not using pre-commit yet, this might be a good time to start.

  2. Check out a new branch off of main.

  3. Remove the module you want to fix from the per-module ignore lists in three places:

    • Inside the file .pre-commit-config.yaml under the numpydoc-validation section.
    • Inside the file .ruff.toml under the header [format].
    • Inside the file .ruff.toml under the header [lint.per-file-ignores].

  4. From the top level of the JWST repository, run the command

    pre-commit run --all-files

    This should show all the failing checks for the module you just removed from the ignore lists, and will fix some of them.

  5. Fix the problems using a normal git workflow (make changes, git add, git commit. Note that the pre-commit hook that runs every time you git commit checks ONLY any files that were modified, so it may not pick up every failing style check in the module the way pre-commit run --all-files will. Note also that if there were any failures, even if they were auto-fixed, the commit will fail and you'll need to stage and commit the changes again.

  6. Submit the PR! The no-changelog-entry-needed tag can be applied, unless you found and fixed a bug while reviewing the code or something like that.

Updating C code style

Use JP-4099 for reference and obtain the clang-format file, a configuration file, attached to this ticket and save it to a convenient location on your computer; typically it is saved as a . file in the home directory. A complete list of the configuration options for the configuration file is here.

Once you have saved the configuration file, format C code according to clang-format do the following:

  • Create a conda environment.
  • Activate the conda environment, then run pip install clang-format.
  • To format a C file, run clang-format --style='file:<configuration file>' -i <C file>

Ignoring style fixes

Sometimes the style checkers find issues that we would like to ignore for some reason. All of the checkers support ignoring specific lines using in-line comments, but these comments have different syntax for the different checkers. Here is a list for convenience:

  • Ruff Linter: Use # noqa: RULEID on the same line as the offending text, e.g. wcsinfo = datamodel.meta.wcsinfo._instance # noqa: SLF001
  • Ruff Formatter: For a single-line skip, use #fmt: skip on the same line as the offending text. For a multi-line skip, surround a block of text with # fmt: off and fmt: on, each on their own lines.
  • Numpydoc: Use # numpydoc ignore=RULEID on the same line as the offending text, e.g. def tikho_mat(self): # numpydoc ignore=RT01
  • Codespell: Use # codespell:ignore varname on the same line as the varname it complained about, e.g. (!(err = ensure_array(erro, &free_err))) || // codespell:ignore erro (note this example is C code). You can also consider whether the word should be added to the "skip" list under [tool.codespell] in pyproject.toml.
  • Mypy: Use type: ignore[rule-name]``on the same line as the offending text, e.g. ``from jwst.datamodels import ImageModel # type: ignore[attr-defined]

IDE-specific style tools

Notes on how to configure Emacs and Vim to conform to PEP8.

Notes on VSCode extensions for linting and formatting.

Clone this wiki locally