Skip to content

Release Procedures

Jump to bottom
Zach Burnett edited this page Jun 11, 2025 · 101 revisions

This guide assumes that you have cloned jwst, and added a remote named upstream pointing to the central repository:

git remote add upstream https://github.com/spacetelescope/jwst.git

Before starting the release process, ensure that the latest regression test run for the main / master branch has passed successfully.

1. Update metadata in the main / master branch

  1. Run towncrier build with the desired release version, to build the next changelog entry in CHANGES.rst from fragments in the changes/ directory:

    pip install towncrier
towncrier build --version 0.16.1

  2. Update the Software vs DMS build version map section of README.md to show the latest release version, date, and the CRDS_CONTEXT this release is being currently tested with (this is usually the current operational context: crds list --operational-context).

  3. Update the release version and date in CITATION.cff, and add any new contributors.

  4. Make sure dependencies in pyproject.toml are updated. In particular, ensure that crds, stcal, stdatamodels, and stpipe include their newest tested versions, and exclude any older versions with incompatible API.

Important

If stcal and stdatamodels have new updates that change the API used by jwst, make sure to use the relevant reference files on Artifactory when testing.

Caution

PyPI will reject any submission with Git dependencies in its metadata (i.e. git+https://github.com/spacetelescope/stcal.git@main). Dependencies must be defined against a released version available on PyPI (i.e. stcal>=1.5.0).

  1. Commit your changes, make a PR, and merge to main / master on spacetelescope/jwst.

If this release is a cherry-picked patch to an release branch, make the changes on the master branch first, then cherry-pick and modify as needed over to the release branch.

2. Create or update the release branch

If you're making a major or minor version release, then the release branch will not yet exist. If you're releasing a patch version, then a release branch will already exist. Select one of the next two sections accordingly.

New major / minor version

  1. Fetch and checkout the upstream main / master:

    git fetch --all --tags
git checkout -t upstream/master

  2. Inspect the log to ensure that no commits have snuck in since your change log update:

    git log

  3. Create a new release branch. The name of the release branch should share the major and minor version of your release version, but the patch version should be x. For example, when releasing 1.8.0, name the branch release/1.8.x.

git checkout -b release/a.b.x
  1. Push the branch to the upstream remote: 
    git push -u upstream HEAD

Patch release of an existing major / minor version

In the case of a patch release, the release branch will already exist.

  1. Checkout and freshen release branch (this assumes that your local branch is already tracking upstream/release/a.b.x):

    git checkout release/a.b.x
git pull

  2. Cherry-pick relevant commits from main / master that should be included in the patch release (including the new changelog commit):

    git cherry-pick ...

Tip

Alternatively, you can pull all changes from master into the release branch:

git fetch upstream
git pull upstream master
  1. Push updates to the upstream remote: 
    git push upstream HEAD

Warning

If the release branch has diverged from master, you will have to run the regression tests on the release branch and have them pass to be able to do the automated release to DMS (see the DMS release section below for details). Additionally, you should tag main / master with a development tag (i.e. 1.8.9dev) so the locally-installed version shows correctly.

Tips for git cherry-pick

Say a bug needs to be fixed in 0.16.0, and is also present in master.

  1. Fix the bug on master through a PR. Merge it and get the hash of the commit. As an example, let's say the commit hash is f41120e.

  2. Cherry pick all commits that need to be included in the patch release.

    git cherry-pick -x f41120e

  3. If there are conflicts, resolve them before you continue with other commits. In general, start with the oldest commit so that conflicts are minimized.

Tip

Sometimes a commit is a merge commit with more than one parent. In that case, the above command will fail with an error message:

error: Commit f41120e is a merge but no -m option was given.
fatal: cherry-pick failed

This discussion might be helpful.

To resolve this, look at the parents of the commit and choose which one is the ancestor:

git cherry-pick -x -m 1 f41120e

(this tells git to use the first parent)

  1. Use git show to verify which changes are going into the release branch.

3. Review the release branch's latest automated test run

The creation or update of the release branch should have triggered a CI job on GitHub actions. Find the latest build on the release branch in the Actions tab: https://github.com/spacetelescope/jwst/actions/workflows/ci.yml

4. Prepare main / master for further development

  1. Remove requirements from requirements-sdp.txt.

  2. If the commit with the release tag is not on main / master, you should tag the next commit on main / master with a development tag (i.e. if we just released version 1.2.1 on release/1.2.x, the development tag should be 1.2.1.dev on the main / master commit directly after the divergence). This allows setuptools-scm to show the correct version when installing the package locally (pip install .).

    git fetch upstream
git checkout upstream/master
git tag -a a.b.c.dev -m "development tag after divergence"
git push upstream a.b.c.dev

5. Tag the desired release commit with a pre-release version

At this point, you should have the release branch checked out and ready to tag.

  1. Create an annotated tag with a name that matches your intended release:

    git tag -a a.b.crc1 -m "JWST DMS Build x.y rc1"

  2. Push the new tag to the upstream remote:

    git push upstream a.b.crc1

8. Deliver the pre-release RCs to DMS

  1. Create a new stasis delivery configuration (.ini) with a new revision number. See the documentation for instructions. Put it on the shared drive, or at a publicly accessible URL.
example of a delivery configuration... 
[default]

[meta]
mission = jwst
name = JWSTDP
version = 1.18.1
rc = 1
final = false
python = 3.11

[conda]
installer_name = Miniforge3
installer_version = 24.9.0-0
installer_platform = {{env:STASIS_CONDA_PLATFORM}}
installer_arch = {{env:STASIS_CONDA_ARCH}}
installer_baseurl = https://github.com/conda-forge/miniforge/releases/download/{{conda.installer_version}}
pip_packages =
    jwst[sdp]==1.18.1

[runtime]
PYTHONUNBUFFERED = 1
TEST_BIGDATA = https://bytesalad.stsci.edu/artifactory
CRDS_CONTEXT = jwst_1364.pmap
CRDS_CACHE = ./crds_cache
CRDS_SERVER_URL = https://jwst-crds.stsci.edu
CRDS_CLIENT_RETRY_COUNT = 3
CRDS_CLIENT_RETRY_DELAY_SECONDS = 20

[test:jwst]
repository = https://github.com/spacetelescope/jwst
script_setup =
    pip install --force-reinstall -U certifi
    pip install -e ".[test]"
    pip install -r requirements-sdp.txt
    pip install pytest-xdist
script =
    pytest -s \
        -v \
        -r=fEsx \
        --bigdata \
        --env="1.18.0" \
        --slow \
        -n $(( $CPU_COUNT )) \
        --dist=loadscope \
        --basetemp="{{ func:basetemp_dir() }}" \
        --junitxml="{{ func:junitxml_file() }}"

[deploy:artifactory:test results]
files =
    {{ storage.results_dir }}/results-*.xml
dest = {{ meta.mission }}/{{ info.build_name }}/results/

[deploy:artifactory:delivery]
files =
    {{ storage.output_dir }}/**
dest = {{ meta.mission }}/{{ info.build_name }}/

[deploy:docker]
registry = bytesalad.stsci.edu
image_compression = zstd -v -9 -c
build_args =
    SNAPSHOT_INPUT={{ info.release_name }}.yml
    SNAPSHOT_PKGDIR=packages
tags =
    {{ meta.name }}:{{ info.build_number }}-py{{ meta.python_compact }}
    {{ deploy.docker.registry }}/{{ meta.name }}:{{ info.build_number }}-py{{ meta.python_compact }}
test_script =
    source /etc/profile
    python -m pip freeze
    mamba info
  1. Run the STASIS Delivery (Public) workflow, pointing the Delivery file box either to the path on the shared drive, or to a publicly available URL.
  2. Check https://ssb.stsci.edu/stasis/releases/jwst/ to see the newest delivered release.

Tip

Delivering new revisions of the same version / build

To deliver another revision of the same version, increment the meta.rc number in the delivery configuration file (.ini), then re-run the STASIS Delivery (Public) workflow.

9. Tag the final release commit with the final release version

Once DMS has identified a commit as suitable for a final release, you can tag that commit with the final version.

  1. Create an annotated tag with a name that matches your intended release:

    git tag -a a.b.c -m "JWST DMS Build x.y"

  2. Push the new tag to the upstream remote:

    git push upstream a.b.c

10. Make a GitHub release and publish the package to PyPI

Create a new GitHub release

  1. Visit the spacetelescope/jwst repository's releases page.

  2. Click Draft a new release.

  3. Select the existing tag that you just created and pushed, and title the release JWST Build x.y rcN, where N is the current release candidate for DMS (we can always remove the rcN part from the title later if this turns out to be the final release).

  4. Publish the release.

Validate that the package did publish to PyPI

Publishing the GitHub release should trigger an automated workflow that should build the wheel and source distribution and publish the package to PyPI.

After this workflow completes, you can confirm that the new release appears on PyPI: https://pypi.org/project/jwst/#history

Additionally, you can test installing the new version with pip:

pip install jwst==a.b.c

11. Update Artifactory regression test data

Go to the Artifactory web interface and copy the passing test input and truth data from

jwst-pipeline/dev

to a new directory named the same as the release tag

jwst-pipeline/a.b.c

12. Update CRDS CALVER and CRDSCFG

CRDS maintains a reference file, CALVER, and metadata that are tied to JWST software releases. Review the two substeps that are part of the CRDS S/W Release Process. If either require updates due to a JWST release, file a CCD issue to update CRDS as needed.

The two subseps are:

Clone this wiki locally