Separate nsclean tests from standard runs

[melanieclarke]

Currently most NIRSpec spec2 tests explicitly turn nsclean on, but it is off by default, it is a deprecated step, and it is a numerically sensitive algorithm. It would be helpful to separate out nsclean test results from standard runs.

Tasks

• If you have a specific reviewer in mind, tag them.
• add a build milestone, i.e. Build 12.0 (use the latest build if not sure)
• Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see changelog readme for instructions)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[melanieclarke]

Initial regtests to generate truth files:
https://github.com/spacetelescope/RegressionTests/actions/runs/15000660549

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.86%. Comparing base (53d69d1) to head (9c837cd).
⚠️ Report is 651 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #9453   +/-   ##
=======================================
  Coverage   77.86%   77.86%           
=======================================
  Files         362      362           
  Lines       36335    36335           
=======================================
  Hits        28294    28294           
  Misses       8041     8041           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[melanieclarke]

Full regtests here:
https://github.com/spacetelescope/RegressionTests/actions/runs/15002661779

The new nsclean tests should pass - I uploaded truth files for them. The spec2 products will need okifying when this goes in.

Diffs are as expected, except that there are some unrelated MIRI LRS diffs for optimal extraction, from a new PSF reference file delivered today.

[emolter]

Is there any easy way to skip some steps, or otherwise reduce the runtime, of the added nsclean tests? It seems to me that these tests are now mostly covering identical functionality, with the exception of that test (and its integration within the pipeline).

The new runtime tracker shows me that the tracked instance of calwebb_spec2 takes just over 2 minutes, so an extremely rough estimate would be that this PR adds 6 minutes of runtime to the regtest suite. Which isn't a lot, but it's probably like 10%.

[melanieclarke]

Is there any easy way to skip some steps, or otherwise reduce the runtime, of the added nsclean tests?

I could just run the nsclean step and check that output if that's preferable. I like running it from the spec2 pipeline because that also checks the integration of the step. I also generally find it useful when the subsequent products are also produced and checked - sometimes it's helpful for diagnosing the impact of a new diff. But maybe none of that is important here, since we expect this step will eventually be removed.

[emolter]

I could just run the nsclean step and check that output if that's preferable. I like running it from the spec2 pipeline because that also checks the integration of the step. I also generally find it useful when the subsequent products are also produced and checked - sometimes it's helpful for diagnosing the impact of a new diff. But maybe none of that is important here, since we expect this step will eventually be removed.

I can see both sides. I'll leave it up to you

[melanieclarke]

I can see both sides. I'll leave it up to you

As long as we're still calling nsclean in the spec2 pipeline, I'd like to make sure we test it in that context. I think the extra runtime is not too extreme.

I filed a ticket to remove nsclean: JP-4005. If we can get that done, the extra tests won't be around much longer.

[emolter]

Both my questions have been answered, thanks Melanie!