Properly show continue-on-error jobs/steps in PR UI

[eregon]

continue-on-error is a very useful part of GitHub Actions, but unfortunately it's reflected extremely poorly (= not at all) in the UI and specifically in the pull request UI.
That means it's near-useless or barely better than || true, which is not the desired feature.
The desired feature is to allow some jobs to fail, and indicate it on the PR UI in a sensible way, much like TravisCI did with allow-failures: true.

continue-on-error:

• if used at the step level, then everything appears green and the only way to notice the failure would be to manually open the step and carefully read the output in details. There is also an annotation Check failure on line 1 in .github but that doesn't link to the proper place.
  That's not enough notification, and there seem to be no reasonable way to see if that job actually passed or failed.
  Example: https://github.com/eregon/sequel/runs/1476292451?check_suite_focus=true#step:7:27
  I believe it would be far more useful if continue-on-error at the step level would behave more like it currently does at the job level, and mark the CI job as failed or failed-but-expected.
  The failed step itself should have a red or yellow mark, so it can be noticed without having to guess which step failed.
  GitHub Actions already knows about the outcome of every step, and that's also in API, the only missing part is showing it in a useful manner (and not exactly the same as a successful step).
• if used at the job level, then the job is marked as failed, which at least can be noticed. And the whole workflow is still marked as successful when looking in the Actions tab. But on the PR it's rather confusing, and it looks like the CI broke when it did not.
  Example: https://github.com/jeremyevans/sequel/pull/1737/checks?check_run_id=1479678900
  
  The big red Some checks were not successful sounds like something broke, but no, an error/failure there is expected so it shouldn't be marked as "CI failed". And on the wheel there is a bit of red.
  The whole area is encircled in yellow (I'm not sure why), which seems OK.
  So there I would propose to mark the failed job in yellow, and possibly use another symbol. And then to use the same color in the wheel for jobs which are continue-on-error: true. And also change the text so it would be something like All expected checks have passed in green (the wording for all checks are green is All checks have passed).
  The whole area should then be encircled in green, because it's safe to merge without looking at logs.

This from issue https://github.com/actions/toolkit/issues/399, which seems to not be getting too little attention from github while there are 100+ comments and 500+ interactions.
Somewhat related discussion: #9875

[ktomk]

This perhaps depends on perspective. I had problems to understand continue-on-error first of all (and why it is not the same as || true which I was looking for using allow-failure on Travis before), which - like in your report - was related to the differences between Github Actions and the display of failed jobs in the PR checks.

At the end of the day I understand it that continue-on-error merely controls the behaviour how the following steps in a job are executed in case of a non-zero exit status. The job is green then in any case.

Why the check is red with the PR then I never fully understood, and I only explain it to myself it is using a steps context outcome and not conclusion but that is a guess, I don't know.

From my perspective that difference (Actions vs. Checks) is the main bummer for continue-on-error.

And if it helps, for seeing if continue-on-error was in action, my suggestion is to not scan the actions' logs but the actions' summary (the thing on top), it lists it. @eregon

[eregon]

• job continue-on-error should be the same as allow-failure, I think that's clear and what everyone wants. It works fine except for the reporting in the UI (the job is not failed and is not passed, it's "expected failure" and needs to be shown as such, e.g., a yellow check mark).
• step continue-on-error seems almost the same as || true, except the extra Check failure but that's only visible on the summary, and it doesn't link to the failed step properly. This step continue-on-error if it fails needs to be shown somehow on the job, otherwise it's just too hidden, and || true would be a better choice if people never care whether the job succeeds or not. It must be visible from the PR UI whether a continue-on-error step failed or not (e.g., via a yellow check mark or so)

[ktomk]

+1 for:

• summary should link to failed step
• yellow icon on the job would be very nice

[jeking3]

This is indeed a common technique I used frequently on Travis CI and Appveyor. Here are links directly to the documentation stating one clear use case that this is for:

https://www.appveyor.com/docs/build-configuration/#allow-failing-jobs

You can configure AppVeyor to allow certain build matrix rows to fail and still continue with the rest of the build. The result of these failing jobs is not counted towards build status. This may be useful if you are experimenting with running tests on the latest version of a platform or framework, e.g. Node.js 0.11.

This is a useful technique for saying "test against the latest language level" which can change over time and can potentially fail, without violating any documented requirements of a project. In this case one does not want the sum of their checks to be "failed" or "failure", but "ok" or "passing". The yellow indicator requests related to this ask I find are not really necessary, but would not be harmful, and could be added at a later time.

https://docs.travis-ci.com/user/customizing-the-build/#jobs-that-are-allowed-to-fail

You can define jobs that are allowed to fail in the build matrix.

Allowed failures are jobs in your build matrix that are allowed to fail without causing the entire build to fail. This lets you add in experimental and preparatory builds, for example to test against runtime versions or configurations that you are not ready to officially support.

Additionally I have another use case, which is to add tests for known failures and run the tests in the future to see if they get fixed:

1. I was adding CI jobs to boostorg/mpl to improve testing.
2. One of the jobs added a new architecture and exposes a known problem already in the issues.
3. I want to leave the job in place, failing, and check it in. Then if someone fixes it, this will verify it.
4. I do not want the job to cause other pull requests or CI workflows on main branches to reflect as "failing" or "failed" because it is a known, documented issue. If this is the only failed job in a workflow, it should not fail the workflow if it were marked as allow-failure.

---

The semantics of continue-on-error are quite different from continue-on-error:

• continue-on-error allows the rest of a job or workflow to run even if something fails (i.e. do not "fail fast") but still fail the whole.
• allow-failure allows a job or step to fail, be marked as failed, but be treated as an acceptable error; any step or job with allow-failure should not contribute to the final calculation of the job or workflow. Additionally, allow-failure implies continue-on-error;
  • If this is on a step, and that step is the only step that fails, the job outcome can be failed but the conclusion should be succeeded.
  • If this is on a job, and that jobis the only job that fails, the workflow outcome can be failed but the conclusion should be succeeded.
  • Additional continuance logic should be based on the conclusion, not the outcome.

All other dependencies such as step and job dependencies should continue to work as normal.

[pboling]

Github pushing back on this (in other threads / discussions / issues, which I won't link to because it isn't the fault of the messenger) makes it seem that Github doesn't understand what developers do.

It's not a good look. 👀

(NOTE: eregon wasn't replying to this comment, but to a later comment of mine, which I've deleted as requested, and now the thread is a bit confusing)

[eregon]

I think this is not helpful, could you delete this last post? It's very unlikely to help the matter, is it?

[hishamhm]

(@pboling Side-note: it is sort of an urban legend that companies must always act in the interest of shareholders. See here, here. The "interest of shareholders" can be used by boards to justify acts, but their acts must not always be motivated by that; ultimately the board's fiduciary duty is with the company's overall value, which may be long-term increased by actions which go against the short-term interest of shareholders — it's complicated).

[dhirschfeld]

Fellas, please take your discussion of the merits (or otherwise) of capitalism offline - this isn't the place. For the benefit of everyone wanting to get useful information/updates please keep the discussion here focused on the topic at hand.

[ioquatix]

Not having a proper allow-failures (same semantics as Travis) wastes so much time and causes so much confusion in the UI. Please fix it GitHub. 100% on board with all of the OP statements.

[ioquatix]

I have just filed a bug report with GitHub Support:

Description

As discussed here, GitHub Actions is missing an "Allow Failure" mode/configuration: https://github.com/orgs/community/discussions/15452

https://github.com/actions/toolkit/issues/399 - in total over 1000 positive reactions.

What do we have to do, as a community, to communicate the importance of this feature and get movement on it?

Reproduction Steps

The desired feature is to allow some jobs to fail, and indicate it on the PR UI in a sensible way, much like TravisCI did with allow-failures: true.

An example use case is where you are creating a library of code, which can work with several different implementations. Let's say, you are targeting several implementations of node.js. Some implementations might not have all the features you need, but you still want to test them. Some builds are expected to fail when the node.js implementation doesn't support the required feature. They should not result in a net failure of the entire build matrix, because they are marked as "allowed to fail".

At some point in the future, those builds may start passing. At that point, calling out that they are passing but also allowed to fail, is useful. Because it tells you, now this particular implementation has been updated and the build is passing. You can remove it from "allowed to fail".

[mcraveiro]

"Allow failures" as an enabler of incremental improvement

(reposting here from 399 as requested by @pboling)

I have a one-person project [1] , so PR checks are not an issue for me. However, as with most in this ticket, I am need of "allow failures". Let me add my use case to the discussion as I think it is slightly different from the ones described so far, but probably a very common one.

At present in my project we do not support clang-cl (a c++ compiler for windows). The clang-cl build fails to link due to some weird link error, and I haven't had time to figure it out. If I add clang-cl to the build matrix, all my commits get marked with a red X, i.e. failed. This, even though the build was marked as experimental [2]. As a consequence, I need to check each and every commit in detail, in case some other build is broken. I could of course remove the build altogether (which is my workaround at present, see commit above). This is less than ideal because I'd rather know if a particular commit makes that build worse, or, in case I get lucky, better. So it would be nice to continue to build clang-cl in my build matrix, without impacting the overall red/green status of each commit.

In fact, by not having this feature, it makes incremental improvements to the build much harder. I now need to enable the build, deal with any rotting that may have happened since the last time it was enabled, make some changes, run the build and then disable it again. The drive-by approach I used to take - whenever I got 20 minutes here and there, do a quick experiment to see if it improved things - has now been rendered unworkable because most of those twenty minutes would be spent on overhead.

All of this to say that, for me, this is an important feature.

Links

[1] https://github.com/MASD-Project/dogen
[2] https://github.com/MASD-Project/dogen/blob/66022e278892f2fedfa44df2b56cb2741e01a28c/.github/workflows/continuous-windows.yml#L30

[marcellodesales]

I have the same problem... My slack action is failing, failing the whole pipeline, although the rest worked... So the PR is locked... At this point, it's hard to believe Github Management is way too far from the community... This feature is definitely a very important ask and would allow us to bypass situations like the one described above...

[rachelruderman]

Plus one for all of the above. I'm working on fixing flaky tests in our code that cause CI to fail, which means I'm running many, many tests and constantly checking if they succeeded or failed. We have a GitHub check for CI and another GitHub check to flag tests that failed once but succeeded on retry. The "succeeded on retry" check is optional. My work would be greatly facilitated by the optional check not causing the build to be marked a fail.

[ross-spencer]

Echoing what others are saying here, our use-case is very similar and connected to https://github.com/actions/toolkit/issues/399#issuecomment-1260017446. allow-failures would enable this.

I want to monitor Python release-candidates and forward compatibility. Just to provide a warning for the roadmap if anything fails. The red failure is a little depressing after a period of time, and not true to the intention of the workflow.

[mtaylor-evidation]

I agree this would be a nice to have. +1 the allow-failure

[johnnyshields]

allow-failure and continue-on-error are two different features which are not mutually exclusive and in fact can be combined to complement each other (see below). As these are common semantics in every other CI system on the planet, Github should implement both, and we should stop squabbling over "whether we should only eat apples or only eat oranges."

• allow-failure (could also be called ignore-failure) permits ignoring a failure of a specific job run in the matrix and still marking the overall CI task as success (green).
  For example, given I have a Ruby gem hosted on Github, and I want to check that it works against multiple Rails versions. If I test against the Rails HEAD (i.e. latest repo master branch), as it may contain unknown breaking changes, I can use allow-failure so this particular HEAD run can fail, but I still pass my CI. In this case, the UI state of the failed job should be indicated as a "Gray X" , i.e. failed, but ignored.
• continue-on-error permits a step within a job run to fail, but still move on to the next step.
  For example, given I have a step that does a report notification, which if failed I still want to run the subsequent steps, I could use continue-on-error. Here the failed step still counts a failure for the overall job run, unless I also combine it with allow-failure as per above. Here a "Yellow X" might make sense UI-wise.

[pboling]

Until Github has both of these features properly implemented (and addresses other serious issues), I will continue investing hours multi-homing my projects onto other services.

[pboling]

Update: The closed issue has been moved to the appropriate, relevant, repository, and is where backlinks should point, even though it remains closed.
actions/runner#2347 (comment)

Now that this is the official, and only, forum for discussion of this issue it is worth point out that - this is a terrible forum for discussing this issue.

Github Discussions do not backlink to commits / PRs / Issues that link to them. GitHub issues, OTOH, do offer backlinks. Many, nay hundreds, nay the majority of interactions on the now-closed-issue were backlinks where people were referencing the issue from all over GitHub precisely in the context where they were having problems that would be solved by this feature.

Now any future context of that sort will be lost, and we will only get a fraction of the context, discussion, and input from the community, and devoid of the hundreds of real world examples that were available before.

Perhaps it is good to have the loudest voice control the discussion (GitHub in this case, speaking over everyone else by saying nothing at all). But if that is the case, don't elfing call it "Community Discussions". It's misleading, as it implies to most reasonable people that GitHub would be participating in the discussion. Call it "The Place To Vent While GitHub Ignores You" instead.

[mrk-han]

+1 for your use of "elfing"

And yes, these are very valid concerns. But let's try to tone down the rhetoric and keep things civil here.

[rivy]

Now that this is the official, and only, forum for discussion of this issue it is worth point out that - this is a terrible forum for discussing this issue.

Github Discussions do not backlink to commits / PRs / Issues that link to them. GitHub issues, OTOH, do offer backlinks. Many, nay hundreds, nay the majority of interactions on the now-closed-issue were backlinks where people were referencing the issue from all over GitHub precisely in the context where they were having problems that would be solved by this feature.

Unfortunately, I also agree that this is more of a "goes to quietly die" area, rather than "discussion of issue" area.
But, at least to maintain some context, here is a backlink to the larger, multi-year issue discussion thread https://github.com/actions/toolkit/issues/399.

[haltman-at]

Regardless of whether continue-on-error is changed to work like allow-failure, or whether allow-failure is introduced as a separate option and the two are kept distinct, ultimately, we need allow-failure in some form. More specfically: An allowed failure is one whose failure does not cause a failure of the level it's contained in. If one job fails, but it's an allowed failure, then that job should display red but the overall PR should display green. Etc.

...of course a ton of people have said this already, I just wanted to add my name to do what I can to underscore the importance of this...

[mansona]

Hey folks 👋 I wrote a post about this on the now infamous closed issue on this topic a while ago where I was sharing a project that I created to bring back some semblance of the "allow failures" feature that we used heavily in the Ember community when Travis was the default CI provider.

I thought I would share a link to the project here in case it helps anyone 👍

https://github.com/mainmatter/continue-on-error-comment

[akrabat]

Seeing a red X against a PR that has successfully passed CI for all shipping versions of the underlying runtime, but failed the nightly is requiring additional training of all project maintainers. I'm very surprised that GitHub closed issue actions/runner#2347 as "Not planned" with no notice of why this UX problem is considered acceptable.

[andrewakim]

Hi all 👋 - I'm a PM for Actions and looking into this issue. I'm hoping to setup 5-10 conversations with folks to dig into the problem and use cases. If you're interested in chatting, please use this Google Form to send over your details. I'll close the form and edit my post once I get enough conversations lined up. Thanks and looking forward to chatting!

Edit on Mar 3, 2023: Thanks for signing up and the responses! I have enough conversations lined up so I've closed the form. I'll reach back out to setup more conversations if needed.

[ricardoboss]

So... it's over a year later and we haven't got an update. Is this still in the planning phase?

[pboling]

@andrewakim A day on Jupiter is only ten Earth hours long. It has been almost 3 Jupiter years Earth-year-Jupiter-days now (I don't think that's a real thing). Any chance we'll get this soon?

[johnnyshields]

Nit: In fairness to Microsoft, a Jupiter year is approximately 12 Earth years, so it hasn’t been a Jupiter year… yet. A year is how long it takes the planet to orbit the sun, a day is how long it takes the planet to rotate on its axis.

[pboling]

Well nitted.

I sumbit that all Microsoft needs to do is address this issue in the next few trillion years, and it can comfortably claim to be moving faster than a photon of light (attempting to visit every Hydrogen molecule of the Sun).

86,400 seconds in a day = 60 * 60 * 24
10^-21 seconds in a zeptosecond (zs)
(8.64 * 10^25 zs) / day = 86,400 seconds per day / 10^-21 seconds per zs

Since issue creation:
• 1080 days
• 9.3312 * 10^28 zs = (8.64 * 10^25 zs / day) * 1,080 days

247 zs per hydrogen-molecule-traversal-by-light

3.77781377 * 10^26 = 9.3312 * 10^28 zs / 247 zs per hydrogen-molecule-traversal-by-light

Light may have traveled across 3.77781377 * 10^26 hydrogen molecules since Microsoft went silent on this issue.

There are about 8.36541 * 10^56 hydrogen molecules in the Sun.

If light was tracing a path across every hydrogen molecule in the Sun, and they are all statically lined up next to each other, (4.51599356 * 10^-29)% of the Hydrogen molecules would be traversed so far, which is a tiny fraction, and it will take bajillions of years to travel the remainder.

[johnnyshields]

The sun will go red giant and engulf the Earth in about 5 billion years, however, the Earth will still appear green in the Github Actions UI because it is marked as “continue-on-error”. There will be no soft failure indication at all.

[pllim]

All I ever wanted is a new yellow icon to show when a job that is allowed to fail failed but does not necessarily block merging. ❌ is too much and ✔️ is misleading.

[tracyhoward]

It can be a green circle with a red X in the middle for all I care, we just need a distinction.

[y-nk]

As a temporary workaround, if that can help anyone:

steps:
  - name: foobar
    run: |
      set +e
      set +o pipefail

      function wrap_command() {
        curl # here, any command which supposedly fails.
        return $?
      }

      wrap_command
      echo $?
      exit 0

I use this in a E2E test logic where we want 3 retries before calling a test "fully failed" such as:

  - name: foobar
    run: |
      set +e
      set +o pipefail

      function yarn_e2e() {
        yarn e2e $1
        return $?
      }

      for attempt in {1..3}; do
        echo '::group:: test: ${{ matrix.testFile }} ($attempt/3)'
        yarn_e2e ${{ matrix.testFile }}
        [ $? -eq 0 ] && exit 0
        echo '::endgroup::'
      done
      exit 1

[ro0NL]

i had clean/simple yaml once 😂

[adamwojt]

This is much needed. There is no way to deem workflow as "skipped" or "neutral" when one of the jobs fails but it is not so important.

[fuch1m]

+1 for this feature

[sjanc]

how hard can it be to implement this? is it being blocked on purpose or what?

[onigoetz]

If you put an AI spin to that feature there is probably a chance to get it implemented ... 🙃

[antonbaliasnikov]

+1 for this feature

[zackees]

How is this feature not supported yet?

[pboling]

This is the right question. The evident answer is that software development is not a priority for Microsoft. Possibly the most upvoted issue in the history of GitHub Actions is still dead after four years.

actions/runner#2347
actions/runner#798
https://github.com/actions/toolkit/issues/399
actions/toolkit#1034
https://github.com/orgs/community/discussions/56441
https://www.reddit.com/r/devops/comments/y2t6b9/the_feature_request_for_a_allowfailure_option_on/

It's a ridiculous foot gun.

Any updates @andrewakim ?

[LeMyst]

Product Manager recently answered: actions/runner#2347 (comment)

[pboling]

I had read that and forgotten about that, as I am currently running into this missing feature again.

@nebuk89 this is the discussion where it has been discussed most recently by a different project manager (@andrewakim), and let us know if there is a new home for this issue!

[sobomax]

[wsanchez]

I got sufficiently tired of this that I came up with a work-around. Posting it here in case others find it useful:

https://github.com/burningmantech/ranger-ims-server/pull/1347/files

What I did gets me:

• a green checkmark for the job (not great, but this is the trade-off we are making)
• a green checkmark for the workflow (the goal here)
• a comment in the PR indicating that an optional build failed (replacement for red checkmark for the job)

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[pboling]

What the hell? 😂😭. Did DOGE cuts hit GitHub?!

[sjanc]