✨ Add terminal error

[alvaroaleman]

This change adds a terminal error into the reconcile package. The purpose of that error is to still be logged and used in metrics but avoid retrying in situations where it is known that that will not help. It also adds a new metric for just terminal errors, as they often indicate that some kind of human intervention is needed.

Fixes #2215

[k8s-ci-robot]

@alvaroaleman: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-controller-runtime-apidiff	d10ae95	link	false	/test pull-controller-runtime-apidiff

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[sbueringer]

/lgtm
from my side

/hold
Not sure if Vince also wants to take a look
/assign @vincepri

[vincepri]

/hold cancel

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: alvaroaleman, vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [alvaroaleman,vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment