Enhance failure detection to also recognize failure status of top level resources

[dgrove-oss]

MCAD provided a mechanism for encoding conditions of the top level resources that if present would indicate success or failure. This was used to augment the pod-level status infromation.

We need to design and Implement a similar mechanism for the v1beta2 AppWrapper.

[dgrove-oss]

We've discussed further and primarily see value in recognizing when a top level resource has failed. This would allow us to transition the AppWrapper to failed immediately (without relying on pod-level health).

There seems to be less value in the resource-level success status because we still would have to wait for all pods to reach a Completed state before we could release resources (if we released eagerly, we could cause quota overages).

[dgrove-oss]

#163 does this for PyTorchJobs

[dgrove-oss]

Closing; we've now implemented for all the supported GVKs that have useful operator-level failure statuses.