Service Controller failing to create k8s Events

[c-kruse]

Describe the bug
The skupper service controller is logging a failure to write events when deployed without the console.

E1214 16:25:45.094731       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)
E1214 16:25:55.096072       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)
E1214 16:26:05.096629       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)
E1214 16:26:15.097808       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)
E1214 16:26:25.098153       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)
E1214 16:26:35.099177       1 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:".17a0bf6df167801f", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Service", Namespace:"", Name:"", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"ServiceSyncEvent", Message:"Service sync receiver connection to amqps://skupper-router-local.withoutconsole.svc.cluster.local:5671 established", Source:v1.EventSource{Component:"skupper", Host:""}, FirstTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), LastTimestamp:time.Date(2023, time.December, 14, 16, 25, 38, 68480031, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'can't create an event with namespace 'default' in namespace 'withoutconsole''(may retry after sleeping)

How To Reproduce
skupper init in some kubernetes namespace besides default.

Wait for init to finish, then watch the logs for the service controller: kubectl logs deployments/skupper-service-controller.

Observe that the ServiceSyncEvents do not get published: kubectl events | grep ServiceSyncEvent

Expected behavior
I would expect ServiceSyncEvents to be published. I think we are just associating them with the wrong resource.

When deployed with the console, events are published associated with service/skupper

$ kubectl -n withconsole events --for Service/skupper
LAST SEEN   TYPE      REASON             OBJECT            MESSAGE
17m         Warning   AllocationFailed   Service/skupper   Failed to allocate IP for "withconsole/skupper": no available IPs
16m         Normal    IPAllocated        Service/skupper   Assigned IP ["192.168.11.21"]
16m         Normal    ServiceSyncEvent   Service/skupper   Service sync receiver connection to amqps://skupper-router-local.withconsole.svc.cluster.local:5671 established
16m         Normal    ServiceSyncEvent   Service/skupper   Service sync sender connection to amqps://skupper-router-local.withconsole.svc.cluster.local:5671 established

Environment details

• Skupper CLI: 1.5.1
• Skupper Operator (if applicable): N/A
• Platform: kubernetes

Additional context
Add any other context about the problem here.

[c-kruse]

Looked into this a little, I don't have the context into our intent here or what the common practices are in the k8s ecosystem, but it appears to me we're in a bit of a pickle from an API stability standpoint here. I couldn't find any evidence that we had any sort of event base choreography in skupper itself (maybe in the operator?) that depends on these events, but that's not to stop someone from rolling their own extension that depends on these events.

If that's not a concern, my take on this is that we should probably stop targeting the Service/skupper object and maybe target the service controller deployment? Or pod? I'd think there'd be a way to just do "self" or "owner". Maybe @nluaces or @grs would have thoughts here?

[nluaces]

@c-kruse I always (wrongfully) thought that the skupper service was going to be available in all occasions. It makes total sense to me to assign the kubernetes events to a resource that is going to be available whether the console is available or not, like the service-controller. In fact, most of the kubernetes events are generated by the service controller pod.