JDK Mission Control to monitor Java Pod in AKS

[dkirrane]

I'm trying to connect JDK Mission Control to a Java app (using microsoft/openjdk) running on AKS v1.27.9.
I have tried the following...

1. I have the following set in the Pod Spec:

          env:
            - name: JAVA_OPTS
              value: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1"

          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: httpmgmt
              containerPort: 8081
              protocol: TCP
            - name: jmx
              containerPort: 9010
              protocol: TCP

2. I can port-forward without issue

kubectl port-forward -n testing pod/<pod_name> 9010:jmx
Forwarding from 127.0.0.1:9010 -> 9010
Forwarding from [::1]:9010 -> 9010

3. From misson control I hit Test Connection

4. port-forward command then fails and exits with connection refused IPv6 dial tcp6:

Handling connection for 9010
E0403 14:12:45.950407    2290 portforward.go:409] an error occurred forwarding 9010 -> 9010: 
error forwarding port 9010 to pod 693cfe5a67379a549c62b23e8b78414e41f9443d9da89225b39e9e7c7891d142, uid : 
failed to execute portforward in network namespace "/var/run/netns/cni-687066f0-fd88-0d91-3af7-2dea8495bb13": 
failed to connect to localhost:9010 inside namespace "693cfe5a67379a549c62b23e8b78414e41f9443d9da89225b39e9e7c7891d142", 
IPv4: dial tcp4 127.0.0.1:9010: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
error: lost connection to pod

[karianna]

@dkirrane Apologies for missing this Q! I've got some generic K8S advice that you can try, but if that doesn't help resolve the issue I'll put you in touch with the AKS team.

To debug and resolve the connection issue you are experiencing when attempting to connect Java Mission Control (JMC) to your JVM running in an Azure Kubernetes Service (AKS) environment, you can follow these troubleshooting steps. These steps are designed to identify common issues related to network configuration, service exposure, and pod accessibility:

1. Validate Pod Status and Logs

First, ensure that your JVM pod is running and healthy:

• Check Pod Status:

  kubectl get pods -n <namespace>

  Replace <namespace> with the appropriate namespace if not using the default. Look for your specific pod's status.

• Check Pod Logs:

  kubectl logs <pod-name> -n <namespace>

  This can provide insights if the JVM or any related service has internal errors.

2. Confirm Port Exposure and Service Configuration

Ensure that the correct port (9010 in your case) is exposed and configured properly in your Kubernetes deployment or service:

• Check Service Configuration:

  kubectl describe svc <service-name> -n <namespace>

  Verify if the target port is set to 9010 and the service is of the correct type (e.g., NodePort, LoadBalancer).

• Port Forwarding Command:
  Ensure you are using the correct syntax for port forwarding. It should look like this:

  kubectl port-forward <pod-name> 9010:9010 -n <namespace>

3. Firewall and Network Policies

Check if there are any network policies or firewalls that might be blocking the connection:

• List Network Policies:

  kubectl get networkpolicies -n <namespace>

  Inspect any policies that might restrict traffic to or from the pod/service.

4. Check IP Address and Connectivity

Since the error indicates issues connecting to localhost (127.0.0.1), it's critical to confirm that the port-forwarding maps correctly:

• Connectivity Test:
  You might want to try connecting to the pod directly through its cluster IP to rule out local localhost issues. This involves finding the pod's IP and then attempting a connection:

  kubectl get pods -o wide -n <namespace>

  Look for the IP and try to ping or connect using a separate tool.

5. Reconfigure and Retry Port Forwarding

Sometimes, specifying the IP explicitly can resolve connection issues:

• Explicit Port Forwarding:

  kubectl port-forward <pod-name> 9010:9010 --address 0.0.0.0 -n <namespace>

  Using --address 0.0.0.0 allows connections from any IP, which can help if there's a local configuration issue blocking localhost.

6. Inspect Kubernetes Events

Kubernetes events can provide clues about what might be going wrong:

• Get Events:

  kubectl get events -n <namespace>

  Look for any errors or warnings that might explain connectivity issues.

7. Use kubectl exec to Test Connectivity

Test connectivity directly from within the Kubernetes cluster:

• Exec Into Pod:

  kubectl exec -it <pod-name> -- /bin/bash

  Once inside the pod, try to curl or telnet to the local port:

  curl -v http://localhost:9010

  or

  telnet localhost 9010

By following these steps systematically, you should be able to pinpoint where the connection issue lies and how to fix it. This process helps in understanding whether the problem is due to Kubernetes configurations, network policies, firewall settings, or something else.