Docker DNS Service Discovery "no such host"

[SIMULATAN]

I'm running two mimir nodes in a stack and want to use DNS Service Discovery to connect the two nodes together.
Therefore, I configured dns+tasks.monitoring_mimir as a join_member in mimir.yml.
Unfortunately, when it tries to do the DNS query, this is printed in the console:

caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=tasks.monitoring_mimir err="lookup tasks.monitoring_mimir on 127.0.0.11:53: no such host"
caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=tasks.monitoring_mimir

However, when running dig tasks.monitoring_mimir, it works just as expected!

(both when doing a docker exec and using that as the entrypoint)

Also, when looking at the dashboard, I can see that the member for some reason uses the wrong interface?
That one is the bridge interface made when publishing ports.

I tried setting the name manually to the correct interface name (eth0) by using instance_interface_names, but it's no use.

Logfile
mimir.yml
docker-compose.yml

I tried multiple combinations of using instance_interface_names, instance_addr, service discovery prefixes, network drivers, nodes, architectures, replication modes and more, but nothing worked, it keeps erroring with the above logs.

Another observation was that it tries to connect to some IPs in the 192.168.x.x subnet, even though I didn't add that range anywhere and the DNS query doesn't return those IPs.

[pracucci]

caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=tasks.monitoring_mimir err="lookup tasks.monitoring_mimir on 127.0.0.11:53: no such host"
caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=tasks.monitoring_mimir

I'm not sure what's happening here. The configuration you shared looks correct to me. Since I'm not familiar with your setup, let me ask a dumb question: can you confirm the host tasks.monitoring_mimir exists before Mimir process starts, right?

Also, when looking at the dashboard, I can see that the member for some reason uses the wrong interface?
That one is the bridge interface made when publishing ports.

The IP address exposed by memberlist can be configured setting -memberlist.advertise-addr or its respective YAML config option (doc). The default is using the address from the same interface where the memberlist listening port was binded, which is configured by -memberlist.bind-addr.

[marvinnitz18]

i seem to have the exact same problem, but eth1 would be wrong for me, eth0 would be correct:

i will try to implement the workaround @SIMULATAN has provided now

[marvinnitz18]

for everyone using docker swarm, this i my solution i put the workaround into a wrapper.sh which i mounted into the container and defined it as the entrypoint in the stackfile:
version: "3.7" services: mimir1: image: grafana/mimir:2.11.0 entrypoint: ["/bin/wrapper.sh"]

wrapper.sh:
#!/bin/sh IP="$(ip addr show eth0 | grep inet | tr -s ' ' | cut -f 3 -d ' ' | cut -f 1 -d '/')" exec /bin/mimir -config.file=/etc/mimir.yaml -memberlist.advertise-addr=$IP

[balzreber]

I have the exact same problem. I have a docker swarm and want to run 3 instances of mimir. So I somehow have to set the advertisement address dynamically. I tried the entripoint/wrapper hack listed here. Unfortunately, without success. I gladly share logs configs and so on, but one question first. Does the container that gets the wrapper mapped need to have /bin/sh installed itself? I assume so? As far as I can see, the current Mimir image (2.16) is bare bones and has no shell installed whatsoever. How do you get the wrapper to work? Any help is much appreciated. Thanks

[davbgallo]

@balzreber have you found a solution? I have the identical issue in docker swarm.

[balzreber]

@davbgallo Unfortunately I haven't. And I have moved on to a full on k8s cluster. So I didn't do any more research into this. I'm sorry.

[ksingh-scogo]

@SIMULATAN @pracucci encountering exact same issue with Mimir running on K8s

ts=2023-12-24T12:18:35.380649131Z caller=log.go:245 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
ts=2023-12-24T12:18:35.381232016Z caller=tcp_transport.go:405 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.244.2.176 advertisePort=7946
ts=2023-12-24T12:18:35.381724232Z caller=tcp_transport.go:405 level=debug component="memberlist TCPTransport" msg=FinalAdvertiseAddr advertiseAddr=10.244.2.176 advertisePort=7946

ts=2023-12-24T12:18:35.38402796Z caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.grafana.svc.cluster.local. err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

ts=2023-12-24T12:18:35.384052997Z caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=mimir-gossip-ring.grafana.svc.cluster.local.

ts=2023-12-24T12:18:35.384066923Z caller=memberlist_client.go:541 level=info msg="memberlist fast-join starting" nodes_found=0 to_join=0
ts=2023-12-24T12:18:35.384078274Z caller=memberlist_client.go:561 level=warn msg="memberlist fast-join finished" joined_nodes=0 elapsed_time=2.201117ms

ts=2023-12-24T12:18:35.384090126Z caller=memberlist_client.go:573 level=info phase=startup msg="joining memberlist cluster" join_members=dns+mimir-gossip-ring.grafana.svc.cluster.local.:7946

ts=2023-12-24T12:18:35.385085489Z caller=resolver.go:87 level=error msg="failed to lookup IP addresses" host=mimir-gossip-ring.grafana.svc.cluster.local. err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

ts=2023-12-24T12:18:35.385100887Z caller=resolver.go:134 level=warn msg="IP address lookup yielded no results. No host found or no addresses found" host=mimir-gossip-ring.grafana.svc.cluster.local.

ts=2023-12-24T12:18:35.3851131Z caller=memberlist_client.go:607 level=warn phase=startup msg="joining memberlist cluster" attempts=1 max_attempts=10 err="found no nodes to join"

[ksingh-scogo]

i am using mimir helm

[pracucci]

From the logs we can see:

err="lookup mimir-gossip-ring.grafana.svc.cluster.local. on 10.0.0.10:53: no such host"

I suggest to run few sanity checks to begin with:

1. Does the mimir-gossip-ring K8S service exists in the grafana namespace? It should exist, as it's expected to be created as part of the Helm chart.
2. If you get a shell in one of the Mimir pods, can they resolve the service host mimir-gossip-ring.grafana.svc.cluster.local?