Pulling local docker image (possible dns issue)

Timothy Sutton

unread,

Jun 16, 2022, 3:36:29 PM6/16/22

to Knative Users

I am using k3s which relies on coredns to do dns look up. I am having a problem where I can not get kn to deploy one of my services on the local registry. It appears to imply lookup is failing, but I don't understand how. Below is a sample busybox run where dns is successful and the output I get from kn when I try creating said service.

[vagrant@master ~]$ kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup docker-registry
Server: 10.43.0.10
Address 1: 10.43.0.10 kube-dns.kube-system.svc.cluster.local

Name: docker-registry
Address 1: 10.43.172.19 docker-registry.default.svc.cluster.local
pod "busybox" deleted
[vagrant@master ~]$ kn service create knative-serving --image=docker-registry:5000/s3_receiverequest:0.0.1
Creating service 'knative-serving' in namespace 'default':

0.072s The Route is still working to reflect the latest desired specification.
0.138s ...
0.165s Configuration "knative-serving" is waiting for a Revision to become ready.
4.523s Revision "knative-serving-00001" failed with message: Unable to fetch image "docker-registry:5000/s3_receiverequest:0.0.1": failed to resolve image to digest: Get "https://docker-registry:5000/v2/": dial tcp: lookup docker-registry on 10.43.0.10:53: server misbehaving.
4.546s Configuration "knative-serving" does not have any ready Revision.
Error: RevisionFailed: Revision "knative-serving-00001" failed with message: Unable to fetch image "docker-registry:5000/s3_receiverequest:0.0.1": failed to resolve image to digest: Get "https://docker-registry:5000/v2/": dial tcp: lookup docker-registry on 10.43.0.10:53: server misbehaving

I did find this which is something similar to what I am doing https://zhimin-wen.medium.com/running-knative-on-on-premise-kubernetes-cluster-ibm-cloud-private-4227ee322aa0, but is fairly outdated going on 4 years old. This also implies that his kubedns couldnt resolve the address, but I have demonstrated that opendns can resolve it.

Timothy Sutton

unread,

Jun 16, 2022, 3:56:00 PM6/16/22

to Knative Users

Additional information that might be of interest is that I installed knative with operator, and when configuring network layer I chose kourier.

Evan Anderson

unread,

Jun 16, 2022, 7:25:01 PM6/16/22

to Timothy Sutton, Knative Users

Does your resis registry have a custom CA that's not part of the common CA roots? If so, you may need to customize the Knative Serving controller Deployment, as described here: https://knative.dev/docs/serving/tag-resolution/

Knative resolves image tags (mutable) to digests (immutable) to ensure that Revisions don't surprisingly change image partway through their lifecycle.

From: knativ...@googlegroups.com <knativ...@googlegroups.com> on behalf of Timothy Sutton <timothy...@saabusa.com>
Sent: Thursday, June 16, 2022 12:55:59 PM
To: Knative Users <knativ...@googlegroups.com>
Subject: [Suspected Spam] Re: Pulling local docker image (possible dns issue)

⚠ External Email

--
You received this message because you are subscribed to the Google Groups "Knative Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to knative-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/knative-users/45a40d65-8220-4409-887a-1e20f6e87d8dn%40googlegroups.com.

⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.

Timothy Sutton

unread,

Jun 17, 2022, 8:03:58 AM6/17/22

to Knative Users

I did have to do this and included both corporate firewall, and my ca cert for my docker registry. I can deploy a service from the outside world but it is claiming to have issues with the ip that is kubedns.

Timothy Sutton

unread,

Jun 17, 2022, 8:50:35 AM6/17/22

to Knative Users

To further back this not being an issue pulling with the CA certificate, I double checked the logs of the docker-registry, and nothing is even reaching the docker registry for the pull. I did try to add manually into opendns an entry following these instructions https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/, but no better resolution.

Timothy Sutton

unread,

Jun 17, 2022, 9:02:18 AM6/17/22

to Knative Users

Solved my issue I believe. I had to further edit my controller yaml (kubectl edit deployment controller --namespace knative-serving)