ubuntu - kubeadm kubedns error. could not access external network or other pods -

when using self hosted kubeadm in ubuntu, not access other pods , external network within k8s pod able access using regular docker containers.

i tried different types of pod network including calico, weave , flannel.

i followed debugging instructinos here without success, below logs.

$ kubectl exec -ti busybox -- nslookup kubernetes.default server:    10.96.0.10 address 1: 10.96.0.10  nslookup: can't resolve 'kubernetes.default'   $ kubectl exec busybox cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local options ndots:5   $ kubectl pods --namespace=kube-system -l k8s-app=kube-dns name                        ready     status    restarts   age kube-dns-2425271678-9zwtd   3/3       running   0          12m   $ kubectl logs --namespace=kube-system $(kubectl pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns i0823 16:02:58.407162       6 dns.go:48] version: 1.14.3-4-gee838f6 i0823 16:02:58.408957       6 server.go:70] using configuration read directory: /kube-dns-config period 10s i0823 16:02:58.409223       6 server.go:113] flag: --alsologtostderr="false" i0823 16:02:58.409248       6 server.go:113] flag: --config-dir="/kube-dns-config" i0823 16:02:58.409288       6 server.go:113] flag: --config-map="" i0823 16:02:58.409301       6 server.go:113] flag: --config-map-namespace="kube-system" i0823 16:02:58.409309       6 server.go:113] flag: --config-period="10s" i0823 16:02:58.409325       6 server.go:113] flag: --dns-bind-address="0.0.0.0" i0823 16:02:58.409333       6 server.go:113] flag: --dns-port="10053" i0823 16:02:58.409370       6 server.go:113] flag: --domain="cluster.local." i0823 16:02:58.409387       6 server.go:113] flag: --federations="" i0823 16:02:58.409401       6 server.go:113] flag: --healthz-port="8081" i0823 16:02:58.409411       6 server.go:113] flag: --initial-sync-timeout="1m0s" i0823 16:02:58.409434       6 server.go:113] flag: --kube-master-url="" i0823 16:02:58.409451       6 server.go:113] flag: --kubecfg-file="" i0823 16:02:58.409458       6 server.go:113] flag: --log-backtrace-at=":0" i0823 16:02:58.409470       6 server.go:113] flag: --log-dir="" i0823 16:02:58.409478       6 server.go:113] flag: --log-flush-frequency="5s" i0823 16:02:58.409489       6 server.go:113] flag: --logtostderr="true" i0823 16:02:58.409496       6 server.go:113] flag: --nameservers="" i0823 16:02:58.409521       6 server.go:113] flag: --stderrthreshold="2" i0823 16:02:58.409533       6 server.go:113] flag: --v="2" i0823 16:02:58.409544       6 server.go:113] flag: --version="false" i0823 16:02:58.409559       6 server.go:113] flag: --vmodule="" i0823 16:02:58.409728       6 server.go:176] starting skydns server (0.0.0.0:10053) i0823 16:02:58.467505       6 server.go:198] skydns metrics enabled (/metrics:10055) i0823 16:02:58.467640       6 dns.go:147] starting endpointscontroller i0823 16:02:58.467810       6 dns.go:150] starting servicecontroller i0823 16:02:58.557166       6 logs.go:41] skydns: ready queries on cluster.local. tcp://0.0.0.0:10053 [rcache 0] i0823 16:02:58.557335       6 logs.go:41] skydns: ready queries on cluster.local. udp://0.0.0.0:10053 [rcache 0] i0823 16:02:58.968454       6 dns.go:174] waiting services , endpoints initialized apiserver... i0823 16:02:59.468406       6 dns.go:171] initialized services , endpoints apiserver i0823 16:02:59.468698       6 server.go:129] setting healthz handler (/readiness) i0823 16:02:59.469064       6 server.go:134] setting cache handler (/cache) i0823 16:02:59.469305       6 server.go:120] status http port 8081   $ kubectl logs --namespace=kube-system $(kubectl pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq i0823 16:02:59.445525      11 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000} i0823 16:02:59.445741      11 nanny.go:86] starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] i0823 16:02:59.820424      11 nanny.go:108] dnsmasq[38]: started, version 2.76 cachesize 1000 i0823 16:02:59.820546      11 nanny.go:108] dnsmasq[38]: compile time options: ipv6 gnu-getopt no-dbus no-i18n no-idn dhcp dhcpv6 no-lua tftp no-conntrack ipset auth no-dnssec loop-detect inotify i0823 16:02:59.820596      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain ip6.arpa  i0823 16:02:59.820623      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain in-addr.arpa  i0823 16:02:59.820659      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain cluster.local  i0823 16:02:59.820736      11 nanny.go:108] dnsmasq[38]: reading /etc/resolv.conf i0823 16:02:59.820762      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain ip6.arpa  i0823 16:02:59.820788      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain in-addr.arpa  i0823 16:02:59.820825      11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 domain cluster.local  i0823 16:02:59.820850      11 nanny.go:108] dnsmasq[38]: using nameserver 8.8.8.8#53 i0823 16:02:59.820928      11 nanny.go:108] dnsmasq[38]: read /etc/hosts - 7 addresses i0823 16:02:59.821193      11 nanny.go:111]  w0823 16:02:59.821212      11 nanny.go:112] got eof stdout  $ kubectl logs --namespace=kube-system $(kubectl pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar error: logging before flag.parse: i0823 16:03:00.789793      26 main.go:48] version v1.14.3-4-gee838f6 error: logging before flag.parse: i0823 16:03:00.790052      26 server.go:45] starting server (options {dnsmasqport:53 dnsmasqaddr:127.0.0.1 dnsmasqpollintervalms:5000 probes:[{label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1}] prometheusaddr:0.0.0.0 prometheusport:10054 prometheuspath:/metrics prometheusnamespace:kubedns}) error: logging before flag.parse: i0823 16:03:00.790121      26 dnsprobe.go:75] starting dnsprobe {label:kubedns server:127.0.0.1:10053 name:kubernetes.default.svc.cluster.local. interval:5s type:1} error: logging before flag.parse: i0823 16:03:00.790419      26 dnsprobe.go:75] starting dnsprobe {label:dnsmasq server:127.0.0.1:53 name:kubernetes.default.svc.cluster.local. interval:5s type:1} 

below etc/resolv.conf master.

$ cat /etc/resolv.conf # dynamic resolv.conf(5) file glibc resolver(3) generated resolvconf(8) #     not edit file hand -- changes overwritten nameserver 8.8.8.8  $ kubeadm version kubeadm version: &version.info{major:"1", minor:"7", gitversion:"v1.7.3", gitcommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", gittreestate:"clean", builddate:"2017-08-03t06:43:48z", goversion:"go1.8.3", compiler:"gc", platform:"linux/amd64"} 

below etc/resolv.conf worker node pod running

# dynamic resolv.conf(5) file glibc resolver(3) generated resolvconf(8) #     not edit file hand -- changes overwritten nameserver 8.8.4.4 nameserver 8.8.8. 

here output of sudo iptables -n -l

chain input (policy accept) target     prot opt source               destination          cali-input   --  0.0.0.0/0            0.0.0.0/0            /* cali:cz_u1iqiximmkd4c */ kube-services   --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ kube-firewall   --  0.0.0.0/0            0.0.0.0/0             chain forward (policy drop) target     prot opt source               destination          cali-forward   --  0.0.0.0/0            0.0.0.0/0            /* cali:wuhhoiayhpho9mso */ docker-user   --  0.0.0.0/0            0.0.0.0/0            docker-isolation   --  0.0.0.0/0            0.0.0.0/0            accept      --  0.0.0.0/0            0.0.0.0/0            ctstate related,established docker      --  0.0.0.0/0            0.0.0.0/0            accept      --  0.0.0.0/0            0.0.0.0/0            accept      --  0.0.0.0/0            0.0.0.0/0            weave-npc   --  0.0.0.0/0            0.0.0.0/0            nflog       --  0.0.0.0/0            0.0.0.0/0            state new nflog-group 86 drop        --  0.0.0.0/0            0.0.0.0/0            accept      --  0.0.0.0/0            0.0.0.0/0            accept      --  0.0.0.0/0            0.0.0.0/0            ctstate related,established  chain output (policy accept) target     prot opt source               destination          cali-output   --  0.0.0.0/0            0.0.0.0/0            /* cali:tvnhkvao15huipy0 */ kube-services   --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */ kube-firewall   --  0.0.0.0/0            0.0.0.0/0             chain docker (1 references) target     prot opt source               destination           chain docker-isolation (1 references) target     prot opt source               destination          return      --  0.0.0.0/0            0.0.0.0/0             chain docker-user (1 references) target     prot opt source               destination          return      --  0.0.0.0/0            0.0.0.0/0             chain kube-firewall (2 references) target     prot opt source               destination          drop        --  0.0.0.0/0            0.0.0.0/0            /* kubernetes firewall dropping marked packets */ mark match 0x8000/0x8000  chain kube-services (2 references) target     prot opt source               destination          reject     tcp  --  0.0.0.0/0            10.96.252.131        /* default/redis-cache-service:redis has no endpoints */ tcp dpt:6379 reject-with icmp-port-unreachable reject     tcp  --  0.0.0.0/0            10.96.252.131        /* default/redis-cache-service:cluster has no endpoints */ tcp dpt:16379 reject-with icmp-port-unreachable reject     tcp  --  0.0.0.0/0            10.105.180.126       /* default/redis-pubsub-service:redis has no endpoints */ tcp dpt:6379 reject-with icmp-port-unreachable reject     tcp  --  0.0.0.0/0            10.105.180.126       /* default/redis-pubsub-service:cluster has no endpoints */ tcp dpt:16379 reject-with icmp-port-unreachable  chain weave-npc (1 references) target     prot opt source               destination          accept      --  0.0.0.0/0            0.0.0.0/0            state related,established accept      --  0.0.0.0/0            224.0.0.0/4          weave-npc-default   --  0.0.0.0/0            0.0.0.0/0            state new weave-npc-ingress   --  0.0.0.0/0            0.0.0.0/0            state new accept      --  0.0.0.0/0            0.0.0.0/0            ! match-set weave-local-pods dst  chain weave-npc-default (1 references) target     prot opt source               destination          accept      --  0.0.0.0/0            0.0.0.0/0            match-set weave-k?z;25^m}|1s7p3|h9i;*;mhg dst accept      --  0.0.0.0/0            0.0.0.0/0            match-set weave-iuzcey(5dexbzgrfs8szo]+@p dst accept      --  0.0.0.0/0            0.0.0.0/0            match-set weave-4vtqmi+kx/2]jd%_c0s%tho%v dst  chain weave-npc-ingress (1 references) target     prot opt source               destination           chain cali-forward (1 references) target     prot opt source               destination          cali-from-wl-dispatch   --  0.0.0.0/0            0.0.0.0/0            /* cali:x3vb2lgcbrfkyquc */ cali-to-wl-dispatch   --  0.0.0.0/0            0.0.0.0/0            /* cali:utj9fnhbnfbyqmvu */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:tt19hcsda5yigssw */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:9lzffcvnpc5_myxm */ mark        --  0.0.0.0/0            0.0.0.0/0            /* cali:7aoflloqcm5j36rm */ mark , 0xf1ffffff cali-from-host-endpoint   --  0.0.0.0/0            0.0.0.0/0            /* cali:qm1_josl7tl76az7 */ mark match 0x0/0x1000000 cali-to-host-endpoint   --  0.0.0.0/0            0.0.0.0/0            /* cali:c1qsog3bk0aykjao */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:dmfipamzcisqzcvo */ /* host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000  chain cali-input (1 references) target     prot opt source               destination          accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:i7okjzps8vxajb3n */ mark match 0x1000000/0x1000000 drop       4    --  0.0.0.0/0            0.0.0.0/0            /* cali:p8wwvr6qydju36aq */ /* drop ipip packets non-calico hosts */ ! match-set cali4-all-hosts src cali-wl-to-host   --  0.0.0.0/0            0.0.0.0/0           [goto]  /* cali:qzt4ptg57_76ngng */ mark        --  0.0.0.0/0            0.0.0.0/0            /* cali:v0veitpvpl5h1xwi */ mark , 0xf0ffffff cali-from-host-endpoint   --  0.0.0.0/0            0.0.0.0/0            /* cali:3r1g0cpvsoblkzvr */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:efxx-pqd4s60wsdl */ /* host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000  chain cali-output (1 references) target     prot opt source               destination          accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:yqssjisrchjfbxai */ mark match 0x1000000/0x1000000 return      --  0.0.0.0/0            0.0.0.0/0            /* cali:krjbsksbcfbykcew */ mark        --  0.0.0.0/0            0.0.0.0/0            /* cali:3vkaqbcyuuw5ks_j */ mark , 0xf0ffffff cali-to-host-endpoint   --  0.0.0.0/0            0.0.0.0/0            /* cali:z1mbcsh1xhm6qq0k */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:n0jywt2rfbedkw3l */ /* host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000  chain cali-failsafe-in (0 references) target     prot opt source               destination          accept     tcp  --  0.0.0.0/0            0.0.0.0/0            /* cali:wwfqm43tju7wwnfz */ multiport dports 22 accept     udp  --  0.0.0.0/0            0.0.0.0/0            /* cali:lwnv--r8mjeuyacw */ multiport dports 68  chain cali-failsafe-out (0 references) target     prot opt source               destination          accept     tcp  --  0.0.0.0/0            0.0.0.0/0            /* cali:73bzkoydfopfwc2t */ multiport dports 2379 accept     tcp  --  0.0.0.0/0            0.0.0.0/0            /* cali:qmfuwo6o-d9yopnm */ multiport dports 2380 accept     tcp  --  0.0.0.0/0            0.0.0.0/0            /* cali:kup7qkrsdmfgx0ul */ multiport dports 4001 accept     tcp  --  0.0.0.0/0            0.0.0.0/0            /* cali:xyyr5peqdf_pqfkv */ multiport dports 7001 accept     udp  --  0.0.0.0/0            0.0.0.0/0            /* cali:nbwbvu4otudvy60q */ multiport dports 53 accept     udp  --  0.0.0.0/0            0.0.0.0/0            /* cali:uxfu5cdk5en6dt3y */ multiport dports 67  chain cali-from-host-endpoint (2 references) target     prot opt source               destination           chain cali-from-wl-dispatch (2 references) target     prot opt source               destination          drop        --  0.0.0.0/0            0.0.0.0/0            /* cali:ztj6p0tigyvgz-md */ /* unknown interface */  chain cali-to-host-endpoint (2 references) target     prot opt source               destination           chain cali-to-wl-dispatch (1 references) target     prot opt source               destination          drop        --  0.0.0.0/0            0.0.0.0/0            /* cali:7knphb1nnhw80nio */ /* unknown interface */  chain cali-wl-to-host (1 references) target     prot opt source               destination          accept     udp  --  0.0.0.0/0            0.0.0.0/0            /* cali:aeompplgak2s0lxs */ multiport sports 68 multiport dports 67 accept     udp  --  0.0.0.0/0            0.0.0.0/0            /* cali:szr8ejpiuxtfms8b */ multiport dports 53 cali-from-wl-dispatch   --  0.0.0.0/0            0.0.0.0/0            /* cali:memlbcdco0fefcrw */ accept      --  0.0.0.0/0            0.0.0.0/0            /* cali:lzboxhdolr3ok4r3 */ /* configured defaultendpointtohostaction */ 

wiki