kubernetes：无法从其他计算机访问NodePort-Java 学习之路

我正在尝试安装kubernetes . 检查在kubernetes上执行的容器时出现问题 . 我将服务类型设置为Nodeport，但我无法从运行容器的节点以外的节点进行访问 . 想要从其他计算机访问它，请告诉我它的不同之处 . 我尝试了externalIPs和LoadBarancer，但这是不可能的 .

环境

操作系统：Ubuntu 16.04 LTS
Kubernetes：1.8
Docker：17.09.0-ce
etcd：3.2.8
法兰绒：0.9.0

网络

物理：10.1.1.0/24
法兰绒：172.16.0.0/16
docker：192.168.0.0/16

机

主节点（2节点）：10.1.1.24,10.1.1.25
工作节点（2节点）：10.1.1.26,10.1.1.27

kubectl描述了svc nginx-cluster

Name:              nginx-cluster
Namespace:         default
Labels:            app=nginx-demo
Annotations:       <none>
Selector:          app=nginx-demo
Type:              ClusterIP
IP:                172.16.236.159
Port:              <unset>  8090/TCP
TargetPort:        80/TCP
Endpoints:         192.168.24.2:80
Session Affinity:  None
Events:            <none>

kubectl描述了svc nginx-service

Name:                     nginx-service
Namespace:                default
Labels:                   app=nginx-demo
Annotations:              <none>
Selector:                 app=nginx-demo
Type:                     NodePort
IP:                       172.16.199.69
Port:                     <unset>  8090/TCP
TargetPort:               80/TCP
NodePort:                 <unset>  31659/TCP
Endpoints:                192.168.24.2:80
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

正在运行的容器工作节点（10.1.1.27）

卷曲10.1.1.27:31659

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
...

Worker 节点（10.1.1.26）

卷曲10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

其他机器（10.1.1.XX）

卷曲10.1.1.27:31659

curl: (7) Failed to connect to 10.1.1.27 port 31659:Connection timed out.

kubectl获得pods -o wide

NAME                          READY     STATUS    RESTARTS   AGE       IP             NODE
echoserver-848b75d85-9fx7r    1/1       Running   3          6d        192.168.70.2   k8swrksv01
nginx-demo-85cc49574c-wv2b9   1/1       Running   3          6d        192.168.2.2    k8swrksv02

kubectl得到svc -o宽

NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE       SELECTOR
clusterip       ClusterIP   172.16.39.77     <none>        80/TCP           6d        run=echoserver
kubernetes      ClusterIP   172.16.0.1       <none>        443/TCP          10d       <none>
nginx-cluster   ClusterIP   172.16.236.159   <none>        8090/TCP         6d        app=nginx-demo
nginx-service   NodePort    172.16.199.69    <none>        8090:31659/TCP   6d        app=nginx-demo
nodeport        NodePort    172.16.38.40     <none>        80:31317/TCP     6d        run=echoserver

netstat -ntlp

tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1963/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2202/kube-proxy
tcp        0      0 127.0.0.1:4243          0.0.0.0:*               LISTEN      1758/dockerd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      996/sshd
tcp6       0      0 :::4194                 :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10250                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::31659                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::10255                :::*                    LISTEN      1963/kubelet
tcp6       0      0 :::10256                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::31317                :::*                    LISTEN      2202/kube-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      996/sshd

iptables-save命令

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-SEP-AZ4EGFEAU4RTSLJO - [0:0]
:KUBE-SEP-C7HQKKO26GIFOZZM - [0:0]
:KUBE-SEP-EWKNS2YCPXGJCXDC - [0:0]
:KUBE-SEP-LQVPUPFGW6BWATIP - [0:0]
:KUBE-SEP-OMMOFZ27GPKZ4OPA - [0:0]
:KUBE-SEP-UD3HOGDD5NDLNY74 - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-CQNAS6RSUGJF2C2D - [0:0]
:KUBE-SVC-GKN7Y2BSGW4NJTYL - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
:KUBE-SVC-XP7QDA4CRQ2QA33W - [0:0]
:KUBE-SVC-Z5P6OMNAEVLAQUTS - [0:0]
-A PREROUTING -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.2.0/24 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 192.168.0.0/16 -d 192.168.0.0/16 -j RETURN
-A POSTROUTING -s 192.168.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.2.0/24 -j RETURN
-A POSTROUTING ! -s 192.168.0.0/16 -d 192.168.0.0/16 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nginx-service:" -m tcp --dport 31659 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/nodeport:" -m tcp --dport 31317 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -s 192.168.70.2/32 -m comment --comment "default/clusterip:" -j KUBE-MARK-MASQ
-A KUBE-SEP-AZ4EGFEAU4RTSLJO -p tcp -m comment --comment "default/clusterip:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-C7HQKKO26GIFOZZM -s 192.168.70.2/32 -m comment --comment "default/nodeport:" -j KUBE-MARK-MASQ
-A KUBE-SEP-C7HQKKO26GIFOZZM -p tcp -m comment --comment "default/nodeport:" -m tcp -j DNAT --to-destination 192.168.70.2:8080
-A KUBE-SEP-EWKNS2YCPXGJCXDC -s 10.1.1.25/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-EWKNS2YCPXGJCXDC -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.25:6443
-A KUBE-SEP-LQVPUPFGW6BWATIP -s 192.168.2.2/32 -m comment --comment "default/nginx-service:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LQVPUPFGW6BWATIP -p tcp -m comment --comment "default/nginx-service:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -s 10.1.1.24/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OMMOFZ27GPKZ4OPA -p tcp -m comment --comment "default/kubernetes:https" -m recent --set --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -m tcp -j DNAT --to-destination 10.1.1.24:6443
-A KUBE-SEP-UD3HOGDD5NDLNY74 -s 192.168.2.2/32 -m comment --comment "default/nginx-cluster:" -j KUBE-MARK-MASQ
-A KUBE-SEP-UD3HOGDD5NDLNY74 -p tcp -m comment --comment "default/nginx-cluster:" -m tcp -j DNAT --to-destination 192.168.2.2:80
-A KUBE-SERVICES -d 172.16.236.159/32 -p tcp -m comment --comment "default/nginx-cluster: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-Z5P6OMNAEVLAQUTS
-A KUBE-SERVICES -d 172.16.199.69/32 -p tcp -m comment --comment "default/nginx-service: cluster IP" -m tcp --dport 8090 -j KUBE-SVC-GKN7Y2BSGW4NJTYL
-A KUBE-SERVICES -d 172.16.38.40/32 -p tcp -m comment --comment "default/nodeport: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XP7QDA4CRQ2QA33W
-A KUBE-SERVICES -d 172.16.39.77/32 -p tcp -m comment --comment "default/clusterip: cluster IP" -m tcp --dport 80 -j KUBE-SVC-CQNAS6RSUGJF2C2D
-A KUBE-SERVICES -d 172.16.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-CQNAS6RSUGJF2C2D -m comment --comment "default/clusterip:" -j KUBE-SEP-AZ4EGFEAU4RTSLJO
-A KUBE-SVC-GKN7Y2BSGW4NJTYL -m comment --comment "default/nginx-service:" -j KUBE-SEP-LQVPUPFGW6BWATIP
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-OMMOFZ27GPKZ4OPA --mask 255.255.255.255 --rsource -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m recent --rcheck --seconds 10800 --reap --name KUBE-SEP-EWKNS2YCPXGJCXDC --mask 255.255.255.255 --rsource -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-OMMOFZ27GPKZ4OPA
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment "default/kubernetes:https" -j KUBE-SEP-EWKNS2YCPXGJCXDC
-A KUBE-SVC-XP7QDA4CRQ2QA33W -m comment --comment "default/nodeport:" -j KUBE-SEP-C7HQKKO26GIFOZZM
-A KUBE-SVC-Z5P6OMNAEVLAQUTS -m comment --comment "default/nginx-cluster:" -j KUBE-SEP-UD3HOGDD5NDLNY74
COMMIT
*filter
:INPUT ACCEPT [40:14606]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42:6275]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:DOCKER-USER - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT

4 回答

3

它是由FORWARD链上的默认DROP引起的（而这又是由docker引起的） .

如果将 iptables -A FORWARD -j ACCEPT 规则添加到节点，则可以再次查看它 .

k8s问题在这里：https://github.com/kubernetes/kubernetes/issues/39823但是实际修复在这里https://github.com/kubernetes/kubernetes/pull/52569（预计在1.9） .

回复于 2024-05-10T20:08:20+08:00

附加信息 . 有什么东西应该阻止端口，但这是未知的......

pod运行节点

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE SERVICE
31317/tcp open  unknown
31659/tcp open  unknown

其他节点

nmap 10.1.1.27 -p31000-32000

Not shown: 999 closed ports
PORT      STATE    SERVICE
31317/tcp filtered unknown
31659/tcp filtered unknown

回复于 2024-05-10T20:08:20+08:00

-1

Farcaller是对的 . 我们在运行firewalld的同时也达到了同样的效果 .

在我们升级到k8s 1.9之前，我们添加了以下firewalld规则 . 该规则类似于k8s 1.9中由kube-proxy创建的规则

#!/bin/bash
# follows https://github.com/kubernetes/kubernetes/pull/52569 introduced in k8s 1.9
# required to support nodeport services routing from all nodes in the cluster when the firewall is turned on.
# KUBE-MARK-MASQ corresponds to kube-proxy --iptables-masquerade-bit=14, which is the default.
KUBE_MARK_MASQ="0x4000/0x4000"
firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD 3 -m mark --mark "$KUBE_MARK_MASQ" -j ACCEPT

回复于 2024-05-10T20:08:20+08:00

0

你可以先检查节点端口端口是否打开，

使用

netstat -ntlp

检查，如果是，可能是iptable或路由的一些问题，

如果不是，请检查防火墙或其他问题

祝好运

回复于 2024-05-10T20:08:20+08:00

kubernetes：无法从其他计算机访问NodePort

4 回答

相关问题