我有一个带有以下(容器化)组件的Kubernetes onebox部署,全部运行为 --net=host
,其中kubelet作为特权Docker容器运行,kubernetes标志 --allow-privileged
设置为true .
gcr.io/google_containers/hyperkube-amd64:v1.7.9 "/bin/bash -c './hype" kubelet
gcr.io/google_containers/hyperkube-amd64:v1.7.9 "/bin/bash -c './hype" kube-proxy
gcr.io/google_containers/hyperkube-amd64:v1.7.9 "/bin/bash -c './hype" kube-scheduler
gcr.io/google_containers/hyperkube-amd64:v1.7.9 "/bin/bash -c './hype" kube-controller-manager
gcr.io/google_containers/hyperkube-amd64:v1.7.9 "/bin/bash -c './hype" kube-apiserver
quay.io/coreos/etcd:v3.1.0 "/usr/local/bin/etcd " etcd
最重要的是,我使用 kubectl create -f https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/resources/manifests/kube-addon-manager.yaml
启用了插件管理器,其中calico 2.6.1和kube-dns 1.14.5的默认yaml清单已安装到 /etc/kubernetes/addons/
. calico pod按预期提供两个节点(install-cni和calico-node) .
但是,kube-dns卡在ContainerCreating或ContainerCannotRun中,尝试启动Kubernetes暂停容器时出现以下错误:
{"log":"I1111 00:35:19.549318 1 manager.go:913] Added container: \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\" (aliases: [k8s_POD_kube-dns-v20-141138543-pmdww_kube-system_3173eef3-c678-11e7-ac4b-e41d2d59689e_0 1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54], namespace: \"docker\")\n","stream":"stderr","time":"2017-11-11T00:35:19.5526284Z"}
{"log":"I1111 00:35:19.549433 1 cni.go:291] About to add CNI network cni-loopback (type=loopback)\n","stream":"stderr","time":"2017-11-11T00:35:19.5526748Z"}
{"log":"I1111 00:35:19.549504 1 handler.go:325] Added event \u0026{/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54 2017-11-11 00:35:19.3931718 +0000 UTC containerCreation {\u003cnil\u003e}}\n","stream":"stderr","time":"2017-11-11T00:35:19.5527217Z"}
{"log":"I1111 00:35:19.551134 1 container.go:407] Start housekeeping for container \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\"\n","stream":"stderr","time":"2017-11-11T00:35:19.5527441Z"}
{"log":"E1111 00:35:19.555099 1 cni.go:294] Error adding network: failed to Statfs \"/proc/54226/ns/net\": no such file or directory\n","stream":"stderr","time":"2017-11-11T00:35:19.5553606Z"}
{"log":"E1111 00:35:19.555122 1 cni.go:237] Error while adding to cni lo network: failed to Statfs \"/proc/54226/ns/net\": no such file or directory\n","stream":"stderr","time":"2017-11-11T00:35:19.5553887Z"}
{"log":"I1111 00:35:19.600281 1 manager.go:970] Destroyed container: \"/kubepods/burstable/pod3173eef3-c678-11e7-ac4b-e41d2d59689e/1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54\" (aliases: [k8s_POD_kube-dns-v20-141138543-pmdww_kube-system_3173eef3-c678-11e7-ac4b-e41d2d59689e_0 1dd57d6f6c996d7abe061f6236fc8a0150cf6f95d16d5c3c462c9ed7158d3c54], namespace: \"docker\")\n","stream":"stderr","time":"2017-11-11T00:35:19.6005722Z"}
我看到\暂停容器不断出现只是为了退出一秒钟,带有一个无害的错误消息(这个是旧的,我停止了群集,所以它不会继续产生更多的容器):
ubuntu@r172-16-6-39:~$ docker ps -a | grep 216e39defa36
216e39defa36 gcr.io/google_containers/pause-amd64:3.0 "/pause" About an hour ago Exited (0) About an hour ago k8s_POD_kube-dns-v20-141138543-xvdmv_kube-system_0594732f-c688-11e7-9da5-e41d2d59689e_17
ubuntu@r172-16-6-39:~$ docker logs 216e39defa36
shutting down, got signal: Terminated
dir /proc/54226
在我的主机上不存在,我认为是CNI抱怨的原因 . 但Calico的暂停容器很好,运行相同的图像,因此必须要么只能在kube-dns的情况下写入,要么在Calico的情况下不能写入 . 我发现了一些类似SELinux-related error on Openshift的引用,但是我运行的是一个没有安装SELinux的裸Ubuntu 14.04 VM .
ubuntu@r172-16-6-39:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.4 LTS
Release: 14.04
Codename: trusty
ubuntu@r172-16-6-39:~$ setenforce
The program 'setenforce' is currently not installed. You can install it by typing:
sudo apt-get install selinux-utils
我的CNI conf也非常简单,由install-cni calico容器生成:
ubuntu@r172-16-6-39:~$ cat /etc/cni/net.d/10-calico.conf
{
"name": "k8s-pod-network",
"cniVersion": "0.1.0",
"type": "calico",
"log_level": "debug",
"datastore_type": "kubernetes",
"nodename": "172.16.6.39",
"mtu": 1500,
"ipam": {
"type": "host-local",
"subnet": "usePodCidr"
},
"policy": {
"type": "k8s",
"k8s_auth_token": "****"
},
"kubernetes": {
"k8s_api_root": "https://168.16.0.1:443",
"kubeconfig": "/etc/kubernetes/kubeconfig"
}
}
有没有人碰过类似的东西?