Pod无限期地处于挂起状态

过去几周我一直在使用6节点集群而没有问题 . 今天早些时候我们遇到了一个打开文件问题（https://github.com/kubernetes/kubernetes/pull/12443/files），我打了补丁并重新启动了kube-proxy .

从那以后，所有rc部署的pod到ALL BUT node-01卡在挂起状态，并且有日志消息说明原因 .

查看节点上的docker守护程序，pod中的容器实际上正在运行，并且删除rc将删除它们 . 根据kubelet和kube-apiserver，这似乎是州之间的某种回调问题 .

群集正在运行v1.0.3

这是一个州的例子

docker run --rm -it lachie83/kubectl:prod get pods --namespace=kube-system -o wide
NAME                READY     STATUS    RESTARTS   AGE       NODE
kube-dns-v8-i0yac   0/4       Pending   0          4s        10.1.1.35
kube-dns-v8-jti2e   0/4       Pending   0          4s        10.1.1.34

得到事件

Wed, 16 Sep 2015 06:25:42 +0000   Wed, 16 Sep 2015 06:25:42 +0000   1         kube-dns-v8                       ReplicationController                                                successfulCreate   {replication-controller }   Created pod: kube-dns-v8-i0yac
Wed, 16 Sep 2015 06:25:42 +0000   Wed, 16 Sep 2015 06:25:42 +0000   1         kube-dns-v8-i0yac                 Pod                                                                  scheduled          {scheduler }                Successfully assigned kube-dns-v8-i0yac to 10.1.1.35
Wed, 16 Sep 2015 06:25:42 +0000   Wed, 16 Sep 2015 06:25:42 +0000   1         kube-dns-v8-jti2e                 Pod                                                                  scheduled          {scheduler }                Successfully assigned kube-dns-v8-jti2e to 10.1.1.34
Wed, 16 Sep 2015 06:25:42 +0000   Wed, 16 Sep 2015 06:25:42 +0000   1         kube-dns-v8                       ReplicationController                                                successfulCreate   {replication-controller }   Created pod: kube-dns-v8-jti2e

调度程序日志

I0916 06:25:42.897814   10076 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-v8-jti2e", UID:"c1cafebe-5c3b-11e5-b3c4-020443b6797d", APIVersion:"v1", ResourceVersion:"670117", FieldPath:""}): reason: 'scheduled' Successfully assigned kube-dns-v8-jti2e to 10.1.1.34
I0916 06:25:42.904195   10076 event.go:203] Event(api.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-dns-v8-i0yac", UID:"c1cafc69-5c3b-11e5-b3c4-020443b6797d", APIVersion:"v1", ResourceVersion:"670118", FieldPath:""}): reason: 'scheduled' Successfully assigned kube-dns-v8-i0yac to 10.1.1.35

在pod创建期间拖尾kubelet日志文件

tail -f kubelet.kube-node-03.root.log.INFO.20150916-060744.10668
I0916 06:25:04.448916   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:25:24.449253   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:25:44.449522   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:26:04.449774   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:26:24.450400   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:26:44.450995   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:27:04.451501   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:27:24.451910   10668 config.go:253] Setting pods for source file : {[] 0 file}
I0916 06:27:44.452511   10668 config.go:253] Setting pods for source file : {[] 0 file}

kubelet过程

root@kube-node-03:/var/log/kubernetes# ps -ef | grep kubelet
root     10668     1  1 06:07 ?        00:00:13 /opt/bin/kubelet --address=10.1.1.34 --port=10250 --hostname_override=10.1.1.34 --api_servers=https://kube-master-01.sj.lithium.com:6443 --logtostderr=false --log_dir=/var/log/kubernetes --cluster_dns=10.1.2.53 --config=/etc/kubelet/conf --cluster_domain=prod-kube-sjc1-1.internal --v=4 --tls-cert-file=/etc/kubelet/certs/kubelet.pem --tls-private-key-file=/etc/kubelet/certs/kubelet-key.pem

节点列表

docker run --rm -it lachie83/kubectl:prod get nodes
NAME            LABELS                                             STATUS
10.1.1.30   kubernetes.io/hostname=10.1.1.30,name=node-1   Ready
10.1.1.32   kubernetes.io/hostname=10.1.1.32,name=node-2   Ready
10.1.1.34   kubernetes.io/hostname=10.1.1.34,name=node-3   Ready
10.1.1.35   kubernetes.io/hostname=10.1.1.35,name=node-4   Ready
10.1.1.42   kubernetes.io/hostname=10.1.1.42,name=node-5   Ready
10.1.1.43   kubernetes.io/hostname=10.1.1.43,name=node-6   Ready