本文共 18284 字，大约阅读时间需要 60 分钟。
参考文档:
感谢作者的无私分享。集群环境已搭建成功跑起来。文章是部署过程中遇到的错误和详细操作步骤记录。如有需要对比参考，请按照顺序阅读和测试。将二进制文件拷贝到所有 master 节点：
wget https://dl.k8s.io/v1.12.3/kubernetes-server-linux-amd64.tar.gztar -xzvf kubernetes-server-linux-amd64.tar.gzcd kubernetestar -xzvf kubernetes-src.tar.gz
将二进制文件拷贝到所有 master 节点：
[root@k8s-master bin]# pwd/root/k8s/kubernetes/kubernetes/server/bin[root@k8s-master bin]#[root@k8s-master bin]# cp kube-apiserver kube-controller-manager kube-scheduler /opt/k8s/bin[root@k8s-master bin]# scp kube-apiserver kube-controller-manager kube-scheduler root@k8s-node1:/opt/k8s/binkube-apiserver 100% 177MB 99.8MB/s 00:01 kube-controller-manager 100% 147MB 104.6MB/s 00:01 kube-scheduler 100% 53MB 96.7MB/s 00:00 [root@k8s-master bin]# scp kube-apiserver kube-controller-manager kube-scheduler root@k8s-node2:/opt/k8s/binkube-apiserver 100% 177MB 88.4MB/s 00:02 kube-controller-manager 100% 147MB 101.9MB/s 00:01 kube-scheduler 100% 53MB 87.4MB/s 00:00 [root@k8s-master bin]#
配置下权限
[root@k8s-master bin]# chmod +x /opt/k8s/bin/*[root@k8s-master bin]# chown k8s /opt/k8s/bin/*[root@k8s-master bin]# ssh k8s-node1 "chmod +x /opt/k8s/bin/* && chown k8s /opt/k8s/bin/*"[root@k8s-master bin]# ssh k8s-node2 "chmod +x /opt/k8s/bin/* && chown k8s /opt/k8s/bin/*"[root@k8s-master bin]#
7.1
创建kubernetes证书和私钥创建证书签名请求
[root@k8s-master ~]# source /opt/k8s/bin/environment.sh [root@k8s-master ~]# echo ${MASTER_VIP}192.168.1.94[root@k8s-master ~]# echo ${CLUSTER_KUBERNETES_SVC_IP}10.254.0.1[root@k8s-master ~]#[root@k8s-master ~]# cat kubernetes-csr.json{"CN": "kubernetes","hosts": ["127.0.0.1","192.168.1.92","192.168.1.93","192.168.1.95","${MASTER_VIP}","${CLUSTER_KUBERNETES_SVC_IP}","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "SZ","L": "SZ","O": "k8s","OU": "4Paradigm"}]} hosts 字段指定授权使用该证书的 IP 或域名列表，这里列出了 VIP 、apiserver 节
点 IP、kubernetes 服务 IP 和域名；域名最后字符不能是 . (如不能为kubernetes.default.svc.cluster.local. )，否则解析时失败，提示： x509:cannot parse dnsName "kubernetes.default.svc.cluster.local." ；
如果使用非 cluster.local 域名，如 opsnull.com ，则需要修改域名列表中的最后两个域名
为： kubernetes.default.svc.opsnull 、 kubernetes.default.svc.opsnull.comkubernetes 服务 IP 是 apiserver 自动创建的，一般是 --service-cluster-iprange参数指定的网段的第一个IP
注意："${MASTER_VIP}","${CLUSTER_KUBERNETES_SVC_IP}", 这两个必须全部改成真实的ip地址，不然后面容易出问题，ip地址参考见下:
[root@k8s-master apiserver]# echo ${MASTER_VIP}192.168.1.94[root@k8s-master apiserver]# echo ${CLUSTER_KUBERNETES_SVC_IP}10.254.0.1 7.2
生成证书和私钥[root@k8s-master apiserver]# cfssl gencert -ca=/etc/kubernetes/cert/ca.pem -ca-key=/etc/kubernetes/cert/ca-key.pem -config=/etc/kubernetes/cert/ca-config.json -profile=kubernetes kubernetes-csr.json | cfssljson -bare kubernetes2018/08/23 09:49:46 [INFO] generate received request2018/08/23 09:49:46 [INFO] received CSR2018/08/23 09:49:46 [INFO] generating key: rsa-20482018/08/23 09:49:46 [INFO] encoded CSR2018/08/23 09:49:46 [INFO] signed certificate with serial number 4077207747297659816588502509943702666856639250942018/08/23 09:49:46 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 ("Information Requirements").[root@k8s-master apiserver]# lskubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem[root@k8s-master apiserver] 7.3
证书拷到各个master节点，并确保k8s用户有权限[root@k8s-master apiserver]# cp kubernetes* /etc/kubernetes/cert/[root@k8s-master apiserver]# scp kubernetes* root@k8s-node1:/etc/kubernetes/cert/kubernetes.csr 100% 1301 1.1MB/s 00:00 kubernetes-csr.json 100% 404 452.9KB/s 00:00 kubernetes-key.pem 100% 1675 1.8MB/s 00:00 kubernetes.pem 100% 1663 2.6MB/s 00:00 [root@k8s-master apiserver]# scp kubernetes* root@k8s-node2:/etc/kubernetes/cert/kubernetes.csr 100% 1301 1.6MB/s 00:00 kubernetes-csr.json 100% 404 657.4KB/s 00:00 kubernetes-key.pem 100% 1675 2.5MB/s 00:00 kubernetes.pem
三个节点都要查看权限
[root@k8s-master apiserver]# chown -R k8s /etc/kubernetes/cert[root@k8s-master apiserver]# chmod +x -R /etc/kubernetes/cert[root@k8s-master apiserver]# ll /etc/kubernetes/cert/总用量 36-rwxr-xr-x 1 k8s root 292 8月 16 16:05 ca-config.json-rwxr-xr-x 1 k8s root 993 8月 16 16:05 ca.csr-rwxr-xr-x 1 k8s root 201 8月 16 16:05 ca-csr.json-rwx--x--x 1 k8s root 1675 8月 16 16:05 ca-key.pem-rwxr-xr-x 1 k8s root 1338 8月 16 16:05 ca.pem-rwxr-xr-x 1 k8s root 1301 8月 23 09:52 kubernetes.csr-rwxr-xr-x 1 k8s root 404 8月 23 09:52 kubernetes-csr.json-rwx--x--x 1 k8s root 1675 8月 23 09:52 kubernetes-key.pem-rwxr-xr-x 1 k8s root 1663 8月 23 09:52 kubernetes.pem[root@k8s-master apiserver]#
7.4
配置加密配置文件[root@k8s-master apiserver]# cat encryption-config.yamlkind: EncryptionConfigapiVersion: v1resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: ${ENCRYPTION_KEY} - identity: {}[root@k8s-master apiserver]# 将加密配置文件拷贝到 master 节点的 /etc/kubernetes 目录下：
[root@k8s-master apiserver]# cp encryption-config.yaml /etc/kubernetes/[root@k8s-master apiserver]# scp encryption-config.yaml root@k8s-node1:/etc/kubernetes/encryption-config.yaml 100% 213 339.8KB/s 00:00 [root@k8s-master apiserver]# scp encryption-config.yaml root@k8s-node2:/etc/kubernetes/encryption-config.yaml 100% 213 274.7KB/s 00:00 [root@k8s-master apiserver]#
7.5
创建 kube-apiserver systemd unit 模板文件[root@k8s-master kubernetes]# source /opt/k8s/bin/environment.sh[root@k8s-master kubernetes]# cat kube-apiserver.service.template[Unit]Description=Kubernetes API ServerDocumentation=https://github.com/GoogleCloudPlatform/kubernetesAfter=network.target[Service]ExecStart=/opt/k8s/bin/kube-apiserver \--enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRest riction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \--anonymous-auth=false \--experimental-encryption-provider-config=/etc/kubernetes/encryption-config.yaml \--advertise-address=##NODE_IP## \--bind-address=##NODE_IP## \--insecure-port=0 \--authorization-mode=Node,RBAC \--runtime-config=api/all \--enable-bootstrap-token-auth \--service-cluster-ip-range=${SERVICE_CIDR} \--service-node-port-range=${NODE_PORT_RANGE} \--tls-cert-file=/etc/kubernetes/cert/kubernetes.pem \--tls-private-key-file=/etc/kubernetes/cert/kubernetes-key.pem \--client-ca-file=/etc/kubernetes/cert/ca.pem \--kubelet-client-certificate=/etc/kubernetes/cert/kubernetes.pem \--kubelet-client-key=/etc/kubernetes/cert/kubernetes-key.pem \--service-account-key-file=/etc/kubernetes/cert/ca-key.pem \--etcd-cafile=/etc/kubernetes/cert/ca.pem \--etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \--etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem \--etcd-servers=${ETCD_ENDPOINTS} \--enable-swagger-ui=true \--allow-privileged=true \--apiserver-count=3 \--audit-log-maxage=30 \--audit-log-maxbackup=3 \--audit-log-maxsize=100 \--audit-log-path=/var/log/kube-apiserver-audit.log \--event-ttl=1h \--alsologtostderr=true \--logtostderr=false \--log-dir=/var/log/kubernetes \--v=2Restart=on-failureRestartSec=5Type=notifyUser=k8sLimitNOFILE=65536[Install]WantedBy=multi-user.target[root@k8s-master kubernetes]# ##--experimental-encryption-provider-config ：启用加密特性；
##--authorization-mode=Node,RBAC ： 开启 Node 和 RBAC 授权模式，拒绝未授权的请求；##--enable-admission-plugins ：启用 ServiceAccount 和NodeRestriction ；##--service-account-key-file ：签名 ServiceAccount Token 的公钥文件，kube-controller-manager 的 --service-account-private-key-file 指定私钥文件，两者配对使用；##--tls--file ：指定 apiserver 使用的证书、私钥和 CA 文件。 ##--client-cafile用于验证 client (kue-controller-manager、kube-scheduler、kubelet、kubeproxy等)请求所带的证书；##--kubelet-client-certificate 、 --kubelet-client-key ：如果指定，则使用 https 访问 kubelet APIs；需要为证书对应的用户(上面 kubernetes.pem 证书的用户为 kubernetes) 用户定义 RBAC 规则，否则访问 kubelet API 时提示未授权##--etcd-cafile=/etc/kubernetes/cert/ca.pem \ --etcd-certfile=/etc/kubernetes/cert/kubernetes.pem \ --etcd-keyfile=/etc/kubernetes/cert/kubernetes-key.pem 指定访问etcd使用的证书、私钥和 CA 文件 --etcd-servers=${ETCD_ENDPOINTS} etcd集群终端地址##--bind-address ： 不能为 127.0.0.1 ，否则外界不能访问它的安全端口6443；##--insecure-port=0 ：关闭监听非安全端口(8080)；##--service-cluster-ip-range ： 指定 Service Cluster IP 地址段；##--service-node-port-range ： 指定 NodePort 的端口范围；##--runtime-config=api/all=true ： 启用所有版本的 APIs，如autoscaling/v2alpha1；##--enable-bootstrap-token-auth ：启用 kubelet bootstrap 的 token 认证；##--apiserver-count=3 ：指定集群运行模式，多台 kube-apiserver 会通过 leader选举产生一个工作节点，其它节点处于阻塞状态；##User=k8s ：使用 k8s 账户运行；为各节点创建和分发 kube-apiserver systemd unit
文件替换模板文件中的变量，为各节点创建 systemd unit 文件：分发
[root@k8s-master apiserver]# cp kube-apiserver.service.template /etc/systemd/system/kube-apiserver.service[root@k8s-master apiserver]# scp kube-apiserver.service.template root@k8s-node1:/etc/systemd/system/kube-apiserver.servicekube-apiserver.service.template 100% 1652 2.2MB/s 00:00 [root@k8s-master apiserver]# scp kube-apiserver.service.template root@k8s-node2:/etc/systemd/system/kube-apiserver.servicekube-apiserver.service.template 100% 1652 2.0MB/s 00:00 [root@k8s-master apiserver]#
修改
[root@k8s-master apiserver]# sed -i -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[0]}/" /etc/systemd/system/kube-apiserver.service[root@k8s-master apiserver]# ssh root@k8s-node1Last login: Thu Aug 23 10:44:39 2018 from k8s-master[root@k8s-node1 ~]# source /opt/k8s/bin/environment.sh[root@k8s-node1 ~]# sed -i -e "s/##NODE_NAME##/${NODE_NAMES[i]}/" -e "s/##NODE_IP##/${NODE_IPS[1]}/" /etc/systemd/system/kube-apiserver.service[root@k8s-node1 ~]# exit登出Connection to k8s-node1 closed.[root@k8s-master apiserver]# ssh root@k8s-node2Last login: Thu Aug 23 09:15:27 2018 from 192.168.1.143[root@k8s-node2 ~]# source /opt/k8s/bin/environment.sh[root@k8s-node2 ~]# sed -i -e "s/##NODE_NAME##/${NODE_NAMES}/" -e "s/##NODE_IP##/${NODE_IPS[2]}/" /etc/systemd/system/kube-apiserver.service[root@k8s-node2 ~]# 检查下权限
[root@k8s-master apiserver]# ll /etc/systemd/system/kube-apiserver.service-rw-r--r-- 1 root root 1653 8月 23 10:43 /etc/systemd/system/kube-apiserver.service[root@k8s-master apiserver]#[root@k8s-master apiserver]# chmod +x /etc/systemd/system/kube-apiserver.service[root@k8s-master apiserver]# ssh root@k8s-node1 "chmod +x /etc/systemd/system/kube-apiserver.service"[root@k8s-master apiserver]# ssh root@k8s-node2 "chmod +x /etc/systemd/system/kube-apiserver.service"
启动前必须先创建日志目录 三节点都操作
[root@k8s-master apiserver]# mkdir -p /var/log/kubernetes && chown -R k8s /var/log/kubernetes
启动 kube-apiserver 服务
把/etc/systemd/system/kube-apiserver.service的变量改成真实的数据，参考见下：
[root@k8s-master bin]# echo ${SERVICE_CIDR}10.254.0.0/16[root@k8s-master bin]# echo ${NODE_PORT_RANGE}8400-9000[root@k8s-master bin]# echo ${ETCD_ENDPOINTS}https://192.168.1.92:2379,https://192.168.1.93:2379,https://192.168.1.95:2379[root@k8s-master bin]# [root@k8s-master bin]# systemctl daemon-reload && systemctl enable kube-apiserver && systemctl restart kube-apiserver
报错
Aug 23 11:40:55 k8s-master kube-apiserver: error: enable-admission-plugins plugin "NodeRest" is unknown
把NodeRest插件删除再启动还是报错
Aug 23 11:56:52 k8s-master kube-apiserver: error: error while parsing encryption provider configuration file "/etc/kubernetes/encryption-config.yaml": could not obtain secret for named key key1: illegal base64 data at input byte 0
把加密里的函数变成真实数字
[root@k8s-master ~]# cat /etc/kubernetes/encryption-config.yamlkind: EncryptionConfigapiVersion: v1resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: ${ENCRYPTION_KEY} - identity: {} [root@k8s-master ~]# echo ${ENCRYPTION_KEY}PhRCSuyQ1rhDom5X/78zHIfdFjyNDd7JjdI0Mx/B924=[root@k8s-master ~]# cat /etc/kubernetes/encryption-config.yamlkind: EncryptionConfigapiVersion: v1resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: PhRCSuyQ1rhDom5X/78zHIfdFjyNDd7JjdI0Mx/B924= - identity: {}[root@k8s-master ~]# 再启动，成功
[root@k8s-master ~]# systemctl status kube-apiserver● kube-apiserver.service - Kubernetes API Server Loaded: loaded (/etc/systemd/system/kube-apiserver.service; enabled; vendor preset: disabled) Active: active (running) since 四 2018-08-23 11:59:46 CST; 1min 10s ago Docs: https://github.com/GoogleCloudPlatform/kubernetesMain PID: 15986 (kube-apiserver) Tasks: 8 Memory: 291.7M CGroup: /system.slice/kube-apiserver.service └─15986 /opt/k8s/bin/kube-apiserver #--enable-admission-plugins=Initializers,NamespaceLifecycle,NodeRest r...8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.282484 15986 store.go:1397] Monitoring networ...cies8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.282504 15986 master.go:418] Enabling API grou...io".8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.304661 15986 store.go:1397] Monitoring poddis...gets8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.331913 15986 store.go:1397] Monitoring podsec...licy8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.331930 15986 master.go:418] Enabling API grou...cy".8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.353019 15986 store.go:1397] Monitoring roles....oles8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.374228 15986 store.go:1397] Monitoring rolebi...ings8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.395413 15986 store.go:1397] Monitoring cluste...oles8月 23 11:59:38 k8s-master kube-apiserver[15986]: I0823 11:59:38.445875 15986 store.go:1397] Monitoring cluste...ings8月 23 11:59:46 k8s-master systemd[1]: Started Kubernetes API Server.Hint: Some lines were ellipsized, use -l to show in full.[root@k8s-master ~]#
其他两个节点也启动
打印 kube-apiserver 写入 etcd 的数据
[root@k8s-master apiserver]# ETCDCTL_API=3 etcdctl --endpoints=${ETCD_ENDPOINTS} --cacert=/etc/kubernetes/cert/ca.pem --cert=/etc/etcd/cert/etcd.pem --key=/etc/etcd/cert/etcd-key.pem get /registry/ --prefix --keys-only 看看集群信息
[root@k8s-master apiserver]# kubectl get csNAME STATUS MESSAGE ERRORscheduler Unhealthy Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused controller-manager Unhealthy Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused etcd-0 Healthy {"health":"true"} etcd-2 Healthy {"health":"true"} etcd-1 Healthy {"health":"true"} [root@k8s-master apiserver]# kubectl get all --all-namespacesNAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEdefault service/kubernetes ClusterIP 10.254.0.1443/TCP 1h[root@k8s-master apiserver]# kubectl cluster-infoKubernetes master is running at https://192.168.1.94:8443To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.[root@k8s-master apiserver]#
遇到的问题：
1:The connection to the server localhost:8080 was refused - did you specify the right host or port?
报做错误是因为，见下：
[root@k8s-master ~]# ls -a. anaconda-ks.cfg .bash_logout .bashrc .config environment.sh .kube .ssh.. .bash_history .bash_profile .cache .cshrc k8s .pki .tcshrc[root@k8s-master ~]# ls .kubecache config http-cache[root@k8s-master ~]#
config这个配置文件的问题，注意名字必须是conifg，
kubectl读取证书和server就是这个文件。2.[root@k8s-master .kube]# kubectl cluster-infoKubernetes master is running at https://192.168.1.94:8443To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.Unable to connect to the server: x509: certificate is valid for 127.0.0.1, 192.168.1.92, 192.168.1.93, 192.168.1.95, not 192.168.1.94
报这个错误是因为
[root@k8s-master apiserver]# cat kubernetes-csr.json{"CN": "kubernetes","hosts": ["127.0.0.1","192.168.1.92","192.168.1.93","192.168.1.95","${MASTER_VIP}","${CLUSTER_KUBERNETES_SVC_IP}","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "SZ","L": "SZ","O": "k8s","OU": "4Paradigm"}]} "${MASTER_VIP}",
"${CLUSTER_KUBERNETES_SVC_IP}",这两个值没修改成正确的值。修改好重新启动后，测试正常。
授予 kubernetes 证书访问 kubelet API 的权限
在执行 kubectl exec、run、logs 等命令时，apiserver 会转发到 kubelet。这里定义RBAC 规则，授权 apiserver 调用 kubelet API。[root@k8s-master ~]# kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetesclusterrolebinding.rbac.authorization.k8s.io "kube-apiserver:kubelet-apis" created
重点提示:
配置文件里的ip地址，请不要用函数，全部用真实地址，减少报错。转载于:https://blog.51cto.com/goome/2164833