阿里云Kubernetes实战1–集群搭建与服务暴露

1.2 NAT网关与EIP打通网络

由于VPC网络内，所有的ECS没有配置外网IP，所以这里要配置NAT网关和弹性IP来打通外网和VPC的通讯。

配置DNAT(外网访问VPC)

配置SNAT(VPC访问外网)

配置完成后，便可以使用绑定DNAT的EIP的映射端口通过SSH访问ECS

1.3 使用Kubeasz部署K8S集群

搭建K8S集群相对比较简单，使用 kubeasz的 AllinOne部署即可

二、部署Gitlab实战

2.1 K8S Dashboard

部署好集群后，我们可以使用DNAT的EIP，通过映射端口23443访问K8S API和Dashboard

https://EIP:Port/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

令牌可在Master节点通过以下命令获取kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

2.2 PV与PVC

K8S中的PV和PVC的概念这里不再多提，引用官方的一段解释：

A PersistentVolume (PV) is a piece of storage in the cluster that has been provisioned by an administrator. It is a resource in the cluster just like a node is a cluster resource. PVs are volume plugins like Volumes, but have a lifecycle independent of any individual pod that uses the PV. This API object captures the details of the implementation of the storage, be that NFS, iSCSI, or a cloud-provider-specific storage system.

A PersistentVolumeClaim (PVC) is a request for storage by a user. It is similar to a pod. Pods consume node resources and PVCs consume PV resources. Pods can request specific levels of resources (CPU and Memory). Claims can request specific size and access modes (e.g., can be mounted once read/write or many times read-only).

在 Gitlab for Docker中，我们看到Volumes 有三个，如下表所示

Local location	Container location	Usage

所以我们也需要给Gitlab for K8S分配3个PV和PVC，这里我们用到了阿里云NAS

给NAS添加挂载点，选择VPC网络和VPC的交换机

查看挂载地址

2.3 K8S部署Gitlab

接下来补全Gitlab的Deployment和Service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitlab
  namespace: gitlab
spec:   
  selector:
    matchLabels:
      app: gitlab
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: gitlab
    spec:
      containers:
      - image: gitlab/gitlab-ce:latest
        name: gitlab
        ports:
        - containerPort: 80
          name: gitlab-http
        - containerPort: 443
          name: gitlab-https
        - containerPort: 22
          name: gitlab-ssh
        volumeMounts:
        - name: gitlab-config
          mountPath: /etc/gitlab
        - name: gitlab-log
          mountPath: /var/log/gitlab
        - name: gitlab-data
          mountPath: /var/opt/gitlab
      volumes:
      - name: gitlab-data
        persistentVolumeClaim:
          claimName: gitlab-data-claim
      - name: gitlab-config
        persistentVolumeClaim:
          claimName: gitlab-config-claim
      - name: gitlab-log
        persistentVolumeClaim:
          claimName: gitlab-log-claim
---
kind: Service
apiVersion: v1
metadata:
  name: gitlab-service
  labels:
    app: gitlab-service
  namespace: gitlab
spec:
  selector:
    app: gitlab
  ports:
  - protocol: TCP
    name: gitlab-https
    port: 443
    targetPort: 443
  - protocol: TCP
    name: gitlab-http
    port: 80
    targetPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: gitlab-ssh-service
  labels:
    app: gitlab-ssh-service
  namespace: gitlab
spec:
  type: NodePort
  selector:
    app: gitlab
  ports:
  - protocol: TCP
    name: gitlab-ssh
    port: 22
    targetPort: 22
    nodePort: 30000

到这里，配置与部署基本完成了，但我们还不能从外网访问Gitlab，不过至少可以在集群内验证配置是否正确。

在Master节点查看Servicekubectl get svc --namespace=gitlab

可以看到443和80端口已经开发给Cluster IP，同时22端口映射到了30000的NodePort上

2.4 使用Ingress-Nginx和阿里云SLB暴露服务

K8S暴露服务的方法有3种：

由于我们的集群搭建在阿里云上，所以第一时间想到的是LoadBalancer方案，但很遗憾，没办法使用，原因如下：

回归到NodePort的方式，目前已有的解决方案是基于Ingress的几款工具，如Ingress-Nginx、Traefik-Ingress，他们的对比如下（注意，目前的版本是IngressNginx 0.13.0、Traefik 1.6）：

最终我们还是选择了Ingress-Nginx，结合阿里云SLB，最终的拓扑图如下所示：

其原理是：

接下来看详细步骤。

2.4.1 部署Ingress-Nginx

主要参考 https://kubernetes.github.io/ingress-nginx/deploy/，并做一些小调整

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    app: default-http-backend
  namespace: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: default-http-backend
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissible as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-shenzhen.aliyuncs.com/heygears/defaultbackend:1.4
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi
---

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: default-http-backend
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
        - events
    verbs:
        - create
        - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx 
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: nginx-ingress-controller
          image: registry.cn-shenzhen.aliyuncs.com/heygears/nginx-ingress-controller:0.13.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --annotations-prefix=nginx.ingress.kubernetes.io
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
---
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-service
  namespace: ingress-nginx
spec:
  selector:
    app: ingress-nginx
  ports:
    - protocol: TCP
      port: 80
      # 从默认20000~40000之间选一个可用端口，让ingress-controller暴露给外部的访问
      nodePort: 23456
  type: NodePort

上传到Master节点后执行命令：

kubectl apply -f ingress-nginx.yaml

2.4.2 给gitlab配置ingress

修改2.3章节的gitlab.yaml，添加

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gitlab-ingress
  namespace: gitlab
  annotations:
      nginx.ingress.kubernetes.io/force-ssl-redirect: "true" # 强制http重定向到https
      nginx.ingress.kubernetes.io/ssl-passthrough: "true" # 将请求时的ssl传递到此，如果后台监听80端口，则无需此配置
      nginx.ingress.kubernetes.io/proxy-body-size: "0" # 设置client_max_body_size为0
spec:
  rules:
  - host: git.xxx.com  # hostname
    http:
      paths:
      - path: /
        backend:
          serviceName: gitlab-service
          servicePort: 443 # 监听443端口

重新执行