Flink on Kubernetes部署详细教程

Flink on Kubernetes部署详细教程一、概念二、准备工作三、在K8s集群上搭建Flink集群3.1 准备配置文件3.2 通过配置文件在 Kubernetes 上部署 Flink 集群3.3 haproxy配置flink集群负载均衡3.4 K8s验证一、概念Flink on Kubernetes将Flink的流处理能力与K8s的资源管理能力深度融合实现“按需分配、自动运维、弹性伸缩”的生产级流处理系统。Flink与K8s的核心对应关系Flink官方提供三种K8s部署模式分别对应不同的业务场景。我们需要根据作业类型、资源需求、运维复杂度选择合适的模式。Session Mode是共享集群模式先启动一个长期运行的Flink集群包含JM和一组TM然后通过Client向集群提交多个作业。所有作业共享集群的资源Task Slot。Job Cluster Mode是专属集群模式为每个作业启动一个独立的Flink集群JM 专属TM。作业完成后集群自动销毁。Application Mode是云原生最优模式将应用代码与Flink集群打包成一个Docker镜像 直接在K8s上运行。作业的生命周期与集群一致启动集群→运行作业→销毁集群。对比如下回到目录二、准备工作参考链接从0到1搭建Flink on K8s集群安装依赖工具Kind本地K8s集群搭建工具替代Minikube启动更快Docker镜像构建与运行工具kubectlK8s命令行工具Flink CLIFlink命令行客户端用于提交作业。搭建本地K8s集群三、在K8s集群上搭建Flink集群3.1 准备配置文件1.K8s集群主节点操作注意必须是主节点从节点不行在K8s集群其中一个主节点上进入主节点查看集群状态kubectl version集群状态正常进入下一步进入主节点目录创建文件夹并创建yaml配置文件#创建文件夹 mkdir /home/flink # 进入文件夹 cd /home/flink # 创建pv.yaml文件 touch pv.yaml # 创建flink.yaml文件 touch flink.yamlpv.yaml文件说明为Flink 集群持久化存储创建两个Kubernetes 存储资源对象分别为PersistentVolume (PV)名称为 flink-data-pv和PersistentVolumeClaim (PVC)名称为 flink-data-pvcflink.yaml文件说明一个用于在 Kubernetes 上部署 Flink 集群的完整配置文件定义了 5 个 Kubernetes 资源Deployment: flink-jobmanager (Flink JobManager 部署)负责作业调度和协调Service: flink-jobmanager (JobManager 服务)Deployment: flink-taskmanager (Flink TaskManager 部署)负责实际的数据处理Service: flink-jobmanager-rest (JobManager REST API 服务)Service: flink-taskmanager-http (TaskManager HTTP 服务)—根据实际情况修改下面2个 yaml 文件中 namespace 的值编辑pv.yaml文件将下面内容复制到文件中vi pv.yamlapiVersion: v1 kind: PersistentVolume metadata: name: flink-data-pv spec: capacity: storage: 10Gi accessModes: - ReadWriteMany hostPath: path: /mnt/data/flink-data type: DirectoryOrCreate --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: flink-data-pvc namespace: test # 添加命名空间,根据实际命名修改 spec: accessModes: - ReadWriteMany resources: requests: storage: 10Gi volumeName: flink-data-pv回到目录编辑flink.yaml文件将下面内容复制到文件中vi flink.yamlapiVersion: apps/v1 kind: Deployment metadata: name: flink-jobmanager namespace: test # 添加命名空间 spec: replicas: 1 selector: matchLabels: app: flink component: jobmanager template: metadata: labels: app: flink component: jobmanager spec: securityContext: runAsUser: 0 runAsGroup: 0 fsGroup: 0 containers: - name: jobmanager image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/flink:1.19.1-java8 args: [jobmanager] ports: - containerPort: 6123 name: rpc - containerPort: 6124 name: blob - containerPort: 8081 name: ui env: - name: JOB_MANAGER_RPC_ADDRESS value: flink-jobmanager - name: FLINK_PROPERTIES value: | jobmanager.memory.process.size: 1g jobmanager.memory.jvm-overhead.max: 256m state.backend: filesystem state.checkpoints.dir: file:///mnt/data/flink-data cluster.evenly-spread-out-slots: true volumeMounts: - name: flink-storage mountPath: /mnt/data/flink-data volumes: - name: flink-storage persistentVolumeClaim: claimName: flink-data-pvc --- apiVersion: v1 kind: Service metadata: name: flink-jobmanager namespace: test # 添加命名空间 spec: ports: - name: rpc port: 6123 - name: blob port: 6124 - name: ui port: 8081 selector: app: flink component: jobmanager type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: name: flink-taskmanager namespace: test # 添加命名空间 spec: replicas: 3 selector: matchLabels: app: flink component: taskmanager template: metadata: labels: app: flink component: taskmanager spec: securityContext: runAsUser: 0 runAsGroup: 0 fsGroup: 0 containers: - name: taskmanager image: swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/apache/flink:1.19.1-java8 args: [taskmanager] ports: - containerPort: 6121 name: data - containerPort: 6122 name: rpc - containerPort: 6666 name: http1 - containerPort: 6667 name: http2 - containerPort: 6668 name: http3 - containerPort: 6669 name: http4 - containerPort: 6670 name: http5 - containerPort: 6671 name: http6 - containerPort: 6672 name: http7 env: - name: JOB_MANAGER_RPC_ADDRESS value: flink-jobmanager - name: FLINK_PROPERTIES value: | taskmanager.bind-host: 0.0.0.0 taskmanager.numberOfTaskSlots: 16 taskmanager.memory.process.size: 1024m taskmanager.memory.heap.size: 512m taskmanager.memory.off-heap.size: 256m taskmanager.memory.managed.size: 256m taskmanager.memory.jvm-metaspace.size: 128m taskmanager.memory.jvm-overhead.min: 64m taskmanager.memory.jvm-overhead.max: 128m cluster.evenly-spread-out-slots: true volumeMounts: - name: flink-storage mountPath: /mnt/data/flink-data volumes: - name: flink-storage persistentVolumeClaim: claimName: flink-data-pvc --- apiVersion: v1 kind: Service metadata: name: flink-jobmanager-rest namespace: test # 添加命名空间 spec: ports: - name: ui port: 8081 targetPort: 8081 nodePort: 30081 selector: app: flink component: jobmanager type: NodePort --- apiVersion: v1 kind: Service metadata: name: flink-taskmanager-http namespace: test # 添加命名空间 spec: type: NodePort selector: app: flink component: taskmanager ports: - name: http1 port: 6666 targetPort: 6666 nodePort: 31001 - name: http2 port: 6667 targetPort: 6667 nodePort: 31002 - name: http3 port: 6668 targetPort: 6668 nodePort: 31003 - name: http4 port: 6669 targetPort: 6669 nodePort: 31004 - name: http5 port: 6670 targetPort: 6670 nodePort: 31005 - name: http6 port: 6671 targetPort: 6671 nodePort: 31006 - name: http7 port: 6672 targetPort: 6672 nodePort: 31007回到目录3.2 通过配置文件在 Kubernetes 上部署 Flink 集群通过配置文件创建资源,执行命令kubectl apply -f pv.yaml执行后通过下面命令判断是否成功创建资源kubectl get pv -n 命名空间名称 kubectl get pvc -n 命名空间名称2. 通过配置文件创建应用,执行命令kubectl apply -f flink.yaml执行后获取集群中核心资源的信息执行命令kubectl get all -n 命名空间名称如果此时STATUS是Creating请稍等一会等待几分钟后重新执行直至看到STATUS变成Running说明资源运行成功flink集群创建成功。回到目录3.3 haproxy配置flink集群负载均衡在haproxy集群的所有节点服务器中找到 /etc/haproxy/haproxy.cfg配置文件在文件末尾加上如下内容vi /etc/haproxy/haproxy.cfg#------- FLink服务 ------- listen Flink bind *:30081 #映射端口 mode tcp #均衡模式 balance roundrobin #均衡算法 option httpclose # 此选项表在客户端和服务器端完成次连接请求后haproxy将主动关闭此TCP连接 option abortonclose # 在服务器负载很⾼的情况下⾃动结束掉当前队列中处理时间⽐较长的链接 server k8smaster01 10.30.2.52:30081 check server k8smaster02 10.30.2.13:30081 check server k8sworker01 10.30.2.25:30081 check server k8sworker02 10.30.2.56:30081 check保存退出然后重启haproxy服务执行命令sudo systemctl restart haproxy查看haproxy状态确认成功运行sudo systemctl status haproxy注意option abortonclose 下的server 节点名称 节点IP:30081根据flink集群实际情况配置所有haproxy集群下的节点服务器都需要按照上面的步骤修改 /etc/haproxy/haproxy.cfg配置文件后重启haproxy服务。所有服务配置结束后查看flink集群是否成功启动浏览器进入Apache Flink DashboardApache Flink 仪表盘链接地址http://10.30.2.150:30081/#/overview这里的IP地址指的是haproxy配置的对外前端地址。出现如下画面说明flink集群负载均衡配置成功。回到目录3.4 K8s验证打开K8s的仪表板Dashboard访问界面查看flink集群服务是否正常启动回到目录