flink + iceberg 快速搭建指南

flink + iceberg 快速搭建

the environment includes:

• minio
• iceberg
• flink

Centos 更换 tencent 的yum源

备份系统旧配置文件

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup 

获取对应版本的CentOS-Base.repo 到/etc/yum.repos.d/目录

各版本源配置列表

CentOS7

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos7_base.repo 

CentOS8

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.cloud.tencent.com/repo/centos8_base.repo 

更新缓存

yum clean all yum makecache 

Centos 安装Java环境（flink使用）

wget --no-check-certificate https://repo.huaweicloud.com/java/jdk/8u202-b08/jdk-8u202-linux-x64.tar.gz  mkdir /usr/local/java/ tar -zxvf jdk-8u202-linux-x64.tar.gz -C /usr/local/java  echo "export JAVA_HOME=/usr/local/java/jdk1.8.0_202" >> /etc/profile echo "export JRE_HOME=${JAVA_HOME}/jre" >> /etc/profile echo "export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib" >> /etc/profile echo "export PATH=${JAVA_HOME}/bin:$PATH" >> /etc/profile  source /etc/profile ln -s /usr/local/java/jdk1.8.0_202/bin/java /usr/bin/java  java -version 

Centos install Docker & Docker Compose

First, To install Docker Engine, you need a maintained version of CentOS 7 or 8.
Archived versions aren't supported or tested.

sudo yum install -y python3-pip yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo sudo yum install docker-ce sudo systemctl start docker 

update docker daemon..

sudo mkdir -p /etc/docker sudo tee /etc/docker/daemon.json <<-'EOF' {   "registry-mirrors": [     "https://docker.fxxk.dedyn.io/", 	"https://dockerproxy.cn"   ] } EOF sudo systemctl daemon-reload sudo systemctl restart docker 

install docker compose..

sudo curl -L "https://mirrors.aliyun.com/docker-toolbox/linux/compose/1.21.2/docker-compose-Linux-x86_64" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose docker-compose --version 

install Minio

使用以下命令下载安装最新版本的稳定 MinIO二进制包， 并设置 $PATH:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio chmod +x minio sudo mv minio /usr/local/bin/ 

设置minio linux服务

sudo tee /etc/systemd/system/minio.service <<-'EOF' [Unit] Description=Minio Service  [Service] Environment="MINIO_ROOT_USER=admin" Environment="MINIO_ROOT_PASSWORD=password" ExecStart=/usr/local/bin/minio server /mnt/minio --console-address ":9001" ExecReload=/bin/kill -s HUP $MAINPID ExecStop=/bin/kill -s QUIT $MAINPID StandardOutput=/mnt/minio/logs/minio.log PrivateTmp=true  [Install] WantedBy=multi-user.target EOF 

启动 minio linux服务

systemctl start minio systemctl enable minio 

install iceberg in docker

save the yaml below into a file named docker-compose.yml:

wget https://dl.minio.org.cn/server/minio/release/linux-amd64/minio version: "3"  services:   rest:     image: tabulario/iceberg-rest:1.5.0     container_name: iceberg-rest     networks:       iceberg_net:     ports:       - 8181:8181     environment:       - AWS_ACCESS_KEY_ID=admin       - AWS_SECRET_ACCESS_KEY=password       - AWS_REGION=us-east-1       - CATALOG_WAREHOUSE=s3://warehouse/       - CATALOG_IO__IMPL=org.apache.iceberg.aws.s3.S3FileIO       - CATALOG_S3_ENDPOINT=<更换本地minio服务地址:9000>   mc:     image: minio/mc     container_name: mc     networks:       iceberg_net:     environment:       - AWS_ACCESS_KEY_ID=admin       - AWS_SECRET_ACCESS_KEY=password       - AWS_REGION=us-east-1     entrypoint: >       /bin/sh -c "       until (/usr/bin/mc config host add minio http://<更换本地minio服务地址:9000> admin password) do echo '...waiting...' && sleep 1; done;       /usr/bin/mc mb minio/warehouse;       /usr/bin/mc policy set public minio/warehouse;       tail -f /dev/null       " networks:   iceberg_net: 

Next, start up the docker containers with this command:

docker-compose up -d 

install flink

download jdk filnk 1.18.1

wget https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.18.1/flink-1.18.1-bin-scala_2.12.tgz  tar zxvf flink-1.18.1-bin-scala_2.12.tgz -C /opt/ 

download dll ，copy to flink-1.18.1/lib folder

wget https://repo1.maven.org/maven2/org/apache/iceberg/iceberg-aws-bundle/1.5.2/iceberg-aws-bundle-1.5.2.jar wget ttps://repo1.maven.org/maven2/org/apache/iceberg/iceberg-flink-runtime-1.18/1.5.2/iceberg-flink-runtime-1.18-1.5.2.jar wget https://repo1.maven.org/maven2/org/apache/flink/flink-connector-jdbc/3.1.2-1.18/flink-connector-jdbc-3.1.2-1.18.jar wget https://repo1.maven.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar wget https://repo1.maven.org/maven2/org/apache/flink/flink-s3-fs-hadoop/1.18.0/flink-s3-fs-hadoop-1.18.0.jar  cp ./*.jar /opt/flink-1.18.1/lib/ 

start flink

cd /opt/flink-1.18.1/  bash ./bin/start-cluster.sh 

flink 创建icberg表

bash flink 客户端：

cd /opt/flink-1.18.1/  bash ./bin/sql-client.sh  SET 'sql-client.execution.result-mode' = 'tableau'; SET 'execution.runtime-mode' = 'batch'; 

创建 Iceberg Catalog

CREATE CATALOG iceberg WITH (   'type'='iceberg',   'catalog-type'='rest',   'uri'='http://rest:8181/',   's3.endpoint'='http://<更换本地minio服务地址:9000>',   'warehouse'='s3://warehouse/wh/' ); 

创建数据库和 Iceberg 表

create database if not exists iceberg.db_iceberg;  CREATE TABLE if not exists iceberg.db_iceberg.tb_iceberg (     id BIGINT,     val string,     PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'write.upsert.enabled'='true', 'upsert-enabled'='true', 'write.delete.mode'='merge-on-read', 'write.update.mode'='merge-on-read' );