基于k8s的CI/CD的实现

综述

首先,本篇文章所介绍的内容,已经有完整的实现,可以参考这里
在微服务、DevOps和云平台流行的当下,使用一个高效的持续集成工具也是一个非常重要的事情。虽然市面上目前已经存在了比较成熟的自动化构建工具,比如jekines,还有一些商业公司推出的自动化构建工具,但他们都不能够很好的和云环境相结合。那么究竟该如何实现一个简单、快速的基于云环境的自动化构建系统呢?我们首先以一个Springboot应用为例来介绍一下整体的发布流程,然后再来看看具体如何实现。不发的步骤大体如下:
1.首先从代码仓库下载代码,比如Gitlab、GitHub等;
2.接着是进行打包,比如使用Maven、Gradle等;
3.如果要使用k8s作为编排,还需要把步骤2产生的包制作成镜像,比如用Docker等;
4.上传步骤3的镜像到远程仓库,比如Harhor、DockerHub等;
5.最后,下载镜像并编写Deployment文件部署到k8s集群;
如图1所示:
基于k8s的CI/CD的实现
图1

从以上步骤可以看出,发布过程中需要的工具和环境至少包括:代码仓库(Gitlab、GitHub等)、打包环境(Maven、Gradle等)、镜像制作(Docker等)、镜像仓库(Harbor、DockerHub等)、k8s集群等;此外,还包括发布系统自身的数据存储等。
可以看出,整个流程里依赖的环境很多,如果发布系统不能与这些环境解耦,那么要想实现一个安装简单、功能快速的系统没有那么容易。那么有没有合理的解决方案来实现与这些环境的解耦呢?答案是有的,下面就分别介绍。

代码仓库

操作代码仓库,一般系统提供的都有对应Restful API,以GitLab系统提供的Java客户端为例,如下代码:

<dependency> 	<groupId>org.gitlab4j</groupId> 	<artifactId>gitlab4j-api</artifactId> 	<version>4.17.0</version> </dependency> 

比如,我们想获取某个项目的分支列表,如下代码所示:

public List<Branch> branchList(CodeRepo codeRepo, BranchListParam param) { 	GitLabApi gitLabApi = gitLabApi(codeRepo); 	List<Branch> list = null; 	try { 		list = gitLabApi.getRepositoryApi().getBranches(param.getProjectIdOrPath(), param.getBranchName()); 	} catch (GitLabApiException e) { 		LogUtils.throwException(logger, e, MessageCodeEnum.PROJECT_BRANCH_PAGE_FAILURE); 	} finally { 		gitLabApi.close(); 	} }  private GitLabApi gitLabApi(CodeRepo codeRepo) { 	GitLabApi gitLabApi = new GitLabApi(codeRepo.getUrl(), codeRepo.getAuthToken()); 	gitLabApi.setRequestTimeout(1000, 5 * 1000); 	try { 		gitLabApi.getVersion(); 	}catch(GitLabApiException e) { 		//如果token无效,则用账号登录 		if(e.getHttpStatus() == 401 && !StringUtils.isBlank(codeRepo.getAuthUser())) { 			gitLabApi = new GitLabApi(codeRepo.getUrl(), codeRepo.getAuthUser(), codeRepo.getAuthPassword()); 			gitLabApi.setRequestTimeout(1000, 5 * 1000); 		} 	} 	 	return gitLabApi; } 

打包环境

我们以Maven为例进行说明,一般情况下,我们使用Maven打包时,需要首先安装Maven环境,接着引入打包插件,然后使用mvn clean package命令就可以打包了。比如springboot自带插件:

<plugin> 	<groupId>org.springframework.boot</groupId> 	<artifactId>spring-boot-maven-plugin</artifactId> 	<version>2.5.6</version> 	<configuration> 		<classifier>execute</classifier> 		<mainClass>com.test.Application</mainClass> 	</configuration> 	<executions> 		<execution> 			<goals> 				<goal>repackage</goal> 			</goals> 		</execution> 	</executions> </plugin> 

再比如,通用的打包插件:

<plugin> 	<groupId>org.apache.maven.plugins</groupId> 	<artifactId>maven-assembly-plugin</artifactId> 	<version>3.8.2</version> 	<configuration> 		<appendAssemblyId>false</appendAssemblyId> 		<descriptors> 			<descriptor>src/main/resources/assemble.xml</descriptor> 		</descriptors> 		<outputDirectory>../target</outputDirectory> 	</configuration> 	<executions> 		<execution> 			<id>make-assembly</id> 			<phase>package</phase> 			<goals> 				<goal>single</goal> 			</goals> 		</execution> 	</executions> </plugin> 

等等。然后再通过运行mvn clean package命令进行打包。那么,在打包时如果要去除对maven环境的依赖,该如何实现呢?
可以使用嵌入式maven插件maven-embedder来实现。
具体可以这样来做,首先在平台项目里引入依赖,如下:

<dependency> 	<groupId>org.apache.maven</groupId> 	<artifactId>maven-embedder</artifactId> 	<version>3.8.1</version> </dependency> <dependency> 	<groupId>org.apache.maven</groupId> 	<artifactId>maven-compat</artifactId> 	<version>3.8.1</version> </dependency> <dependency> 	<groupId>org.apache.maven.resolver</groupId> 	<artifactId>maven-resolver-connector-basic</artifactId> 	<version>1.7.1</version> </dependency> <dependency> 	<groupId>org.apache.maven.resolver</groupId> 	<artifactId>maven-resolver-transport-http</artifactId> 	<version>1.7.1</version> </dependency> 

运行如下代码,就可以对项目进行打包了:

String[] commands = new String[] { "clean", "package", "-Dmaven.test.skip" }; String pomPath = "D:/hello/pom.xml"; MavenCli cli = new MavenCli(); try { 	cli.doMain(commands, pomPath, System.out, System.out); } catch (Exception e) { 	e.printStackTrace(); } 

但是,一般情况下,我们通过maven的settings文件还会做一些配置,比如配置工作目录、nexus私服地址、Jdk版本、编码方式等等,如下:

<?xml version="1.0" encoding="UTF-8"?> <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0" 	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 	xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd"> 	<localRepository>C:/m2/repository</localRepository> 	<profiles> 		<profile> 			<id>myNexus</id> 			<repositories> 				<repository> 					<id>nexus</id> 					<name>nexus</name> 					<url>https://repo.maven.apache.org/maven2</url> 					<releases> 						<enabled>true</enabled> 					</releases> 					<snapshots> 						<enabled>true</enabled> 					</snapshots> 				</repository> 			</repositories> 			<pluginRepositories> 				<pluginRepository> 					<id>nexus</id> 					<name>nexus</name> 					<url>https://repo.maven.apache.org/maven2</url> 					<releases> 						<enabled>true</enabled> 					</releases> 					<snapshots> 						<enabled>true</enabled> 					</snapshots> 				</pluginRepository> 			</pluginRepositories> 		</profile>  		<profile> 			<id>java11</id> 			<activation> 				<activeByDefault>true</activeByDefault> 				<jdk>11</jdk> 			</activation> 			<properties> 				<maven.compiler.source>11</maven.compiler.source> 				<maven.compiler.target>11</maven.compiler.target> 				<maven.compiler.compilerVersion>11</maven.compiler.compilerVersion> 				<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 				<project.build.outputEncoding>UTF-8</project.build.outputEncoding> 			</properties> 		</profile> 	</profiles> 	<activeProfiles> 		<activeProfile>myNexus</activeProfile> 	</activeProfiles> </settings> 

通过查看MavenCli类发现,doMain(CliRequest cliRequest)方法有比较丰富的参数,CliRequest的代码如下:

package org.apache.maven.cli;  public class CliRequest {     String[] args;      CommandLine commandLine;      ClassWorld classWorld;      String workingDirectory;      File multiModuleProjectDirectory;      boolean debug;      boolean quiet;      boolean showErrors = true;      Properties userProperties = new Properties();      Properties systemProperties = new Properties();      MavenExecutionRequest request;      CliRequest( String[] args, ClassWorld classWorld )     {         this.args = args;         this.classWorld = classWorld;         this.request = new DefaultMavenExecutionRequest();     }      public String[] getArgs()     {         return args;     }      public CommandLine getCommandLine()     {         return commandLine;     }      public ClassWorld getClassWorld()     {         return classWorld;     }      public String getWorkingDirectory()     {         return workingDirectory;     }      public File getMultiModuleProjectDirectory()     {         return multiModuleProjectDirectory;     }      public boolean isDebug()     {         return debug;     }      public boolean isQuiet()     {         return quiet;     }      public boolean isShowErrors()     {         return showErrors;     }      public Properties getUserProperties()     {         return userProperties;     }      public Properties getSystemProperties()     {         return systemProperties;     }      public MavenExecutionRequest getRequest()     {         return request;     }      public void setUserProperties( Properties properties )      {         this.userProperties.putAll( properties );           } } 

可以看出,这些参数非常丰富,也许可以满足我们的需求,但是CliRequest只有一个默认修饰符的构造方法,也就说只有位于org.apache.maven.cli包下的类才有访问CliRequest构造方法的权限,我们可以在平台项目里新建一个包org.apache.maven.cli,然后再创建一个类(如:DefaultCliRequest)继承自CliRequest,然后实现一个public的构造方法,就可以在任何包里使用该类了,如下代码:

package org.apache.maven.cli;  import org.codehaus.plexus.classworlds.ClassWorld;  public class DefaultCliRequest extends CliRequest{  	public DefaultCliRequest(String[] args, ClassWorld classWorld) { 		super(args, classWorld); 	} 	 	public void setWorkingDirectory(String directory) { 		this.workingDirectory = directory; 	} } 

定义好参数类型DefaultCliRequest后,我们再来看看打包的代码:

public void doPackage() { 	String[] commands = new String[] { "clean", "package", "-Dmaven.test.skip" }; 	DefaultCliRequest request = new DefaultCliRequest(commands, null); 	request.setWorkingDirectory("D:/hello/pom.xml");  	Repository repository = new Repository(); 	repository.setId("nexus"); 	repository.setName("nexus"); 	repository.setUrl("https://repo.maven.apache.org/maven2"); 	RepositoryPolicy policy = new RepositoryPolicy(); 	policy.setEnabled(true); 	policy.setUpdatePolicy("always"); 	policy.setChecksumPolicy("fail"); 	repository.setReleases(policy); 	repository.setSnapshots(policy);  	String javaVesion = "11"; 	Profile profile = new Profile(); 	profile.setId("java11"); 	Activation activation = new Activation(); 	activation.setActiveByDefault(true); 	activation.setJdk(javaVesion); 	profile.setActivation(activation); 	profile.setRepositories(Arrays.asList(repository)); 	profile.setPluginRepositories(Arrays.asList(repository));  	Properties properties = new Properties(); 	properties.put("java.home", "D:/java/jdk-11.0.16.2"); 	properties.put("java.version", javaVesion); 	properties.put("maven.compiler.source", javaVesion); 	properties.put("maven.compiler.target", javaVesion); 	properties.put("maven.compiler.compilerVersion", javaVesion); 	properties.put("project.build.sourceEncoding", "UTF-8"); 	properties.put("project.reporting.outputEncoding", "UTF-8"); 	profile.setProperties(properties); 	MavenExecutionRequest executionRequest = request.getRequest(); 	executionRequest.setProfiles(Arrays.asList(profile));  	MavenCli cli = new MavenCli(); 	try { 		cli.doMain(request); 	} catch (Exception e) { 		e.printStackTrace(); 	} } 

如果需要设置其他参数,也可以通过以上参数自行添加。

镜像制作

一般情况下,我们在Docker环境中通过Docker命令来制作镜像,过程如下:
1.首先编写Dockerfile文件;
2.通过docker build制作镜像;
3.通过docker push上传镜像;
可以看出,如果要使用docker制作镜像的话,必须要有docker环境,而且需要编写Dockerfile文件。当然,也可以不用安装docker环境,直接使用doker的远程接口:post/build。但是,在远程服务器中仍然需要安装doker环境和编写Dockerfile。在不依赖Docker环境的情况下,仍然可以制作镜像,下面就介绍一款工具Jib的用法。
Jib是谷歌开源的一套工具,github地址,它是一个无需Docker守护进程——也无需深入掌握Docker最佳实践的情况下,为Java应用程序构建Docker和OCI镜像, 它可以作为Maven和Gradle的插件,也可以作为Java库。

比如,使用jib-maven-plugin插件构建镜像的代码如下:

<plugin> 	<groupId>com.google.cloud.tools</groupId> 	<artifactId>jib-maven-plugin</artifactId> 	<version>3.3.0</version> 	<configuration> 		<from> 			<image>openjdk:13-jdk-alpine</image> 		</from> 		<to> 			<image>gcr.io/dhorse/client</image> 			<tags> 				<tag>102</tag> 			</tags> 			<auth> 				<!--连接镜像仓库的账号和密码 --> 				<username>username</username> 				<password>password</password> 			</auth> 		</to> 		<container> 			<ports> 				<port>8080</port> 			</ports> 		</container> 	</configuration> 	<executions> 		<execution> 			<phase>package</phase> 			<goals> 				<goal>build</goal> 			</goals> 		</execution> 	</executions> </plugin> 

然后使用命令进行构建:

mvn compile jib:build 

可以看出,无需docker环境就可以实现镜像的构建。但是,要想通过平台类型的系统去为每个系统构建镜像,显然通过插件的方式,不太合适,因为需要每个被构建系统引入jib-maven-plugin插件才行,也就是需要改造每一个系统,这样就会带来一定的麻烦。那么有没有不需要改造系统的方式直接进行构建镜像呢?答案是通过Jib-core就可以实现。

首先,在使用Jib-core的项目中引入依赖,maven如下:

<dependency> 	<groupId>com.google.cloud.tools</groupId> 	<artifactId>jib-core</artifactId> 	<version>0.22.0</version> </dependency> 

然后就可以直接使用Jib-core的API来进行制作镜像,如下代码:

try { 	JibContainerBuilder jibContainerBuilder = null; 	if (StringUtils.isBlank(context.getProject().getBaseImage())) { 		jibContainerBuilder = Jib.fromScratch(); 	} else { 		jibContainerBuilder = Jib.from(context.getProject().getBaseImage()); 	} 	//连接镜像仓库5秒超时 	System.setProperty("jib.httpTimeout", "5000"); 	System.setProperty("sendCredentialsOverHttp", "true"); 	String fileNameWithExtension = targetFiles.get(0).toFile().getName(); 	List<String> entrypoint = Arrays.asList("java", "-jar", fileNameWithExtension); 	RegistryImage registryImage = RegistryImage.named(context.getFullNameOfImage()).addCredential( 			context.getGlobalConfigAgg().getImageRepo().getAuthUser(), 			context.getGlobalConfigAgg().getImageRepo().getAuthPassword()); 	jibContainerBuilder.addLayer(targetFiles, "/") 		.setEntrypoint(entrypoint) 		.addVolume(AbsoluteUnixPath.fromPath(Paths.get("/etc/localtime"))) 		.containerize(Containerizer.to(registryImage) 				.setAllowInsecureRegistries(true) 				.addEventHandler(LogEvent.class, logEvent -> logger.info(logEvent.getMessage()))); } catch (Exception e) { 	logger.error("Failed to build image", e); 	return false; } 

其中,targetFiles是要构建镜像的目标文件,比如springboot打包后的jar文件。
通过Jib-core,可以很轻松的实现镜像构建,而不需要依赖任何其他环境,也不需要被构建系统做任何改造,非常方便。

镜像仓库

类似代码仓库提供的Restful API,也可以通过Restful API来操作镜像仓库,以Harbor创建一个项目为例,代码如下:

public void createProject(ImageRepo imageRepo) { 	String uri = "api/v2.0/projects"; 	if(!imageRepo.getUrl().endsWith("/")) { 		uri = "/" + uri; 	} 	HttpPost httpPost = new HttpPost(imageRepo.getUrl() + uri); 	RequestConfig requestConfig = RequestConfig.custom() 			.setConnectionRequestTimeout(5000) 			.setConnectTimeout(5000) 			.setSocketTimeout(5000) 			.build(); 	httpPost.setConfig(requestConfig); 	httpPost.setHeader("Content-Type", "application/json;charset=UTF-8"); 	httpPost.setHeader("Authorization", "Basic "+ Base64.getUrlEncoder().encodeToString((imageRepo.getAuthUser() + ":" + imageRepo.getAuthPassword()).getBytes())); 	ObjectNode objectNode = JsonUtils.getObjectMapper().createObjectNode(); 	objectNode.put("project_name", "dhorse"); 	//1:公有类型 	objectNode.put("public", 1); 	httpPost.setEntity(new StringEntity(objectNode.toString(),"UTF-8")); 	try (CloseableHttpResponse response = createHttpClient(imageRepo.getUrl()).execute(httpPost)){ 		if (response.getStatusLine().getStatusCode() != 201 				&& response.getStatusLine().getStatusCode() != 409) { 			LogUtils.throwException(logger, response.getStatusLine().getReasonPhrase(), 					MessageCodeEnum.IMAGE_REPO_PROJECT_FAILURE); 		} 	} catch (IOException e) { 		LogUtils.throwException(logger, e, MessageCodeEnum.IMAGE_REPO_PROJECT_FAILURE); 	} } 

k8s集群

同样,k8s也提供了Restful API。同时,官方也提供了各种语言的客户端,下面以Java语言的客户端为例,来创建一个deployment。
首先,引入Maven依赖:

<dependency> 	<groupId>io.kubernetes</groupId> 	<artifactId>client-java</artifactId> 	<version>13.0.0</version> </dependency> 

然后,使用如下代码:

public boolean createDeployment(DeployContext context) { 	V1Deployment deployment = new V1Deployment(); 	deployment.apiVersion("apps/v1"); 	deployment.setKind("Deployment"); 	deployment.setMetadata(deploymentMetaData(context.getDeploymentAppName())); 	deployment.setSpec(deploymentSpec(context)); 	ApiClient apiClient = this.apiClient(context.getCluster().getClusterUrl(), 			context.getCluster().getAuthToken(), 1000, 1000); 	AppsV1Api api = new AppsV1Api(apiClient); 	CoreV1Api coreApi = new CoreV1Api(apiClient); 	String namespace = context.getProjectEnv().getNamespaceName(); 	String labelSelector = K8sUtils.getDeploymentLabelSelector(context.getDeploymentAppName()); 	try { 		V1DeploymentList oldDeployment = api.listNamespacedDeployment(namespace, null, null, null, null, 				labelSelector, null, null, null, null, null); 		if (CollectionUtils.isEmpty(oldDeployment.getItems())) { 			deployment = api.createNamespacedDeployment(namespace, deployment, null, null, null); 		} else { 			deployment = api.replaceNamespacedDeployment(context.getDeploymentAppName(), namespace, deployment, null, null, 					null); 		} 	} catch (ApiException e) { 		if (!StringUtils.isBlank(e.getMessage())) { 			logger.error("Failed to create k8s deployment, message: {}", e.getMessage()); 		} else { 			logger.error("Failed to create k8s deployment, message: {}", e.getResponseBody()); 		} 		return false; 	} 	return true; }  private ApiClient apiClient(String basePath, String accessToken, int connectTimeout, int readTimeout) { 	ApiClient apiClient = new ClientBuilder().setBasePath(basePath).setVerifyingSsl(false) 			.setAuthentication(new AccessTokenAuthentication(accessToken)).build(); 	apiClient.setConnectTimeout(connectTimeout); 	apiClient.setReadTimeout(readTimeout); 	return apiClient; } 

至此,关键的技术点已经介绍完了。

也可以参考其他文章:
DHorse系列文章之操作手册

DHorse系列文章之镜像制作

DHorse系列文章之maven打包

DHorse系列文章之Dubbo项目解决方案

发表评论

相关文章