一种通过nacos动态配置实现多租户的log4j2日志物理隔离的设计

1、背景

1.1、背景

旧服务改造为多租户服务后,log4j日志打印在一起不能区分是哪个租户的,日志太多,太杂,不好定位排除问题,排查问题较难。

1.2、前提

不改动以前的日志代码(工作量太大)

1.3、打印日志示例

package com.cherf.sauth.controller;  import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.cherf.common.ResultVo; import io.swagger.annotations.ApiOperation; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController;  /**  * @author cherf  * @description: test  * @date 2023/03/01 17:33  **/ @RestController @RequestMapping("/v1") public class TestController {  	private static Logger log = LoggerFactory.getLogger(TestController.class);  	        @PostMapping("/test")     @ApiOperation(value = "test", notes = "test")     public ResultVo<?> authorization() {         log.trace("test:{}", "trace");         log.debug("test:{}", "debug");         log.info("test:{}", "info");         log.warn("test:{}", "warn");         log.error("test:{}", "error");         return ResultVo.ok();     }  }  

2、实现

2.1、版本依赖

nacos: 2.1.0

slf4j-api: 1.7.36

slf4j-log4j12: 1.7.36

spring-boot:2.6.14

spring-cloud:2021.0.1

spring-cloud-alibaba:2021.0.1.0

注: log4j-apilogback-core用的是 spring-boot-starter-test 2.6.14中的版本分别是 2.17.2 1.2.11

2.2、实现思路

2.2.1、日志分租户打印

logback通过加载ogback.xml 配置,通过 Appender 接口来实现打印,原理请看:logback

通过跟踪源码,可以找到AppenderAttachableImpl 这个类,其中通过Appender.doAppend 方法实现根据logabck配置中的类(默认是:RollingFileAppender)来打印日志,我们需要自定义重写doAppend方法来实现一种通过nacos动态配置实现多租户的log4j2日志物理隔离的设计

2.2.2、logback配置动态生效

通过JoranConfiguration来实现 。Joranlogback 使用的一个配置加载库,动态生效 logback 的配置可以通过joranConfigurator.doConfigure方法实现,(实现代码在下面);

2.2.3、新增租户时新增logback配置

主要思路是通过nacos来动态修改和发布配置从而实现logback.xml动态修改;XML格式比较烦,如果配置较多可以使用DOM4J来实现修改XML;(我们的较为简单,所以只是通过字符串替换来实现~)

2.3、实现

2.3.1、日志分离打印

2.3.1.1 重写doAppend方法

日志分离打印主要的实现就是重写doAppend方法,示例如下:
其中TenantContextHolder是用来存放租户id本地变量,实现可参考:多租户改造(字段隔离和表隔离混合模式)(也可使用自己的方法)

package com.cherf.common.logback;  import ch.qos.logback.core.rolling.RollingFileAppender; import com.cherf.common.context.TenantContextHolder; import com.cherf.common.util.StringUtil;   /**  * @author cherf  * @description:logback复写  * @date 2023/02/24 11:49  **/ public class TenantRollingFileAppender<E> extends RollingFileAppender<E> {      /**      * 日志打印会调用此方法,进行复写,判断租户,根据租户打印到不同日志文件      *      * @param eventObject      */     public void doAppend(E eventObject) {         String tenantId = TenantContextHolder.getTenantId();         if (StringUtil.isBlank(tenantId)) {             //没有租户id的日志,打印到public下面             tenantId = "public";         }         //  this.getName() 是在logback.xml中配置的<appender name="appenderName" class="com.cherf.common.logback.TenantRollingFileAppender">         // 只打印当前租户的Append,RollingFileAppender追加器以租户类型标识开头的执行追加         if (this.getName().startsWith(tenantId)) {             super.doAppend(eventObject);         }     } }  

2.3.1.2 logback配置示例

logback.xml需要将配置中的appender标签的class属性修改为刚刚重写的方法全限定类名:com.cherf.common.logback.TenantRollingFileAppender

<?xml version="1.0" encoding="UTF-8"?> <configuration> 	<property name="log.path" value="../../logs/system" /> 	<property name="log.pattern" value="%date [%level] [%thread] %logger{80} [%file : %line] %msg%n" /> 	<appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT"> 		<encoder> 			<pattern>${log.pattern}</pattern> 		</encoder> 	</appender> 	<!--    public 公共--> 	<appender class="com.cherf.common.logback.TenantRollingFileAppender" name="public"> 		<file>${log.path}/public/sys-api.log</file> 		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 			<fileNamePattern>${log.path}/public/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern> 			<!-- 日志最大的历史 60天 --> 			<maxHistory>60</maxHistory> 		</rollingPolicy> 		<encoder> 			<pattern>${log.pattern}</pattern> 		</encoder> 		<!-- 保留INFO级别及以上的日志 --> 		<filter class="ch.qos.logback.classic.filter.ThresholdFilter"> 			<level>INFO</level> 		</filter> 	</appender> 	<!--    默认租户 --> 	<appender class="com.cherf.common.logback.TenantRollingFileAppender" name="tid20220831114008942"> 		<file>${log.path}/tid20220831114008942/sys-api.log</file> 		<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> 			<fileNamePattern>${log.path}/tid20220831114008942/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern> 			<!-- 日志最大的历史 60天 --> 			<maxHistory>60</maxHistory> 		</rollingPolicy> 		<encoder> 			<pattern>${log.pattern}</pattern> 		</encoder> 		<!-- 保留INFO级别及以上的日志 --> 		<filter class="ch.qos.logback.classic.filter.ThresholdFilter"> 			<level>INFO</level> 		</filter> 	</appender> 	<logger additivity="false" name="com.cherf.system"> 		<appender-ref ref="public" /> 		<appender-ref ref="STDOUT" /> 		<appender-ref ref="tid20220831114008942" /> 	</logger> 	<root level="INFO"> 		<appender-ref ref="public" /> 		<appender-ref ref="STDOUT" /> 		<appender-ref ref="tid20220831114008942" /> 	</root> </configuration> 

2.3.2、配置动态生效

2.3.2.1、项目启动读取nacos配置

服务启动时logback默认加载 classpath:logback.xml 配置,需要在yml中指定logback配置 (logging.config后配置)
配置如下:

spring:   cloud:     nacos:       discovery:         # 不使用nacos的配置         # enabled: false         server-addr: 127.0.0.1:8848   #日志打印 logging:   config: http://${spring.cloud.nacos.discovery.server-addr}/nacos/v1/cs/configs?group=${logback.group}&tenant=public&dataId=${logback.systemDataId}   level:     com.cherf: info     com.cherf.mapper: info     org.springframework: info     org.spring.springboot.dao: info  logback:   group: logback   systemDataId: system-logback.xml 

2.3.2.2、nacos配置监听+logback动态加载配置

主要采用nacos ConfigServiceaddListener方法来监听;
注意:网上很多直接通过 NacosFactory.createConfigService()来创建ConfigService的方法可能会重复创建实例,导致CPU上升,详情可参考:记一次CPU占用持续上升问题排查(Nacos动态路由引起)

1、nacos动态配置监听器

package com.cherf.common.nacos;  import ch.qos.logback.classic.LoggerContext; import ch.qos.logback.classic.joran.JoranConfigurator; import com.alibaba.nacos.api.config.ConfigService; import com.alibaba.nacos.api.config.ConfigType; import com.alibaba.nacos.api.config.listener.Listener; import com.alibaba.nacos.api.exception.NacosException; import com.cherf.common.nacos.NacosConfigService; import com.cherf.common.util.StringUtil; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component;  import javax.annotation.PostConstruct; import javax.annotation.Resource; import java.io.ByteArrayInputStream; import java.io.InputStream; import java.util.concurrent.Executor;  /**  * @author cherf  * @description: nacos监听器,修改logback.xml后动态生效  * @date 2023/03/04 16:35  **/ @Component public class NacosDynamicLogbackService {      private static final Logger log = LoggerFactory.getLogger(NacosDynamicLogbackService.class);      /**      * 配置 ID      */     @Value("${logback.systemDataId}")     private String dataId;      /**      * 配置 分组      */     @Value("${logback.group}")     private String group;     @Resource     private NacosConfigService nacosConfigService;       @PostConstruct     public void dynamicLogbackByNacosListener() {         try {             ConfigService configService = nacosConfigService.getInstance();             if (configService != null) {                 configService.getConfig(dataId, group, 5000);                 configService.addListener(dataId, group, new Listener() {                     @Override                     public void receiveConfigInfo(String configInfo) {                         if (StringUtil.isNotBlank(configInfo)) {                             System.out.println("configInfo=============================>" + configInfo);                             try {                                 LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();                                 JoranConfigurator configurator = new JoranConfigurator();                                 configurator.setContext(context);                                 context.reset();                                 //获取nacos配置,生成inputStream                                 InputStream inputStreamRoute = new ByteArrayInputStream(new String(configInfo).getBytes());                                 //configurator.doConfigure("/logback.xml");                                 configurator.doConfigure(inputStreamRoute);                                  context.start();                             } catch (Exception e) {                                 log.error("加载logback.xml配置发生错误", e);                             }                         }                     }                      @Override                     public Executor getExecutor() {                         return null;                     }                 });             }         } catch (NacosException e) {             log.error("获取logback.xml配置发生错误", e);         }     }      /**      * 获取配置文件内容      *      * @return      */     public String getLogBackConfig(String dataId, String group) {         try {             ConfigService configService = nacosConfigService.getInstance();             // 根据dataId、group定位到具体配置文件,获取其内容. 方法中的三个参数分别是: dataId, group, 超时时间             String content = configService.getConfig(dataId, group, 5000L);             return content;         } catch (NacosException e) {             log.error(e.getErrMsg());         }         return null;     }      /**      * 发布配置      *      * @param logbackXml      * @return      */     public boolean publishLogBackConfig(String dataId, String group,String logbackXml) {         try {             ConfigService configService = nacosConfigService.getInstance();             boolean isPublishOk = configService.publishConfig(dataId, group, logbackXml, ConfigType.XML.getType());             return isPublishOk;         } catch (Exception e) {             log.error(e.getMessage());         }         return false;     }  } 

2、 ConfigService单例

package com.cherf.common.nacos;  import cn.hutool.log.Log; import cn.hutool.log.LogFactory; import com.alibaba.nacos.api.NacosFactory; import com.alibaba.nacos.api.PropertyKeyConst; import com.alibaba.nacos.api.config.ConfigService; import com.alibaba.nacos.api.exception.NacosException; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.stereotype.Component;  import java.util.Properties;  /**  * @author cherf  * @description: NacosConfigservice单例  * @date 2023/03/4 16:35  **/ @Component public class NacosConfigService {     private static final Log log = LogFactory.get(NacosConfigService.class);     /**      * nacos地址      */     @Value("${spring.cloud.nacos.discovery.server-addr}")     private  String ipAddress;      //声明变量, 使用volatile关键字确保绝对线程安全     private volatile  ConfigService configService = null;      @Bean     public  ConfigService getInstance() throws NacosException {         if (configService == null) {             //对单例类进行加锁             synchronized (NacosConfigService.class) {                 if (configService == null) {                     Properties properties = new Properties();                     // nacos服务器地址,127.0.0.1:8848                     properties.put(PropertyKeyConst.SERVER_ADDR, ipAddress);                     //创建实例                     configService = NacosFactory.createConfigService(properties);                     log.info("==========创建configService实例===============");                 }             }         }         return configService;     } }  

2.3.3、logback配置动态修改

新增租户后,需要在logback.xml里添加新增租户的配置信息,以我们的为例是在其中添加如下三段配置
一种通过nacos动态配置实现多租户的log4j2日志物理隔离的设计

中间较大的这一段可以写在Resource目录下,然后读出来替换{tenantId}即可使用,配置如下

    <!-- {tenantId} 租户日志 -->     <!--system日志-->     <appender class="com.cherf.common.logback.TenantRollingFileAppender" name="{tenantId}">         <file>${log.path}/{tenantId}/sys-api.log</file>         <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">             <fileNamePattern>${log.path}/{tenantId}/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern>             <!-- 日志最大的历史 60天 -->             <maxHistory>60</maxHistory>         </rollingPolicy>         <encoder>             <pattern>${log.pattern}</pattern>         </encoder>         <!-- 保留INFO级别及以上的日志 -->         <filter class="ch.qos.logback.classic.filter.ThresholdFilter">             <level>INFO</level>         </filter>     </appender> 

研究了半天DOM4J用法,感觉很麻烦,为了省事就直接使用字符串替换来完成了;先压缩读到的XML配置,然后替换需要新增或删除的配置信息,再格式化XML,最后再去发布
可以参考下面代码,包括了读取ini配置,XML压缩,XML格式化,从nacos获取配置到发布配置到nacos都有示例;(使用了最笨的方法来实现,有好的思路大家可以发出来探讨探讨)。

package com.cherf.common.logback;  import cn.hutool.core.io.IoUtil; import cn.hutool.core.io.resource.ClassPathResource; import cn.hutool.core.io.resource.Resource; import cn.hutool.log.Log; import cn.hutool.log.LogFactory; import com.cherf.common.constant.StringPool; import com.cherf.common.context.TenantContextHolder; import com.cherf.common.nacos.NacosDynamicLogbackService; import com.cherf.common.util.StringUtil; import com.isearch.common.logback.LogbackXmlContent; import com.sun.org.apache.xml.internal.serialize.OutputFormat; import com.sun.org.apache.xml.internal.serialize.XMLSerializer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import org.w3c.dom.Document; import org.xml.sax.InputSource; import org.xml.sax.SAXException;   import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import javax.xml.parsers.ParserConfigurationException; import java.io.*;  /**  * @author cherf  * @description:  * @date 2023/03/04 11:29  **/ @Component public class DynamicModifyLogback {     private static final Log log = LogFactory.get(DynamicModifyLogback.class);     //group     @Value("${logback.group}")     private String group;     //dataId     @Value("${logback.systemDataId}")     private String systemDataId;      @Autowired     private NacosDynamicLogbackService nacosDynamicLogbackService;      /**      * 新增配置      */     public void addLogbackXml() {         String logBackConfig = this.getLogBackConfig(systemDataId);         String addSystemXml = addSystemXml(logBackConfig);         //发布配置         publishLogBackConfig(systemDataId, addSystemXml);     }      /**      * 删除配置      */     public void removeLogbackXml() {         String logBackConfig = this.getLogBackConfig(systemDataId);         String removeSystemXml = removeSystemXml(logBackConfig);         //发布配置         publishLogBackConfig(systemDataId, removeSystemXml);     }       public static String addSystemXml(String logBackXml) {         String systemAppender = getIniResourec("logback/system-logback.ini").replace("{tenantId}", TenantContextHolder.getTenantId());         log.info("appender:", systemAppender);         String systemRef = "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId());         log.info("ref", systemRef);         return replaceLogBack(logBackXml, systemAppender);     }       public static String removeSystemXml(String logBackXml) {         String systemAppender = getIniResourec("logback/system-logback.ini").replace("{tenantId}", TenantContextHolder.getTenantId());         log.info("appender:", systemAppender);         //logBackXml = format(logBackXml);         //压缩xml         return packXml(logBackXml, systemAppender);     }       private static String replaceLogBack(String logBackXml, String appender) {         String appenderRep = LogbackXmlContent.appenderRep;         logBackXml = StringUtil.replaceLast(logBackXml, appenderRep, LogbackXmlContent.NULL + appenderRep + appender);         logBackXml = logBackXml.replace("<appender-ref ref="tid20220831114008942"/>", "<appender-ref ref="tid20220831114008942"/>" + StringPool.NEWLINE + "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId()));         logBackXml = format(logBackXml);         log.info(logBackXml);         return logBackXml;     }      private static String packXml(String logBackXml, String appender) {         logBackXml = convertFromXml(logBackXml).replace(StringPool.ELEVE_SPACE, StringPool.EMPTY).replace(StringPool.NEWLINE, StringPool.EMPTY).trim();         appender = convertFromXml(appender).trim();         String systemRef = "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId());         log.info("ref", systemRef);         logBackXml = StringUtil.replaceLast(logBackXml, appender, StringPool.EMPTY);         logBackXml = logBackXml.replace(systemRef, StringPool.EMPTY);         logBackXml = format(logBackXml);         log.info(logBackXml);         return logBackXml;     }       /**      * 获取配置      */     public String getLogBackConfig(String dataId) {         return nacosDynamicLogbackService.getLogBackConfig(dataId, group);     }      /**      * 发布      */     public Boolean publishLogBackConfig(String dataId, String logbackXml) {         return nacosDynamicLogbackService.publishLogBackConfig(dataId, group, logbackXml);     }      /**      * 格式化xml      *      * @param unformattedXml      * @return      */     public static String format(String unformattedXml) {         try {             final Document document = parseXmlFile(unformattedXml);             OutputFormat format = new OutputFormat(document);             format.setLineWidth(256);             format.setIndenting(true);             format.setIndent(2);             Writer out = new StringWriter();             XMLSerializer serializer = new XMLSerializer(out, format);             serializer.serialize(document);             return out.toString();         } catch (IOException e) {             throw new RuntimeException(e);         }     }      private static Document parseXmlFile(String in) {         try {             DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();             DocumentBuilder db = dbf.newDocumentBuilder();             InputSource is = new InputSource(new StringReader(in));             return db.parse(is);         } catch (ParserConfigurationException e) {             throw new RuntimeException(e);         } catch (SAXException e) {             throw new RuntimeException(e);         } catch (IOException e) {             throw new RuntimeException(e);         }     }      /**      * 获取配置      *      * @param fileName      * @return      */     private static String getIniResourec(String fileName) {         String xml = StringPool.EMPTY;         Resource resource = new ClassPathResource(fileName);         InputStream is = resource.getStream();         xml = IoUtil.readUtf8(is);         return xml;     }      /**      * 压缩xml      *      * @param str      * @return      */     public static String convertFromXml(String str) {         boolean flag = true;         boolean quotesFlag = true;         StringBuffer ans = new StringBuffer();         String tmp = "";         for (int i = 0; i < str.length(); i++) {             if ('"' == str.charAt(i)) {                 ans.append(str.charAt(i));                 quotesFlag = !quotesFlag;             } else if ('<' == str.charAt(i)) {                 tmp = tmp.trim();                 ans.append(tmp);                 flag = true;                 ans.append(str.charAt(i));             } else if ('>' == str.charAt(i)) {                 if (quotesFlag) {                     flag = false;                     ans.append(str.charAt(i));                     tmp = "";                 } else {                     ans.append("&gt;");                 }             } else if (flag) {                 ans.append(str.charAt(i));             } else {                 tmp += str.charAt(i);             }         }         return ans.toString();     } }  

3、总结

前面都还可以,只是由于时间关系,XML修改的方法确实有点挫,后期有时间再研究着改吧!

其中TenantContextHolder实现可以参考另一篇文章多租户改造(字段隔离和表隔离混合模式)

日志分离打印部分参考大佬实现:springboot logback多租户根据请求打印日志到不同文件

发表评论

评论已关闭。

相关文章