1、背景
1.1、背景
旧服务改造为多租户服务后,log4j日志打印在一起不能区分是哪个租户的,日志太多,太杂,不好定位排除问题,排查问题较难。
1.2、前提
不改动以前的日志代码(工作量太大)
1.3、打印日志示例
package com.cherf.sauth.controller; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import com.cherf.common.ResultVo; import io.swagger.annotations.ApiOperation; import org.springframework.web.bind.annotation.PostMapping; import org.springframework.web.bind.annotation.RequestMapping; import org.springframework.web.bind.annotation.RestController; /** * @author cherf * @description: test * @date 2023/03/01 17:33 **/ @RestController @RequestMapping("/v1") public class TestController { private static Logger log = LoggerFactory.getLogger(TestController.class); @PostMapping("/test") @ApiOperation(value = "test", notes = "test") public ResultVo<?> authorization() { log.trace("test:{}", "trace"); log.debug("test:{}", "debug"); log.info("test:{}", "info"); log.warn("test:{}", "warn"); log.error("test:{}", "error"); return ResultVo.ok(); } }
2、实现
2.1、版本依赖
nacos: 2.1.0
slf4j-api: 1.7.36
slf4j-log4j12: 1.7.36
spring-boot:2.6.14
spring-cloud:2021.0.1
spring-cloud-alibaba:2021.0.1.0
(注: log4j-api和 logback-core用的是 spring-boot-starter-test 2.6.14中的版本分别是 2.17.2 1.2.11)
2.2、实现思路
2.2.1、日志分租户打印
logback通过加载ogback.xml 配置,通过 Appender 接口来实现打印,原理请看:logback
通过跟踪源码,可以找到AppenderAttachableImpl 这个类,其中通过Appender.doAppend 方法实现根据logabck配置中的类(默认是:RollingFileAppender)来打印日志,我们需要自定义重写doAppend方法来实现
2.2.2、logback配置动态生效
通过JoranConfiguration来实现 。Joran 是 logback 使用的一个配置加载库,动态生效 logback 的配置可以通过joranConfigurator.doConfigure方法实现,(实现代码在下面);
2.2.3、新增租户时新增logback配置
主要思路是通过nacos来动态修改和发布配置从而实现logback.xml动态修改;XML格式比较烦,如果配置较多可以使用DOM4J来实现修改XML;(我们的较为简单,所以只是通过字符串替换来实现~)
2.3、实现
2.3.1、日志分离打印
2.3.1.1 重写doAppend方法
日志分离打印主要的实现就是重写doAppend方法,示例如下:
其中TenantContextHolder是用来存放租户id本地变量,实现可参考:多租户改造(字段隔离和表隔离混合模式)(也可使用自己的方法)
package com.cherf.common.logback; import ch.qos.logback.core.rolling.RollingFileAppender; import com.cherf.common.context.TenantContextHolder; import com.cherf.common.util.StringUtil; /** * @author cherf * @description:logback复写 * @date 2023/02/24 11:49 **/ public class TenantRollingFileAppender<E> extends RollingFileAppender<E> { /** * 日志打印会调用此方法,进行复写,判断租户,根据租户打印到不同日志文件 * * @param eventObject */ public void doAppend(E eventObject) { String tenantId = TenantContextHolder.getTenantId(); if (StringUtil.isBlank(tenantId)) { //没有租户id的日志,打印到public下面 tenantId = "public"; } // this.getName() 是在logback.xml中配置的<appender name="appenderName" class="com.cherf.common.logback.TenantRollingFileAppender"> // 只打印当前租户的Append,RollingFileAppender追加器以租户类型标识开头的执行追加 if (this.getName().startsWith(tenantId)) { super.doAppend(eventObject); } } }
2.3.1.2 logback配置示例
logback.xml需要将配置中的appender标签的class属性修改为刚刚重写的方法全限定类名:com.cherf.common.logback.TenantRollingFileAppender
<?xml version="1.0" encoding="UTF-8"?> <configuration> <property name="log.path" value="../../logs/system" /> <property name="log.pattern" value="%date [%level] [%thread] %logger{80} [%file : %line] %msg%n" /> <appender class="ch.qos.logback.core.ConsoleAppender" name="STDOUT"> <encoder> <pattern>${log.pattern}</pattern> </encoder> </appender> <!-- public 公共--> <appender class="com.cherf.common.logback.TenantRollingFileAppender" name="public"> <file>${log.path}/public/sys-api.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${log.path}/public/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern> <!-- 日志最大的历史 60天 --> <maxHistory>60</maxHistory> </rollingPolicy> <encoder> <pattern>${log.pattern}</pattern> </encoder> <!-- 保留INFO级别及以上的日志 --> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>INFO</level> </filter> </appender> <!-- 默认租户 --> <appender class="com.cherf.common.logback.TenantRollingFileAppender" name="tid20220831114008942"> <file>${log.path}/tid20220831114008942/sys-api.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${log.path}/tid20220831114008942/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern> <!-- 日志最大的历史 60天 --> <maxHistory>60</maxHistory> </rollingPolicy> <encoder> <pattern>${log.pattern}</pattern> </encoder> <!-- 保留INFO级别及以上的日志 --> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>INFO</level> </filter> </appender> <logger additivity="false" name="com.cherf.system"> <appender-ref ref="public" /> <appender-ref ref="STDOUT" /> <appender-ref ref="tid20220831114008942" /> </logger> <root level="INFO"> <appender-ref ref="public" /> <appender-ref ref="STDOUT" /> <appender-ref ref="tid20220831114008942" /> </root> </configuration>
2.3.2、配置动态生效
2.3.2.1、项目启动读取nacos配置
服务启动时logback默认加载 classpath:logback.xml 配置,需要在yml中指定logback配置 (logging.config后配置)
配置如下:
spring: cloud: nacos: discovery: # 不使用nacos的配置 # enabled: false server-addr: 127.0.0.1:8848 #日志打印 logging: config: http://${spring.cloud.nacos.discovery.server-addr}/nacos/v1/cs/configs?group=${logback.group}&tenant=public&dataId=${logback.systemDataId} level: com.cherf: info com.cherf.mapper: info org.springframework: info org.spring.springboot.dao: info logback: group: logback systemDataId: system-logback.xml
2.3.2.2、nacos配置监听+logback动态加载配置
主要采用nacos ConfigService的addListener方法来监听;
注意:网上很多直接通过 NacosFactory.createConfigService()来创建ConfigService的方法可能会重复创建实例,导致CPU上升,详情可参考:记一次CPU占用持续上升问题排查(Nacos动态路由引起)
1、nacos动态配置监听器
package com.cherf.common.nacos; import ch.qos.logback.classic.LoggerContext; import ch.qos.logback.classic.joran.JoranConfigurator; import com.alibaba.nacos.api.config.ConfigService; import com.alibaba.nacos.api.config.ConfigType; import com.alibaba.nacos.api.config.listener.Listener; import com.alibaba.nacos.api.exception.NacosException; import com.cherf.common.nacos.NacosConfigService; import com.cherf.common.util.StringUtil; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import javax.annotation.PostConstruct; import javax.annotation.Resource; import java.io.ByteArrayInputStream; import java.io.InputStream; import java.util.concurrent.Executor; /** * @author cherf * @description: nacos监听器,修改logback.xml后动态生效 * @date 2023/03/04 16:35 **/ @Component public class NacosDynamicLogbackService { private static final Logger log = LoggerFactory.getLogger(NacosDynamicLogbackService.class); /** * 配置 ID */ @Value("${logback.systemDataId}") private String dataId; /** * 配置 分组 */ @Value("${logback.group}") private String group; @Resource private NacosConfigService nacosConfigService; @PostConstruct public void dynamicLogbackByNacosListener() { try { ConfigService configService = nacosConfigService.getInstance(); if (configService != null) { configService.getConfig(dataId, group, 5000); configService.addListener(dataId, group, new Listener() { @Override public void receiveConfigInfo(String configInfo) { if (StringUtil.isNotBlank(configInfo)) { System.out.println("configInfo=============================>" + configInfo); try { LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory(); JoranConfigurator configurator = new JoranConfigurator(); configurator.setContext(context); context.reset(); //获取nacos配置,生成inputStream InputStream inputStreamRoute = new ByteArrayInputStream(new String(configInfo).getBytes()); //configurator.doConfigure("/logback.xml"); configurator.doConfigure(inputStreamRoute); context.start(); } catch (Exception e) { log.error("加载logback.xml配置发生错误", e); } } } @Override public Executor getExecutor() { return null; } }); } } catch (NacosException e) { log.error("获取logback.xml配置发生错误", e); } } /** * 获取配置文件内容 * * @return */ public String getLogBackConfig(String dataId, String group) { try { ConfigService configService = nacosConfigService.getInstance(); // 根据dataId、group定位到具体配置文件,获取其内容. 方法中的三个参数分别是: dataId, group, 超时时间 String content = configService.getConfig(dataId, group, 5000L); return content; } catch (NacosException e) { log.error(e.getErrMsg()); } return null; } /** * 发布配置 * * @param logbackXml * @return */ public boolean publishLogBackConfig(String dataId, String group,String logbackXml) { try { ConfigService configService = nacosConfigService.getInstance(); boolean isPublishOk = configService.publishConfig(dataId, group, logbackXml, ConfigType.XML.getType()); return isPublishOk; } catch (Exception e) { log.error(e.getMessage()); } return false; } }
2、 ConfigService单例
package com.cherf.common.nacos; import cn.hutool.log.Log; import cn.hutool.log.LogFactory; import com.alibaba.nacos.api.NacosFactory; import com.alibaba.nacos.api.PropertyKeyConst; import com.alibaba.nacos.api.config.ConfigService; import com.alibaba.nacos.api.exception.NacosException; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.stereotype.Component; import java.util.Properties; /** * @author cherf * @description: NacosConfigservice单例 * @date 2023/03/4 16:35 **/ @Component public class NacosConfigService { private static final Log log = LogFactory.get(NacosConfigService.class); /** * nacos地址 */ @Value("${spring.cloud.nacos.discovery.server-addr}") private String ipAddress; //声明变量, 使用volatile关键字确保绝对线程安全 private volatile ConfigService configService = null; @Bean public ConfigService getInstance() throws NacosException { if (configService == null) { //对单例类进行加锁 synchronized (NacosConfigService.class) { if (configService == null) { Properties properties = new Properties(); // nacos服务器地址,127.0.0.1:8848 properties.put(PropertyKeyConst.SERVER_ADDR, ipAddress); //创建实例 configService = NacosFactory.createConfigService(properties); log.info("==========创建configService实例==============="); } } } return configService; } }
2.3.3、logback配置动态修改
新增租户后,需要在logback.xml里添加新增租户的配置信息,以我们的为例是在其中添加如下三段配置

中间较大的这一段可以写在Resource目录下,然后读出来替换{tenantId}即可使用,配置如下
<!-- {tenantId} 租户日志 --> <!--system日志--> <appender class="com.cherf.common.logback.TenantRollingFileAppender" name="{tenantId}"> <file>${log.path}/{tenantId}/sys-api.log</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>${log.path}/{tenantId}/sys-api.%d{yyyy-MM-dd}.log</fileNamePattern> <!-- 日志最大的历史 60天 --> <maxHistory>60</maxHistory> </rollingPolicy> <encoder> <pattern>${log.pattern}</pattern> </encoder> <!-- 保留INFO级别及以上的日志 --> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>INFO</level> </filter> </appender>
研究了半天DOM4J用法,感觉很麻烦,为了省事就直接使用字符串替换来完成了;先压缩读到的XML配置,然后替换需要新增或删除的配置信息,再格式化XML,最后再去发布
可以参考下面代码,包括了读取ini配置,XML压缩,XML格式化,从nacos获取配置到发布配置到nacos都有示例;(使用了最笨的方法来实现,有好的思路大家可以发出来探讨探讨)。
package com.cherf.common.logback; import cn.hutool.core.io.IoUtil; import cn.hutool.core.io.resource.ClassPathResource; import cn.hutool.core.io.resource.Resource; import cn.hutool.log.Log; import cn.hutool.log.LogFactory; import com.cherf.common.constant.StringPool; import com.cherf.common.context.TenantContextHolder; import com.cherf.common.nacos.NacosDynamicLogbackService; import com.cherf.common.util.StringUtil; import com.isearch.common.logback.LogbackXmlContent; import com.sun.org.apache.xml.internal.serialize.OutputFormat; import com.sun.org.apache.xml.internal.serialize.XMLSerializer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; import org.w3c.dom.Document; import org.xml.sax.InputSource; import org.xml.sax.SAXException; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import javax.xml.parsers.ParserConfigurationException; import java.io.*; /** * @author cherf * @description: * @date 2023/03/04 11:29 **/ @Component public class DynamicModifyLogback { private static final Log log = LogFactory.get(DynamicModifyLogback.class); //group @Value("${logback.group}") private String group; //dataId @Value("${logback.systemDataId}") private String systemDataId; @Autowired private NacosDynamicLogbackService nacosDynamicLogbackService; /** * 新增配置 */ public void addLogbackXml() { String logBackConfig = this.getLogBackConfig(systemDataId); String addSystemXml = addSystemXml(logBackConfig); //发布配置 publishLogBackConfig(systemDataId, addSystemXml); } /** * 删除配置 */ public void removeLogbackXml() { String logBackConfig = this.getLogBackConfig(systemDataId); String removeSystemXml = removeSystemXml(logBackConfig); //发布配置 publishLogBackConfig(systemDataId, removeSystemXml); } public static String addSystemXml(String logBackXml) { String systemAppender = getIniResourec("logback/system-logback.ini").replace("{tenantId}", TenantContextHolder.getTenantId()); log.info("appender:", systemAppender); String systemRef = "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId()); log.info("ref", systemRef); return replaceLogBack(logBackXml, systemAppender); } public static String removeSystemXml(String logBackXml) { String systemAppender = getIniResourec("logback/system-logback.ini").replace("{tenantId}", TenantContextHolder.getTenantId()); log.info("appender:", systemAppender); //logBackXml = format(logBackXml); //压缩xml return packXml(logBackXml, systemAppender); } private static String replaceLogBack(String logBackXml, String appender) { String appenderRep = LogbackXmlContent.appenderRep; logBackXml = StringUtil.replaceLast(logBackXml, appenderRep, LogbackXmlContent.NULL + appenderRep + appender); logBackXml = logBackXml.replace("<appender-ref ref="tid20220831114008942"/>", "<appender-ref ref="tid20220831114008942"/>" + StringPool.NEWLINE + "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId())); logBackXml = format(logBackXml); log.info(logBackXml); return logBackXml; } private static String packXml(String logBackXml, String appender) { logBackXml = convertFromXml(logBackXml).replace(StringPool.ELEVE_SPACE, StringPool.EMPTY).replace(StringPool.NEWLINE, StringPool.EMPTY).trim(); appender = convertFromXml(appender).trim(); String systemRef = "<appender-ref ref="tid20220831114008942"/>".replace("tid20220831114008942", TenantContextHolder.getTenantId()); log.info("ref", systemRef); logBackXml = StringUtil.replaceLast(logBackXml, appender, StringPool.EMPTY); logBackXml = logBackXml.replace(systemRef, StringPool.EMPTY); logBackXml = format(logBackXml); log.info(logBackXml); return logBackXml; } /** * 获取配置 */ public String getLogBackConfig(String dataId) { return nacosDynamicLogbackService.getLogBackConfig(dataId, group); } /** * 发布 */ public Boolean publishLogBackConfig(String dataId, String logbackXml) { return nacosDynamicLogbackService.publishLogBackConfig(dataId, group, logbackXml); } /** * 格式化xml * * @param unformattedXml * @return */ public static String format(String unformattedXml) { try { final Document document = parseXmlFile(unformattedXml); OutputFormat format = new OutputFormat(document); format.setLineWidth(256); format.setIndenting(true); format.setIndent(2); Writer out = new StringWriter(); XMLSerializer serializer = new XMLSerializer(out, format); serializer.serialize(document); return out.toString(); } catch (IOException e) { throw new RuntimeException(e); } } private static Document parseXmlFile(String in) { try { DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance(); DocumentBuilder db = dbf.newDocumentBuilder(); InputSource is = new InputSource(new StringReader(in)); return db.parse(is); } catch (ParserConfigurationException e) { throw new RuntimeException(e); } catch (SAXException e) { throw new RuntimeException(e); } catch (IOException e) { throw new RuntimeException(e); } } /** * 获取配置 * * @param fileName * @return */ private static String getIniResourec(String fileName) { String xml = StringPool.EMPTY; Resource resource = new ClassPathResource(fileName); InputStream is = resource.getStream(); xml = IoUtil.readUtf8(is); return xml; } /** * 压缩xml * * @param str * @return */ public static String convertFromXml(String str) { boolean flag = true; boolean quotesFlag = true; StringBuffer ans = new StringBuffer(); String tmp = ""; for (int i = 0; i < str.length(); i++) { if ('"' == str.charAt(i)) { ans.append(str.charAt(i)); quotesFlag = !quotesFlag; } else if ('<' == str.charAt(i)) { tmp = tmp.trim(); ans.append(tmp); flag = true; ans.append(str.charAt(i)); } else if ('>' == str.charAt(i)) { if (quotesFlag) { flag = false; ans.append(str.charAt(i)); tmp = ""; } else { ans.append(">"); } } else if (flag) { ans.append(str.charAt(i)); } else { tmp += str.charAt(i); } } return ans.toString(); } }
3、总结
前面都还可以,只是由于时间关系,XML修改的方法确实有点挫,后期有时间再研究着改吧!
其中TenantContextHolder实现可以参考另一篇文章多租户改造(字段隔离和表隔离混合模式)
日志分离打印部分参考大佬实现:springboot logback多租户根据请求打印日志到不同文件