Kafka不定期挂掉原因分析

journalctl -u kafka |grep ERROR

根据错误日志筛选分析 最近的错误信息如下：

Feb 25 10:03:54 iZ8vb9f7hiv5r7dqd2ub3xZ kafka-server-start.sh[22495]: [2021-02-25 10:03:54,957] ERROR Failed to clean up log for __consumer_offsets-8 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
Feb 25 10:03:54 iZ8vb9f7hiv5r7dqd2ub3xZ kafka-server-start.sh[22495]: [2021-02-25 10:03:54,984] ERROR Failed to clean up log for __consumer_offsets-8 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
Feb 25 10:03:55 iZ8vb9f7hiv5r7dqd2ub3xZ kafka-server-start.sh[22495]: [2021-02-25 10:03:55,039] ERROR Failed to clean up log for __consumer_offsets-8 in dir /tmp/kafka-logs due to IOException (kafka.server.LogDirFailureChannel)
Feb 25 10:03:55 iZ8vb9f7hiv5r7dqd2ub3xZ kafka-server-start.sh[22495]: [2021-02-25 10:03:55,036] ERROR Shutdown broker because all log dirs in /tmp/kafka-logs have failed (kafka.log.LogManager)

日志分析：

没找到 /tmp/kafka-logs 下面的某个文件

查找思路：

vi /opt/kafka/config/server.properties 

根据 kafka配置信息搜索这个目录；发现是/tmp/kafka-logs  是日志存储位置

问题原因：

/tmp目录可能被系统不定时清理；清理后会丢失文件；kafka报错

解决方案：

1.修改 systemd的配置文件 忽略kafka日志目录

vi /usr/lib/tmpfiles.d/tmp.conf

添加一个忽略路径

X /tmp/kafka-logs

2.修改kafka 配置文件中 log.dirs