Lettuce闲置一段时间后连接失败 #

问题描述 #

环境：

java 8
springboot 2.5.5 (其中 redis 客户端默认使用 lettuce 6.1.5)
redis 5.0 (官方 docker 镜像，默认配置)

应用配置：

yml
spring:
  redis:
    host: xxx
    port: xxx
    password: xxx
    database: xxx

现象：

使用 redis 连接之后闲置一段时间（大概20分钟），再次连接时报错：远程主机强迫关闭了一个现有的连接。

text
2024-01-16 22:43:24.429 ERROR 15328 --- [nio-8082-exec-5] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

org.springframework.data.redis.RedisSystemException: Redis exception; nested exception is io.lettuce.core.RedisException: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
	at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:74) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceExceptionConverter.convert(LettuceExceptionConverter.java:41) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.PassThroughExceptionTranslationStrategy.translate(PassThroughExceptionTranslationStrategy.java:44) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.FallbackExceptionTranslationStrategy.translate(FallbackExceptionTranslationStrategy.java:42) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.convertLettuceAccessException(LettuceConnection.java:271) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1062) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.lambda$doInvoke$4(LettuceConnection.java:919) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceInvoker$Synchronizer.invoke(LettuceInvoker.java:665) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceInvoker.just(LettuceInvoker.java:94) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.lettuce.LettuceStringCommands.get(LettuceStringCommands.java:55) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:267) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:57) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:60) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:222) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:189) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:96) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:53) ~[spring-data-redis-2.5.5.jar:2.5.5]
	at com.jeeplus.common.redis.RedisUtils.get(RedisUtils.java:314) ~[classes/:na]
	......
	......
Caused by: io.lettuce.core.RedisException: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
	at io.lettuce.core.internal.Exceptions.bubble(Exceptions.java:83) ~[lettuce-core-6.1.5.RELEASE.jar:6.1.5.RELEASE]
	at io.lettuce.core.internal.Futures.awaitOrCancel(Futures.java:250) ~[lettuce-core-6.1.5.RELEASE.jar:6.1.5.RELEASE]
	at io.lettuce.core.LettuceFutures.awaitOrCancel(LettuceFutures.java:74) ~[lettuce-core-6.1.5.RELEASE.jar:6.1.5.RELEASE]
	at org.springframework.data.redis.connection.lettuce.LettuceConnection.await(LettuceConnection.java:1060) ~[spring-data-redis-2.5.5.jar:2.5.5]
	... 54 common frames omitted
Caused by: java.io.IOException: 远程主机强迫关闭了一个现有的连接。
	at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_171]
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_171]
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_171]
	at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_171]
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_171]
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) ~[netty-buffer-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132) ~[netty-buffer-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) ~[netty-transport-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.68.Final.jar:4.1.68.Final]
	... 1 common frames omitted

2024-01-16 22:43:24.510  INFO 15328 --- [xecutorLoop-1-3] i.l.core.protocol.ConnectionWatchdog     : Reconnecting, last destination was /192.168.1.100:6379
2024-01-16 22:43:24.576  INFO 15328 --- [ioEventLoop-4-2] i.l.core.protocol.ReconnectionHandler    : Reconnected to 192.168.1.100:6379

出现原因 #

网络波动导致服务器主动断开连接，而 lettuce 客户端没有发现连接异常，导致下一次从连接池获取连接时报错。

详见博客： https://www.cnblogs.com/songjiyang/p/16809694.html 详见博客： https://www.cnblogs.com/wingcode/p/14527107.html

解决方案 A #

应用添加配置（仅对集群有效，要求 springboot 2.3 及以上）：

yml
spring:
  redis:
    cluster:
      refresh:
        # 启用自适应拓扑刷新
        adaptive: true
        # 集群拓扑刷新间隔
        period: 20s

解决方案 B #

应用启用验证连接这项配置（每次使用连接时都会验证该连接是否可用，会影响性能）：

java
@Bean
public RedisTemplate<String, Object> redisTemplate(LettuceConnectionFactory factory) {
    RedisTemplate<String, Object> redisTemplate = new RedisTemplate<>();
    // 启用验证连接
    factory.setValidateConnection(true);
    redisTemplate.setConnectionFactory(factory);
    ...
    return redisTemplate;
}

解决方案 C #

使用 jedis 而不是默认的 lettuce（需要注意把 lettuce.pool 连接池配置改成 jedis.pool）：

xml
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
    <exclusions>
        <exclusion>
            <groupId>io.lettuce</groupId>
            <artifactId>lettuce-core</artifactId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>redis.clients</groupId>
    <artifactId>jedis</artifactId>
</dependency>

解决方案 D（待验证） #

给 lettuce 依赖的底层 netty 添加心跳机制（好像只是超时断连不是心跳？？？）：

java
@Bean
public ClientResources clientResources() {
    NettyCustomizer nettyCustomizer = new NettyCustomizer() {
        @Override
        public void afterChannelInitialized(Channel channel) {
            // 此处必须小于超时时间
            int readerIdleTimeSeconds = 40;
            // 0表示禁用
            int writerIdleTimeSeconds = 0;
            int allIdleTimeSeconds = 0;
            channel.pipeline().addLast(
                new IdleStateHandler(readerIdleTimeSeconds, writerIdleTimeSeconds, allIdleTimeSeconds));
            channel.pipeline().addLast(
                new ChannelDuplexHandler() {
                    @Override
                    public void userEventTriggered(ChannelHandlerContext ctx, Object evt) throws Exception {
                        if (evt instanceof IdleStateEvent) {
                            ctx.disconnect();
                        }
                    }
                }
            );
        }
    };

    return ClientResources.builder().nettyCustomizer(nettyCustomizer).build();
}

解决方案 E #

定时发送ping命令：

java
@Scheduled(fixedDelay = 60000)
public void redisPingTask() {
    redisTemplate.execute(new RedisCallback<Object>() {
        @Override
        public Object doInRedis(@NotNull RedisConnection connection) throws DataAccessException {
            return connection.ping();
        }
    });
}

或者定时验证连接：

java
@Scheduled(fixedDelay = 60000)
public void redisValidateConnectionTask() {
    lettuceConnectionFactory.validateConnection();
}