nginx temp_proxy 权限导致报错

Posted on:September 19, 2023 at 06:47 PM

背景

线上有个服务,本来请求路径是 PHP -> NGINX -> PHP 需要做项目迁移,迁移到 PHP-> NGINX -> JAVA, 也就是将请求的服务从php改成请求java

排查

错误复现

上线后php请求java的接口报错cURL error 18: transfer closed with outstanding read data remaining , 排查之后发现java有个socket rest

排查java 错误

查看java日志,线上发现有rst错误,请求只有不够1秒,但是被rst掉了,所以排除了接口太慢导致超时的问题

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer

        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:353)

        at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:783)

        at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:688)

        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:388)

        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:366)

        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:96)

        at org.springframework.util.StreamUtils$NonClosingOutputStream.write(StreamUtils.java:287)

        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._flushBuffer(UTF8JsonGenerator.java:2161)

        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment2(UTF8JsonGenerator.java:1476)

        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegment(UTF8JsonGenerator.java:1423)

        at com.fasterxml.jackson.core.json.UTF8JsonGenerator._writeStringSegments(UTF8JsonGenerator.java:1306)

        at com.fasterxml.jackson.core.json.UTF8JsonGenerator.writeString(UTF8JsonGenerator.java:502)

        at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)

        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)

        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)

        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)

        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:726)

        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeWithoutTypeInfo(MapSerializer.java:681)

        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:637)

        at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)

        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)

        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)

        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serializeContents(IndexedListSerializer.java:119)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:79)

        at com.fasterxml.jackson.databind.ser.impl.IndexedListSerializer.serialize(IndexedListSerializer.java:18)

        at com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:728)

        at com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:755)

        at com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:178)

        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)

        at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)

        at com.fasterxml.jackson.databind.ObjectWriter$Prefetch.serialize(ObjectWriter.java:1516)

        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1006)

        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:346)

        at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104)

        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:277)

        at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:181)

        at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:82)

        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:123)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:878)

        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:792)

        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)

        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)

        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)

        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:652)

        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:733)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)

        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)

        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)

        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)

        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1707)

        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.base/java.lang.Thread.run(Thread.java:834)

Caused by: java.io.IOException: Connection reset by peer

        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)

        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)

        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)

        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)

        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)

        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)

        at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135)

        at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:118)

        at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:151)

        at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1367)

        at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:766)

        at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:586)

        at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:530)

        at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:546)

        at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:84)

        at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:193)

        at org.apache.coyote.Response.doWrite(Response.java:606)

        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:340)

        ... 73 common frames omitted

排查nginx日志

proxy_temp permission denied , 所以问题找到了,就是nginx权限有问题,没有权限创建proxy_temp 文件

原因: 每个请求都会分配一个页的缓冲区.如果超过一个页8kb/16kb,就会将内容存到proxy_temp文件里面,因为没有权限创建这个文件,导致请求直接被断开

解决方式

当前方案: 修改到当前nginx的用户组可以创建proxy_temp文件

可选方案: 如果不想写文件,嫌弃写文件有性能问题,可以调节

When buffering is enabled, nginx receives a response from the proxied server as soon as possible, saving it into the buffers set by the proxy_buffer_size and proxy_buffers directives. If the whole response does not fit into memory, a part of it can be saved to a temporary file on the disk. Writing to temporary files is controlled by the proxy_max_temp_file_size and proxy_temp_file_write_size directives.

上线后:

相关阅读