Enabling cache results in IOException: Error writing request body to server (OD-2043)

wojtek opened 1 year ago

For our android app build I tried setting up cache (first step)😃

  - !SetupCacheStep
    name: gradle-home-cache
    key: gradle-home-cache
    paths:
    - /root/.gradle/
    - /root/.m2/repository/
    uploadStrategy: UPLOAD_IF_CHANGED
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL

But build fails with the IOException. Unfortunatelly there is no info what the actual problem was:

18:17:01 Cache file changed: /agent/work/temp/onedev-build-1152-291/cache/1/wrapper/dists/gradle-8.4-all/56r6xik2f6skrm47et0ibifug/gradle-8.4-all.zip.lck
18:18:46 Error running job: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Error writing request body to server
    	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    	at io.onedev.server.job.JobExecution.check(JobExecution.java:55)
    	at io.onedev.server.job.DefaultJobManager.lambda$run$12(DefaultJobManager.java:1240)
    	at io.onedev.server.persistence.DefaultSessionManager.lambda$run$0(DefaultSessionManager.java:108)
    	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90)
    	at io.onedev.server.persistence.DefaultSessionManager.run(DefaultSessionManager.java:107)
    	at io.onedev.server.job.DefaultJobManager.run(DefaultJobManager.java:1228)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Error writing request body to server
    	at io.onedev.server.job.DefaultJobManager.runJob(DefaultJobManager.java:1332)
    	at io.onedev.server.job.DefaultResourceAllocator.runAgentJob(DefaultResourceAllocator.java:321)
    	at io.onedev.server.plugin.executor.remotedocker.RemoteDockerExecutor.execute(RemoteDockerExecutor.java:143)
    	at io.onedev.server.job.DefaultJobManager.lambda$execute$3(DefaultJobManager.java:585)
    	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    	at io.onedev.server.security.SecurityUtils.lambda$inheritSubject$1(SecurityUtils.java:506)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	... 1 more
    Caused by: java.util.concurrent.ExecutionException: javax.ws.rs.ProcessingException: java.io.IOException: Error writing request body to server
    	at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    	at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    	at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:278)
    	at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    	at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
    	at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
    Caused by: javax.ws.rs.ProcessingException: java.io.IOException: Error writing request body to server
    	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:312)
    	at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
    	at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
    	at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
    	at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
    	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
    	at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
    	at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
    	at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:439)
    	at org.glassfish.jersey.client.JerseyInvocation$Builder.post(JerseyInvocation.java:345)
    	at io.onedev.k8shelper.KubernetesHelper.uploadCache(KubernetesHelper.java:1198)
    	at io.onedev.agent.AgentCacheHelper.uploadCache(AgentCacheHelper.java:36)
    	at io.onedev.k8shelper.CacheHelper.uploadCacheThenLog(CacheHelper.java:88)
    	at io.onedev.k8shelper.CacheHelper.buildFinished(CacheHelper.java:105)
    	at io.onedev.agent.AgentSocket.executeDockerJob(AgentSocket.java:814)
    	at io.onedev.agent.AgentSocket.service(AgentSocket.java:974)
    	at io.onedev.agent.AgentSocket.lambda$onMessage$2(AgentSocket.java:206)
    	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    	at java.base/java.lang.Thread.run(Thread.java:829)
    Caused by: java.lang.RuntimeException: java.io.IOException: Error writing request body to server
    	at io.onedev.commons.utils.TarUtils.tar(TarUtils.java:69)
    	at io.onedev.k8shelper.CacheHelper.tar(CacheHelper.java:129)
    	at io.onedev.k8shelper.KubernetesHelper.lambda$uploadCache$5(KubernetesHelper.java:1197)
    	at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:55)
    	at org.glassfish.jersey.message.internal.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:37)
    	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:242)
    	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:227)
    	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:139)
    	at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1116)
    	at org.glassfish.jersey.client.ClientRequest.doWriteEntity(ClientRequest.java:461)
    	at org.glassfish.jersey.client.ClientRequest.writeEntity(ClientRequest.java:443)
    	at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:371)
    	at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:268)
    	at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:300)
    	... 21 more
    Caused by: java.io.IOException: Error writing request body to server
    	at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3685)
    	at java.base/sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3668)
    	at org.glassfish.jersey.message.internal.CommittingOutputStream.write(CommittingOutputStream.java:200)
    	at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$UnCloseableOutputStream.write(WriterInterceptorExecutor.java:276)
    	at com.google.common.io.CountingOutputStream.write(CountingOutputStream.java:54)
    	at java.base/java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81)
    	at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:127)
    	at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:262)
    	at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211)
    	at java.base/java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:146)
    	at org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:147)
    	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream$BufferAtATimeOutputChannel.write(FixedLengthBlockOutputStream.java:93)
    	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.writeBlock(FixedLengthBlockOutputStream.java:266)
    	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.maybeFlush(FixedLengthBlockOutputStream.java:176)
    	at org.apache.commons.compress.utils.FixedLengthBlockOutputStream.write(FixedLengthBlockOutputStream.java:213)
    	at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.write(TarArchiveOutputStream.java:669)
    	at io.onedev.commons.utils.TarUtils.addTarEntry(TarUtils.java:98)
    	at io.onedev.commons.utils.TarUtils$1.visitFile(TarUtils.java:53)
    	at io.onedev.commons.utils.TarUtils$1.visitFile(TarUtils.java:36)
    	at java.base/java.nio.file.Files.walkFileTree(Files.java:2725)
    	at java.base/java.nio.file.Files.walkFileTree(Files.java:2797)
    	at io.onedev.commons.utils.TarUtils.tar(TarUtils.java:36)
    	... 34 more

Activities

Robin Shen commented 1 year ago

Most probably the upload file size is too large to be uploaded. Please configure your frondend proxy to increase max accepted body size, for instance, client_max_body_size in Nginx can be used to control this.
wojtek commented 1 year ago

Indeed, it seemed to be the problem and the cache size totalled to ~2G.

Shouldn't the IOException contain the information what the issue was (response from the server) as a hint?
Robin Shen commented 1 year ago

Seems that the stack is too long to be captured fully.
wojtek commented 1 year ago

Understood.
wojtek changed state to 'Closed' 1 year ago
Previous Value Current Value
Open

Closed
Login to comment

Previous Value	Current Value
Open	Closed

Type	Question
Priority	Normal
Assignee	Robin Shen
Labels	No labels

Issue Votes (0)

Watchers (2)

Reference

OD-2043