Can't publish artifacts bigger than 50M (OD-916)

Closed

Activities

wojtek opened 2 years ago

When tryinh to publish artifact bigger than 50M I'm getting error "413 Request Entity Too Large" from nginx

15:38:28java.lang.RuntimeException: Http request failed (status code: 413, error message: <html>
15:38:28    <head><title>413 Request Entity Too Large</title></head>
15:38:28    <body>
15:38:28    <center><h1>413 Request Entity Too Large</h1></center>
15:38:28    <hr><center>nginx</center>
15:38:28    </body>
15:38:28    </html>
15:38:28    )
15:38:28    	at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:607)
15:38:28    	at io.onedev.k8shelper.KubernetesHelper.runServerSideStep(KubernetesHelper.java:1181)
15:38:28    	at io.onedev.k8shelper.KubernetesHelper.runServerStep(KubernetesHelper.java:1138)
15:38:28    	at io.onedev.k8shelper.RunServerSideStep.main(RunServerSideStep.java:25)
15:38:28    
15:38:29Step "Publish artifacts is failed: Command failed with exit code 1

We are running custom k8s chart, so I already increased nginx's maximum body size in ingress: nginx.ingress.kubernetes.io/proxy-body-size: 200m but still the issue persists.

I dug a bit in the sources of PublishArtifact step (https://github.com/theonedev/onedev/blob/main/server-core/src/main/java/io/onedev/server/buildspec/step/PublishArtifactStep.java#L76-L76) and it seems like a regular copy… (though, we are using Kubernetes Executor, so there's that).

I though that file size limit from the performance options may be the problem (though publication of artifacts smaller than 50M works just fine) but after increasing this option from 20M to 100M it still doesn't work. However, manual upload now doesn't complain about file size limit, though it fails when trying to submit form just the same:

org.apache.wicket.WicketRuntimeException: Method onRequest of interface org.apache.wicket.behavior.IBehaviorListener targeted at org.apache.wicket.ajax.markup.html.form.AjaxButton$1@53cd7c6b on component [AjaxButton [Component id = upload]] threw an exception
	at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:268)
	at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241)
	at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248)
	at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234)
	at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:917)
	at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64)
	at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:274)
	at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:231)
	at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:302)
	at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70)
	at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203)
	at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$4c7ce51.CGLIB$service$0(<generated>)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$4c7ce51$$FastClassByGuice$$e35a994d.invoke(<generated>)
	at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:76)
	at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23)
	at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:79)
	at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18)
	at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:78)
	at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:54)
	at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$4c7ce51.service(<generated>)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
	at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:440)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:336)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at io.onedev.server.util.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258)
	... 89 more
Caused by: java.lang.NullPointerException
	at io.onedev.server.web.page.project.builds.detail.artifacts.ArtifactUploadPanel$2.onSubmit(ArtifactUploadPanel.java:81)
	at org.apache.wicket.ajax.markup.html.form.AjaxButton$1.onSubmit(AjaxButton.java:113)
	at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218)
	at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312)
	at org.apache.wicket.markup.html.form.Form.process(Form.java:976)
	at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797)
	at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:174)
	at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155)
	at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601)
	... 93 more

Any hint where this 50M could be configured would be highly appreciated.

Robin Shen changed state to 'Closed' 2 years ago

Previous Value	Current Value
Open	Closed

Robin Shen commented 2 years ago

I am using below values.yaml and it works. Please check your custom deploy config to make sure your setting is taking effect.

...
ingress:
  ## Optionally specify dns name to access OneDev via ingress controller
  host: "test.onedev.io"
  ## Whether or not to enable TLS for above host
  tls: false
  ## Will be used as ingressClassName of ingress spec to match controller
  class: nginx
  ## Set to false to disable the default certificate issuer from this chart if you want to use a custom one
  enableDefaultIssuer: true
  ## Custom annotations for the ingress
  annotations:
    # kubernetes.io/tls-acme: "true"
    # cert-manager.io/issuer: ""
    # cert-manager.io/cluster-issuer: ""
    nginx.ingress.kubernetes.io/proxy-body-size: "100m"
...

The NPE when manually upload an artifact is a bug though which will be fixed in issue #917

wojtek commented 2 years ago

I am using below values.yaml and it works. Please check your custom deploy config to make sure your setting is taking effect.

OK, the problem was... weird and I think it boils down to 1dev sending everything as one request hence somewhat "arbitrary" 50M limit, which didn't match anything... but - we had nginx.ingress.kubernetes.io/proxy-body-size: "200m", and we were producing 4 files that were subsequently deployed thus in the end we hit that 200M limit as a sum of size of all files...

Type	Bug
Priority	Normal
Assignee	Robin Shen
Affected Versions	Not Found

Issue Votes (0)

Watchers (3)

Reference

OD-916