#1677  Unable to Push to Onedev Server (Reverse Proxy) (UPDATE)
Closed
Maximus opened 3 months ago

I have combed every other setting possible in the reverse proxy. Could it be an issue where OneDev is expecting a local url for an access attempt (push, pull) example: 127.0.0.1:6610. Like is there a place in a config file that states the url needed for an access attempt that I have to change to the domain? I am using Cloudflare if this helps.

Robin Shen commented 3 months ago

Are you able to access OneDev UI via your reverse proxy?

Robin Shen commented 3 months ago

Please make sure server url in administration / system settings is specified correctly.

Maximus commented 3 months ago

Yes, I am able to access the UI and have just switched the root URL to my domain. I still get a 403 returned immediately. Is a restart of the service required to take effect? Or might I have to set the localhost URL to redirect to the domain in my Hosts file?

Robin Shen commented 3 months ago

No need to restart and no change to hosts file is necessary.

I've never used Cloudflare before. If you can provide detailed instructions on how you've set up reverse proxy with that, I will go ahead to give it a try when I am free.

Maximus commented 3 months ago

Of course. In Cloudflare, you link your domain. Then, under "access", you launch the Zero Trust portal. Then, go to Access > Tunnels and create a tunnel. Install the tunnel on wherever you are running OneDev and under Public Hostname, choose a domain or subdomain to route your machine to (specify 127.0.0.1:6610 as the URL). You should then be able to access the UI online at your domain, but if you open a terminal with git on another machine and attempt to remotely access any git repo, a 403 is immediately returned. Let me know if you have any questions about replicating this.

Robin Shen commented 3 months ago

Just set up a test instance and configured http tunnel to OneDev and pull/push works as expected.

For this 403 error, is it possible that the user you are using to push does not actually have write permision over OneDev repository?

Maximus commented 3 months ago

That's the thing, it doesn't appear to even check my credentials. There is no option to enter them, but they are configured in my system anyways. Did you attempt to push/pull from another machine using the domain you tunneled to?

Maximus commented 3 months ago

UPDATE: So now I can enter my credentials, that works for some reason. No idea what I did. However, I now receive a 520 error and the message "send-pack: unexpected disconnect while reading sideband packet" after it writes objects. Image attached. Please advise, as I'm trying to get this up as soon as possible. Screenshot 2023-12-06 163129.png

Robin Shen commented 3 months ago

Yes I am pulling/pushing from a different machine. OneDev itself installed in bare metal mode on an ubuntu machine.

Maximus commented 3 months ago

Did you change any settings in Cloudflare? I do not believe I did, but I see from my research that a 520 is often associated with Cloudflare. Have you ever seen this error before and if so, do you have any insight into fixing it? Thank you

Maximus commented 3 months ago

Note: the error above only occurs when pushing. Pulling works fine.

Robin Shen commented 3 months ago

Just followed your suggestion to add a domain and create a tunnel. Did not touch any other settings.

Before trying with the tunnel option, I set up a nginx to forward traffic from port 80 to 6610 on the test instance. And add dns entry at Cloudflare side to forward https traffic to nginx. It also works without any issues.

Then I created a tunnel to map the host name directly to http://localhost:6610 without using nginx.

Robin Shen commented 3 months ago

You may try with the non-tunnel option (install nginx to forward traffic) to see if it works.

Maximus commented 3 months ago

I have set up Cloudflare Tunnel to talk to nginx on the local system (I don't want to port forward). I can still access the UI and pull, but not push. I've also gotten a couple different errors, which is odd. 413, 520, 500.

Maximus commented 3 months ago

I have also just discovered that I am not able to upload files from the UI. So it may be some storage issue. Error log attached below.

org.apache.wicket.WicketRuntimeException: Method onRequest of interface org.apache.wicket.behavior.IBehaviorListener targeted at org.apache.wicket.ajax.markup.html.form.AjaxButton$1@7d60e909 on component [AjaxButton [Component id = upload]] threw an exception at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:268) at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241) at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248) at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234) at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:917) at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64) at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:274) at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:231) at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:302) at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70) at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203) at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43) at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$177928374.GUICE$TRAMPOLINE() at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74) at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23) at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90) at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75) at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55) at io.onedev.server.web.DefaultWicketServlet$$EnhancerByGuice$$177928374.service() at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:458) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:372) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.lang.reflect.InvocationTargetException at jdk.internal.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258) ... 87 more Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at io.onedev.server.ee.clustering.DefaultClusterManager.getResult(DefaultClusterManager.java:289) at io.onedev.server.ee.clustering.DefaultClusterManager.runOnServer(DefaultClusterManager.java:295) at jdk.internal.reflect.GeneratedMethodAccessor157.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at com.sun.proxy.$Proxy31.runOnServer(Unknown Source) at io.onedev.server.entitymanager.impl.DefaultProjectManager.runOnActiveServer(DefaultProjectManager.java:1224) at jdk.internal.reflect.GeneratedMethodAccessor159.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:50) at com.sun.proxy.$Proxy29.runOnActiveServer(Unknown Source) at io.onedev.server.git.service.DefaultGitService.runOnProjectServer(DefaultGitService.java:95) at io.onedev.server.git.service.DefaultGitService.commit(DefaultGitService.java:612) at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23) at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90) at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18) at io.onedev.server.web.page.project.blob.ProjectBlobPage.uploadFiles(ProjectBlobPage.java:1546) at io.onedev.server.web.page.project.blob.BlobUploadPanel$2.onSubmit(BlobUploadPanel.java:84) at org.apache.wicket.ajax.markup.html.form.AjaxButton$1.onSubmit(AjaxButton.java:113) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior$AjaxFormSubmitter.onSubmit(AjaxFormSubmitBehavior.java:218) at org.apache.wicket.markup.html.form.Form.delegateSubmit(Form.java:1312) at org.apache.wicket.markup.html.form.Form.process(Form.java:976) at org.apache.wicket.markup.html.form.Form.onFormSubmitted(Form.java:797) at org.apache.wicket.ajax.form.AjaxFormSubmitBehavior.onEvent(AjaxFormSubmitBehavior.java:174) at org.apache.wicket.ajax.AjaxEventBehavior.respond(AjaxEventBehavior.java:155) at org.apache.wicket.ajax.AbstractDefaultAjaxBehavior.onRequest(AbstractDefaultAjaxBehavior.java:601) at jdk.internal.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.wicket.RequestListenerInterface.internalInvoke(RequestListenerInterface.java:258) at org.apache.wicket.RequestListenerInterface.invoke(RequestListenerInterface.java:241) at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.invokeListener(ListenerInterfaceRequestHandler.java:248) at org.apache.wicket.core.request.handler.ListenerInterfaceRequestHandler.respond(ListenerInterfaceRequestHandler.java:234) at org.apache.wicket.request.cycle.RequestCycle$HandlerExecutor.respond(RequestCycle.java:917) at org.apache.wicket.request.RequestHandlerStack.execute(RequestHandlerStack.java:64) at org.apache.wicket.request.cycle.RequestCycle.execute(RequestCycle.java:274) at org.apache.wicket.request.cycle.RequestCycle.processRequest(RequestCycle.java:231) at org.apache.wicket.request.cycle.RequestCycle.processRequestAndDetach(RequestCycle.java:302) at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:70) at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203) at org.apache.wicket.protocol.http.WicketServlet.doPost(WicketServlet.java:159) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43) at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23) at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90) at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18) ... 65 more Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.IOException: No space left on device at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:278) at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:102) Caused by: java.lang.RuntimeException: java.io.IOException: No space left on device at io.onedev.server.git.service.DefaultGitService$13.insertTree(DefaultGitService.java:731) at io.onedev.server.git.service.DefaultGitService$13.call(DefaultGitService.java:757) at io.onedev.server.git.service.DefaultGitService$13.call(DefaultGitService.java:612) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at com.hazelcast.executor.impl.DistributedExecutorService$Processor.run(DistributedExecutorService.java:276) ... 6 more Caused by: java.io.IOException: No space left on device at java.base/java.io.FileOutputStream.writeBytes(Native Method) at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354) at java.base/java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:262) at java.base/java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:211) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.toTemp(ObjectDirectoryInserter.java:217) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.insert(ObjectDirectoryInserter.java:79) at org.eclipse.jgit.internal.storage.file.ObjectDirectoryInserter.insert(ObjectDirectoryInserter.java:56) at org.eclipse.jgit.lib.ObjectInserter.insert(ObjectInserter.java:333) at io.onedev.server.git.service.DefaultGitService$13.insertTree(DefaultGitService.java:686) ... 10 more

Robin Shen commented 3 months ago

I noticed this error:

No space left on device

Seems that some file system is running out of space.

Maximus commented 3 months ago

The drive is 100GB and not nearly full. Does the default database have a storage limit?

Maximus commented 3 months ago

I just discovered the hard disk wasn't mounted. How can I change where OneDev stores files?

Robin Shen commented 3 months ago

If you are using internal database, database files reside under internaldb sub directory. All other files including git/index/build artifacts/issue attachment reside under site directory.

Maximus commented 3 months ago

I suppose the question I was asking is how would I go about changing those locations. Is there a command I have to run or just move the folders? Sorry, I haven't done self-hosting before.

Robin Shen commented 3 months ago

You can not change that. However you may create a symbol link for site pointing to other file system if necessary.

Maximus commented 3 months ago

I have created the link to the moved "site" folder, as shown below. However, I still receive the error "error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 send-pack: unexpected disconnect while reading sideband packet" when attempting to push from Terminal and now get the error "file is required" when attempting to upload manually. Screenshot 2023-12-07 195611.png

Robin Shen commented 3 months ago

I noticed you only have 1.5M free space on the OneDev file system. Can you move to a different file system with enough space? As even if site is point to other places, OneDev still needs some other space to store temp files to operate on.

Maximus commented 3 months ago

I have just re-started the server on the larger drive and still get an error message: "error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413 send-pack: unexpected disconnect while reading sideband packet fatal: the remote end hung up unexpectedly Everything up-to-date"

Maximus commented 3 months ago

But I am able to manually upload files now.

Robin Shen commented 3 months ago

Check nginx config to set a large value for property client_max_body_size or set to 0 for unlimited:

https://docs.onedev.io/administration-guide/reverse-proxy-setup#nginx

Maximus commented 3 months ago

Yes, the nginx file is configured in that manner.

Robin Shen commented 3 months ago

Are you able to push without using Cloudflare?

Maximus commented 3 months ago

I think I am going to switch to Tailscale. The website won't be public, but I won't have to worry about Cloudflare (it has a 100MB throttle)

Maximus commented 3 months ago

Tailscale appears to work. I believe it was Cloudflare that was hampering the file transfers. Thank you so much for your help.

Maximus changed state to 'Closed' 3 months ago
Previous Value Current Value
Open
Closed
issue 1 of 1
Type
Question
Priority
Critical
Assignee
Labels
No labels
Issue Votes (0)
Watchers (4)
Reference
onedev/server#1677
Please wait...
Page is in error, reload to recover