Ngnix Reverse proxy env : Error serving request, should have either 'pageId' (OD-1060)

Jiung opened 3 years ago

I am using Nginx Proxy Manager to connect domain with OneDev Server (7.8.15) But now I am facing very annoying error at every 30s below.

ERROR io.onedev.server.util.ExceptionUtils - Error serving request java.lang.IllegalArgumentException: The request should have either 'pageId' or 'resourceName' parameter! (at every 30s) at org.apache.wicket.protocol.ws.api.AbstractWebSocketProcessor.(AbstractWebSocketProcessor.java:122) at io.onedev.server.web.websocket.WebSocketProcessor.(WebSocketProcessor.java:61) at io.onedev.server.web.websocket.WebSocketFilter$1.createWebSocket(WebSocketFilter.java:64) at org.eclipse.jetty.websocket.server.WebSocketServerFactory.acceptWebSocket(WebSocketServerFactory.java:230) at org.eclipse.jetty.websocket.server.WebSocketServerFactory.acceptWebSocket(WebSocketServerFactory.java:215) at io.onedev.server.web.websocket.WebSocketFilter.acceptWebSocket(WebSocketFilter.java:81) at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:66) at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203) at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137)

My environment is little bit different from official documentaion (A: Proxy Server, B: GIT server) I thought that problem would be Websocket issue, so trying to forward real IP by adding settings like below

     server_name xxx.yyy.com;
     
     # maximum size of uploaded file. Increase this if your repository is very large
     client_max_body_size 200M; 
     
proxy_hide_header X-Powered-By;
add_header Content-Security-Policy "upgrade-insecure-requests";
add_header X-Frame-Options "SAMEORIGIN";
add_header X-XSS-Protection "1; mode=block" always;
add_header X-Content-Type-Options "nosniff" always;
add_header Cache-Control "no-transform" always;
add_header Referrer-Policy no-referrer always;
add_header X-Robots-Tag none;

#Forwading websocket call 
location /wicket/websocket {
    try_files $uri @wss;
}

location @wss {
    proxy_pass http://serverIP:6610;
    proxy_http_version 1.1;
    
    # To forward realIP to GIT server instead of Proxy IP 
    proxy_set_header Host $server;
    proxy_set_header X-Forwarded-Scheme $scheme;
    proxy_set_header X-Forwarded-Proto  $scheme;
    proxy_set_header X-Forwarded-For    $remote_addr;
   
    proxy_read_timeout     60;
    proxy_connect_timeout  60;
    proxy_redirect         off;

 
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
}



     location /~server {
             proxy_pass http://serverIP/~server;
             proxy_http_version 1.1;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";
     }
     
     location / {
             proxy_pass http://$server:6610/;
     }

But it isnt working till now. Still got same error messages at every 30s. Could you help me to fix it too?

Reference Ngnix setting

https://docs.onedev.io/administration-guide/reverse-proxy-setup/

     listen 80;
     listen [::]:80;

     server_name onedev.example.com;
     
     # maximum size of uploaded file. Increase this if your repository is very large
     client_max_body_size 100M; 
     
     location /wicket/websocket {
             proxy_pass http://localhost:6610/wicket/websocket;
             proxy_http_version 1.1;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";                
     }

     location /~server {
             proxy_pass http://localhost:6610/~server;
             proxy_http_version 1.1;
             proxy_set_header Upgrade $http_upgrade;
             proxy_set_header Connection "upgrade";
     }
     
     location / {
             proxy_pass http://localhost:6610/;
     }
}

Activities

Jiung changed fields 3 years ago
Name Previous Value Current Value
Type

Bug

Support Request
Robin Shen commented 3 years ago

I am not very familiar with Nginx. Does it has some log displaying which url it is connecting to OneDev every 30s?
Jiung commented 3 years ago

Yep, exactly same as I showed you before

2022-12-26 21:55:20,264 ERROR [qtp229791798-38] io.onedev.server.util.ExceptionUtils Error serving request java.lang.IllegalArgumentException: The request should have either 'pageId' or 'resourceName' parameter! at org.apache.wicket.protocol.ws.api.AbstractWebSocketProcessor.(AbstractWebSocketProcessor.java:122) at io.onedev.server.web.websocket.WebSocketProcessor.(WebSocketProcessor.java:61) at io.onedev.server.web.websocket.WebSocketFilter$1.createWebSocket(WebSocketFilter.java:64) at org.eclipse.jetty.websocket.server.WebSocketServerFactory.acceptWebSocket(WebSocketServerFactory.java:230) at org.eclipse.jetty.websocket.server.WebSocketServerFactory.acceptWebSocket(WebSocketServerFactory.java:215) at io.onedev.server.web.websocket.WebSocketFilter.acceptWebSocket(WebSocketFilter.java:81) at org.apache.wicket.protocol.ws.AbstractUpgradeFilter.processRequestCycle(AbstractUpgradeFilter.java:66) at org.apache.wicket.protocol.http.WicketFilter.processRequest(WicketFilter.java:203) at org.apache.wicket.protocol.http.WicketServlet.doGet(WicketServlet.java:137) at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at io.onedev.server.web.DefaultWicketServlet.service(DefaultWicketServlet.java:43) at io.onedev.server.persistence.SessionInterceptor$1.call(SessionInterceptor.java:23) at io.onedev.server.persistence.DefaultSessionManager.call(DefaultSessionManager.java:90) at io.onedev.server.persistence.SessionInterceptor.invoke(SessionInterceptor.java:18) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626) at com.google.inject.servlet.DefaultFilterPipeline.dispatch(DefaultFilterPipeline.java:47) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at io.onedev.server.git.GoGetFilter.doFilter(GoGetFilter.java:87) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at io.onedev.server.git.GitLfsFilter.doFilter(GitLfsFilter.java:511) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at io.onedev.server.git.GitFilter.doFilter(GitFilter.java:394) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370) at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at io.onedev.server.jetty.DisableTraceFilter.doFilter(DisableTraceFilter.java:28) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:833)
Robin Shen commented 3 years ago

I mean does Nginx has somewhere to check what is the exact url it is using to connect to OneDev? I want to check that url for sure.
Robin Shen commented 3 years ago
For websocket url, there is also some query string appended at my side, for instance:

/wicket/websocket?pageId=3&wicket-ajax-baseurl=&wicket-app-name=DefaultWicketServlet%24%24EnhancerByGuice%24%24160010585

So please make sure your nginx rule forward query string part as well.
Jiung commented 3 years ago

Yep. Here is svr URL
Robin Shen commented 3 years ago

This info does not help. Please check/enable nginx log and check what url it is using when connect to OneDev backend
Robin Shen commented 3 years ago

I guess some query string is trimmed by your Nginx rule
Jiung commented 3 years ago

Okay, I will investigate it.
Jiung commented 3 years ago

There aren't so many error logs that I was expected

Access log [26/Dec/2022:13:17:27 +0000] - 500 500 - GET http gitb.madorca.com "/server" [Client 110.234.32.229] [Length 646] [Gzip -] [Sent-to 192.168.0.109] "Jetty/9.4.43.v20210629" "-" 2022-12-26 22:17:27,439 ERROR [qtp229791798-38] io.onedev.server.util.ExceptionUtils Error serving request java.lang.IllegalArgumentException: The request should have either 'pageId' or 'resourceName' parameter!

[26/Dec/2022:13:17:42 +0000] - 500 500 - GET http gitb.madorca.com "/server" [Client 211.58.154.16] [Length 646] [Gzip -] [Sent-to 192.168.0.109] "Jetty/9.4.43.v20210629" "-" 2022-12-26 22:17:42,977 ERROR [qtp229791798-38] io.onedev.server.util.ExceptionUtils Error serving request java.lang.IllegalArgumentException: The request should have either 'pageId' or 'resourceName' parameter!

I don't know why Jetty keep requests, but if Jetty sent then those network error would happen considering its error loging time. Jetty is requesting every 30s, so it would be cause of this issue.

But cant find its exact parameter.. Hm For the error log of ngninx, can't find relevant things except several error messages which would not be related considering its querying time.

2022/12/26 11:09:32 [error] 2880#2880: *3312002 connect() failed (111: Connection refused) while connecting to upstream, client: 211.58.154.16, server: gitb.madorca.com, request: "GET /server HTTP/1.1", upstream: "http://192.168.0.109:6610/server", host: "gitb.madorca.com:80"
Robin Shen commented 3 years ago

So why there is stillhttp://192.168.0.109:6610/server? It should be http://192.168.0.109:6610/~server instead.
Robin Shen commented 3 years ago

I guess some of your agent is not upgraded yet.
Jiung commented 3 years ago

Ah okay, I will check agent setting. And will check ~server too.
Jiung commented 3 years ago
Error message is finally stopped.

redirect every /server request to /~server

delete line below from custom code

location / { proxy_pass http://$server:6610/; }

removing every agents added

Greatly Appreciated.
Robin Shen changed state to 'Closed' 3 years ago
Previous Value Current Value
Open

Closed
Login to comment

Name	Previous Value	Current Value
Type	Bug	Support Request

Previous Value	Current Value
Open	Closed

Type	Question
Priority	Normal
Assignee	Robin Shen

Issue Votes (0)

Watchers (4)

Reference

OD-1060