org.eclipse.jetty.io.EofException: null, Docker compose with Onedev UI behind HAProxy. (OD-2068)
brixomatic opened 1 year ago

Clicking through projects and builds, opening a build, my server logs gets spammed with errors like these:

org.eclipse.jetty.io.EofException: null
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:280)
	at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
	at org.eclipse.jetty.websocket.common.io.FrameFlusher.flush(FrameFlusher.java:264)
	at org.eclipse.jetty.websocket.common.io.FrameFlusher.process(FrameFlusher.java:193)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:248)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:229)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.outgoingFrame(AbstractWebSocketConnection.java:581)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.close(AbstractWebSocketConnection.java:181)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:510)
	at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.java:440)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
	at java.base/sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:182)
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130)
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:493)
	at java.base/java.nio.channels.SocketChannel.write(SocketChannel.java:507)
	at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:274)
	... 22 common frames omitted

HProxy config (parts):

global
    # intermediate configuration
    ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers no-tls-tickets ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3

    ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options no-tls-tickets ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3

    # curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam
    ssl-dh-param-file /usr/local/etc/haproxy/haproxy_dhparams.pem

    maxconn 2304

defaults
    # respond to any clients that spend more than five seconds from the first byte of the request to the last
    # with an HTTP 408 Request Timeout error. Normally, this only applies to the HTTP request and its headers
    # and doesn’t include the body of the request.
    timeout http-request 5s
    # store the request body in a buffer and apply the http-request timeout to it.
    option http-buffer-request

    timeout connect 5s
    timeout client 30s
    timeout server 30s

frontend onedevfrontend
  mode  http
  bind  *:6444 ssl crt /usr/local/etc/ssl/mycertificate.pem
  http-request redirect scheme https unless { ssl_fc }
  # A number of attacks use HTTP/1.0 as the protocol version because that’s the version supported by some bots.
  http-request deny if HTTP_1.0
  # curl, phantomjs and slimerjs are scriptable, headless browsers that could be used to automate an attack
  http-request deny if { req.hdr(user-agent) -i -m sub curl phantomjs slimerjs }
  # an attacker who is using an automated tool might send requests that don’t contain a User-Agent header at all.
  http-request deny unless { req.hdr(user-agent) -m found }
  default_backend onedevbackend
backend onedevbackend
  mode http
  option forwarded proto host by by_port for
  option forwardfor
  http-request set-header X-Forwarded-Proto https if { ssl_fc }
  server server1 onedevserver1:6610 maxconn 2048

docker compose.yaml:

services:
  onedev:
    image: 'docker.io/1dev/server:latest'
    container_name: 'onedevserver1'
    hostname: 'onedevserver1'
    networks:
      - my_network
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /opt/onedev:/opt/onedev
      - /etc/timezone:/etc/timezone:ro
    ports:
      - '6511:6511'
  mproxy:
    image: haproxy:3.0-alpine
    container_name: 'loadbalancer'
    networks:
      - my_network
    restart: unless-stopped
    volumes:
      - /etc/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
      - /etc/haproxy/haproxy_dhparams.pem:/usr/local/etc/haproxy/haproxy_dhparams.pem:ro
      - /etc/ssl/mycertificate.pem:/usr/local/etc/ssl/mycertificate.pem:ro
      - /etc/timezone:/etc/timezone:ro
    ports:
      - '6444:6444'

networks:
  my_network:
    driver: bridge

System settings are set to:
Server URL:
https://my_domain.example:6444
(using my real domain of course)
SSH root URL:
ssh://my_domain.example:6511
Security settings set to:
CORS Allowed Origin:
https://my_domain.example:6444

I don't see any obvious mistake in my setup. Requests are handled almost immediately, so don't think HAProxy's timeout is the problem. Also, as I'm still setting up the system, I'm the only user and there is no load on the System. What would cause this problem?

  • Robin Shen commented 1 year ago

    Never tested with HAProxy. Please disable SSL temporarily to see if the problem still exists.

  • brixomatic commented 1 year ago

    I have tried various things now, like disabling all the options and timeouts I made in the haproxy.cfg, and the problem occurs only for HTTPS -> HTTP. The error will not occur, if I change the configuration to to HTTP -> HTTP, like this

    # bind  *:6444 ssl crt /usr/local/etc/ssl/mycertificate.pem
      bind :6444
    

    Of course I need to have an SSL connection, so I can't really run with this settings. But it might be valuable to find the root cause of this.

    Also I've learned that the error can be triggered by just reloading the Server Log view.

    I will investigate further and update my results here, shall I find any setting to solve this for future reference. Any help is of course greatly appreciated, that's why I posted the docker-compose file along with the config, so a lab can be quickly set up (just requires downloading the 'ssl-dh-param-file', as described, and putting it into its place at /etc/haproxy/haproxy_dhparams.pem. A self-signed certificate for testing will probably suffice.

  • brixomatic commented 1 year ago

    For future reference: I did solve the problem eventually by creating a second backend to handle websockets:

      acl hdr_connection_upgrade hdr(Connection)  -i upgrade
      acl hdr_upgrade_websocket  hdr(Upgrade)     -i websocket
      acl websocket_url path_beg -i /wicket/websocket
    
      http-request set-header Upgrade websocket if hdr_upgrade_websocket
      http-request set-header Connection upgrade if hdr_upgrade_websocket
    
      use_backend onedev_websocket if hdr_connection_upgrade hdr_upgrade_websocket websocket_url
    
      default_backend onedev  
    
    backend onedev
      mode http
      option forwarded proto host by by_port for
      option forwardfor
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      server onedev_http_server onedev:6610 maxconn 2048
    
    backend onedev_websocket
      mode http
      option forwarded proto host by by_port for
      option forwardfor
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      server onedev_http_server onedev:6610 maxconn 2048 ws h2
    

    The "ws h2" setting was the vital part.

    Now I just need to clean the config up.

  • Robin Shen changed state to 'Closed' 1 year ago
    Previous Value Current Value
    Open
    Closed
  • Robin Shen commented 1 year ago

    Thanks for sharing! Definitely useful for users running OneDev behind HAProxy.

issue 1/1
Type
Question
Priority
Normal
Assignee
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-2068
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover