#601  Make OneDev Agent docker image compatible with docker swarm
Released
jbauer opened 2 years ago

Hi,

first of all thanks for providing a docker image for OneDev agent! It works pretty well, when deploying as a standalone container.

I am wondering if the image can be made compatible to docker swarm. Currently I see two issues:

  1. Only one agent token can be provided and thus all agents will use the same token. Only one wins, all other fail to register. So if you have 3 replicas of the agent, only 1 is available.
  2. If all agents would start, they need some storage. This will be a docker volume and in my case is mounted via NFS so that the data can move with the container from host to host. Because all agents share the same volume they might override each others data?
  3. Not a blocker, but for security: Because docker swarm supports secrets and the token is a secret it would be nice fo have an ENV variable passing in a file (path) which contains the token.

I can see the following potential fixes:

  1. Allow agents to share tokens. Maybe via an explicit switch that needs to be turned on. Alternatively allow custom tokens. With custom tokens I could for example use hostnames as tokens, e.g. "onedev-agent-1", "onedev-agent-2" in case of two replicas.
  2. The agent could create a sub folder in its working directory (/agent/work) using its own hostname. That way all agents have their own folder, even if the volume is shared because it belongs to a service and is provided via NFS
  3. Many images provide two Env variables, e.g. PASSWORD=agenttoken and PASSWORD_FILE=/run/secrets/agenttoken

Current docker-compose.yaml with some notes to show the above issues within the file:

version: '3.9'

services:
  onedev:
    hostname: "onedev-{{.Task.Slot}}"
    image: 1dev/server:6.3.10
    deploy:
      mode: replicated
      replicas: 1
      update_config:
        parallelism: 1
        order: stop-first
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: volume
        source: onedev
        target: /opt/onedev
    networks:
      - dev
    # make sure linux sends tcp keep alive in case of long idle connections
    sysctls:
      - net.ipv4.tcp_keepalive_time=600
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=10
    ports:
      - target: 6610
        published: 6610
        protocol: tcp
        mode: ingress
      - target: 6611
        published: 6611
        protocol: tcp
        mode: ingress
        
  onedev-agent:
    hostname: "onedev-agent-{{.Task.Slot}}"
    image: 1dev/agent:latest
    deploy:
      mode: replicated
      replicas: 3
      update_config:
        parallelism: 1
        order: stop-first
    # make sure linux sends tcp keep alive in case of long idle connections
    sysctls:
      - net.ipv4.tcp_keepalive_time=600
      - net.ipv4.tcp_keepalive_intvl=30
      - net.ipv4.tcp_keepalive_probes=10
    environment:
      serverUrl: http://onedev:6610/
      # Same Token for all replicas of the agent. Only one of three agents will work.
      agentToken: xxxxxxx-xxxxxx-xxxxx-xxxxxxx
      # Better security, but image needs to support it
      agentToken_file: /run/secrets/agenttoken
      # Unique Token per agent replica could be achieved using docker template. Requires custom tokens in OneDev server.
      agentToken: "xxxx-xxxx-xxx-xxx-{{ .Task.Slot }}"
      agentToken: "onedev-agent-{{ .Task.Slot }}"  # token mimics hostname
      agentToken_file: "/run/secrets/agenttoken-{{ .Task.Slot }}"
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: volume
        source: onedev-agent
        # Because we have a NFS volume here, all agents work in the same directory. Might be better to use /agent/work/<hostname>. Agent needs to create that folder.
        target: /agent/work
    networks:
      - dev
 
volumes:
  onedev:
    driver_opts:
      type: "nfs"
      o: "addr=10.10.10.10,nolock,soft,rw,nfsvers=4,sync"
      device: ":/dev/onedev"
  onedev-agent:
    driver_opts:
      type: "nfs"
      o: "addr=10.10.10.10,nolock,soft,rw,nfsvers=4,sync"
      device: ":/dev/onedev-agent"
OneDev changed state to 'Closed' 2 years ago
Previous Value Current Value
Open
Closed
OneDev commented 2 years ago

State changed as code fixing the issue is committed

OneDev changed state to 'Released' 2 years ago
Previous Value Current Value
Closed
Released
OneDev commented 2 years ago

State changed as build #2338 is successful

Robin Shen commented 2 years ago

Path to file containing agent token can be specified via environment agentTokenFile as documented here:

https://code.onedev.io/projects/162/blob/6.3/pages/agent-management.md#agent-propertiesenvironments

OneDev referenced from other issue 7 months ago
issue 1 of 1
Type
Improvement
Priority
Normal
Assignee
Issue Votes (0)
Watchers (4)
Reference
onedev/server#601
Please wait...
Page is in error, reload to recover