This is not a credential or nginx upload-size problem. Both failing steps never complete a TCP handshake:

dial tcp <server public IP>:443: i/o timeout

Why it fails in the job, but works on the agent CLI

• Build Image always uses the onedev buildx builder. That builder is a BuildKit container, not the default Docker driver.
• Push Image runs crane inside the 1dev/crane container.
• Those containers resolve the registry host to the public IP and then cannot open port 443.

The same docker buildx build without --builder onedev uses the host / default builder network, which can reach the registry. Isolated job/builder containers often cannot. That is typical hairpin NAT or Docker-bridge routing when the agent is on the same host or LAN as OneDev.

Administration → System Settings → server URL (and the host in image tags / registry logins) must be reachable from Docker containers, not only from the agent host. See working with container images.

Confirm on the agent

docker run --rm curlimages/curl -I https://<onedev-host>/v2/
docker run --rm --network host curlimages/curl -I https://<onedev-host>/v2/

If the first times out and the second works, this diagnosis is confirmed.

What to change

1. Point jobs at a name or private IP that containers can route to (split-horizon DNS or an internal hostname). Tags and registry login must use that same host. Prefer @server@ for tags. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission.

2. Recreate the onedev builder on the agent with host networking (same network as the command that already works):

docker buildx rm onedev
docker ps -a | grep buildx_buildkit_
# remove any leftover builder container
docker buildx create --name onedev --driver docker-container --driver-opt network=host

3. For Push Image, add a host mapping in the docker executor Run Options, for example --add-host <onedev-host>:<ip-reachable-from-containers>.

Do not use localhost or 127.0.0.1 as the registry host from these steps.

Nginx client_max_body_size is not involved here (the connection never reaches nginx). After containers can connect, keep a large body size and generous proxy_read_timeout / proxy_send_timeout for layer uploads.

If the agent host only reaches that public IP via an HTTP forward proxy, also configure the builder and crane as in docker registry proxy. The log shows a direct TCP dial to the public IP, so missing proxy settings are less likely than container routing.

If you later use HTTP or a self-signed cert, follow insecure docker registry.

Thanks for the fast and elaborate reply.

Confirm on the agent

docker run --rm curlimages/curl -I https:///v2/ docker run --rm --network host curlimages/curl -I https:///v2/

I started an SSH session into the bare-metal host that hosts the dockerized OneDev agent and executed both commands. -> Both commands succeed with a 401 status

From there I started a bash session in the dockerized OneDev agent using: docker exec -it CodeSpaceAgent1 /bin/bash

From this bash prompt I executed both commands again. -> Both commands succeed with a 401 status.

My OneDev dockerized agent is on a different machine and different LAN than my OneDev server instance. The agent is running on an internal (behind firewall and NAT) machine. The OneDev server is on a publicly reachable VPS with a routable internet IPv4 address. No forward proxies in play at the agent side. Certificates are current LetsEncrypt certificates.

Those results rule out hairpin NAT. With the agent on a different LAN and both host-network and default-bridge containers getting HTTP 401, the host can reach the public registry.

Those two docker run commands still do not use the networks the failing steps use:

1. Push Image runs crane in a job step container on a per-job user-defined Docker network (build-<executor>-…), not on the default docker0 bridge and not on host network.
2. Build Image always uses --builder onedev. That builder is a BuildKit container (docker-container driver). The push is made from that container’s network. The same docker buildx build without --builder onedev uses the default Docker driver and pushes from the daemon/host namespace, which is why it works from the agent CLI.

docker run without --network uses the default bridge. That is a different path.

Please run these on the agent host (same Docker daemon the jobs use):

# 1) User-defined network (same class of network as job step containers)
docker network create onedev-egress-test
docker run --rm --network onedev-egress-test curlimages/curl -v -I https://<onedev-host>/v2/
docker network rm onedev-egress-test

# 2) IPv4 vs IPv6 (crane and buildkit are Go; they do not Happy-Eyeballs like curl)
getent ahosts <onedev-host>
docker run --rm curlimages/curl -4 -I https://<onedev-host>/v2/
docker run --rm curlimages/curl -6 -I https://<onedev-host>/v2/

# 3) Existing onedev builder container
docker ps -a | grep buildx_buildkit

If (1) times out, job networks cannot egress. Typical causes are firewall/UFW/firewalld masquerading only docker0, a DROP FORWARD policy, or a user-defined subnet overlapping a VPN/LAN route to the VPS.

If (2) IPv6 times out while IPv4 returns 401, that matches the ~30s dial tcp … i/o timeout from crane and buildkit.

Workarounds

Recreate the builder with host networking (same network as the CLI command that already works):

docker buildx rm onedev
docker ps -a | grep buildx_buildkit
# remove any leftover builder container
docker buildx create --name onedev --driver docker-container --driver-opt network=host

If the agent runs in Docker, create this builder from inside the agent container (or mount ~/.docker/buildx as in insecure docker registry) so the agent sees the same builder.

For Push Image, either add --network host to the remote docker executor Run Options (this overrides the job network; job services will not see each other), or fix NAT/FORWARD so user-defined bridges can reach the VPS.

Do not use localhost as the registry host. Prefer @server@ for tags. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission.

Please paste the output of tests (1) and (2).

Those results close the last two connectivity guesses:

1. A user-defined Docker network can reach the registry (HTTP 401, TLS OK, source 172.23.0.2). Job-network egress is not blocked.
2. The hostname is IPv4-only. curl -6 failing to resolve is expected (no AAAA). This is not an IPv6 Happy-Eyeballs timeout.

A generic curl container is still not the process that fails. Build Image pushes from the persistent onedev BuildKit container (docker-container driver). That network is independent of onedev-egress-test. Push Image runs crane in 1dev/crane:1.0.0 on the per-job build-<executor>-… network. Test (3) was not pasted.

Please run these inside the agent container (same Docker daemon the jobs use):

# Builder used by Build Image
docker buildx ls
docker ps -a | grep buildx_buildkit

# If a buildkit container exists, use ITS network namespace
docker run --rm --network container:<buildkit-container-name> curlimages/curl -v -I https://<onedev-host>/v2/

# Same image as Push Image
docker run --rm --network onedev-egress-test 1dev/crane:1.0.0 catalog <onedev-host>

crane catalog only needs to get past /v2/. If that prints dial tcp … i/o timeout, the problem is the crane/BuildKit runtime, not the job bridge.

Workaround that matches the CLI path that already works

Recreate the builder with host networking from inside the agent (so the agent sees the same builder). If you create it only on the host, mount ~/.docker/buildx into the agent as in insecure docker registry.

docker buildx rm onedev
docker ps -a | grep buildx_buildkit
# remove any leftover builder container
docker buildx create --name onedev --driver docker-container --driver-opt network=host --use

Then retry Build Image.

For Push Image, add --network host to the remote docker executor Run Options. That overrides the job network; job services will not see each other.

Prefer @server@ in tags. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission.

Please paste docker buildx ls, the buildkit docker ps line, the shared-namespace curl if a builder exists, and the crane catalog output.

Test 3 yielded no results:

root@CodeSpaceAgent1:/agent/bin# docker ps -a | grep buildx_buildkit
root@CodeSpaceAgent1:/agent/bin#

Your test output:

root@CodeSpaceAgent1:/agent/bin# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker
 \_ default    \_ default       running   v0.29.0    linux/amd64 (+3)
root@CodeSpaceAgent1:/agent/bin# docker ps -a | grep buildx_buildkit
root@CodeSpaceAgent1:/agent/bin# docker run --rm --network onedev-egress-test 1dev/crane:1.0.0 catalog [URL]
Unable to find image '1dev/crane:1.0.0' locally
1.0.0: Pulling from 1dev/crane
9d95e77b6d6f: Pull complete
1e3d9b7d1452: Pull complete
4aa0ea1413d3: Pull complete
96266735468f: Pull complete
a7ca0d9ba68f: Pull complete
fe5ca62666f0: Pull complete
b02a7525f878: Pull complete
fcb6f6d2c998: Pull complete
2758d0c31c8c: Pull complete
5627a970d25e: Pull complete
7f1acc538748: Pull complete
7c881f9ab25e: Pull complete
08553ba93cfe: Pull complete
e8c73c638ae9: Pull complete
250c06f7c38e: Pull complete
71b42c1dc2b2: Download complete
Digest: sha256:2d21b3c0d5533d4f6e722a05f1ccb078d4a414a4547ccdf881122cd2c6eb3103
Status: Downloaded newer image for 1dev/crane:1.0.0
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "catalog": executable file not found in $PATH

Run 'docker run --help' for more information
root@CodeSpaceAgent1:/agent/bin# docker run --rm --network onedev-egress-test 1dev/crane:1.0.0 crane catalog [URL]
docker: Error response from daemon: failed to set up container networking: network onedev-egress-test not found

Run 'docker run --help' for more information
root@CodeSpaceAgent1:/agent/bin# docker network create onedev-egress-test
c94cfbd795f84bc16d47d00479938bd12135bf3adbee8589c59d2572530b8b2f
root@CodeSpaceAgent1:/agent/bin# docker run --rm --network onedev-egress-test 1dev/crane:1.0.0 crane catalog [URL]
Error: reading tags for [URL]: GET https://[URL]/v2/_catalog?n=1000: UNSUPPORTED:
root@CodeSpaceAgent1:/agent/bin#

I skipped test #2, because there was no buildkit found. I also suspected a small error in your 3rd command since it threw an error. Hence, I added the 'crane' command before the 'catalog'. I also re-added the onedev-egress-test network since it was deleted as part of earlier tests.

Those results close the crane/job-network question.

UNSUPPORTED means crane reached OneDev

GET /v2/_catalog is not implemented by the built-in registry, so UNSUPPORTED is the expected HTTP response. Crane finished DNS, TCP, TLS, and HTTP on a user-defined network. That is not a dial tcp … i/o timeout.

The extra crane prefix was correct: 1dev/crane:1.0.0 does not treat catalog as the entrypoint.

No onedev builder right now

docker buildx ls showing only default, and no buildx_buildkit container, means the persistent builder is not present at the moment. Build Image creates it on first use with docker buildx create --name onedev (docker-container driver, default container network — not host, not the job build-… network). It should remain after the step. If it is missing, Build Image has not run since a prune/reboot, the builder was removed, or the interactive shell’s $HOME/.docker/buildx is not the one the agent process uses.

The #16 exporting to image / pushing layers / dial tcp … i/o timeout log is buildx. That is Build Image with “Push to container registry”, not Push Image (crane push logs differently).

What is still open

Every single-request test from the agent works (default bridge, user-defined network, host network, crane). A registry push is not a single request: BuildKit (and a real crane push) open many parallel HEAD /v2/…/blobs/sha256:… connections. A firewall, conntrack table, or SYN limit that allows one curl but drops a burst of dials matches the 30s timeouts on those blob HEADs.

What to do

1. Recreate the builder with host networking inside the agent container (same Docker CLI the jobs use), then retry Build Image:

docker buildx rm onedev || true
docker ps -a | grep buildx_buildkit
# remove leftover builder containers if any
docker buildx create --name onedev --driver docker-container --driver-opt network=host --use
docker buildx ls

If docker buildx ls still does not show onedev after create, the agent is using another buildx state (different $HOME). Create the builder as the same user the agent runs as, or mount that user’s ~/.docker/buildx as in insecure docker registry.

2. After Build Image runs (even if it fails), immediately:

docker ps -a | grep buildx_buildkit
docker run --rm --network container:<buildkit-container-name> curlimages/curl -v -I https://<onedev-host>/v2/

3. Retry Push Image. Crane already reached /v2/. If push still times out, add --network host to the remote docker executor Run Options (job services will no longer share a network).

4. Prefer @server@ in the image tag. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission. Do not put https:// in the image tag host.

5. On nginx, keep a large client_max_body_size and generous proxy_read_timeout / proxy_send_timeout once TCP works. Also check whether the VPS firewall rate-limits concurrent connections from the agent.

Please paste docker buildx ls after the create, the Build Image retry result, and (if the builder container exists) the shared-namespace curl.

After creation:

root@CodeSpaceAgent1:/agent/bin# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT                   STATUS     BUILDKIT   PLATFORMS
onedev*       docker-container
 \_ onedev0    \_ unix:///var/run/docker.sock   inactive
default       docker
 \_ default    \_ default                       running    v0.29.0    linux/amd64 (+3)

Build fails the same way as before.

During the build, docker ps -a | grep buildx_buildkit run from inside the agent container yields zero results. After the build the docker buildx ls shows:

root@CodeSpaceAgent1:/agent/work# docker buildx ls
NAME/NODE     DRIVER/ENDPOINT                   STATUS     BUILDKIT   PLATFORMS
onedev*       docker-container
 \_ onedev0    \_ unix:///var/run/docker.sock   inactive
default       docker
 \_ default    \_ default                       running    v0.29.0    linux/amd64 (+3)

The builder you created was never used by the job.

inactive after docker buildx create is expected. The first docker buildx build --builder onedev should start a BuildKit container (buildx_buildkit_onedev0 or similar) and flip that node to running. Yours stayed inactive, and docker ps showed no BuildKit container while the job was failing. So Build Image did not use this builder on the Docker daemon you are inspecting. That is why host networking changed nothing.

Most likely: the job is not running on CodeSpaceAgent1

Single-request tests on this agent already succeed (curl HTTP 401, crane UNSUPPORTED). A CLI docker buildx build without --builder onedev also works here. The failure is still dial tcp <public IP>:443: i/o timeout from whatever actually runs the step.

Please paste the first job-log line that looks like one of these:

• Executing job (executor: …, agent: …) — Remote Docker Executor. Check the agent name. If it is not this agent, recreate the host-network builder on that agent.
• Executing job (executor: …, server: …, network: …) — Server Docker Executor on the OneDev VPS. Then this agent’s builder is irrelevant. The push runs in a BuildKit container on the VPS. A container on that VPS dialing the VPS’s own public IP is classic hairpin NAT and matches this timeout.

If it is the server executor, run this on the OneDev server (not this agent), as the OS user that runs the OneDev process (or mount that user’s ~/.docker/buildx into the server container as in insecure docker registry):

docker buildx rm onedev || true
docker buildx create --name onedev --driver docker-container --driver-opt network=host --use
docker buildx ls

Then retry Build Image. Prefer a hostname those containers can route to (internal name or a container-reachable IP), not only the public one. Prefer @server@ in tags. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission.

For Push Image on that same executor, add --network host to Run Options (job services will no longer share a network).

If the log shows this agent

OneDev copies $HOME/.docker/buildx from the agent JVM (user.home / DOCKER_CONFIG), not necessarily your interactive root shell, into a temporary DOCKER_CONFIG before docker buildx create --name onedev and docker buildx build --builder onedev.

Inside the agent container:

echo "shell HOME=$HOME"
ls -la /root/.docker/buildx /agent/.docker/buildx 2>/dev/null
tr '\0' '\n' < /proc/1/environ | grep -E 'HOME|DOCKER'
ps -o pid,user,args -C java

Create the host-network builder with that same HOME (example if the agent home is /agent):

HOME=/agent docker buildx rm onedev || true
HOME=/agent docker buildx create --name onedev --driver docker-container --driver-opt network=host --use
HOME=/agent docker buildx ls

While Build Image is in #16 pushing layers (not after it fails, and not during Push Image):

docker ps -a --format '{{.Names}} {{.Image}} {{.Status}}'

Please paste the Executing job line, the HOME/process output, and that docker ps during the push.

I am starting to feel stupid. I first tried creating an agent on the same server as the OneDev server, but that threw an error message (I do not remember which on). Then I decided to create an agent on another machine, assuming this one would be used for all subsequent work. But from the first lines of the log...

10:36:28 No job executor defined, auto-discovering...
10:36:28 Discovered server docker executor
10:36:28 Pending resource allocation...
10:36:28 Executing job (executor: auto-discovered, server: 127.0.0.1:5710, network: build-auto-discovered-34-13-3)...

This is a common setup mix-up, not something you did wrong.

Connecting an agent does not make OneDev run jobs on it. An agent is only a worker that a Remote Docker Executor can select. With no job executor defined, OneDev auto-discovers what can run on the OneDev server itself. Docker is available there, so it picked Server Docker Executor.

That is what this line means:

Executing job (executor: auto-discovered, server: 127.0.0.1:5710, network: build-auto-discovered-34-13-3)...

server: 127.0.0.1:5710 is the OneDev process on the VPS, not CodeSpaceAgent1. The job network was created on the server Docker daemon. That is why every test on the remote agent succeeded, the builder you created there stayed inactive, and the push still timed out: Build Image / Push Image never ran on that agent.

An agent on the same machine as OneDev often fails if serverUrl is localhost or 127.0.0.1 (the agent container cannot reach the server). Use a hostname or IP the agent container can route to. You do not need an agent on the OneDev host to run jobs there; that is what Server Docker Executor already does.

So the timeout is hairpin NAT: a BuildKit (or crane) container on the VPS is dialing the VPS public IP on port 443.

Path 1 — run jobs on CodeSpaceAgent1 (what you intended)

Follow Build farm with agents:

1. Administration → Job Executors
2. Add Remote Docker Executor and give it a name
3. Optionally set Agent Selector to this agent
4. Test and save

Once any executor is defined, auto-discover stops. If you add only this Remote Docker Executor, jobs go to the agent. The next job log should start with Executing job (executor: …, agent: …). Your earlier curl/crane tests already show that agent can reach the registry.

Path 2 — keep using the server executor (no agent required)

Do this on the OneDev server, as the OS user that runs OneDev (or mount that user’s ~/.docker/buildx into the server container as in insecure docker registry):

docker buildx rm onedev || true
docker buildx create --name onedev --driver docker-container --driver-opt network=host --use
docker buildx ls

Then retry Build Image. Prefer a hostname those containers can route to (internal name or a container-reachable IP), not only the public one. Prefer @server@ in tags. For built-in registry login use @server_url@, user @job_token@, and an access-token secret with package write permission.

For Push Image on that executor, add --network host to Run Options (job services will no longer share a network).

Please say which path you want, then paste the new Executing job line and the push result.

Bingo! Success!

Thanks you so much for your support.

Glad that is working. Closing this question.