Different agent should be using different agent, and if a token is regenerated, you need to copy it to /conf/agent.properties, followed by restart of the agent.

I'm trying to create a single agent. Copying the docker run command from the create an agent menu will give me that error without having another agent. I have deleted the bind that it has but it doesn't change anything.

So from a new install going to "Administration -> Agents -> Clicking the + -> Running the docker run command" Will immediately give me Agent token rejected by server

Works at my side. Make sure you are connecting to the right server by checking the serverUrl environment.

The url is correct. however, it is being proxied by cloudflare tunnels, I'm not sure if that matters

Most probably that is the reason. You may connect to OneDev server directly to see if it works.

Yeah connecting to the container directly seemed to work. So I guess it is a cloudflared issue, but wouldn't it give a timed out or some other error instead of rejected agent?

OneDev agent authenticates itself via header Authorization: Bearer <agent token>. Please make sure this header is preserved across cloudflare.