Agent: heartbeat thread spins at ~100% CPU forever in AgentSocket.onClose after a send failure #3182
skudr opened 3 hours ago

Environment: OneDev server 16.7.3, agent 3.3.1 (macOS arm64, Temurin 27, Tanuki wrapper). The same code is present in agent 3.3.9 and current main (AgentSocket.java).

Symptom: after the agent's websocket connection drops (laptop sleep / network change), the agent java process keeps one core busy (80–90%) indefinitely. The agent itself reconnects and keeps working, but the CPU never goes down until the agent is restarted. Over 3 days we measured ~66 h of CPU time on a single thread.

Thread dump (two samples, same thread):

"Thread-439" cpu=239470399ms elapsed=248339s RUNNABLE
  at java.lang.Thread.sleep(Thread.java:593)
  at io.onedev.agent.AgentSocket.onClose(AgentSocket.java:555)
  at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.notifyOnClose(...)
  at org.eclipse.jetty.websocket.core.WebSocketCoreSession.closeConnection(...)
  at org.eclipse.jetty.websocket.core.WebSocketCoreSession.processConnectionError(...)
  at org.eclipse.jetty.websocket.core.WebSocketConnection$Flusher.onFailure(...)
  ...
  at io.onedev.agent.Message.sendBy(Message.java:50)
  at io.onedev.agent.AgentSocket.run(AgentSocket.java:1587)
  at java.lang.Thread.run(Thread.java:1527)

The second sample shows the same thread in InterruptedException.<init>.

Cause (as we read the code): the heartbeat thread (new Thread(this) in onConnect, running AgentSocket.run) sends a message; the send fails on a closed channel, and Jetty invokes onClose synchronously on that same thread. onClose does

while (!stopped) {
    thread.interrupt();
    try { Thread.sleep(1000); } catch (InterruptedException e) {}
}

Here thread is the current thread, so it interrupts itself, sleep throws immediately, and stopped is only set by run() – which is blocked inside onClose. The loop never exits and never sleeps.

Suggested fix: in onClose, skip the interrupt/wait loop when Thread.currentThread() == thread (or set stopped = true before returning in that case / in a finally of run()).

Workaround we use: restarting the agent (launchctl kickstart -k) once a day.

1/1
Type
Bug
Priority
Major
Assignee
Affected Versions
16.7.3
Labels
No labels
Issue Votes (0)
Watchers (2)
Reference
OD-3182
Please wait...
Connection lost or session expired, reload to recover
Page is in error, reload to recover