Agent: heartbeat thread spins at ~100% CPU forever in AgentSocket.onClose after a send failure
#3182
-
State changed as build OD-8716 (16.9.0) is successful
-
OneDev
changed state to 'Closed' 3 weeks ago
Previous Value Current Value Open
Closed
| Type |
Bug
|
| Priority |
Major
|
| Assignee | |
| Affected Versions |
16.7.3
|
| Labels |
No labels
|
Issue Votes (0)
Environment: OneDev server 16.7.3, agent 3.3.1 (macOS arm64, Temurin 27, Tanuki wrapper). The same code is present in agent 3.3.9 and current main (AgentSocket.java).
Symptom: after the agent's websocket connection drops (laptop sleep / network change), the agent java process keeps one core busy (80–90%) indefinitely. The agent itself reconnects and keeps working, but the CPU never goes down until the agent is restarted. Over 3 days we measured ~66 h of CPU time on a single thread.
Thread dump (two samples, same thread):
The second sample shows the same thread in
InterruptedException.<init>.Cause (as we read the code): the heartbeat thread (
new Thread(this)inonConnect, runningAgentSocket.run) sends a message; the send fails on a closed channel, and Jetty invokesonClosesynchronously on that same thread.onClosedoesHere
threadis the current thread, so it interrupts itself,sleepthrows immediately, andstoppedis only set byrun()– which is blocked insideonClose. The loop never exits and never sleeps.Suggested fix: in
onClose, skip the interrupt/wait loop whenThread.currentThread() == thread(or setstopped = truebefore returning in that case / in afinallyofrun()).Workaround we use: restarting the agent (
launchctl kickstart -k) once a day.