Problem

In a cluster with more than one server, server steps of a running job fail with this error:

Running step "coverage"...
java.lang.RuntimeException: Http request failed (status code: 500, error message: No job context found for specified job token)
    at io.onedev.k8shelper.KubernetesHelper.checkStatus(KubernetesHelper.java:138)
    at io.onedev.k8shelper.JobHelper.runServerStep(JobHelper.java:509)
    ...
Step "coverage" is failed

The server log shows the failure at the first check in getJobContext, so the jobServers entry of the job is missing:

ERROR i.o.s.rest.JerseyExceptionMapper - Error processing api request
io.onedev.commons.utils.ExplicitException: No job context found for specified job token
    at io.onedev.server.job.DefaultJobService.getJobContext(DefaultJobService.java:724)
    at io.onedev.server.rest.resource.WorkerResource.runServerStep(WorkerResource.java:128)

The job continues to run and to send its log. Each server step after the first failure also fails. All other steps are successful. All types of server step fail, for example publish of a report and a step template that publishes an image reference.

What we saw

• The failures occur once each hour, at the same minute of the hour.
• A job fails only if it runs when that minute starts. Server steps of the job before that minute are successful.
• Only some of the jobs that run at that minute fail. Other jobs that run at the same time are successful.
• No server left the cluster or joined the cluster at these times.

Cause

The cause is the hourly maintenance task in DefaultJobService (17.0.2, line 1124 to 1126). The leader server removes each jobServers entry that is not in getActiveJobTokens():

var activeJobTokens = getActiveJobTokens();
jobServers.removeAll(it -> !activeJobTokens.contains(it.getKey()));

getActiveJobTokens() (line 1596) uses a bound method reference:

for (var value: clusterService.runOnAllServers(jobContexts::keySet).values()) {

jobContexts::keySet binds the jobContexts map of the leader when the leader makes the task. The task is serialized with that map. Thus each server returns the job tokens of the leader, not its own job tokens. The sweep then removes the jobServers entries of all jobs that run on the other servers. After that, getJobContext cannot find these jobs.

The method reference came in with commit 75ded79a ("chore: Log dependency waiting message and cancel waiting builds"). The current main branch still has it.